cur4g16
Step 1
| Rollout | Reward | Test | Edited accuracy | Hit cap | Tensors edited | Turns | Minutes | Forward passes | Python calls | Ended |
|---|
| 1 | +18.4 | – | 25.8% | 73.0% | 24 | 68 | 99.9 | 11 | 55 | editor finished |
| 2 | +22.3 | – | 29.7% | 69.5% | 48 | 83 | 122.6 | 13 | 68 | editor finished |
| 3 | +2.3 | – | 9.8% | 90.6% | 7 | 113 | 135.7 | 9 | 99 | time limit |
| 4 | +0.4 | – | 7.8% | 93.8% | 1 | 79 | 133.7 | 13 | 64 | editor finished |
| 5 | +12.5 | – | 19.9% | 80.9% | 1 | 109 | 110.9 | 17 | 88 | editor finished |
| 6 | +16.4 | – | 23.8% | 77.0% | 7 | 102 | 75.9 | 7 | 91 | editor finished |
| 7 | -6.2 | – | 1.2% | 100.0% | 8 | 86 | 29.4 | 0 | 66 | editor finished |
| 8 | +35.2 | – | 42.6% | 46.5% | 24 | 87 | 45.7 | 11 | 70 | editor finished |
| 9 | +17.2 | – | 24.6% | 73.0% | 1 | 79 | 64.3 | 14 | 62 | editor finished |
| 10 | +1.2 | – | 8.6% | 91.8% | 24 | 69 | 30.1 | 0 | 30 | editor finished |
| 11 | +14.8 | – | 22.3% | 78.1% | 4 | 85 | 88.6 | 10 | 73 | editor finished |
| 12 | +12.9 | – | 20.3% | 79.7% | 1 | 67 | 103.3 | 10 | 54 | editor finished |
| 13 | +26.2 | – | 33.6% | 62.9% | 24 | 66 | 114.1 | 11 | 53 | editor finished |
| 14 | +34.8 | – | 42.2% | 37.1% | 16 | 90 | 106.1 | 15 | 72 | editor finished |
| 15 | +27.7 | – | 35.2% | 58.6% | 22 | 78 | 120.6 | 12 | 62 | editor finished |
| 16 | +22.3 | – | 29.7% | 67.6% | 25 | 97 | 110.2 | 12 | 82 | editor finished |
| Mean | +16.1 | – | 23.6% | 73.8% | 14.8 | 84.9 | 93.2 | 10.3 | 68.1 | |
Step 2
| Rollout | Reward | Test | Edited accuracy | Hit cap | Tensors edited | Turns | Minutes | Forward passes | Python calls | Ended |
|---|
| 1 | +4.7 | – | 12.1% | 85.9% | 48 | 85 | 107.0 | 9 | 73 | editor finished |
| 2 | +7.8 | – | 15.2% | 85.9% | 1 | 83 | 103.1 | 16 | 64 | editor finished |
| 3 | +21.9 | – | 29.3% | 70.7% | 48 | 78 | 128.4 | 13 | 61 | time limit |
| 4 | +8.2 | – | 15.6% | 87.9% | 24 | 66 | 121.5 | 11 | 53 | editor finished |
| 5 | +19.1 | – | 26.6% | 71.9% | 30 | 72 | 118.1 | 11 | 57 | editor finished |
| 6 | +7.0 | – | 14.5% | 84.8% | 48 | 70 | 119.2 | 10 | 58 | editor finished |
| 7 | +13.7 | – | 21.1% | 77.3% | 10 | 103 | 125.7 | 24 | 77 | editor finished |
| 8 | +14.8 | – | 22.3% | 75.4% | 13 | 78 | 113.6 | 13 | 60 | editor finished |
| 9 | +25.4 | – | 32.8% | 58.2% | 1 | 74 | 90.0 | 10 | 60 | editor finished |
| 10 | +27.3 | – | 34.8% | 61.7% | 48 | 79 | 111.7 | 13 | 63 | editor finished |
| 11 | +17.6 | – | 25.0% | 74.2% | 30 | 75 | 125.5 | 15 | 59 | editor finished |
| 12 | +5.5 | – | 12.9% | 87.9% | 1 | 63 | 132.8 | 12 | 49 | time limit |
| 13 | +18.0 | – | 25.4% | 75.4% | 1 | 87 | 124.1 | 14 | 68 | editor finished |
| 14 | +14.1 | – | 21.5% | 79.3% | 1 | 70 | 127.2 | 9 | 58 | editor finished |
| 15 | +18.0 | – | 25.4% | 72.3% | 24 | 90 | 121.9 | 17 | 70 | editor finished |
| 16 | +19.5 | – | 27.0% | 73.0% | 5 | 72 | 114.7 | 12 | 58 | editor finished |
| Mean | +15.2 | – | 22.6% | 76.4% | 20.8 | 77.8 | 117.8 | 13.1 | 61.8 | |
Step 3
| Rollout | Reward | Test | Edited accuracy | Hit cap | Tensors edited | Turns | Minutes | Forward passes | Python calls | Ended |
|---|
| 1 | +6.6 | – | 14.1% | 86.3% | 48 | 85 | 112.2 | 16 | 67 | editor finished |
| 2 | +2.3 | – | 9.8% | 92.6% | 1 | 85 | 92.5 | 14 | 69 | editor finished |
| 3 | +13.7 | – | 21.1% | 77.0% | 4 | 108 | 114.0 | 21 | 84 | editor finished |
| 4 | +10.2 | – | 17.6% | 87.5% | 8 | 105 | 90.1 | 13 | 87 | editor finished |
| 5 | +13.7 | – | 21.1% | 77.0% | 48 | 65 | 66.0 | 10 | 53 | editor finished |
| 6 | +16.8 | – | 24.2% | 77.0% | 6 | 68 | 80.5 | 14 | 51 | editor finished |
| 7 | +14.8 | – | 22.3% | 78.1% | 7 | 71 | 84.3 | 13 | 56 | editor finished |
| 8 | +24.2 | – | 31.6% | 65.2% | 48 | 68 | 83.2 | 13 | 52 | editor finished |
| 9 | +12.9 | – | 20.3% | 80.5% | 3 | 72 | 88.9 | 18 | 52 | editor finished |
| 10 | +13.3 | – | 20.7% | 77.3% | 4 | 122 | 121.2 | 23 | 95 | editor finished |
| 11 | +15.2 | – | 22.7% | 75.8% | 1 | 79 | 72.6 | 14 | 63 | editor finished |
| 12 | +37.5 | – | 44.9% | 35.5% | 24 | 73 | 47.4 | 10 | 61 | editor finished |
| 13 | +16.4 | – | 23.8% | 76.6% | 8 | 89 | 91.1 | 17 | 69 | editor finished |
| 14 | +19.9 | – | 27.3% | 72.7% | 48 | 67 | 61.3 | 11 | 54 | editor finished |
| 15 | +30.9 | – | 38.3% | 42.2% | 24 | 107 | 91.5 | 17 | 87 | editor finished |
| 16 | +23.4 | – | 30.9% | 71.1% | 1 | 82 | 71.4 | 17 | 63 | editor finished |
| Mean | +17.0 | – | 24.4% | 73.3% | 17.7 | 84.1 | 85.5 | 15.1 | 66.4 | |
Step 4
| Rollout | Reward | Test | Edited accuracy | Hit cap | Tensors edited | Turns | Minutes | Forward passes | Python calls | Ended |
|---|
| 1 | +19.5 | – | 27.0% | 72.7% | 1 | 73 | 117.4 | 13 | 57 | editor finished |
| 2 | +14.8 | – | 22.3% | 75.0% | 1 | 77 | 118.2 | 16 | 57 | editor finished |
| 3 | +14.4 | – | 21.9% | 77.0% | 4 | 89 | 113.8 | 19 | 68 | editor finished |
| 4 | +19.9 | – | 27.3% | 73.4% | 48 | 104 | 106.5 | 15 | 86 | editor finished |
| 5 | +19.1 | – | 26.6% | 73.4% | 3 | 89 | 94.4 | 12 | 75 | editor finished |
| 6 | +3.9 | – | 11.3% | 89.8% | 1 | 87 | 99.4 | 15 | 70 | editor finished |
| 7 | +24.6 | – | 32.0% | 69.1% | 48 | 53 | 61.9 | 9 | 41 | editor finished |
| 8 | +18.8 | – | 26.2% | 72.3% | 30 | 89 | 95.7 | 14 | 73 | editor finished |
| 9 | +23.4 | – | 30.9% | 68.4% | 5 | 57 | 87.7 | 13 | 42 | editor finished |
| 10 | +18.4 | – | 25.8% | 73.8% | 4 | 77 | 86.7 | 10 | 63 | editor finished |
| 11 | +27.0 | – | 34.4% | 62.1% | 2 | 72 | 91.8 | 13 | 58 | editor finished |
| 12 | +22.3 | – | 29.7% | 58.2% | 48 | 79 | 64.9 | 10 | 67 | editor finished |
| 13 | +30.1 | – | 37.5% | 53.5% | 12 | 100 | 95.6 | 10 | 86 | editor finished |
| 14 | +19.9 | – | 27.3% | 71.9% | 24 | 62 | 95.8 | 13 | 45 | editor finished |
| 15 | +17.2 | – | 24.6% | 75.4% | 1 | 104 | 121.6 | 18 | 84 | editor finished |
| 16 | +21.5 | – | 28.9% | 71.1% | 48 | 83 | 112.4 | 15 | 65 | editor finished |
| Mean | +19.7 | – | 27.1% | 71.1% | 17.5 | 80.9 | 97.7 | 13.4 | 64.8 | |
Step 5
| Rollout | Reward | Test | Edited accuracy | Hit cap | Tensors edited | Turns | Minutes | Forward passes | Python calls | Ended |
|---|
| 1 | +0.0 | – | 44.5% | 37.1% | 329 | 102 | 51.9 | 10 | 90 | editor finished |
| 2 | +0.0 | – | – | – | 0 | 62 | 39.2 | 7 | 53 | editor finished |
| 3 | +0.8 | – | 45.3% | 40.2% | 10 | 79 | 55.6 | 13 | 62 | editor finished |
| 4 | +3.1 | – | 47.7% | 17.2% | 11 | 64 | 57.3 | 14 | 47 | editor finished |
| 5 | +0.0 | – | – | – | 0 | 92 | 41.3 | 8 | 83 | editor finished |
| 6 | +3.5 | – | 48.0% | 32.4% | 1 | 97 | 59.6 | 16 | 79 | editor finished |
| 7 | +7.0 | – | 51.6% | 22.3% | 8 | 73 | 37.5 | 11 | 58 | editor finished |
| 8 | +0.0 | – | – | – | 0 | 92 | 55.7 | 13 | 77 | editor finished |
| 9 | -2.3 | – | 42.2% | 45.3% | 24 | 83 | 59.6 | 15 | 65 | editor finished |
| 10 | +0.0 | – | – | – | 0 | 92 | 46.5 | 13 | 75 | editor finished |
| 11 | +0.0 | – | – | – | 0 | 97 | 37.9 | 8 | 87 | editor finished |
| 12 | +2.7 | – | 47.3% | 17.2% | 48 | 67 | 50.6 | 14 | 51 | editor finished |
| 13 | +0.0 | – | – | – | 0 | 98 | 41.5 | 10 | 86 | editor finished |
| 14 | +0.0 | – | – | – | 0 | 75 | 50.5 | 9 | 64 | editor finished |
| 15 | +0.0 | – | – | – | 0 | 126 | 31.4 | 8 | 117 | editor finished |
| 16 | -3.5 | – | 41.0% | 35.9% | 9 | 89 | 51.5 | 15 | 71 | editor finished |
| Mean | +0.7 | – | 45.9% | 31.0% | 27.5 | 86.8 | 48.0 | 11.5 | 72.8 | |
Step 6
| Rollout | Reward | Test | Edited accuracy | Hit cap | Tensors edited | Turns | Minutes | Forward passes | Python calls | Ended |
|---|
| 1 | +0.8 | – | 45.3% | 30.1% | 24 | 102 | 58.3 | 14 | 86 | editor finished |
| 2 | +0.4 | – | 44.9% | 23.0% | 8 | 75 | 73.1 | 17 | 55 | editor finished |
| 3 | +0.0 | – | – | – | 0 | 109 | 67.1 | 17 | 90 | editor finished |
| 4 | +0.0 | – | – | – | 0 | 87 | 50.3 | 15 | 70 | editor finished |
| 5 | +0.0 | – | – | – | 0 | 91 | 49.1 | 11 | 77 | editor finished |
| 6 | +0.0 | – | – | – | 0 | 50 | 33.2 | 6 | 41 | editor finished |
| 7 | +0.0 | – | – | – | 0 | 83 | 40.9 | 7 | 74 | editor finished |
| 8 | +2.7 | – | 47.3% | 30.5% | 24 | 90 | 60.1 | 16 | 69 | editor finished |
| 9 | +1.9 | – | 46.5% | 38.3% | 1 | 99 | 59.3 | 10 | 83 | editor finished |
| 10 | +1.6 | – | 46.1% | 36.7% | 24 | 80 | 62.3 | 13 | 65 | editor finished |
| 11 | +0.0 | – | – | – | 0 | 87 | 47.0 | 8 | 76 | editor finished |
| 12 | +6.6 | – | 51.2% | 17.2% | 24 | 77 | 62.1 | 15 | 59 | editor finished |
| 13 | +0.0 | – | – | – | 0 | 84 | 45.4 | 9 | 73 | editor finished |
| 14 | -0.8 | – | 43.8% | 33.6% | 9 | 78 | 57.9 | 14 | 62 | editor finished |
| 15 | running | – | – | – | – | 1 | 9.9 | None | – | |
| 16 | +0.0 | – | – | – | 0 | 55 | 49.4 | 11 | 42 | editor finished |
| Mean of 15 finished | +0.9 | – | 46.4% | 29.9% | 7.6 | 83.1 | 54.4 | 12.2 | 68.1 | |
Step 7
| Rollout | Reward | Test | Edited accuracy | Hit cap | Tensors edited | Turns | Minutes | Forward passes | Python calls | Ended |
|---|
| 1 | +0.0 | – | – | – | 0 | 76 | 40.9 | 8 | 65 | editor finished |
| 2 | +0.0 | – | – | – | 0 | 84 | 51.2 | 12 | 69 | editor finished |
| 3 | +0.0 | – | – | – | 0 | 79 | 39.6 | 9 | 67 | editor finished |
| 4 | +0.0 | – | – | – | 0 | 90 | 53.6 | 16 | 73 | editor finished |
| 5 | +0.0 | – | – | – | 0 | 96 | 69.6 | 16 | 78 | editor finished |
| 6 | +0.0 | – | – | – | 0 | 92 | 40.0 | 10 | 80 | editor finished |
| 7 | -1.9 | – | 42.6% | 39.1% | 24 | 78 | 51.8 | 17 | 58 | editor finished |
| 8 | +0.0 | – | – | – | 0 | 85 | 24.3 | 6 | 77 | editor finished |
| 9 | +0.0 | – | – | – | 0 | 60 | 33.5 | 7 | 52 | editor finished |
| 10 | +0.0 | – | – | – | 0 | 64 | 39.6 | 10 | 52 | editor finished |
| 11 | +0.0 | – | – | – | 0 | 79 | 57.8 | 9 | 68 | editor finished |
| 12 | +1.2 | – | 45.7% | 30.1% | 24 | 91 | 64.1 | 13 | 74 | editor finished |
| 13 | +0.0 | – | – | – | 0 | 76 | 47.5 | 10 | 65 | editor finished |
| 14 | +1.9 | – | 46.5% | 24.2% | 4 | 106 | 53.3 | 14 | 87 | editor finished |
| 15 | +0.0 | – | – | – | 0 | 91 | 57.0 | 13 | 76 | editor finished |
| 16 | -1.9 | – | 42.6% | 36.3% | 24 | 114 | 51.5 | 12 | 98 | editor finished |
| Mean | -0.0 | – | 44.3% | 32.4% | 4.8 | 85.1 | 48.5 | 11.4 | 71.2 | |
Step 8
| Rollout | Reward | Test | Edited accuracy | Hit cap | Tensors edited | Turns | Minutes | Forward passes | Python calls | Ended |
|---|
| 1 | +3.1 | – | 47.7% | 33.6% | 4 | 82 | 66.2 | 15 | 64 | editor finished |
| 2 | running | – | – | – | – | 68 | 40.0 | None | – | |
| 3 | -2.7 | – | 41.8% | 38.3% | 24 | 87 | 68.8 | 16 | 69 | editor finished |
| 4 | +2.7 | – | 47.3% | 16.0% | 49 | 58 | 44.2 | 12 | 44 | editor finished |
| 5 | +0.0 | – | – | – | 0 | 117 | 95.9 | 19 | 93 | editor finished |
| 6 | +0.0 | – | – | – | 0 | 85 | 45.0 | 10 | 72 | editor finished |
| 7 | +0.0 | – | – | – | 0 | 72 | 44.2 | 9 | 60 | editor finished |
| 8 | -1.2 | – | 43.4% | 34.0% | 4 | 72 | 54.8 | 15 | 54 | editor finished |
| 9 | -1.9 | – | 42.6% | 36.3% | 24 | 67 | 63.0 | 8 | 57 | editor finished |
| 10 | +3.9 | – | 48.4% | 18.0% | 48 | 87 | 65.0 | 14 | 70 | editor finished |
| 11 | +3.9 | – | 48.4% | 28.5% | 24 | 70 | 48.9 | 14 | 54 | editor finished |
| 12 | +0.0 | – | – | – | 0 | 67 | 41.3 | 8 | 57 | editor finished |
| 13 | +6.6 | – | 51.2% | 12.1% | 8 | 96 | 72.2 | 18 | 75 | editor finished |
| 14 | +2.7 | – | 47.3% | 16.0% | 48 | 73 | 60.2 | 16 | 55 | editor finished |
| 15 | +0.0 | – | – | – | 0 | 75 | 30.8 | 6 | 67 | editor finished |
| 16 | +0.0 | – | – | – | 0 | 79 | 44.7 | 10 | 66 | editor finished |
| Mean of 15 finished | +1.1 | – | 46.4% | 25.9% | 15.5 | 79.1 | 56.3 | 12.7 | 63.8 | |
Reward: edited minus original accuracy on the reward set (256 answers, original 7.4%), in points. Test: the same on MATH-500 (original 6.6%), not used for training; scored after a step finishes, – until then. Hit cap: share of answers reaching 8,192 tokens (original 93%).