Round l2-plan-specificity
L2 plan specificity × builder strength
Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.
Round source README · rounds/l2-plan-specificity/README.md
Round date: HISTORICAL (before 2026-10-09) Publication badge: VALID; KEPT FOR AUDIT Recomputation status: NOT RECOMPUTABLE
Status
VALID
Why not VALID
- None: the README documents a pre-result registration, execution under its rule, reconciled denominators, recomputable results, matched conditions, and retained evidence.
STATUS: VALID
Data completeness: 78/78 current assigned cells have latest exact-cell official rows and are included as VALID. The separate historical Luna-max no-plan baseline contains 32 valid official rows.
Pre-registered: yes, 7 October 2026, before the first scored cell; Amendment 1 was recorded before rep 2.
Label: DESCRIPTIVE
Question: Does a more specific plan help a weaker builder more?
n: 78 assigned cells: 72 factorial cells (6 variants × 3 plan levels × 2 builders × 2 reps) plus 6 Luna-low no-plan cells.
Headline: Luna-low steps vs criteria: 3 steps-only wins, 1 criteria-only win, 8 ties; net +2/12, below the preregistered +3 adoption threshold. Do not adopt steps for cheap builders; report descriptively.
Configuration: Six fixed R70 task variants; three frozen plan levels; Luna-low and Luna-max builders; two reps per planned condition. Eighteen plans were generated once with Luna-max. Direct Codex builder CLI 0.160.0; 3,600-second cap and zero retries; planner CLI 0.160.1; Python 3.14.7. Elixir/Rust ran on kogen-bench-eu; Go/TypeScript-Bun ran on kogen-bench-us. Runner tree SHA-256: 3a6638ea3d6967f03770478039bf1f32e522ab393c32feb5df85d1b923e1ca19; wrapper SHA-256: 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300. Mac Studio remained reserved.
Limit: Pilot on 18 frozen plans, one plan per task × level; plan content and specificity are inseparable. The observed one-sided sign test is p=5/16=0.3125 over four discordant pairs; even the minimal +3 screen case has p=1/8=0.125. Low-vs-max results and the historical no-plan baseline are descriptive only; there is no interaction estimator. Confirmation needs independent plans on new tasks and contemporaneous no-plan arms.
Sources: pre-registration; decision rule; results; official cell data; planner receipts; historical no-plan baseline; reproducer.
Lifecycle counts
| Lifecycle count | Count | Scope |
|---|---|---|
| Planned | 78 | All assigned cells |
| Started | 78 | All gate and bulk cells |
| Finished | 78 | All completed runs |
| Officially graded | 78 | Latest official row per exact cell ID |
| ITT denominator | 78 | All assigned cells |
- Lifecycle counts: planned 78 / started 78 / finished 78 / graded 78 / ITT 78.
- Headline: Luna-low steps vs criteria: 3 steps-only wins, 1 criteria-only win, 8 ties; net +2/12, below the preregistered +3 adoption threshold. Do not adopt steps for cheap builders; report descriptively.
- Luna-low descriptive full-pass rates: none 1/6; criteria 4/12; approach 4/12; steps 6/12.
- Luna-max descriptive full-pass rates: criteria 6/12; approach 7/12; steps 6/12; historical no-plan baseline 20/32.
- Sign-test caveat: observed one-sided exact p=5/16=0.3125 over 4 discordant pairs; the minimal +3 case (3 wins, 0 losses) gives p=1/8=0.125.
- Grading provenance: the final 2 EU rep-2 cells (
r70-4-elixir-fe2-l2criteria,r70-4-elixir-fe2-l2steps) used grade_worker SHA 0e6f8aa0 instead of 73dc09af; the change was inventory-glob-only and scoring code was byte-identical.
Reproduce
All exact task IDs are in the cell table in RESULTS.md. The six base task IDs are r70-2-elixir, r70-2-go, r70-7-rust, r70-4-elixir-fe2, r70-4-go-fe2, and r70-4-ts-bun-fe2.
Kogen commit: the task-specific commit map below lists every base commit used by the current cells; the first linked commit is 470ba65c1d3c19191a2190a032cd9ebf24e71975.
Harness commit: not recorded as a Git commit. Runner tree SHA-256: 3a6638ea3d6967f03770478039bf1f32e522ab393c32feb5df85d1b923e1ca19; wrapper SHA-256: 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300.
Model and effort: direct Codex gpt-6-luna at low and max for builders; gpt-6-luna at max for plan generation. Builder CLI codex-cli 0.160.0; planner CLI codex-cli 0.160.1; Python 3.14.7.
Task IDs: six base task IDs above and 18 plan-bearing task IDs in the commit map.
Command: plan generation used the command template below; scored dispatches used the four gate/bulk commands below; official grading used the operator-run MacBook route r70-macbook-window-v1. Analysis command: python3 reproduce/l2-plan-specificity.py.
Raw records: sanitized official rows and planner receipts are included in the data files below; latest row per exact cell ID is used.
Plan-generation command template (run from each public input directory):
codex exec --ephemeral --ignore-user-config --skip-git-repo-check --sandbox read-only --model gpt-6-luna -c model_reasoning_effort="max" --cd <public-input-dir> --json --output-last-message <plan.md> -
Scored dispatch commands (run from the L2 lane directory):
python3 dispatch.py us gate
python3 dispatch.py eu gate
python3 dispatch.py us bulk
python3 dispatch.py eu bulk
Data files: rounds/l2-plan-specificity/data/official_cells.csv, rounds/l2-plan-specificity/data/planner_usage.csv, and rounds/l2-plan-specificity/data/historical_no_plan_max.csv. The standard-library reproducer recomputes the result page and checks the summary and commit map in this README.
| Task ID | Plan level | Kogen commit | Plan SHA-256 |
|---|---|---|---|
r70-2-elixir |
none | `470ba65c1d3c19191a2190a032cd9ebf24e71975` | — |
r70-2-elixir-l2approach |
approach | `0c6f95804e901912ba40c256105c49650dade6bd` | 4986e20ea03a06466030cd3e40a613a34f6fd7f541d4fb92e103439ea2505f2f |
r70-2-elixir-l2criteria |
criteria | `2c6ee03a749151a28aa0e908de4f1474017b411f` | e3f054f0bf8335d286581ea716825cb540e31d6333a78935e759feab3ccd1e2d |
r70-2-elixir-l2steps |
steps | `0cabbafd45cf324219747a2fa97968ae262ab8b2` | 75b11ce37398535be44d17272c0dbcf55fa7791f93863f4f7bd603df82250a46 |
r70-2-go |
none | `13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82` | — |
r70-2-go-l2approach |
approach | `1db582020ff97c48c854feee9b4588161323cef4` | 707ff42bb366fad3e83a1c9ba929f7ad15613195e06c078c6678c05e81affcd6 |
r70-2-go-l2criteria |
criteria | `48672b705c40c6232023a63a697f44e12fd7a7d0` | 92799bbf4b92d915b52b4b3b787dab9bb659380eb196998c41955887fd19e511 |
r70-2-go-l2steps |
steps | `d08c7c9d38b87416c47fb648f472d7524d1ff9d0` | 9f7dfe458cf2d0e7ce83747a8a1f6bd657f23b467c5173ef5e4c656448ebdf87 |
r70-4-elixir-fe2 |
none | `5ccd0bf1e816e2b2f5b2694861f8596402409f52` | — |
r70-4-elixir-fe2-l2approach |
approach | `f9a5948042d525d20729f9346770ca185bcecb9d` | c8963e6d6421b153ce56a9c1389ab1af5fb0c434413d14505dda63c143fc207d |
r70-4-elixir-fe2-l2criteria |
criteria | `634d71701ded3a7e4f698650e5bb9fea71fce589` | 950ac4033b51b7cdffc9b7f4cf1e161d984acf84c83cc7f3949a43b773be6497 |
r70-4-elixir-fe2-l2steps |
steps | `3d1e48fab23a0e2b01fa5051cc1c31a2f8611052` | 2515e7b8cec8a619dd955401c2b74a9aa9ed3c68a0b450aa1068b43f4f83eadc |
r70-4-go-fe2 |
none | `21d21a2f420a387c921f998e5b4d16efc8f80904` | — |
r70-4-go-fe2-l2approach |
approach | `9c2f20f38404d4cc5906268c5b68357e11902c97` | dd12e95f67c0388c51a1626bc0d46f1850face51b0209a279624a0baf0013cb5 |
r70-4-go-fe2-l2criteria |
criteria | `1524f0aeb36233c80dd86e635a550dc2b8762995` | 90f4b65162e54d422cb26bd5c2890eb13224d92d60678e5ff95f11c75e6d1d80 |
r70-4-go-fe2-l2steps |
steps | `9c40d5ead25b9bda63c12fb4919c836d12c415ec` | 104853e7dc867ae0e2e41c7c3abc095d97b1d5398a27da4c590fd94da793ab6f |
r70-4-ts-bun-fe2 |
none | `cc3a831a167020538c4ca2ad90185606a2e4ad6a` | — |
r70-4-ts-bun-fe2-l2approach |
approach | `de4007f44b3115aa79c94bf62c09067d839ef7ec` | ddb365943f53b51f148cad60d5c4679d3dba69d398cf63e7b9ddbd8c79dc24b1 |
r70-4-ts-bun-fe2-l2criteria |
criteria | `c977e8ea8ce3cd817a87fb32c005cf65a2e7c718` | e0c0149b8a6c11a0ff745ae7218dade16ae9ea58420a392a3d17ee680e7e2a41 |
r70-4-ts-bun-fe2-l2steps |
steps | `2dbbffb9759ff1372ff1087951c3332aafd57bf4` | b19f9b6e352b9ab3e099ab510eddd14cd6743c0d2765d99df43282d9bba71ce8 |
r70-7-rust |
none | `111a0c776ae6d93f6e7fccd3fc694d5f4fa26838` | — |
r70-7-rust-l2approach |
approach | `7eb195d8474713c66fcd3173b064d801876c367e` | d14c4377010fd3ccb8e5688528d2d36454821e0db9f559dd41d301cc527e2e50 |
r70-7-rust-l2criteria |
criteria | `538de74aefb8bd061438a00da684aeec2b3e8a5b` | 6415b150f9a16886a7328d4d847228c5335e5cb26c39d590276f12e3b182eef2 |
r70-7-rust-l2steps |
steps | `c70927abbf58f038226c0f83a24da9edae0c00d0` | 53ae845814f28c4758e3ddf7d2f7c0fee434b7e490c69b1964afc1d6f6b7c7e5 |
Historical Luna-max no-plan baseline commits:
| Task ID | Official cell IDs | Kogen commit |
|---|---|---|
r70-2-elixir |
codex__gpt-6-luna__max__default__r70-2-elixir__r31, codex__gpt-6-luna__max__default__r70-2-elixir__r32, codex__gpt-6-luna__max__default__r70-2-elixir__r33, codex__gpt-6-luna__max__default__r70-2-elixir__r910 |
`470ba65c1d3c19191a2190a032cd9ebf24e71975` |
r70-2-go |
codex__gpt-6-luna__max__default__r70-2-go__r31, codex__gpt-6-luna__max__default__r70-2-go__r32, codex__gpt-6-luna__max__default__r70-2-go__r33, codex__gpt-6-luna__max__default__r70-2-go__r911 |
`13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82` |
r70-4-elixir-fe2 |
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10, codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13, codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9 |
`5ccd0bf1e816e2b2f5b2694861f8596402409f52` |
r70-4-elixir-fe2 |
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r909, codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r910 |
`a43dbb032d795157c5a93976947637502b30fece` |
r70-4-go-fe2 |
codex__gpt-6-luna__max__default__r70-4-go-fe2__r10, codex__gpt-6-luna__max__default__r70-4-go-fe2__r13, codex__gpt-6-luna__max__default__r70-4-go-fe2__r9 |
`21d21a2f420a387c921f998e5b4d16efc8f80904` |
r70-4-go-fe2 |
codex__gpt-6-luna__max__default__r70-4-go-fe2__r820, codex__gpt-6-luna__max__default__r70-4-go-fe2__r823, codex__gpt-6-luna__max__default__r70-4-go-fe2__r919 |
`748ebe3a3bc03a112dab83e785159654e0709f9e` |
r70-4-ts-bun-fe2 |
codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r819, codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r920, codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r923 |
`880afb00474fe33e9cf38a6d70016ddcd4e38e81` |
r70-4-ts-bun-fe2 |
codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r10, codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r13, codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9 |
`cc3a831a167020538c4ca2ad90185606a2e4ad6a` |
r70-7-rust |
codex__gpt-6-luna__max__default__r70-7-rust__r1, codex__gpt-6-luna__max__default__r70-7-rust__r2, codex__gpt-6-luna__max__default__r70-7-rust__r3, codex__gpt-6-luna__max__default__r70-7-rust__r31, codex__gpt-6-luna__max__default__r70-7-rust__r32, codex__gpt-6-luna__max__default__r70-7-rust__r33, codex__gpt-6-luna__max__default__r70-7-rust__r905 |
`111a0c776ae6d93f6e7fccd3fc694d5f4fa26838` |
Recomputed from raw records
`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 39 rows (fail 22, pass 17); overall pass rate is 43.6% (17/39) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.
No shared cell IDs are present in the committed public grade export; the source-only cohort is checked against the page's reported figures below. No shared public-export cell IDs are available for a paired outcome comparison. No complete published metadata/manifest token pairs are available for this round (0 missing manifest usage; 39 missing published metadata). Missing counters remain unknown, not zero. This round also has 39 raw-only or non-public rows; they are kept separate from the published-export denominator.
Round results · rounds/l2-plan-specificity/RESULTS.md
STATUS: VALID
The current cohort has 78/78 officially graded exact cell IDs: 42 gate cells and 36 rep-2 bulk cells. It includes 72 planned factorial cells and six Luna-low no-plan cells. Every included current row is labeled VALID; the historical Luna-max no-plan baseline is a separate descriptive cohort.
Per-cell outcomes and builder tokens
Token rule: uncached input + cached input + output; total = the sum of those three components. Builder tokens below include the provided frozen plan in the builder prompt. No hidden grader details are included.
| Task / variant | Level | Builder | Rep | Stage | Status | Outcome | Cell ID | Uncached | Cached | Output | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|
r70-2-elixir |
none | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-elixir__r1 |
31,788 | 231,168 | 4,187 | 267,143 |
r70-2-elixir-l2approach |
approach | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-elixir-l2approach__r1 |
20,935 | 88,064 | 1,858 | 110,857 |
r70-2-elixir-l2approach |
approach | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-elixir-l2approach__r2 |
15,882 | 233,472 | 3,849 | 253,203 |
r70-2-elixir-l2approach |
approach | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-2-elixir-l2approach__r1 |
209,777 | 9,861,120 | 84,230 | 10,155,127 |
r70-2-elixir-l2approach |
approach | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-2-elixir-l2approach__r2 |
317,097 | 8,334,848 | 106,857 | 8,758,802 |
r70-2-elixir-l2criteria |
criteria | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-elixir-l2criteria__r1 |
83,829 | 1,040,896 | 7,757 | 1,132,482 |
r70-2-elixir-l2criteria |
criteria | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-elixir-l2criteria__r2 |
97,041 | 1,399,552 | 13,443 | 1,510,036 |
r70-2-elixir-l2criteria |
criteria | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-2-elixir-l2criteria__r1 |
177,269 | 7,560,704 | 75,061 | 7,813,034 |
r70-2-elixir-l2criteria |
criteria | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-2-elixir-l2criteria__r2 |
312,914 | 18,456,064 | 123,669 | 18,892,647 |
r70-2-elixir-l2steps |
steps | Luna-low | 1 | gate | VALID | PASS | codex__gpt-6-luna__low__default__r70-2-elixir-l2steps__r1 |
57,881 | 576,512 | 8,200 | 642,593 |
r70-2-elixir-l2steps |
steps | Luna-low | 2 | bulk | VALID | PASS | codex__gpt-6-luna__low__default__r70-2-elixir-l2steps__r2 |
64,917 | 714,752 | 15,237 | 794,906 |
r70-2-elixir-l2steps |
steps | Luna-max | 1 | gate | VALID | FAIL | codex__gpt-6-luna__max__default__r70-2-elixir-l2steps__r1 |
260,423 | 8,961,792 | 113,232 | 9,335,447 |
r70-2-elixir-l2steps |
steps | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-2-elixir-l2steps__r2 |
194,177 | 9,452,288 | 78,703 | 9,725,168 |
r70-2-go |
none | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-go__r1 |
15,864 | 153,600 | 3,204 | 172,668 |
r70-2-go-l2approach |
approach | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-go-l2approach__r1 |
16,611 | 142,592 | 2,700 | 161,903 |
r70-2-go-l2approach |
approach | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-go-l2approach__r2 |
19,744 | 224,256 | 3,866 | 247,866 |
r70-2-go-l2approach |
approach | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-2-go-l2approach__r1 |
66,021 | 916,480 | 35,961 | 1,018,462 |
r70-2-go-l2approach |
approach | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-2-go-l2approach__r2 |
47,881 | 578,304 | 31,512 | 657,697 |
r70-2-go-l2criteria |
criteria | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-go-l2criteria__r1 |
14,676 | 83,200 | 2,430 | 100,306 |
r70-2-go-l2criteria |
criteria | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-go-l2criteria__r2 |
20,598 | 105,216 | 2,620 | 128,434 |
r70-2-go-l2criteria |
criteria | Luna-max | 1 | gate | VALID | FAIL | codex__gpt-6-luna__max__default__r70-2-go-l2criteria__r1 |
47,033 | 777,216 | 29,716 | 853,965 |
r70-2-go-l2criteria |
criteria | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-2-go-l2criteria__r2 |
52,090 | 684,032 | 27,313 | 763,435 |
r70-2-go-l2steps |
steps | Luna-low | 1 | gate | VALID | PASS | codex__gpt-6-luna__low__default__r70-2-go-l2steps__r1 |
45,413 | 204,800 | 4,095 | 254,308 |
r70-2-go-l2steps |
steps | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-2-go-l2steps__r2 |
17,136 | 147,712 | 4,358 | 169,206 |
r70-2-go-l2steps |
steps | Luna-max | 1 | gate | VALID | FAIL | codex__gpt-6-luna__max__default__r70-2-go-l2steps__r1 |
52,186 | 292,352 | 21,989 | 366,527 |
r70-2-go-l2steps |
steps | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-2-go-l2steps__r2 |
65,768 | 644,096 | 28,362 | 738,226 |
r70-4-elixir-fe2 |
none | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-elixir-fe2__r1 |
25,119 | 185,856 | 3,184 | 214,159 |
r70-4-elixir-fe2-l2approach |
approach | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2approach__r1 |
32,734 | 432,640 | 4,628 | 470,002 |
r70-4-elixir-fe2-l2approach |
approach | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2approach__r2 |
18,192 | 229,632 | 3,824 | 251,648 |
r70-4-elixir-fe2-l2approach |
approach | Luna-max | 1 | gate | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2approach__r1 |
107,325 | 1,471,744 | 35,803 | 1,614,872 |
r70-4-elixir-fe2-l2approach |
approach | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2approach__r2 |
71,983 | 592,896 | 19,273 | 684,152 |
r70-4-elixir-fe2-l2criteria |
criteria | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2criteria__r1 |
57,723 | 331,264 | 5,585 | 394,572 |
r70-4-elixir-fe2-l2criteria |
criteria | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2criteria__r2 |
31,917 | 281,088 | 3,612 | 316,617 |
r70-4-elixir-fe2-l2criteria |
criteria | Luna-max | 1 | gate | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2criteria__r1 |
48,351 | 548,864 | 24,040 | 621,255 |
r70-4-elixir-fe2-l2criteria |
criteria | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2criteria__r2 |
58,537 | 911,360 | 28,170 | 998,067 |
r70-4-elixir-fe2-l2steps |
steps | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2steps__r1 |
25,233 | 159,744 | 2,769 | 187,746 |
r70-4-elixir-fe2-l2steps |
steps | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2steps__r2 |
117,662 | 1,199,872 | 13,422 | 1,330,956 |
r70-4-elixir-fe2-l2steps |
steps | Luna-max | 1 | gate | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2steps__r1 |
60,230 | 881,408 | 27,388 | 969,026 |
r70-4-elixir-fe2-l2steps |
steps | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2steps__r2 |
59,480 | 869,120 | 26,787 | 955,387 |
r70-4-go-fe2 |
none | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-go-fe2__r1 |
16,741 | 106,240 | 2,315 | 125,296 |
r70-4-go-fe2-l2approach |
approach | Luna-low | 1 | gate | VALID | PASS | codex__gpt-6-luna__low__default__r70-4-go-fe2-l2approach__r1 |
15,838 | 177,920 | 4,281 | 198,039 |
r70-4-go-fe2-l2approach |
approach | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-go-fe2-l2approach__r2 |
14,730 | 140,544 | 2,722 | 157,996 |
r70-4-go-fe2-l2approach |
approach | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-4-go-fe2-l2approach__r1 |
59,420 | 863,488 | 25,418 | 948,326 |
r70-4-go-fe2-l2approach |
approach | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-4-go-fe2-l2approach__r2 |
36,397 | 399,616 | 20,173 | 456,186 |
r70-4-go-fe2-l2criteria |
criteria | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-go-fe2-l2criteria__r1 |
4,732 | 33,024 | 216 | 37,972 |
r70-4-go-fe2-l2criteria |
criteria | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-go-fe2-l2criteria__r2 |
21,856 | 184,832 | 2,994 | 209,682 |
r70-4-go-fe2-l2criteria |
criteria | Luna-max | 1 | gate | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-go-fe2-l2criteria__r1 |
89,064 | 1,393,664 | 41,067 | 1,523,795 |
r70-4-go-fe2-l2criteria |
criteria | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-4-go-fe2-l2criteria__r2 |
47,152 | 708,096 | 26,708 | 781,956 |
r70-4-go-fe2-l2steps |
steps | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-go-fe2-l2steps__r1 |
24,968 | 285,696 | 3,857 | 314,521 |
r70-4-go-fe2-l2steps |
steps | Luna-low | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-go-fe2-l2steps__r2 |
13,047 | 174,848 | 2,341 | 190,236 |
r70-4-go-fe2-l2steps |
steps | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-4-go-fe2-l2steps__r1 |
55,099 | 691,968 | 25,790 | 772,857 |
r70-4-go-fe2-l2steps |
steps | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-4-go-fe2-l2steps__r2 |
71,562 | 908,288 | 32,930 | 1,012,780 |
r70-4-ts-bun-fe2 |
none | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2__r1 |
22,848 | 103,424 | 1,757 | 128,029 |
r70-4-ts-bun-fe2-l2approach |
approach | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2approach__r1 |
11,670 | 85,248 | 1,873 | 98,791 |
r70-4-ts-bun-fe2-l2approach |
approach | Luna-low | 2 | bulk | VALID | PASS | codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2approach__r2 |
27,467 | 258,560 | 3,861 | 289,888 |
r70-4-ts-bun-fe2-l2approach |
approach | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2approach__r1 |
48,507 | 518,656 | 28,377 | 595,540 |
r70-4-ts-bun-fe2-l2approach |
approach | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2approach__r2 |
58,375 | 718,592 | 28,704 | 805,671 |
r70-4-ts-bun-fe2-l2criteria |
criteria | Luna-low | 1 | gate | VALID | PASS | codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2criteria__r1 |
18,685 | 231,424 | 2,714 | 252,823 |
r70-4-ts-bun-fe2-l2criteria |
criteria | Luna-low | 2 | bulk | VALID | PASS | codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2criteria__r2 |
20,882 | 130,560 | 2,598 | 154,040 |
r70-4-ts-bun-fe2-l2criteria |
criteria | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2criteria__r1 |
50,249 | 235,776 | 16,232 | 302,257 |
r70-4-ts-bun-fe2-l2criteria |
criteria | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2criteria__r2 |
50,576 | 558,592 | 25,881 | 635,049 |
r70-4-ts-bun-fe2-l2steps |
steps | Luna-low | 1 | gate | VALID | FAIL | codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2steps__r1 |
26,853 | 103,424 | 1,780 | 132,057 |
r70-4-ts-bun-fe2-l2steps |
steps | Luna-low | 2 | bulk | VALID | PASS | codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2steps__r2 |
17,101 | 198,400 | 2,792 | 218,293 |
r70-4-ts-bun-fe2-l2steps |
steps | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2steps__r1 |
35,638 | 237,824 | 21,900 | 295,362 |
r70-4-ts-bun-fe2-l2steps |
steps | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2steps__r2 |
46,346 | 620,032 | 25,218 | 691,596 |
r70-7-rust |
none | Luna-low | 1 | gate | VALID | PASS | codex__gpt-6-luna__low__default__r70-7-rust__r1 |
27,266 | 149,760 | 3,384 | 180,410 |
r70-7-rust-l2approach |
approach | Luna-low | 1 | gate | VALID | PASS | codex__gpt-6-luna__low__default__r70-7-rust-l2approach__r1 |
13,258 | 119,552 | 3,324 | 136,134 |
r70-7-rust-l2approach |
approach | Luna-low | 2 | bulk | VALID | PASS | codex__gpt-6-luna__low__default__r70-7-rust-l2approach__r2 |
33,481 | 166,144 | 5,522 | 205,147 |
r70-7-rust-l2approach |
approach | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-7-rust-l2approach__r1 |
46,385 | 220,928 | 15,984 | 283,297 |
r70-7-rust-l2approach |
approach | Luna-max | 2 | bulk | VALID | FAIL | codex__gpt-6-luna__max__default__r70-7-rust-l2approach__r2 |
27,476 | 220,160 | 13,087 | 260,723 |
r70-7-rust-l2criteria |
criteria | Luna-low | 1 | gate | VALID | PASS | codex__gpt-6-luna__low__default__r70-7-rust-l2criteria__r1 |
24,048 | 189,184 | 3,503 | 216,735 |
r70-7-rust-l2criteria |
criteria | Luna-low | 2 | bulk | VALID | PASS | codex__gpt-6-luna__low__default__r70-7-rust-l2criteria__r2 |
18,204 | 123,648 | 2,959 | 144,811 |
r70-7-rust-l2criteria |
criteria | Luna-max | 1 | gate | VALID | FAIL | codex__gpt-6-luna__max__default__r70-7-rust-l2criteria__r1 |
41,237 | 293,376 | 16,574 | 351,187 |
r70-7-rust-l2criteria |
criteria | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-7-rust-l2criteria__r2 |
65,051 | 375,552 | 15,298 | 455,901 |
r70-7-rust-l2steps |
steps | Luna-low | 1 | gate | VALID | PASS | codex__gpt-6-luna__low__default__r70-7-rust-l2steps__r1 |
31,485 | 222,720 | 3,563 | 257,768 |
r70-7-rust-l2steps |
steps | Luna-low | 2 | bulk | VALID | PASS | codex__gpt-6-luna__low__default__r70-7-rust-l2steps__r2 |
29,076 | 147,712 | 4,382 | 181,170 |
r70-7-rust-l2steps |
steps | Luna-max | 1 | gate | VALID | PASS | codex__gpt-6-luna__max__default__r70-7-rust-l2steps__r1 |
43,185 | 441,344 | 19,205 | 503,734 |
r70-7-rust-l2steps |
steps | Luna-max | 2 | bulk | VALID | PASS | codex__gpt-6-luna__max__default__r70-7-rust-l2steps__r2 |
39,815 | 371,200 | 16,648 | 427,663 |
Descriptive full-task outcomes
- Luna-low no-plan cells: 1/6.
- Luna-low criteria: 4/12; approach: 4/12; steps: 6/12.
- Luna-max criteria: 6/12; approach: 7/12; steps: 6/12.
- Historical Luna-max no-plan baseline: 20/32 valid official cells; this is unrandomized historical context.
Primary Luna-low paired comparison
| Measure | Steps vs criteria |
|---|---|
| Steps-only wins | 3 |
| Criteria-only wins | 1 |
| Ties | 8 |
| Net (wins − losses) | 2 |
| Source task | Rep | Steps | Criteria | Pair result |
|---|---|---|---|---|
r70-2-elixir |
1 | PASS | FAIL | steps win |
r70-2-elixir |
2 | PASS | FAIL | steps win |
r70-2-go |
1 | PASS | FAIL | steps win |
r70-2-go |
2 | FAIL | FAIL | tie |
r70-4-elixir-fe2 |
1 | FAIL | FAIL | tie |
r70-4-elixir-fe2 |
2 | FAIL | FAIL | tie |
r70-4-go-fe2 |
1 | FAIL | FAIL | tie |
r70-4-go-fe2 |
2 | FAIL | FAIL | tie |
r70-4-ts-bun-fe2 |
1 | FAIL | PASS | criteria win |
r70-4-ts-bun-fe2 |
2 | PASS | PASS | tie |
r70-7-rust |
1 | PASS | PASS | tie |
r70-7-rust |
2 | PASS | PASS | tie |
The preregistered adoption screen required net +3 across 12 pairs and lower total tokens per official pass than criteria. The observed net was +2; steps do not meet the adoption screen. The observed one-sided exact sign test, excluding ties, is p = 5/16 = 0.3125 (3 wins among 4 discordant pairs). The minimal +3 screen outcome of three wins and zero losses would still have p = 1/8 = 0.125; treat this as a pilot screen, not confirmatory evidence.
Planner receipts
The 18 frozen-plan generations used Luna-max. Receipt components sum to uncached 138,159, cached 694,784, output 54,278, total 887,221 tokens.
Tokens per official pass
Each plan receipt is included in the cost view. Amortized charges allocate one plan receipt over its four planned uses (two builder efforts × two reps). One-off charges assign the full receipt to each task-level scored result. Cost totals keep uncached input, cached input, and output separate; no dollar conversion or wall-time combination is used.
Amortized planning cost
- criteria / Luna-low: 4 of 12 cells passed; builder uncached/cached/output/total = 414,191/4,133,888/50,431/4,598,510; plan charge uncached/cached/output/total = 21,434.5/116,992/5,899.5/144,326; combined charged uncached/cached/output/total = 435,625.5/4,250,880/56,330.5/4,742,836; tokens per official full-task pass = 1,185,709.
- criteria / Luna-max: 6 of 12 cells passed; builder uncached/cached/output/total = 1,039,523/32,503,296/449,729/33,992,548; plan charge uncached/cached/output/total = 21,434.5/116,992/5,899.5/144,326; combined charged uncached/cached/output/total = 1,060,957.5/32,620,288/455,628.5/34,136,874; tokens per official full-task pass = 5,689,479.
- approach / Luna-low: 4 of 12 cells passed; builder uncached/cached/output/total = 240,542/2,298,624/42,308/2,581,474; plan charge uncached/cached/output/total = 27,432.5/112,384/7,984/147,800.5; combined charged uncached/cached/output/total = 267,974.5/2,411,008/50,292/2,729,274.5; tokens per official full-task pass = 682,318.6.
- approach / Luna-max: 7 of 12 cells passed; builder uncached/cached/output/total = 1,096,644/24,696,832/445,379/26,238,855; plan charge uncached/cached/output/total = 27,432.5/112,384/7,984/147,800.5; combined charged uncached/cached/output/total = 1,124,076.5/24,809,216/453,363/26,386,655.5; tokens per official full-task pass = 3,769,522.2.
- steps / Luna-low: 6 of 12 cells passed; builder uncached/cached/output/total = 470,772/4,136,192/66,796/4,673,760; plan charge uncached/cached/output/total = 20,212.5/118,016/13,255.5/151,484; combined charged uncached/cached/output/total = 490,984.5/4,254,208/80,051.5/4,825,244; tokens per official full-task pass = 804,207.3.
- steps / Luna-max: 6 of 12 cells passed; builder uncached/cached/output/total = 983,909/24,371,712/438,152/25,793,773; plan charge uncached/cached/output/total = 20,212.5/118,016/13,255.5/151,484; combined charged uncached/cached/output/total = 1,004,121.5/24,489,728/451,407.5/25,945,257; tokens per official full-task pass = 4,324,209.5.
One-off planning cost
- criteria / Luna-low: 4 of 12 cells passed; builder uncached/cached/output/total = 414,191/4,133,888/50,431/4,598,510; plan charge uncached/cached/output/total = 85,738/467,968/23,598/577,304; combined charged uncached/cached/output/total = 499,929/4,601,856/74,029/5,175,814; tokens per official full-task pass = 1,293,953.5.
- criteria / Luna-max: 6 of 12 cells passed; builder uncached/cached/output/total = 1,039,523/32,503,296/449,729/33,992,548; plan charge uncached/cached/output/total = 85,738/467,968/23,598/577,304; combined charged uncached/cached/output/total = 1,125,261/32,971,264/473,327/34,569,852; tokens per official full-task pass = 5,761,642.
- approach / Luna-low: 4 of 12 cells passed; builder uncached/cached/output/total = 240,542/2,298,624/42,308/2,581,474; plan charge uncached/cached/output/total = 109,730/449,536/31,936/591,202; combined charged uncached/cached/output/total = 350,272/2,748,160/74,244/3,172,676; tokens per official full-task pass = 793,169.
- approach / Luna-max: 7 of 12 cells passed; builder uncached/cached/output/total = 1,096,644/24,696,832/445,379/26,238,855; plan charge uncached/cached/output/total = 109,730/449,536/31,936/591,202; combined charged uncached/cached/output/total = 1,206,374/25,146,368/477,315/26,830,057; tokens per official full-task pass = 3,832,865.3.
- steps / Luna-low: 6 of 12 cells passed; builder uncached/cached/output/total = 470,772/4,136,192/66,796/4,673,760; plan charge uncached/cached/output/total = 80,850/472,064/53,022/605,936; combined charged uncached/cached/output/total = 551,622/4,608,256/119,818/5,279,696; tokens per official full-task pass = 879,949.3.
- steps / Luna-max: 6 of 12 cells passed; builder uncached/cached/output/total = 983,909/24,371,712/438,152/25,793,773; plan charge uncached/cached/output/total = 80,850/472,064/53,022/605,936; combined charged uncached/cached/output/total = 1,064,759/24,843,776/491,174/26,399,709; tokens per official full-task pass = 4,399,951.5.
No-plan cost context
Current Luna-low no-plan cells: 1/6 passed; uncached/cached/output/total = 139,626/930,048/18,031/1,087,705; tokens per official full-task pass = 1,087,705. No planner charge applies, so both planning-cost views coincide. The historical Luna-max no-plan baseline is reported for outcome context only and is not mixed into the L2 planning-cost comparison.
Grading provenance note
The final two EU rep-2 cells, r70-4-elixir-fe2-l2criteria and r70-4-elixir-fe2-l2steps, were officially graded with grade_worker SHA prefix 0e6f8aa0 rather than 73dc09af. The difference was an inventory-glob-only change; scoring code was byte-identical. The result rows above use the latest official row for each exact cell ID.
Captured run records
Standard-schema records assigned to this round in the indexed snapshot.
No per-round file is listed. Check the run-record index for the complete partition list.
Reproduction inventory
Published analysis replay and full experiment replay are reported separately.
- Replay type
analysis-only- Script
- reproduce/l2-plan-specificity.py
python3 reproduce/l2-plan-specificity.py - Inputs
- Full execution replay
- Unavailable
- Scope
- Recomputes all published numeric results from the sanitized official-cell, planner-receipt, and historical-baseline CSVs. Sandbox execution and official grading cannot be replayed from this public bundle.
View the complete per-round reproduction inventory · Repository reproduction guide