Round l2-plan-specificity

L2 plan specificity × builder strength

Status · VALIDReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/l2-plan-specificity/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: VALID; KEPT FOR AUDIT Recomputation status: NOT RECOMPUTABLE

Status

VALID

Why not VALID

  • None: the README documents a pre-result registration, execution under its rule, reconciled denominators, recomputable results, matched conditions, and retained evidence.

STATUS: VALID

Data completeness: 78/78 current assigned cells have latest exact-cell official rows and are included as VALID. The separate historical Luna-max no-plan baseline contains 32 valid official rows.

Pre-registered: yes, 7 October 2026, before the first scored cell; Amendment 1 was recorded before rep 2. Label: DESCRIPTIVE Question: Does a more specific plan help a weaker builder more? n: 78 assigned cells: 72 factorial cells (6 variants × 3 plan levels × 2 builders × 2 reps) plus 6 Luna-low no-plan cells. Headline: Luna-low steps vs criteria: 3 steps-only wins, 1 criteria-only win, 8 ties; net +2/12, below the preregistered +3 adoption threshold. Do not adopt steps for cheap builders; report descriptively. Configuration: Six fixed R70 task variants; three frozen plan levels; Luna-low and Luna-max builders; two reps per planned condition. Eighteen plans were generated once with Luna-max. Direct Codex builder CLI 0.160.0; 3,600-second cap and zero retries; planner CLI 0.160.1; Python 3.14.7. Elixir/Rust ran on kogen-bench-eu; Go/TypeScript-Bun ran on kogen-bench-us. Runner tree SHA-256: 3a6638ea3d6967f03770478039bf1f32e522ab393c32feb5df85d1b923e1ca19; wrapper SHA-256: 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300. Mac Studio remained reserved. Limit: Pilot on 18 frozen plans, one plan per task × level; plan content and specificity are inseparable. The observed one-sided sign test is p=5/16=0.3125 over four discordant pairs; even the minimal +3 screen case has p=1/8=0.125. Low-vs-max results and the historical no-plan baseline are descriptive only; there is no interaction estimator. Confirmation needs independent plans on new tasks and contemporaneous no-plan arms. Sources: pre-registration; decision rule; results; official cell data; planner receipts; historical no-plan baseline; reproducer.

Lifecycle counts

Lifecycle count Count Scope
Planned 78 All assigned cells
Started 78 All gate and bulk cells
Finished 78 All completed runs
Officially graded 78 Latest official row per exact cell ID
ITT denominator 78 All assigned cells
  • Lifecycle counts: planned 78 / started 78 / finished 78 / graded 78 / ITT 78.
  • Headline: Luna-low steps vs criteria: 3 steps-only wins, 1 criteria-only win, 8 ties; net +2/12, below the preregistered +3 adoption threshold. Do not adopt steps for cheap builders; report descriptively.
  • Luna-low descriptive full-pass rates: none 1/6; criteria 4/12; approach 4/12; steps 6/12.
  • Luna-max descriptive full-pass rates: criteria 6/12; approach 7/12; steps 6/12; historical no-plan baseline 20/32.
  • Sign-test caveat: observed one-sided exact p=5/16=0.3125 over 4 discordant pairs; the minimal +3 case (3 wins, 0 losses) gives p=1/8=0.125.
  • Grading provenance: the final 2 EU rep-2 cells (r70-4-elixir-fe2-l2criteria, r70-4-elixir-fe2-l2steps) used grade_worker SHA 0e6f8aa0 instead of 73dc09af; the change was inventory-glob-only and scoring code was byte-identical.

Reproduce

All exact task IDs are in the cell table in RESULTS.md. The six base task IDs are r70-2-elixir, r70-2-go, r70-7-rust, r70-4-elixir-fe2, r70-4-go-fe2, and r70-4-ts-bun-fe2.

Kogen commit: the task-specific commit map below lists every base commit used by the current cells; the first linked commit is 470ba65c1d3c19191a2190a032cd9ebf24e71975. Harness commit: not recorded as a Git commit. Runner tree SHA-256: 3a6638ea3d6967f03770478039bf1f32e522ab393c32feb5df85d1b923e1ca19; wrapper SHA-256: 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300. Model and effort: direct Codex gpt-6-luna at low and max for builders; gpt-6-luna at max for plan generation. Builder CLI codex-cli 0.160.0; planner CLI codex-cli 0.160.1; Python 3.14.7. Task IDs: six base task IDs above and 18 plan-bearing task IDs in the commit map. Command: plan generation used the command template below; scored dispatches used the four gate/bulk commands below; official grading used the operator-run MacBook route r70-macbook-window-v1. Analysis command: python3 reproduce/l2-plan-specificity.py. Raw records: sanitized official rows and planner receipts are included in the data files below; latest row per exact cell ID is used.

Plan-generation command template (run from each public input directory):

codex exec --ephemeral --ignore-user-config --skip-git-repo-check --sandbox read-only --model gpt-6-luna -c model_reasoning_effort="max" --cd <public-input-dir> --json --output-last-message <plan.md> -

Scored dispatch commands (run from the L2 lane directory):

python3 dispatch.py us gate
python3 dispatch.py eu gate
python3 dispatch.py us bulk
python3 dispatch.py eu bulk

Data files: rounds/l2-plan-specificity/data/official_cells.csv, rounds/l2-plan-specificity/data/planner_usage.csv, and rounds/l2-plan-specificity/data/historical_no_plan_max.csv. The standard-library reproducer recomputes the result page and checks the summary and commit map in this README.

Task ID Plan level Kogen commit Plan SHA-256
r70-2-elixir none `470ba65c1d3c19191a2190a032cd9ebf24e71975` —
r70-2-elixir-l2approach approach `0c6f95804e901912ba40c256105c49650dade6bd` 4986e20ea03a06466030cd3e40a613a34f6fd7f541d4fb92e103439ea2505f2f
r70-2-elixir-l2criteria criteria `2c6ee03a749151a28aa0e908de4f1474017b411f` e3f054f0bf8335d286581ea716825cb540e31d6333a78935e759feab3ccd1e2d
r70-2-elixir-l2steps steps `0cabbafd45cf324219747a2fa97968ae262ab8b2` 75b11ce37398535be44d17272c0dbcf55fa7791f93863f4f7bd603df82250a46
r70-2-go none `13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82` —
r70-2-go-l2approach approach `1db582020ff97c48c854feee9b4588161323cef4` 707ff42bb366fad3e83a1c9ba929f7ad15613195e06c078c6678c05e81affcd6
r70-2-go-l2criteria criteria `48672b705c40c6232023a63a697f44e12fd7a7d0` 92799bbf4b92d915b52b4b3b787dab9bb659380eb196998c41955887fd19e511
r70-2-go-l2steps steps `d08c7c9d38b87416c47fb648f472d7524d1ff9d0` 9f7dfe458cf2d0e7ce83747a8a1f6bd657f23b467c5173ef5e4c656448ebdf87
r70-4-elixir-fe2 none `5ccd0bf1e816e2b2f5b2694861f8596402409f52` —
r70-4-elixir-fe2-l2approach approach `f9a5948042d525d20729f9346770ca185bcecb9d` c8963e6d6421b153ce56a9c1389ab1af5fb0c434413d14505dda63c143fc207d
r70-4-elixir-fe2-l2criteria criteria `634d71701ded3a7e4f698650e5bb9fea71fce589` 950ac4033b51b7cdffc9b7f4cf1e161d984acf84c83cc7f3949a43b773be6497
r70-4-elixir-fe2-l2steps steps `3d1e48fab23a0e2b01fa5051cc1c31a2f8611052` 2515e7b8cec8a619dd955401c2b74a9aa9ed3c68a0b450aa1068b43f4f83eadc
r70-4-go-fe2 none `21d21a2f420a387c921f998e5b4d16efc8f80904` —
r70-4-go-fe2-l2approach approach `9c2f20f38404d4cc5906268c5b68357e11902c97` dd12e95f67c0388c51a1626bc0d46f1850face51b0209a279624a0baf0013cb5
r70-4-go-fe2-l2criteria criteria `1524f0aeb36233c80dd86e635a550dc2b8762995` 90f4b65162e54d422cb26bd5c2890eb13224d92d60678e5ff95f11c75e6d1d80
r70-4-go-fe2-l2steps steps `9c40d5ead25b9bda63c12fb4919c836d12c415ec` 104853e7dc867ae0e2e41c7c3abc095d97b1d5398a27da4c590fd94da793ab6f
r70-4-ts-bun-fe2 none `cc3a831a167020538c4ca2ad90185606a2e4ad6a` —
r70-4-ts-bun-fe2-l2approach approach `de4007f44b3115aa79c94bf62c09067d839ef7ec` ddb365943f53b51f148cad60d5c4679d3dba69d398cf63e7b9ddbd8c79dc24b1
r70-4-ts-bun-fe2-l2criteria criteria `c977e8ea8ce3cd817a87fb32c005cf65a2e7c718` e0c0149b8a6c11a0ff745ae7218dade16ae9ea58420a392a3d17ee680e7e2a41
r70-4-ts-bun-fe2-l2steps steps `2dbbffb9759ff1372ff1087951c3332aafd57bf4` b19f9b6e352b9ab3e099ab510eddd14cd6743c0d2765d99df43282d9bba71ce8
r70-7-rust none `111a0c776ae6d93f6e7fccd3fc694d5f4fa26838` —
r70-7-rust-l2approach approach `7eb195d8474713c66fcd3173b064d801876c367e` d14c4377010fd3ccb8e5688528d2d36454821e0db9f559dd41d301cc527e2e50
r70-7-rust-l2criteria criteria `538de74aefb8bd061438a00da684aeec2b3e8a5b` 6415b150f9a16886a7328d4d847228c5335e5cb26c39d590276f12e3b182eef2
r70-7-rust-l2steps steps `c70927abbf58f038226c0f83a24da9edae0c00d0` 53ae845814f28c4758e3ddf7d2f7c0fee434b7e490c69b1964afc1d6f6b7c7e5

Historical Luna-max no-plan baseline commits:

Task ID Official cell IDs Kogen commit
r70-2-elixir codex__gpt-6-luna__max__default__r70-2-elixir__r31, codex__gpt-6-luna__max__default__r70-2-elixir__r32, codex__gpt-6-luna__max__default__r70-2-elixir__r33, codex__gpt-6-luna__max__default__r70-2-elixir__r910 `470ba65c1d3c19191a2190a032cd9ebf24e71975`
r70-2-go codex__gpt-6-luna__max__default__r70-2-go__r31, codex__gpt-6-luna__max__default__r70-2-go__r32, codex__gpt-6-luna__max__default__r70-2-go__r33, codex__gpt-6-luna__max__default__r70-2-go__r911 `13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82`
r70-4-elixir-fe2 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10, codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13, codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9 `5ccd0bf1e816e2b2f5b2694861f8596402409f52`
r70-4-elixir-fe2 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r909, codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r910 `a43dbb032d795157c5a93976947637502b30fece`
r70-4-go-fe2 codex__gpt-6-luna__max__default__r70-4-go-fe2__r10, codex__gpt-6-luna__max__default__r70-4-go-fe2__r13, codex__gpt-6-luna__max__default__r70-4-go-fe2__r9 `21d21a2f420a387c921f998e5b4d16efc8f80904`
r70-4-go-fe2 codex__gpt-6-luna__max__default__r70-4-go-fe2__r820, codex__gpt-6-luna__max__default__r70-4-go-fe2__r823, codex__gpt-6-luna__max__default__r70-4-go-fe2__r919 `748ebe3a3bc03a112dab83e785159654e0709f9e`
r70-4-ts-bun-fe2 codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r819, codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r920, codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r923 `880afb00474fe33e9cf38a6d70016ddcd4e38e81`
r70-4-ts-bun-fe2 codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r10, codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r13, codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9 `cc3a831a167020538c4ca2ad90185606a2e4ad6a`
r70-7-rust codex__gpt-6-luna__max__default__r70-7-rust__r1, codex__gpt-6-luna__max__default__r70-7-rust__r2, codex__gpt-6-luna__max__default__r70-7-rust__r3, codex__gpt-6-luna__max__default__r70-7-rust__r31, codex__gpt-6-luna__max__default__r70-7-rust__r32, codex__gpt-6-luna__max__default__r70-7-rust__r33, codex__gpt-6-luna__max__default__r70-7-rust__r905 `111a0c776ae6d93f6e7fccd3fc694d5f4fa26838`

Recomputed from raw records

`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 39 rows (fail 22, pass 17); overall pass rate is 43.6% (17/39) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.

No shared cell IDs are present in the committed public grade export; the source-only cohort is checked against the page's reported figures below. No shared public-export cell IDs are available for a paired outcome comparison. No complete published metadata/manifest token pairs are available for this round (0 missing manifest usage; 39 missing published metadata). Missing counters remain unknown, not zero. This round also has 39 raw-only or non-public rows; they are kept separate from the published-export denominator.

Round results · rounds/l2-plan-specificity/RESULTS.md

STATUS: VALID

The current cohort has 78/78 officially graded exact cell IDs: 42 gate cells and 36 rep-2 bulk cells. It includes 72 planned factorial cells and six Luna-low no-plan cells. Every included current row is labeled VALID; the historical Luna-max no-plan baseline is a separate descriptive cohort.

Per-cell outcomes and builder tokens

Token rule: uncached input + cached input + output; total = the sum of those three components. Builder tokens below include the provided frozen plan in the builder prompt. No hidden grader details are included.

Task / variant Level Builder Rep Stage Status Outcome Cell ID Uncached Cached Output Total
r70-2-elixir none Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-2-elixir__r1 31,788 231,168 4,187 267,143
r70-2-elixir-l2approach approach Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-2-elixir-l2approach__r1 20,935 88,064 1,858 110,857
r70-2-elixir-l2approach approach Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-2-elixir-l2approach__r2 15,882 233,472 3,849 253,203
r70-2-elixir-l2approach approach Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-2-elixir-l2approach__r1 209,777 9,861,120 84,230 10,155,127
r70-2-elixir-l2approach approach Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-2-elixir-l2approach__r2 317,097 8,334,848 106,857 8,758,802
r70-2-elixir-l2criteria criteria Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-2-elixir-l2criteria__r1 83,829 1,040,896 7,757 1,132,482
r70-2-elixir-l2criteria criteria Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-2-elixir-l2criteria__r2 97,041 1,399,552 13,443 1,510,036
r70-2-elixir-l2criteria criteria Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-2-elixir-l2criteria__r1 177,269 7,560,704 75,061 7,813,034
r70-2-elixir-l2criteria criteria Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-2-elixir-l2criteria__r2 312,914 18,456,064 123,669 18,892,647
r70-2-elixir-l2steps steps Luna-low 1 gate VALID PASS codex__gpt-6-luna__low__default__r70-2-elixir-l2steps__r1 57,881 576,512 8,200 642,593
r70-2-elixir-l2steps steps Luna-low 2 bulk VALID PASS codex__gpt-6-luna__low__default__r70-2-elixir-l2steps__r2 64,917 714,752 15,237 794,906
r70-2-elixir-l2steps steps Luna-max 1 gate VALID FAIL codex__gpt-6-luna__max__default__r70-2-elixir-l2steps__r1 260,423 8,961,792 113,232 9,335,447
r70-2-elixir-l2steps steps Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-2-elixir-l2steps__r2 194,177 9,452,288 78,703 9,725,168
r70-2-go none Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-2-go__r1 15,864 153,600 3,204 172,668
r70-2-go-l2approach approach Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-2-go-l2approach__r1 16,611 142,592 2,700 161,903
r70-2-go-l2approach approach Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-2-go-l2approach__r2 19,744 224,256 3,866 247,866
r70-2-go-l2approach approach Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-2-go-l2approach__r1 66,021 916,480 35,961 1,018,462
r70-2-go-l2approach approach Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-2-go-l2approach__r2 47,881 578,304 31,512 657,697
r70-2-go-l2criteria criteria Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-2-go-l2criteria__r1 14,676 83,200 2,430 100,306
r70-2-go-l2criteria criteria Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-2-go-l2criteria__r2 20,598 105,216 2,620 128,434
r70-2-go-l2criteria criteria Luna-max 1 gate VALID FAIL codex__gpt-6-luna__max__default__r70-2-go-l2criteria__r1 47,033 777,216 29,716 853,965
r70-2-go-l2criteria criteria Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-2-go-l2criteria__r2 52,090 684,032 27,313 763,435
r70-2-go-l2steps steps Luna-low 1 gate VALID PASS codex__gpt-6-luna__low__default__r70-2-go-l2steps__r1 45,413 204,800 4,095 254,308
r70-2-go-l2steps steps Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-2-go-l2steps__r2 17,136 147,712 4,358 169,206
r70-2-go-l2steps steps Luna-max 1 gate VALID FAIL codex__gpt-6-luna__max__default__r70-2-go-l2steps__r1 52,186 292,352 21,989 366,527
r70-2-go-l2steps steps Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-2-go-l2steps__r2 65,768 644,096 28,362 738,226
r70-4-elixir-fe2 none Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-elixir-fe2__r1 25,119 185,856 3,184 214,159
r70-4-elixir-fe2-l2approach approach Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2approach__r1 32,734 432,640 4,628 470,002
r70-4-elixir-fe2-l2approach approach Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2approach__r2 18,192 229,632 3,824 251,648
r70-4-elixir-fe2-l2approach approach Luna-max 1 gate VALID FAIL codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2approach__r1 107,325 1,471,744 35,803 1,614,872
r70-4-elixir-fe2-l2approach approach Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2approach__r2 71,983 592,896 19,273 684,152
r70-4-elixir-fe2-l2criteria criteria Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2criteria__r1 57,723 331,264 5,585 394,572
r70-4-elixir-fe2-l2criteria criteria Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2criteria__r2 31,917 281,088 3,612 316,617
r70-4-elixir-fe2-l2criteria criteria Luna-max 1 gate VALID FAIL codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2criteria__r1 48,351 548,864 24,040 621,255
r70-4-elixir-fe2-l2criteria criteria Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2criteria__r2 58,537 911,360 28,170 998,067
r70-4-elixir-fe2-l2steps steps Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2steps__r1 25,233 159,744 2,769 187,746
r70-4-elixir-fe2-l2steps steps Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-4-elixir-fe2-l2steps__r2 117,662 1,199,872 13,422 1,330,956
r70-4-elixir-fe2-l2steps steps Luna-max 1 gate VALID FAIL codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2steps__r1 60,230 881,408 27,388 969,026
r70-4-elixir-fe2-l2steps steps Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l2steps__r2 59,480 869,120 26,787 955,387
r70-4-go-fe2 none Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-go-fe2__r1 16,741 106,240 2,315 125,296
r70-4-go-fe2-l2approach approach Luna-low 1 gate VALID PASS codex__gpt-6-luna__low__default__r70-4-go-fe2-l2approach__r1 15,838 177,920 4,281 198,039
r70-4-go-fe2-l2approach approach Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-4-go-fe2-l2approach__r2 14,730 140,544 2,722 157,996
r70-4-go-fe2-l2approach approach Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-4-go-fe2-l2approach__r1 59,420 863,488 25,418 948,326
r70-4-go-fe2-l2approach approach Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-4-go-fe2-l2approach__r2 36,397 399,616 20,173 456,186
r70-4-go-fe2-l2criteria criteria Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-go-fe2-l2criteria__r1 4,732 33,024 216 37,972
r70-4-go-fe2-l2criteria criteria Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-4-go-fe2-l2criteria__r2 21,856 184,832 2,994 209,682
r70-4-go-fe2-l2criteria criteria Luna-max 1 gate VALID FAIL codex__gpt-6-luna__max__default__r70-4-go-fe2-l2criteria__r1 89,064 1,393,664 41,067 1,523,795
r70-4-go-fe2-l2criteria criteria Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-4-go-fe2-l2criteria__r2 47,152 708,096 26,708 781,956
r70-4-go-fe2-l2steps steps Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-go-fe2-l2steps__r1 24,968 285,696 3,857 314,521
r70-4-go-fe2-l2steps steps Luna-low 2 bulk VALID FAIL codex__gpt-6-luna__low__default__r70-4-go-fe2-l2steps__r2 13,047 174,848 2,341 190,236
r70-4-go-fe2-l2steps steps Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-4-go-fe2-l2steps__r1 55,099 691,968 25,790 772,857
r70-4-go-fe2-l2steps steps Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-4-go-fe2-l2steps__r2 71,562 908,288 32,930 1,012,780
r70-4-ts-bun-fe2 none Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2__r1 22,848 103,424 1,757 128,029
r70-4-ts-bun-fe2-l2approach approach Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2approach__r1 11,670 85,248 1,873 98,791
r70-4-ts-bun-fe2-l2approach approach Luna-low 2 bulk VALID PASS codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2approach__r2 27,467 258,560 3,861 289,888
r70-4-ts-bun-fe2-l2approach approach Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2approach__r1 48,507 518,656 28,377 595,540
r70-4-ts-bun-fe2-l2approach approach Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2approach__r2 58,375 718,592 28,704 805,671
r70-4-ts-bun-fe2-l2criteria criteria Luna-low 1 gate VALID PASS codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2criteria__r1 18,685 231,424 2,714 252,823
r70-4-ts-bun-fe2-l2criteria criteria Luna-low 2 bulk VALID PASS codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2criteria__r2 20,882 130,560 2,598 154,040
r70-4-ts-bun-fe2-l2criteria criteria Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2criteria__r1 50,249 235,776 16,232 302,257
r70-4-ts-bun-fe2-l2criteria criteria Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2criteria__r2 50,576 558,592 25,881 635,049
r70-4-ts-bun-fe2-l2steps steps Luna-low 1 gate VALID FAIL codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2steps__r1 26,853 103,424 1,780 132,057
r70-4-ts-bun-fe2-l2steps steps Luna-low 2 bulk VALID PASS codex__gpt-6-luna__low__default__r70-4-ts-bun-fe2-l2steps__r2 17,101 198,400 2,792 218,293
r70-4-ts-bun-fe2-l2steps steps Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2steps__r1 35,638 237,824 21,900 295,362
r70-4-ts-bun-fe2-l2steps steps Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l2steps__r2 46,346 620,032 25,218 691,596
r70-7-rust none Luna-low 1 gate VALID PASS codex__gpt-6-luna__low__default__r70-7-rust__r1 27,266 149,760 3,384 180,410
r70-7-rust-l2approach approach Luna-low 1 gate VALID PASS codex__gpt-6-luna__low__default__r70-7-rust-l2approach__r1 13,258 119,552 3,324 136,134
r70-7-rust-l2approach approach Luna-low 2 bulk VALID PASS codex__gpt-6-luna__low__default__r70-7-rust-l2approach__r2 33,481 166,144 5,522 205,147
r70-7-rust-l2approach approach Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-7-rust-l2approach__r1 46,385 220,928 15,984 283,297
r70-7-rust-l2approach approach Luna-max 2 bulk VALID FAIL codex__gpt-6-luna__max__default__r70-7-rust-l2approach__r2 27,476 220,160 13,087 260,723
r70-7-rust-l2criteria criteria Luna-low 1 gate VALID PASS codex__gpt-6-luna__low__default__r70-7-rust-l2criteria__r1 24,048 189,184 3,503 216,735
r70-7-rust-l2criteria criteria Luna-low 2 bulk VALID PASS codex__gpt-6-luna__low__default__r70-7-rust-l2criteria__r2 18,204 123,648 2,959 144,811
r70-7-rust-l2criteria criteria Luna-max 1 gate VALID FAIL codex__gpt-6-luna__max__default__r70-7-rust-l2criteria__r1 41,237 293,376 16,574 351,187
r70-7-rust-l2criteria criteria Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-7-rust-l2criteria__r2 65,051 375,552 15,298 455,901
r70-7-rust-l2steps steps Luna-low 1 gate VALID PASS codex__gpt-6-luna__low__default__r70-7-rust-l2steps__r1 31,485 222,720 3,563 257,768
r70-7-rust-l2steps steps Luna-low 2 bulk VALID PASS codex__gpt-6-luna__low__default__r70-7-rust-l2steps__r2 29,076 147,712 4,382 181,170
r70-7-rust-l2steps steps Luna-max 1 gate VALID PASS codex__gpt-6-luna__max__default__r70-7-rust-l2steps__r1 43,185 441,344 19,205 503,734
r70-7-rust-l2steps steps Luna-max 2 bulk VALID PASS codex__gpt-6-luna__max__default__r70-7-rust-l2steps__r2 39,815 371,200 16,648 427,663

Descriptive full-task outcomes

  • Luna-low no-plan cells: 1/6.
  • Luna-low criteria: 4/12; approach: 4/12; steps: 6/12.
  • Luna-max criteria: 6/12; approach: 7/12; steps: 6/12.
  • Historical Luna-max no-plan baseline: 20/32 valid official cells; this is unrandomized historical context.

Primary Luna-low paired comparison

Measure Steps vs criteria
Steps-only wins 3
Criteria-only wins 1
Ties 8
Net (wins − losses) 2
Source task Rep Steps Criteria Pair result
r70-2-elixir 1 PASS FAIL steps win
r70-2-elixir 2 PASS FAIL steps win
r70-2-go 1 PASS FAIL steps win
r70-2-go 2 FAIL FAIL tie
r70-4-elixir-fe2 1 FAIL FAIL tie
r70-4-elixir-fe2 2 FAIL FAIL tie
r70-4-go-fe2 1 FAIL FAIL tie
r70-4-go-fe2 2 FAIL FAIL tie
r70-4-ts-bun-fe2 1 FAIL PASS criteria win
r70-4-ts-bun-fe2 2 PASS PASS tie
r70-7-rust 1 PASS PASS tie
r70-7-rust 2 PASS PASS tie

The preregistered adoption screen required net +3 across 12 pairs and lower total tokens per official pass than criteria. The observed net was +2; steps do not meet the adoption screen. The observed one-sided exact sign test, excluding ties, is p = 5/16 = 0.3125 (3 wins among 4 discordant pairs). The minimal +3 screen outcome of three wins and zero losses would still have p = 1/8 = 0.125; treat this as a pilot screen, not confirmatory evidence.

Planner receipts

The 18 frozen-plan generations used Luna-max. Receipt components sum to uncached 138,159, cached 694,784, output 54,278, total 887,221 tokens.

Tokens per official pass

Each plan receipt is included in the cost view. Amortized charges allocate one plan receipt over its four planned uses (two builder efforts × two reps). One-off charges assign the full receipt to each task-level scored result. Cost totals keep uncached input, cached input, and output separate; no dollar conversion or wall-time combination is used.

Amortized planning cost

  • criteria / Luna-low: 4 of 12 cells passed; builder uncached/cached/output/total = 414,191/4,133,888/50,431/4,598,510; plan charge uncached/cached/output/total = 21,434.5/116,992/5,899.5/144,326; combined charged uncached/cached/output/total = 435,625.5/4,250,880/56,330.5/4,742,836; tokens per official full-task pass = 1,185,709.
  • criteria / Luna-max: 6 of 12 cells passed; builder uncached/cached/output/total = 1,039,523/32,503,296/449,729/33,992,548; plan charge uncached/cached/output/total = 21,434.5/116,992/5,899.5/144,326; combined charged uncached/cached/output/total = 1,060,957.5/32,620,288/455,628.5/34,136,874; tokens per official full-task pass = 5,689,479.
  • approach / Luna-low: 4 of 12 cells passed; builder uncached/cached/output/total = 240,542/2,298,624/42,308/2,581,474; plan charge uncached/cached/output/total = 27,432.5/112,384/7,984/147,800.5; combined charged uncached/cached/output/total = 267,974.5/2,411,008/50,292/2,729,274.5; tokens per official full-task pass = 682,318.6.
  • approach / Luna-max: 7 of 12 cells passed; builder uncached/cached/output/total = 1,096,644/24,696,832/445,379/26,238,855; plan charge uncached/cached/output/total = 27,432.5/112,384/7,984/147,800.5; combined charged uncached/cached/output/total = 1,124,076.5/24,809,216/453,363/26,386,655.5; tokens per official full-task pass = 3,769,522.2.
  • steps / Luna-low: 6 of 12 cells passed; builder uncached/cached/output/total = 470,772/4,136,192/66,796/4,673,760; plan charge uncached/cached/output/total = 20,212.5/118,016/13,255.5/151,484; combined charged uncached/cached/output/total = 490,984.5/4,254,208/80,051.5/4,825,244; tokens per official full-task pass = 804,207.3.
  • steps / Luna-max: 6 of 12 cells passed; builder uncached/cached/output/total = 983,909/24,371,712/438,152/25,793,773; plan charge uncached/cached/output/total = 20,212.5/118,016/13,255.5/151,484; combined charged uncached/cached/output/total = 1,004,121.5/24,489,728/451,407.5/25,945,257; tokens per official full-task pass = 4,324,209.5.

One-off planning cost

  • criteria / Luna-low: 4 of 12 cells passed; builder uncached/cached/output/total = 414,191/4,133,888/50,431/4,598,510; plan charge uncached/cached/output/total = 85,738/467,968/23,598/577,304; combined charged uncached/cached/output/total = 499,929/4,601,856/74,029/5,175,814; tokens per official full-task pass = 1,293,953.5.
  • criteria / Luna-max: 6 of 12 cells passed; builder uncached/cached/output/total = 1,039,523/32,503,296/449,729/33,992,548; plan charge uncached/cached/output/total = 85,738/467,968/23,598/577,304; combined charged uncached/cached/output/total = 1,125,261/32,971,264/473,327/34,569,852; tokens per official full-task pass = 5,761,642.
  • approach / Luna-low: 4 of 12 cells passed; builder uncached/cached/output/total = 240,542/2,298,624/42,308/2,581,474; plan charge uncached/cached/output/total = 109,730/449,536/31,936/591,202; combined charged uncached/cached/output/total = 350,272/2,748,160/74,244/3,172,676; tokens per official full-task pass = 793,169.
  • approach / Luna-max: 7 of 12 cells passed; builder uncached/cached/output/total = 1,096,644/24,696,832/445,379/26,238,855; plan charge uncached/cached/output/total = 109,730/449,536/31,936/591,202; combined charged uncached/cached/output/total = 1,206,374/25,146,368/477,315/26,830,057; tokens per official full-task pass = 3,832,865.3.
  • steps / Luna-low: 6 of 12 cells passed; builder uncached/cached/output/total = 470,772/4,136,192/66,796/4,673,760; plan charge uncached/cached/output/total = 80,850/472,064/53,022/605,936; combined charged uncached/cached/output/total = 551,622/4,608,256/119,818/5,279,696; tokens per official full-task pass = 879,949.3.
  • steps / Luna-max: 6 of 12 cells passed; builder uncached/cached/output/total = 983,909/24,371,712/438,152/25,793,773; plan charge uncached/cached/output/total = 80,850/472,064/53,022/605,936; combined charged uncached/cached/output/total = 1,064,759/24,843,776/491,174/26,399,709; tokens per official full-task pass = 4,399,951.5.

No-plan cost context

Current Luna-low no-plan cells: 1/6 passed; uncached/cached/output/total = 139,626/930,048/18,031/1,087,705; tokens per official full-task pass = 1,087,705. No planner charge applies, so both planning-cost views coincide. The historical Luna-max no-plan baseline is reported for outcome context only and is not mixed into the L2 planning-cost comparison.

Grading provenance note

The final two EU rep-2 cells, r70-4-elixir-fe2-l2criteria and r70-4-elixir-fe2-l2steps, were officially graded with grade_worker SHA prefix 0e6f8aa0 rather than 73dc09af. The difference was an inventory-glob-only change; scoring code was byte-identical. The result rows above use the latest official row for each exact cell ID.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
reproduce/l2-plan-specificity.py
python3 reproduce/l2-plan-specificity.py
Inputs
Full execution replay
Unavailable
Scope
Recomputes all published numeric results from the sanitized official-cell, planner-receipt, and historical-baseline CSVs. Sandbox execution and official grading cannot be replayed from this public bundle.

View the complete per-round reproduction inventory · Repository reproduction guide