Round l1-task-grader-trust

L1 — Task and grader trust

Status · VALIDReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/l1-task-grader-trust/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: VALID; KEPT FOR AUDIT Recomputation status: OBSERVED SOURCE ONLY

Recomputation update: The mined raw records now provide per-cell outcome, manifest token-counter, and wall-time recomputation for the observed cohort. This does not reconstruct missing planned cells, a full intention-to-treat denominator, or model execution.

Status

VALID

Why not VALID

  • None: the README documents a pre-result registration, execution under its rule, reconciled denominators, recomputable results, matched conditions, and retained evidence.

STATUS: VALID

Required reproduction metadata

  • Kogen commit: not applicable as a single round-level value; the eight task variant commits are listed below.
  • Harness commit: not recorded as a Git commit; available wrapper fingerprints are SHA-256 values, not Git commits.
  • Model and effort: selected pool configuration is Codex gpt-6-luna at max; no scored model cell was launched.
  • Task IDs: see the exact task variant IDs and task commits below.
  • Official replacement grading: operator grade windows are recorded in the L1 source ledger; exact control commands are listed below.
  • Raw records: grades.jsonl, windows.jsonl, POOLS.json, the prep snapshot in CONTROL-MANIFEST.md, and the sanitized data/controls.csv receipt.

Pre-registered: yes; design dated 7 October 2026, before any scored cell. Label: VALID model-free grader-control audit; no scored model cells were launched. Question: Do the official task/grader controls reject prompt-violating reference mutants and accept an allowed alternative implementation across selected r70 variants? n: 8 original task variants; 24 original staged / 21 graded / 3 withdrawn; 2 replacement mutants graded; 7 admitted variants / 21 original graded controls. The original 37-row ledger has 34 graded control rows and 3 withdrawal audit rows, including 11 admission grades. The separate L1-EU cohort adds 10 grades; the combined CSV has 47 rows. Headline: 14 behavioral mutant detections across the 7 admitted variants (two per variant) + 7 passing alternatives. Configuration: Pool selection uses Codex gpt-6-luna at max, default. Control authoring is model-free; Rust and Elixir variants use kogen-bench-eu, and Go and TypeScript/Bun variants use kogen-bench-us. Control manifests identify these as NOT a model cell. Limit: The original 24 controls retain their recorded disposition: 21 officially graded and 3 withdrawn. Original r711/r714 remain labelled as build rejections; replacements r720/r719 are officially graded behavioral failures and supersede those build rejections for the two-mutant criterion. The controls support only these seeded examples and do not establish general grader fairness or build success. No scored model cells were launched.

Reproduction

Kogen task commits:

Harness and CLI: contestant ledger rows record Codex CLI 0.160.0. The bench-codex wrapper SHA-256 is 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300. An official harness version label is not recorded; the selected baseline rows record grader SHA-256 616b1d53fe6ee56376134f0cf6a9350e94b5ecb7ec5b84f46a93e3c677356416, with 5fe9ea3fa0cb748e3a60d4e9153e41c61ec735bd3b5d6590dc7708e7343ae91f on the selected r70-4-rust row. The lane controller and inventory helper fingerprints are recorded below.

Data files: grades.jsonl (SHA-256 39bdd350fe6fca7c5c941b415f3ef91f6b6c9d444c40d96d1426018f3702d69a); windows.jsonl (SHA-256 7de4ad159d388c4b084f8c6a3c8599f71b3dde22c60aa39865fdba4e06163428); POOLS.json (SHA-256 662a097ede933ba115464f238f060de0be5ad46134b9631d6bdef2a848cae020); controls.csv (SHA-256 1d799c14750878dbce58dee45b979e3de0cb4e1beb03606c0fd9279bab8e3482); reproduction checker reproduce/l1_controls.py. The selection uses the latest grade row per exact cell ID, removes invalidated IDs, excludes contestant x-controls from Luna-max rates, and excludes task 8.

Public control-ledger reproduction command: python3 reproduce/l1_controls.py. It reads only data/controls.csv and verifies the public control-count claims and the staged/replacement IDs against CONTROL-MANIFEST.md.

Run these commands from the lane workspace root. They are recorded operator commands; this page-update job did not run any grades:

python3 levers/lanes-2026-10-07/l1/grade_window.py kogen-bench-us --admission-controls --controls-only --pairs 2:go,5:go
python3 levers/lanes-2026-10-07/l1/grade_window.py kogen-bench-us --admission-controls --controls-only --fixed-variants r70-4-go-fe2
python3 levers/lanes-2026-10-07/l1/grade_window.py kogen-bench-eu --cell-ids codex__gpt-6-luna__max__default__r70-2-elixir__r701,codex__gpt-6-luna__max__default__r70-2-elixir__r702,codex__gpt-6-luna__max__default__r70-2-elixir__r703,codex__gpt-6-luna__max__default__r70-4-rust__r707,codex__gpt-6-luna__max__default__r70-4-rust__r708,codex__gpt-6-luna__max__default__r70-4-rust__r709,codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r710,codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r711,codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r712,codex__gpt-6-luna__max__default__r70-7-rust__r722,codex__gpt-6-luna__max__default__r70-7-rust__r723,codex__gpt-6-luna__max__default__r70-7-rust__r724
python3 levers/lanes-2026-10-07/l1/grade_window.py kogen-bench-us --cell-ids codex__gpt-6-luna__max__default__r70-2-go__r704,codex__gpt-6-luna__max__default__r70-2-go__r705,codex__gpt-6-luna__max__default__r70-2-go__r706,codex__gpt-6-luna__max__default__r70-4-go-fe2__r713,codex__gpt-6-luna__max__default__r70-4-go-fe2__r714,codex__gpt-6-luna__max__default__r70-4-go-fe2__r715,codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r716,codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r717,codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r718,codex__gpt-6-luna__max__default__r70-5-go__r719,codex__gpt-6-luna__max__default__r70-5-go__r720,codex__gpt-6-luna__max__default__r70-5-go__r721

The lane-local grade_window.py SHA-256 is ad286b1059b23967c9bd034108acf9e2095699665e3606aeaa10418f5e558930; inventory_l1.py SHA-256 is 689babb23b3d8d467cb9915f2600e2ebdff7b3049144dd9083d2209173553618.

Sources: decision rule; results; control manifest prep snapshot; exact sanitized control ledger; pool-selection context; pool and staging records are POOLS.json and the lane staging records.

Replacement-mutant amendment — 7 October 2026

Status: VALID. The original 35 ledger rows are unchanged: 21 of 24 original staged controls were graded and 3 were withdrawn. Replacements r719/r720 have official grades; both failed behaviorally with 16/18 tests passing and make_check true. They complete two behavioral mutants for each of the seven admitted variants. Original r711/r714 remain labelled as build rejections and are superseded by r720/r719, respectively, for the two-mutant criterion.

Behavior class for both replacements: rebase-success outcome misclassification. The public mutant amendment records each patch SHA-256 and behavior class; it does not publish patch contents or hidden/reference material.

Host Task Cell ID Patch SHA-256 Behavior class Official grade Grade window
kogen-bench-us r70-4-go-fe2 codex__gpt-6-luna__max__default__r70-4-go-fe2__r719 71ad31d37bdf1745dbf6a39efa8c1dbf6ab138f7432ef4c017c03c51a25fd1fc rebase-success outcome misclassification fail, 16/18; make_check true 2026-10-07T14:55:34Z
kogen-bench-eu r70-4-elixir-fe2 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r720 f95c4624becd7edd205d1b62b801dd7ac1207255bcd7afd0c012b98e19321fc7 rebase-success outcome misclassification fail, 16/18; make_check true 2026-10-07T16:40:48Z

The official grades were recorded in the operator windows on the assigned hosts. Exact task commits, harness/CLI versions, commands, and public data files are recorded in the reproduction block above. No scored model cells were launched.

Addendum: L1-EU (7 Oct 2026)

STATUS: VALID. This model-free extension records 10 official grades: four task-6 admission controls and six mutant/alternative X-controls across two EU Elixir task variants. It changes no part of the original seven-variant headline above. No scored model cells were launched.

Pre-registered: yes; the L1-EU selection and control design were recorded on 7 October before these control grades. See the sanitized dated registration receipt, which records the source hashes and pre-grade chronology. Label: VALID control audit; r70-6-elixir is NOT ADMITTED and r70-7-elixir is ADMITTED. Question: Do the official controls establish admission and detect the seeded prompt violations for the two selected EU Elixir variants? n: 2 task variants; 4 task-6 admission grades; 6 X-control grades; 10 appended rows in data/controls.csv. Headline: Task 6 reference/no-op admission passed, but both prompt-length mutants passed, so task 6 is not admitted. Task 7's two mutants failed and its alternative passed, so task 7 is admitted. Configuration: Elixir on kogen-bench-eu. The task-6 baseline was 3/6; the task-7 baseline was 5/6. Task 7 already had two reference PASS and two no-op FAIL admission controls in the R70 ledger. Limit: These seeded controls support only the reported behavior classes. The task-6 suite did not detect the two prompt-length violations. The planned l4b-confirm-eu is NOT-RUN for insufficient power: only one variant was admitted, its baseline was 5/6, and no cells were spent.

Task 6 admission controls: references r1/r2 passed 24/24; no-ops r1/r2 failed 0/24. X-control outcomes and public patch metadata:

Task Kind Cell ID Patch SHA-256 Behavior class Official grade Grade window
r70-6-elixir mutant codex__gpt-6-luna__max__default__r70-6-elixir__r730 a7098c0191e08eedc35e062bd7122c9e3c0b157856e7590cd6c9dc301ecf4981 Event IDs longer than the specified limit are accepted. pass, 24/24 2026-10-07T20:42:58Z
r70-6-elixir mutant codex__gpt-6-luna__max__default__r70-6-elixir__r731 f0aadd52c1dc42f068476927d2201912fa7aaa2f260ef917f81f92848313e218 Job slugs longer than the specified limit are accepted. pass, 24/24 2026-10-07T20:44:13Z
r70-6-elixir alternative codex__gpt-6-luna__max__default__r70-6-elixir__r732 cba3ae7bca030c3741416ba022b2a20dac88a607ec576d7c2f3d4c11c687a4e3 Behavior-equivalent alternative implementation. pass, 24/24 2026-10-07T20:45:28Z
r70-7-elixir mutant codex__gpt-6-luna__max__default__r70-7-elixir__r733 b0f3e4fd6d189c556a521a5e730dc77d95e3c8c8234af6c1efa652caa127c818 Scale parsing accepts values above the specified ceiling. fail, 23/24 2026-10-07T20:46:43Z
r70-7-elixir mutant codex__gpt-6-luna__max__default__r70-7-elixir__r734 a72692f3eb6686acd0ce1b87bba70d894de6f6b9fe024e320f5bd59ea08f021b Status IDs may begin with uppercase letters. fail, 23/24 2026-10-07T20:48:11Z
r70-7-elixir alternative codex__gpt-6-luna__max__default__r70-7-elixir__r735 0cc7c063795b669918bb4579231c6ae7421609166a81dcf3bec926f91aacc40d Behavior-equivalent alternative implementation. pass, 24/24 2026-10-07T20:49:39Z

Only each staged patch's SHA-256 and behavior class are published; patch contents are omitted. All six staged patches passed git apply --check, public setup/build, and public make check in a temporary task copy before grading; author verification did not run hidden tests.

L1-EU reproduction

Kogen task commits: `r70-6-elixir`; `r70-7-elixir`. Harness/CLI and toolchain: bench-codex wrapper SHA-256 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300; codex-cli 0.160.0; Elixir 1.20.2; Erlang/OTP 29.0.3; official L1-EU grades use grader SHA-256 0e6f8aa06dce3b42813e326a85c2248115bf95bc127449f8fb99e3929bec0843. Task IDs: r70-6-elixir, r70-7-elixir.

Recorded public patch checks in a temporary task copy: git apply --check patch.diff; ./setup.sh && make build && make check.

Recorded official commands:

python3 levers/lanes-2026-10-07/l1/grade_window.py kogen-bench-eu --admission-controls --controls-only --pairs 6:elixir
python3 levers/lanes-2026-10-07/l1/grade_window.py kogen-bench-eu --cell-ids codex__gpt-6-luna__max__default__r70-6-elixir__r730,codex__gpt-6-luna__max__default__r70-6-elixir__r731,codex__gpt-6-luna__max__default__r70-6-elixir__r732,codex__gpt-6-luna__max__default__r70-7-elixir__r733,codex__gpt-6-luna__max__default__r70-7-elixir__r734,codex__gpt-6-luna__max__default__r70-7-elixir__r735
python3 reproduce/l1_controls.py

L1-EU source data files: levers/lanes-2026-10-07/l1/grades.jsonl, levers/lanes-2026-10-07/l1-eu/SELECTION.md, and levers/lanes-2026-10-07/l1-eu/STAGED.md. The public receipt is data/controls.csv; it records cohort, kind, official outcome, aggregate tests, cause class, task commit, grader fingerprint, grade-window timestamp, and only the patch SHA-256 and behavior class for X-controls.

Recomputed from raw records

`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 15 rows (fail 9, pass 6); overall pass rate is 40.0% (6/15) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.

No shared cell IDs are present in the committed public grade export; the source-only cohort is checked against the page's reported figures below. No shared public-export cell IDs are available for a paired outcome comparison. No complete published metadata/manifest token pairs are available for this round (0 missing manifest usage; 15 missing published metadata). Missing counters remain unknown, not zero. This round also has 15 raw-only or non-public rows; they are kept separate from the published-export denominator.

Round results · rounds/l1-task-grader-trust/RESULTS.md

STATUS: VALID — complete; replacements r719/r720 are officially graded. No scored model cells were launched.

Control counts

24 original staged / 21 graded / 3 withdrawn; 2 replacement mutants graded; 7 admitted variants / 21 original graded controls. The original 21 graded staged controls remain two controls per admitted variant, with the two original build rejections retained in the ledger. The original 37-row public ledger contains 34 graded control rows and 3 explicit withdrawal audit rows. The separate L1-EU cohort adds 10 grades, bringing the combined CSV to 47 rows. Of the 34 grades, 11 are admission controls: 5 references and 1 skeleton-frontend control passed; 5 no-ops failed.

Headline: 14 behavioral mutant detections across the 7 admitted variants (two per variant) + 7 passing alternatives.

The original r711/r714 mutants remain labelled as build rejections and are superseded for the two-mutant criterion by replacements r720/r719, respectively. Retained build-rejection cell IDs: codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r711 and codex__gpt-6-luna__max__default__r70-4-go-fe2__r714. All 7 valid alternatives passed, one for each admitted variant. These seeded controls do not establish general grader fairness.

Admitted variant Host Mutant 1 Mutant 2 Alternative
r70-2-elixir kogen-bench-eu behavioral failure behavioral failure pass
r70-2-go kogen-bench-us behavioral failure behavioral failure pass
r70-4-elixir-fe2 kogen-bench-eu behavioral failure behavioral failure (replacement r720; supersedes build rejection r711) pass
r70-4-go-fe2 kogen-bench-us behavioral failure behavioral failure (replacement r719; supersedes build rejection r714) pass
r70-4-ts-bun-fe2 kogen-bench-us behavioral failure behavioral failure pass
r70-5-go kogen-bench-us behavioral failure behavioral failure pass
r70-7-rust kogen-bench-eu behavioral failure behavioral failure pass

The exact outcome, task base commit, grader SHA-256, aggregate test count, phase, grade-window timestamp, and status for every grade are in data/controls.csv. The three staged-but-withdrawn controls are also listed there with their disposition and reason: codex__gpt-6-luna__max__default__r70-4-rust__r707, codex__gpt-6-luna__max__default__r70-4-rust__r708, and codex__gpt-6-luna__max__default__r70-4-rust__r709. The original task-4 Rust variant was removed from the pools before scoring because its recorded failures traced to the front-end exit-code trap.

Reproduce

Run from this round directory:

python3 reproduce/l1_controls.py

The standard-library checker reads data/controls.csv and verifies the counts and exact staged/replacement IDs in this page and CONTROL-MANIFEST.md. It does not grade controls or access hidden-suite material. Pool-selection context and historical summaries are preserved in POOL-SELECTION.md.

Replacement-mutant amendment — 7 October 2026

The original 35 ledger rows, including r711/r714 build rejections, remain unchanged. Replacement mutants r719/r720 have official grades: both failed behaviorally with 16/18 tests passing and make_check true. Each admitted variant now has two behavioral mutant failures. Behavior class for both replacements: rebase-success outcome misclassification.

Host Cell ID Patch SHA-256 Behavior class Official grade Grade window
kogen-bench-us codex__gpt-6-luna__max__default__r70-4-go-fe2__r719 71ad31d37bdf1745dbf6a39efa8c1dbf6ab138f7432ef4c017c03c51a25fd1fc rebase-success outcome misclassification fail, 16/18; make_check true 2026-10-07T14:55:34Z
kogen-bench-eu codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r720 f95c4624becd7edd205d1b62b801dd7ac1207255bcd7afd0c012b98e19321fc7 rebase-success outcome misclassification fail, 16/18; make_check true 2026-10-07T16:40:48Z

The exact official grade-window commands are listed in README.md. No scored model cells were launched.

Addendum: L1-EU (7 Oct 2026)

STATUS: VALID. The addendum records four task-6 admission controls and six X-controls. Task 6 reference r1/r2 passed 24/24 and no-op r1/r2 failed 0/24. Despite the admission pair passing, both task-6 prompt-length mutants passed 24/24, showing that the hidden suite does not detect these seeded limits; r70-6-elixir is NOT ADMITTED. Its equivalent alternative r732 passed 24/24. All six staged patches had passed public apply, setup/build, and make check verification before official grading.

Task 7 already had two reference PASS and two no-op FAIL admission controls in the R70 ledger. Its two seeded mutants, r733 and r734, each failed 23/24, and equivalent alternative r735 passed 24/24; r70-7-elixir is ADMITTED. The addendum does not alter the original seven-variant headline above.

Task Kind Cell ID Patch SHA-256 Behavior class Official grade
r70-6-elixir mutant codex__gpt-6-luna__max__default__r70-6-elixir__r730 a7098c0191e08eedc35e062bd7122c9e3c0b157856e7590cd6c9dc301ecf4981 Event IDs longer than the specified limit are accepted. pass, 24/24
r70-6-elixir mutant codex__gpt-6-luna__max__default__r70-6-elixir__r731 f0aadd52c1dc42f068476927d2201912fa7aaa2f260ef917f81f92848313e218 Job slugs longer than the specified limit are accepted. pass, 24/24
r70-6-elixir alternative codex__gpt-6-luna__max__default__r70-6-elixir__r732 cba3ae7bca030c3741416ba022b2a20dac88a607ec576d7c2f3d4c11c687a4e3 Behavior-equivalent alternative implementation. pass, 24/24
r70-7-elixir mutant codex__gpt-6-luna__max__default__r70-7-elixir__r733 b0f3e4fd6d189c556a521a5e730dc77d95e3c8c8234af6c1efa652caa127c818 Scale parsing accepts values above the specified ceiling. fail, 23/24
r70-7-elixir mutant codex__gpt-6-luna__max__default__r70-7-elixir__r734 a72692f3eb6686acd0ce1b87bba70d894de6f6b9fe024e320f5bd59ea08f021b Status IDs may begin with uppercase letters. fail, 23/24
r70-7-elixir alternative codex__gpt-6-luna__max__default__r70-7-elixir__r735 0cc7c063795b669918bb4579231c6ae7421609166a81dcf3bec926f91aacc40d Behavior-equivalent alternative implementation. pass, 24/24

The planned l4b-confirm-eu is NOT-RUN for insufficient power: only one variant was admitted, with a 5/6 baseline, and no cells were spent. Only patch SHA-256 values and behavior classes are disclosed; patch contents are omitted. Full grade provenance is in data/controls.csv.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
unavailable
Script
No per-round analysis script is published.
Inputs
No identified public input bundle.
Full execution replay
Unavailable
Scope
No public per-round analysis script with a complete, identified input bundle is published. See the round README, MEASURED.md, and MISSING.md for recorded scope and gaps; historical execution inputs/toolchains/grading are not bundled.

View the complete per-round reproduction inventory · Repository reproduction guide