Round l3-repair

L3 actionable self-verification repair

Status · DESCRIPTIVEReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/l3-repair/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Why not VALID: The historical inventory retains DESCRIPTIVE because of REGISTERED_THRESHOLD_NOT_MET. See the round evidence and limitations below. Recomputation status: FULLY RECOMPUTABLE

Required reproduction metadata

  • Kogen commit: not applicable; the scored arm is direct Codex. The public task bases are Kogen-ex commits listed below.
  • Harness commit: no harness Git commit is recorded. The runner tree fingerprint on both workers is 7e7a8d406e99436bce20de4f1f1347e9dfa91e73b9b89759e8887cd073faf6d4; contestant wrapper SHA-256 is 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300.
  • Model and effort: direct Codex gpt-6-luna at max; Codex CLI 0.160.0.
  • Runner: Python 3.14.7; 3,600-second cap; zero retries.
  • Task IDs: r70-2-elixir-l3r31, r70-2-elixir-l3r33, r70-2-go-l3r32, r70-5-go-l3r1, r70-5-go-l3r32, and r70-7-rust-l3r31.
  • Host assignment: EU for Elixir and Rust; US for Go.
  • Variant base provenance: the operator created and signed each variant commit from its registered public base plus the delivered failed patch. Exact parent and variant base SHAs are linked below.

STATUS: DESCRIPTIVE — six graded repairs; the registered decision is not fully supported.

Pre-registered: operator-reported before the first scored cell; the public bundle does not independently establish the exact commitment time. Label: DESCRIPTIVE — raw captures not retained; results verified against official grades Question: When direct Codex gpt-6-luna at max resumes a prior failed implementation with a fixed self-verification instruction, how often does it repair an official hidden-suite failure without losing checks passed by the prior implementation? n: 6 matched failures. Headline: 2/6 repairs rescued; the ≥3/6 threshold was not met, so repair-with-verification remains budget-limited. Configuration: same direct Codex harness, runner, and sandbox; official grade route r70-macbook-window-v1 (the MacBook-driven grade window that drains the host and runs the pinned grade worker). The fixed instruction asks Codex to check every prompt requirement against the code, write and run its own tests for each, and fix any gap. No hidden-test names or grader output were given to the model. Limit: six selected failures across tasks 2, 5, and 7; one model; descriptive rescue rule. The comparator is the official pass rate among each original cell's other reps. Aggregate test counts do not establish per-test pass-set preservation; that registered condition is unverifiable from the public data.

Why not VALID

Evidence links: H12 — checks, review, and actionable retries; F01 findings summary; claim CL-L3-REPAIR-RESCUE-RATE (STATUS_ONLY).

Reproduce

Public task base commits

Operator-created variant base commits

Commands and data files

The worker dispatcher sequence was python3 dispatch.py repair --batch first, then python3 dispatch.py repair --batch next-two, then python3 dispatch.py repair --batch final-three, with the preregistered release gates between batches. Variant admission controls used the documented commands:

python3 grade_window_l3.py kogen-bench-eu --admission-controls --controls-only --fixed-variants r70-2-elixir-l3r31,r70-2-elixir-l3r33,r70-7-rust-l3r31
python3 grade_window_l3.py kogen-bench-us --admission-controls --controls-only --fixed-variants r70-2-go-l3r32,r70-5-go-l3r1,r70-5-go-l3r32

From the repository root, reproduce the public results and validate the page with:

python3 rounds/l3-repair/reproduce/l3_repair.py
python3 reproduce/validate_repo.py

The reproducer uses only the standard library and reads rounds/l3-repair/data/cells.csv. The CSV records cell IDs, aggregate test counts, wall seconds, uncached/cached/output token counts, independent base pass rates, and operator variant SHAs. No hidden-test names or reference content are included. The frozen pre-registration bundle preserves the design-time inputs; the final grades and deployment SHAs are recorded in this page and the CSV.

Recomputed from raw records

`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 3 rows (fail 2, pass 1); overall pass rate is 33.3% (1/3) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.

No shared cell IDs are present in the committed public grade export; the source-only cohort is checked against the page's reported figures below. No shared public-export cell IDs are available for a paired outcome comparison. No complete published metadata/manifest token pairs are available for this round (0 missing manifest usage; 3 missing published metadata). Missing counters remain unknown, not zero. This round also has 3 raw-only or non-public rows; they are kept separate from the published-export denominator.

Round results · rounds/l3-repair/RESULTS.md

STATUS: DESCRIPTIVE — the registered three-rescue threshold failed and preservation is unmeasured.

Release label: DESCRIPTIVE — raw captures not retained; results verified against official grades

All reported outcomes and aggregates below are recomputed from data/cells.csv by reproduce/l3_repair.py.

Matched pair results

Task ID Exact arm Original cell ID Original tests Repair cell ID Outcome Repair tests wall_s Uncached Cached Output Total tokens
r70-2-elixir-l3r31 Arm A → Arm B codex__gpt-6-luna__max__default__r70-2-elixir__r31 24/25 codex__gpt-6-luna__max__default__r70-2-elixir-l3r31__r1 PASS 25/25 1435.708 145,573 4,334,848 71,901 4,552,322
r70-2-elixir-l3r33 Arm A → Arm B codex__gpt-6-luna__max__default__r70-2-elixir__r33 21/25 codex__gpt-6-luna__max__default__r70-2-elixir-l3r33__r1 FAIL 21/25 2753.349 384,209 9,389,568 113,206 9,886,983
r70-2-go-l3r32 Arm A → Arm B codex__gpt-6-luna__max__default__r70-2-go__r32 21/25 codex__gpt-6-luna__max__default__r70-2-go-l3r32__r1 FAIL 21/25 1020.239 121,775 2,307,072 50,768 2,479,615
r70-5-go-l3r1 Arm A → Arm B codex__gpt-6-luna__max__default__r70-5-go__r1 23/24 codex__gpt-6-luna__max__default__r70-5-go-l3r1__r1 FAIL 23/24 489.475 76,408 503,808 24,511 604,727
r70-5-go-l3r32 Arm A → Arm B codex__gpt-6-luna__max__default__r70-5-go__r32 23/24 codex__gpt-6-luna__max__default__r70-5-go-l3r32__r1 PASS 24/24 232.210 72,800 931,584 23,072 1,027,456
r70-7-rust-l3r31 Arm A → Arm B codex__gpt-6-luna__max__default__r70-7-rust__r31 23/24 codex__gpt-6-luna__max__default__r70-7-rust-l3r31__r1 FAIL 23/24 471.463 80,040 409,600 24,696 514,336
Total — — — — 2/6 rescues — 6402.444 880,805 17,876,480 308,154 19,065,439

Independent base pass rates

Each rate uses the same task and stack's other official reps, excluding that pair's original cell; invalidated rows and reps at or above 700 are excluded.

  • r70-2-elixir-l3r31 / original codex__gpt-6-luna__max__default__r70-2-elixir__r31: 1/2 across 2 other official reps.
  • r70-2-elixir-l3r33 / original codex__gpt-6-luna__max__default__r70-2-elixir__r33: 1/2 across 2 other official reps.
  • r70-2-go-l3r32 / original codex__gpt-6-luna__max__default__r70-2-go__r32: 2/2 across 2 other official reps.
  • r70-5-go-l3r1 / original codex__gpt-6-luna__max__default__r70-5-go__r1: 5/6 across 6 other official reps.
  • r70-5-go-l3r32 / original codex__gpt-6-luna__max__default__r70-5-go__r32: 5/6 across 6 other official reps.
  • r70-7-rust-l3r31 / original codex__gpt-6-luna__max__default__r70-7-rust__r31: 5/5 across 5 other official reps.

Decision

Rescues: 2/6. The preregistered threshold is at least 3/6; it was not met, so repair-with-verification remains budget-limited. Per-test no-regression requirement: UNVERIFIABLE; per-test grade results are not present in this public bundle. Aggregate pass counts are a separate diagnostic and cannot establish that every previously passed check was preserved. Aggregate count diagnostic: PASS; these counts do not determine the no-regression condition. Token definition: uncached input + cached input + output; total is their sum.

Process log

  • Admission controls, initial attempt: Invalid: reference controls passed, while no-op controls were infrastructure-invalid because deployed variant skeletons lacked repository history.
  • Control deployment correction: The lane-local deployer was corrected to preserve repository history; stray prepared skeletons were archived, hosts cleaned and redeployed, and variant base archives verified.
  • Admission controls, retry: Valid: every reference passed and each no-op reproduced its paired original pass count.
  • Initial scored-cell launch attempt: Voided before any model call after a runner workdir error.
  • Task-record deployment correction: Task records were corrected to point to each variant skeleton; the host variants were redeployed.
  • Official release and grading sequence: Official grades were returned through the MacBook-driven official grade window (route r70-macbook-window-v1); the first repair received its clean grade before the next release gate.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
rounds/l3-repair/reproduce/l3_repair.py
python3 rounds/l3-repair/reproduce/l3_repair.py
Inputs
Full execution replay
Unavailable
Scope
Recomputes aggregate repair counts from six published repair rows; per-test pass sets, grade receipts, and historical model executions are unavailable.

View the complete per-round reproduction inventory · Repository reproduction guide