Round l3b-repair-vs-continue

L3b repair vs continue

Status · VALIDReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/l3b-repair-vs-continue/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: VALID; NARROW / HISTORICAL; KEPT FOR AUDIT Publication limits: Validity applies only to the eight selected failures; 193 missing Standard capture fields, incomplete raw captures, and execution-time evidence limits bound this historical result. Recomputation status: FULLY RECOMPUTABLE

STATUS: VALID — registered rule met (difference 2, threshold ≥2); KEEP applies to these eight selected failures only, with per-test evidence published.

Label: VALID for the registered rule — raw captures not retained; results verified against official grades

Publication: release-included with this label under reproduce/release-rounds.json. The strict run-record gap is declared in MISSING.md.

Reproduction metadata

  • Kogen commit: not applicable to direct Codex scored cells; public task base commits are listed below.
  • Harness commit: no harness Git commit is recorded; bench-codex wrapper SHA-256 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300.
  • Model and effort: direct Codex gpt-6-luna at max; Codex CLI 0.160.0.
  • Task IDs: r70-2-elixir, r70-2-go, r70-4-elixir-fe2, r70-4-ts-bun-fe2, r70-5-go, r70-7-rust.
  • Command: python3 rounds/l3b-repair-vs-continue/reproduce/l3b_repair_vs_continue.py
  • Raw records: model transcripts and raw grader captures are not retained; sanitized official-grade rows and the three available pilot Standard 1.2 records are published with this round. Rebuild the records with python3 reproduce/build_l3b_records.py.

Question

For an official Luna-max failure with its delivered patch applied, does REPAIR with the L3 self-verification instruction rescue more official full-suite failures than neutral CONTINUE, without regression versus the original failure? RESTART measures fresh performance from the original task base.

Pre-registered design and closeout

Design SHA-256: 38acbba12ad264a058592b494cc315b89e3cadd89b65006420a39796dcb5abb3. The design file is a private operator record that is not published here, so this hash is source-reported. That it was frozen before the first scored cell is operator-reported and cannot be established from this repository alone. The selected sample is eight paired failures, with 24 planned scored cells. Three pilot cells were officially graded on 2026-10-08 before bulk release; all 24 planned scored cells now have official grades.

Cell accounting

Population Count
Planned scored cells 24
Identified scored starts 24
Finished cells 24
Officially graded cells 24
ITT denominator 24

Selected original failures

POOL-L3 had 12 entries. The r70-4-rust exclusion and two source-base mismatches left nine initially eligible failures. An admission amendment then excluded r70-2-elixir r31 after its no-op control passed 25/25 against an expected 24/25, leaving eight paired failures. All selected tasks are L1-admitted. No task-8 or r70-4-rust failure is selected.

Original failure cell ID Tests passed / total Original source base Failed-state base commit Delivered patch SHA-256 Host
codex__gpt-6-luna__max__default__r70-2-elixir__r33 21/25 470ba65c1d3c bcf8ab895355cfe822b08d187373dbc6faccb36c d1b79a381c4f526b4b4cdb0c6d3526d417245e856414653706e1c2ed721e3f89 kogen-bench-eu
codex__gpt-6-luna__max__default__r70-2-go__r32 21/25 13a08875ea5e a25ca44fcd42ac0f98b47de2361517171775480a 44995e2f56574eaa2c94c15eb76d88a06ba458d8f4b10b030f28af1bf629c1c3 kogen-bench-us
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9 17/18 5ccd0bf1e816 123de8a8f281a2c1ae5bdd07c47a5a90383d249e 9e6c42031d4716aadff5283855b92ac8222404b4621560decfc7661fbfcf11da kogen-bench-eu
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10 17/18 5ccd0bf1e816 a06ebe6275402f8bc55e7d84f973ecbaf22145c8 f85028bdb5082885b678f7577f23f77cdca74b16f347d5164ef6c4327580fc09 kogen-bench-eu
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13 17/18 5ccd0bf1e816 d3c4244af14088776ea541cfca512c928847dda5 57c2437118828a5723b658ab7a38430127c888657b65d3762303f305ae29ead3 kogen-bench-eu
codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9 17/18 cc3a831a1670 d7a7c0f26f1871c900f5986653fce91aaaa33736 eaed02b9e5c7cf04714145c224ea16687f45023112ac0fad0aae7d3cfb3272e5 kogen-bench-us
codex__gpt-6-luna__max__default__r70-5-go__r32 23/24 e0b4a2478a17 54091c79f09540e6d773be15dcc54df6232a48d8 87b9707bc9d8728c14079e9e95872574d59513cd85d1b8720ba222a6c9483183 kogen-bench-us
codex__gpt-6-luna__max__default__r70-7-rust__r31 23/24 111a0c776ae6 ec773802393fe9d7cd19e493a5e749809470cb03 a1e123612f79ae599035f01085462c6941473bf21df7593f7a82bed682119536 kogen-bench-eu

Excluded candidates:

  • codex__gpt-6-luna__max__default__r70-4-rust__r10: explicitly excluded by design.
  • codex__gpt-6-luna__max__default__r70-4-go-fe2__r820: recorded base 748ebe3a3bc03a112dab83e785159654e0709f9e is absent from the available public task snapshot.
  • codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r819: recorded base 880afb00474fe33e9cf38a6d70016ddcd4e38e81 is absent from the available public task snapshot.
  • codex__gpt-6-luna__max__default__r70-2-elixir__r31: admission-inadmissible after no-op passed 25/25 at the failed-state base; expected 24/25. Admission controls are outcome-independent.

Results

Rescues: REPAIR 3/8; CONTINUE 1/8; RESTART 2/8. The observed difference for these eight selected failures is 2 (threshold ≥2). The public per-test pass/fail sets cover all 8 original failures and 24 scored cells. Per-test no-regression is recomputed from the public pass/fail sets: no REPAIR cell adds a failure absent from its original. Observed KEEP for these eight selected failures; this is not a general repair policy. The full 24-cell table and recomputed totals are in RESULTS.md; MEASURED.md summarizes usage and limitations.

Limits

  • Restored to VALID 2026-10-08: the round executed its registered rule as registered and its numbers recompute; the points below are limits on public evidence, not contradictions in the round's own files.

  • Only 3 of 24 scored cells have Standard records; the public outcome CSV supplies aggregates for the others.

  • Both dated host captures in ENVIRONMENT.md were taken at 05:51Z, after contestant execution ended at 05:47Z, so execution-time conditions are not established by those captures.

  • The rule's public Git commit follows the first scored start. Matching private design bytes and an earlier filesystem time do not independently establish pre-registration.

  • DEV-1 records concurrent admission-control grading with another round's cell. A later hidden-suite exposure reached a non-contestant job after contestant execution; the chronology and lack of contestant access are operator-reported. The exposed per-test names and outcomes are now published so the registered no-regression rule can be recomputed.

Private check script (operator record, not in this repository; hash source-reported): levers/lanes-2026-10-07/l3b/l3b_rule_check.py, SHA-256 17ebe3ef4a67bd09c83c8a8568505e8c3e42ea31981afc80ae7bcdc5c046d4a2.

Configuration

  • REPAIR uses the source base plus the delivered patch, original prompt, and L3 self-verification instruction verbatim.
  • CONTINUE uses the same failed-state base and original prompt plus exactly: “A previous attempt is already in this repository. Continue and finish the task.”
  • RESTART uses the original task base and prompt, fresh and without a prior attempt.
  • Direct Codex, gpt-6-luna at max; Codex CLI 0.160.0; bench-codex wrapper SHA-256 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300; 3,600-second cap; zero retries; same runner and sandbox.
  • Rust and Elixir run on EU; Go and TS-Bun run on US. Scored cells were scheduled one at a time per host, and no two L3b scored cells overlapped on a host. See DEV-1 for an L3b admission-control grading window that overlapped another round's cell.
  • One random.Random(20261009) shuffle covers all cells. Host queues are stable filters of that order. Reps r9000–r9026 were checked against every grade file and both hosts’ result directories.

Deviation DEV-1

Procedural deviation, not an exclusion. On kogen-bench-us, the official grade window for the 12 L3b US admission controls (reference and no-op controls for r70-2-go-l3b32c/r, r70-4-ts-bun-fe2-l3b9c/r and r70-5-go-l3b32c/r; no model involved) ran 03:24:53–03:30:06Z. That overlapped the last ~214 seconds of a scored cell from a different round (a Kogen tier-1 syn-31 cell, 03:00:38–03:28:27Z). Cause: an operator watcher meant to start the window after that round finished matched an earlier log word instead of an exact completion marker. Effect on L3b: the admission controls were graded on a host also running one other cell. All 12 came out as expected: references passed in full, and every no-op reproduced its original failure count. No L3b scored cell ran during the window, so the 24 scored cells are unaffected and stay in the ITT denominator. Limitation: CPU contention during those ~214 seconds can't be ruled out for the control grading itself. The interval and scope come from private operator records (operator-reported). Fix: operator watchers now trigger only on exact completion markers.

Limit

n=8 paired failures; 24 planned cells, below the approved ceiling of 36. The pool is heterogeneous across six task/stack combinations and does not support general model ranking. Two base-mismatched entries were excluded to preserve exact starting states. The strict-validation exception was declared in MISSING.md before the bulk release (about 04:09Z host time; operator-reported). DEV-1 was recorded in private operator records at 03:33Z, also before the bulk release, and is described publicly in the deviation section (timing operator-reported). The result does not support a causal claim beyond the registered sample and decision rule.

Hidden-suite exposure (2026-10-08): an operator documentation job that ran 05:51:38–06:22:34Z printed L3b grade rows, so 20 failed-test names from the r70-2, r70-4, r70-5 and r70-7 hidden suites reached that non-contestant job and its model provider. All 24 L3b contestant executions had finished by 05:47:42Z, before the job began. The last 14 official grades finished at 06:06:26Z, inside the job's interval. Retained timestamps don't establish whether the printing commands came before or after that grading. Grading applies each fixed patch to the hidden suite and takes no input from the job. No contestant cell or agent session had access to the names (operator-reported). Future rounds that use these suites should treat them as exposed. Details and the guard added since are in the task-suite contamination log.

Sources and reproduction

Recompute the 24-cell table, arm rescue counts, usage totals, and rule result with:

python3 rounds/l3b-repair-vs-continue/reproduce/l3b_repair_vs_continue.py

The reproducer reads only the published aggregate CSV and boolean evidence JSON. It does not replay model execution or official grading. The source grade rows were filtered to scored reps at or above 9000 and sanitized with safe_rows.py before the public exports were written; admission-control rows were excluded.

Recomputed from raw records

`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 9 rows (fail 5, pass 4); overall pass rate is 44.4% (4/9) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.

No shared cell IDs are present in the committed public grade export; the source-only cohort is checked against the page's reported figures below. No shared public-export cell IDs are available for a paired outcome comparison. No complete published metadata/manifest token pairs are available for this round (0 missing manifest usage; 9 missing published metadata). Missing counters remain unknown, not zero. This round also has 9 raw-only or non-public rows; they are kept separate from the published-export denominator.

Round results · rounds/l3b-repair-vs-continue/RESULTS.md

STATUS: VALID — registered rule met for the eight selected failures; per-test no-regression is recomputed from public pass/fail sets.

Label: VALID for the registered rule — raw captures not retained; results verified against official grades

The table below is regenerated from data/scored-cells.csv, data/original-failures.csv, and data/per-test-results.json. The public pass/fail sets reproduce the per-test no-regression comparison used by the registered rule.

Arm totals

Arm Cells Full-suite rescues Uncached input Cached input Output Total tokens Wall seconds
REPAIR 8 3 796,472 16,684,800 288,379 17,769,651 3227.436
CONTINUE 8 1 288,227 2,626,048 118,358 3,032,633 1512.299
RESTART 8 2 451,463 5,742,848 200,515 6,394,826 2288.914

Rescues: REPAIR 3/8; CONTINUE 1/8; RESTART 2/8. Rule: REPAIR − CONTINUE = 2 (threshold ≥2); per-test no-regression recomputed true. Result: observed KEEP for these eight selected failures. The no-regression comparison is recomputed from per-test-results.json; the public sets include names because the selected suites are exposed.

Scored cells

Cell ID Original failure Arm Outcome Test fraction Patch SHA-256 Started at (UTC) Finished at (UTC) Uncached input Cached input Output Total tokens Wall seconds
codex__gpt-6-luna__max__default__r70-2-elixir-l3b33r__r9004 codex__gpt-6-luna__max__default__r70-2-elixir__r33 REPAIR fail 21/25 d1b79a381c4f526b4b4cdb0c6d3526d417245e856414653706e1c2ed721e3f89 2026-10-08T05:08:28.922+00:00 2026-10-08T05:24:30.762+00:00 188,958 8,343,808 75,918 8,608,684 947.088
codex__gpt-6-luna__max__default__r70-2-elixir-l3b33c__r9005 codex__gpt-6-luna__max__default__r70-2-elixir__r33 CONTINUE fail 21/25 d1b79a381c4f526b4b4cdb0c6d3526d417245e856414653706e1c2ed721e3f89 2026-10-08T05:35:23.092+00:00 2026-10-08T05:41:12.534+00:00 53,598 570,368 25,639 649,605 334.672
codex__gpt-6-luna__max__default__r70-2-elixir__r9003 codex__gpt-6-luna__max__default__r70-2-elixir__r33 RESTART pass 25/25 — 2026-10-08T04:16:32.244+00:00 2026-10-08T04:29:27.619+00:00 133,454 2,858,752 58,736 3,050,942 760.470
codex__gpt-6-luna__max__default__r70-2-go-l3b32r__r9007 codex__gpt-6-luna__max__default__r70-2-go__r32 REPAIR fail 21/25 44995e2f56574eaa2c94c15eb76d88a06ba458d8f4b10b030f28af1bf629c1c3 2026-10-08T04:28:02.873+00:00 2026-10-08T04:34:01.673+00:00 65,034 900,352 34,866 1,000,252 351.479
codex__gpt-6-luna__max__default__r70-2-go-l3b32c__r9008 codex__gpt-6-luna__max__default__r70-2-go__r32 CONTINUE fail 21/25 44995e2f56574eaa2c94c15eb76d88a06ba458d8f4b10b030f28af1bf629c1c3 2026-10-08T04:22:38.838+00:00 2026-10-08T04:25:30.404+00:00 30,198 331,776 14,919 376,893 164.337
codex__gpt-6-luna__max__default__r70-2-go__r9006 codex__gpt-6-luna__max__default__r70-2-go__r32 RESTART fail 21/25 — 2026-10-08T04:10:38.804+00:00 2026-10-08T04:13:52.872+00:00 50,381 212,992 17,515 280,888 186.926
codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l3b10r__r9013 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10 REPAIR fail 17/18 f85028bdb5082885b678f7577f23f77cdca74b16f347d5164ef6c4327580fc09 2026-10-08T04:46:19.011+00:00 2026-10-08T04:52:50.330+00:00 129,425 1,950,976 34,112 2,114,513 376.476
codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l3b10c__r9014 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10 CONTINUE fail 17/18 f85028bdb5082885b678f7577f23f77cdca74b16f347d5164ef6c4327580fc09 2026-10-08T04:40:46.918+00:00 2026-10-08T04:45:26.457+00:00 47,549 556,032 17,195 620,776 264.680
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9012 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10 RESTART fail 17/18 — 2026-10-08T05:41:14.120+00:00 2026-10-08T05:46:24.775+00:00 57,274 669,696 27,266 754,236 295.980
codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l3b13r__r9016 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13 REPAIR fail 17/18 57c2437118828a5723b658ab7a38430127c888657b65d3762303f305ae29ead3 2026-10-08T03:32:22.354+00:00 2026-10-08T03:38:19.395+00:00 70,238 1,262,080 28,653 1,360,971 341.955
codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l3b13c__r9017 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13 CONTINUE fail 17/18 57c2437118828a5723b658ab7a38430127c888657b65d3762303f305ae29ead3 2026-10-08T05:05:13.904+00:00 2026-10-08T05:08:27.385+00:00 41,380 203,776 11,635 256,791 178.812
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9015 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13 RESTART fail 17/18 — 2026-10-08T05:24:31.997+00:00 2026-10-08T05:29:12.139+00:00 36,934 380,416 16,988 434,338 265.299
codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l3b9r__r9010 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9 REPAIR fail 17/18 9e6c42031d4716aadff5283855b92ac8222404b4621560decfc7661fbfcf11da 2026-10-08T04:10:35.451+00:00 2026-10-08T04:16:29.204+00:00 95,642 947,712 24,276 1,067,630 338.766
codex__gpt-6-luna__max__default__r70-4-elixir-fe2-l3b9c__r9011 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9 CONTINUE fail 17/18 9e6c42031d4716aadff5283855b92ac8222404b4621560decfc7661fbfcf11da 2026-10-08T05:33:05.065+00:00 2026-10-08T05:35:20.353+00:00 22,071 221,696 7,497 251,264 120.759
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9009 codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9 RESTART fail 17/18 — 2026-10-08T04:33:04.574+00:00 2026-10-08T04:37:14.628+00:00 48,617 403,200 18,321 470,138 235.390
codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l3b9r__r9019 codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9 REPAIR pass 18/18 eaed02b9e5c7cf04714145c224ea16687f45023112ac0fad0aae7d3cfb3272e5 2026-10-08T04:13:53.820+00:00 2026-10-08T04:20:24.438+00:00 136,239 2,039,040 40,064 2,215,343 389.022
codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2-l3b9c__r9020 codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9 CONTINUE pass 18/18 eaed02b9e5c7cf04714145c224ea16687f45023112ac0fad0aae7d3cfb3272e5 2026-10-08T03:37:24.495+00:00 2026-10-08T03:41:36.057+00:00 55,769 515,840 24,569 596,178 250.208
codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9018 codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9 RESTART pass 18/18 — 2026-10-08T03:32:24.577+00:00 2026-10-08T03:37:23.281+00:00 65,662 677,120 31,788 774,570 297.295
codex__gpt-6-luna__max__default__r70-5-go-l3b32r__r9022 codex__gpt-6-luna__max__default__r70-5-go__r32 REPAIR pass 24/24 87b9707bc9d8728c14079e9e95872574d59513cd85d1b8720ba222a6c9483183 2026-10-08T04:34:02.899+00:00 2026-10-08T04:38:25.242+00:00 55,794 692,480 25,028 773,302 255.329
codex__gpt-6-luna__max__default__r70-5-go-l3b32c__r9023 codex__gpt-6-luna__max__default__r70-5-go__r32 CONTINUE fail 23/24 87b9707bc9d8728c14079e9e95872574d59513cd85d1b8720ba222a6c9483183 2026-10-08T04:20:26.830+00:00 2026-10-08T04:22:38.118+00:00 18,186 165,376 9,789 193,351 124.142
codex__gpt-6-luna__max__default__r70-5-go__r9021 codex__gpt-6-luna__max__default__r70-5-go__r32 RESTART fail 10/24 — 2026-10-08T04:25:32.856+00:00 2026-10-08T04:27:59.630+00:00 30,032 289,280 17,394 336,706 139.895
codex__gpt-6-luna__max__default__r70-7-rust-l3b31r__r9025 codex__gpt-6-luna__max__default__r70-7-rust__r31 REPAIR pass 24/24 a1e123612f79ae599035f01085462c6941473bf21df7593f7a82bed682119536 2026-10-08T05:29:14.085+00:00 2026-10-08T05:33:02.803+00:00 55,142 548,352 25,462 628,956 227.321
codex__gpt-6-luna__max__default__r70-7-rust-l3b31c__r9026 codex__gpt-6-luna__max__default__r70-7-rust__r31 CONTINUE fail 23/24 a1e123612f79ae599035f01085462c6941473bf21df7593f7a82bed682119536 2026-10-08T05:46:26.170+00:00 2026-10-08T05:47:42.188+00:00 19,476 61,184 7,115 87,775 74.689
codex__gpt-6-luna__max__default__r70-7-rust__r9024 codex__gpt-6-luna__max__default__r70-7-rust__r31 RESTART fail 23/24 — 2026-10-08T04:56:37.279+00:00 2026-10-08T04:58:26.917+00:00 29,109 251,392 12,507 293,008 107.659

Scored attempts total: 27,197,110 tokens; 7028.649 seconds. Including the eight original attempts once: 38,997,124 tokens; 12072.850 seconds.

Patch hashes identify the delivered public patch for each matched failure; RESTART has no patch.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
rounds/l3b-repair-vs-continue/reproduce/l3b_repair_vs_continue.py
python3 rounds/l3b-repair-vs-continue/reproduce/l3b_repair_vs_continue.py
Inputs
Full execution replay
Unavailable
Scope
Recomputes the public 24-cell aggregate outcome table, rescue counts, usage totals, and registered rule result, including per-test no-regression from the published pass/fail sets. Model execution and official grading are not replayed.

View the complete per-round reproduction inventory · Repository reproduction guide