Round l0-reconcile

L0 historical claim reconciliation

Status · VALIDReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/l0-reconcile/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: VALID; KEPT FOR AUDIT Recomputation status: NOT RECOMPUTABLE

Status

VALID — reconciliation only

Limits

  • Restored to VALID 2026-10-08: the round executed its registered rule as registered and its numbers recompute; the points below are limits on public evidence, not contradictions in the round's own files.
  • PRE_REGISTRATION_NOT_APPLICABLE: This retrospective reconciliation has no pre-registered result rule.
  • EVIDENCE_INCOMPLETE: The full R74 scored cohort is not reconstructable from retained public records.

Lifecycle counts for this lane

This reconciliation added no benchmark cells; the historical cohort counts above belong to their source rounds.

Lifecycle field n Basis
Planned 0 No new benchmark cells were planned in this reconciliation lane.
Started 0 No new benchmark cells were launched.
Finished 0 No new benchmark cells were run.
Officially graded 0 No new benchmark cells were graded.
ITT denominator 0 This lane generated no new benchmark outcomes.

Required reproduction metadata

  • Kogen commit: 260a73bc06be16e563cb4e56a168ffd078cec677 (the exact pin recorded in the cited round source, resolved against local Kogen history).
  • Harness commit: not recorded in the cited round source as a Git commit; any adapter or runtime fingerprints are not Git commits.
  • Historical execution command: not recorded in the cited round source.
  • Raw records: the cited sources do not provide a complete public round-specific request and grade bundle; public delivery and outcome exports are linked above where applicable.

STATUS: VALID (reconciliation only)

Pre-registered: no new benchmark cells; this lane reconciles frozen historical receipts.

Label: DESCRIPTIVE

Question: Do the R72, R74, R69, and R57 historical claims match their recoverable cell level receipts?

n: R72 P1 168 cells and partial P3 24; R74 167 scored rows plus 9 smoke attempts; R69 400 assigned rows; R57 pooled 311 scored cells plus one unrecovered planned slot.

Headline: R72 P1 and R69 counts reproduce exactly; R57 counts and published interim bound reproduce; R74 has 167 internal scored grades but no public scored export, and its full retained cohort is not reconstructable.

Configuration: Historical configurations are retained per row in the cohort CSVs. R72 direct Codex CLI 0.160.0; R74 direct Codex CLI 0.160.0. R69 and R57 used kh-gpt; no separate harness version or Codex CLI version was recorded. Effective model and effort are listed per cell; NOT-RECORDED and NOT-RUN are explicit.

Limit: This reconciliation does not establish new performance estimates. R74 has 231 retained slots without recovered cell IDs or terminal statuses. R57 has one missing shell-only slot, excluded from the scored denominator. R57’s source declaration specified stratification but not an estimator; this page recomputes the analyst’s published estimator. The R74 cohort is withdrawn and mixed across time, venue, and configuration.

Sources: ROUNDS.md; LATEST.md; R72 round README and decision rule; R74 results/run-records/index.json, CODEX-POOL.md, amendment, status, and exclusion receipts; R69 itt.jsonl, decision rule, and verdict; R57-POOLED.md; official grades.final.jsonl and cells.jsonl; the probe grade ledger; and R2-EXPERIMENTS-LEVERS.md, GAPS.md, and MASTER-MAP.md. See reconstructed cohort data and the calculation script.

Reproduce

Run from the kogen-bench-sot repository root:

python3 reproduce/reconcile_l0.py

The standard-library-only script reads only the sanitized files below. Cell IDs, arms, tasks, reps, outcomes, stop causes, effective model/effort, and public-grade matches are in these files:

  • rounds/l0-reconcile/data/r72-p1.csv
  • rounds/l0-reconcile/data/r72-p3-partial.csv
  • rounds/l0-reconcile/data/r74-graded-subset.csv
  • rounds/l0-reconcile/data/r74-smoke-attempts.csv
  • rounds/l0-reconcile/data/r74-coverage.csv
  • rounds/l0-reconcile/data/r69-itt.csv
  • rounds/l0-reconcile/data/r57-pooled-itt.csv

Recorded Kogen commits:

Harness/CLI receipts: R72 and R74 direct Codex CLI 0.160.0; Kogen versions are pinned by the commits above. R69 and R57 kh-gpt; a separate version string was not recorded. The per-cell CSVs contain the observed effective model and effort, including role maps for Kogen cells.

Task IDs represented in the recovered data:

  • R72 P1 (12): elx-02-ingest-supervision, elx-04-queue-backpressure, elx-07-cli-stats, elx-12-retry-api-deprecation, elx-port-board-publish-unpublish-public-boundary, elx-port-erase-account, syn-01-live-ticket-filters, syn-06-migration-ticket-numbers, syn-15-validation-ticket-form, syn-20-email-invite-flow, syn-24-csv-import, syn-31-inbound-email-webhook.
  • R72 partial P3 (9): elx-02-ingest-supervision, elx-04-queue-backpressure, elx-12-retry-api-deprecation, syn-01-live-ticket-filters, syn-06-migration-ticket-numbers, syn-15-validation-ticket-form, syn-20-email-invite-flow, syn-24-csv-import, syn-31-inbound-email-webhook.
  • R74 scored subset (12): elx-02-ingest-supervision, elx-04-queue-backpressure, elx-07-cli-stats, elx-12-retry-api-deprecation, elx-port-board-publish-unpublish-public-boundary, elx-port-erase-account, syn-01-live-ticket-filters, syn-06-migration-ticket-numbers, syn-15-validation-ticket-form, syn-20-email-invite-flow, syn-24-csv-import, syn-31-inbound-email-webhook. This is the recovered scored subset; task IDs for the 231 ungraded retained slots are not recovered.
  • R69 (25): elx-02-ingest-supervision, elx-04-queue-backpressure, rails-ac-throttle-search, rails-aj-enqueue-after-commit, rails-ar-archive-book-access, rails-ar-bulk-access-grants, rails-as-variant-processed-once, rails-ft-active-storage-tracking, rails-ft-activity-feed-api, rails-ft-board-publish-unpublish-public-boundary, rails-ft-cancelled-account-cleanup, rails-ft-entropy-sweep, rails-ft-i18n-support, rails-ft-mysql-fulltext-search-foundation, rails-ft-notification-bundle-window-overlap, rails-ft-saas-billing, rails-hw-scoped-broadcast, rails-sec-audit-sweep, rails-sup-legacy-conversions, syn-01-live-ticket-filters, syn-06-migration-ticket-numbers, syn-15-validation-ticket-form, syn-20-email-invite-flow, syn-24-csv-import, syn-31-inbound-email-webhook.
  • R57 pooled (11): elx-05-cache-single-flight, elx-07-cli-stats, elx-12-retry-api-deprecation, rails-ac-throttle-search, rails-aj-enqueue-after-commit, rails-ar-bulk-access-grants, rails-as-variant-processed-once, rails-ft-mysql-fulltext-search-foundation, rails-hw-scoped-broadcast, syn-13-bug-empty-filter-crash, syn-14-bug-sla-business-hours.

Round results · rounds/l0-reconcile/RESULTS.md

Exact cell IDs, task IDs, reps, arm assignments, outcomes, stop causes, effective model/effort, and public grade matches are recorded in the linked sanitized CSVs.

R72: 64/84 versus 60/84; stopped early

Recovered cohort Cell outcomes Public export Reconciliation
P1 Kogen Luna ladder, n=84 64 PASS, 19 FAIL, 1 INVALID 0 matching public scored rows Reproduced exactly: 64/84. Invalid remains in denominator and is not a pass.
P1 Codex Luna max, n=84 60 PASS, 24 FAIL 0 matching public scored rows Reproduced exactly: 60/84.
P3 partial, n=24 Kogen ladder 12/13; Codex Sol high 10/11 0 matching public scored rows Partial follow-on only; not included in the P1 claim.

The stopped-early description is consistent with the recovered record. One invalid Kogen P1 row has no effective model/effort receipt; its row is marked NOT-RECORDED in P1 data. The cell-level P1 and partial P3 rows are here and here. Source cohort label: INTERIM.

R74: stopped; no public scored export

Recovered cohort Count Reconciliation
Amendment C retained specs 398 Receipt count reproduced.
Internal scored grades 167 Recovered in the internal grade ledger; 37/57 Codex Luna max, 11/11 Codex Sol high, 51/60 Kogen Luna ladder, 33/36 Kogen Sol high ladder, 3/3 Kogen Sol medium ladder.
Smoke attempt grade rows 9 8 selected smoke rows and 1 withdrawn environment fault; excluded from the 167-cell scored subset.
Public scored export 0 Claim reproduced as a statement about public publication.
Retained slots without scored grade 231 Exact cell IDs and individual terminal statuses are not recovered.

The claim is true for the public export, while internal scored grades exist. Full retained-cohort ITT is not reconstructable from the captured receipts. This source round was withdrawn. See graded subset, smoke attempts, and coverage record. Source cohort label: WITHDRAWN.

R69: SPEC / APPROACH / ACCEPT / NONE

Source-reported, not reproducible from the public export: PASS / retained ITT was SPEC 63/99, APPROACH 66/99, ACCEPT 66/98, and NONE 59/100. The listed intervals are source-reported in the linked cohort reconciliation.

Pairwise comparison Difference 95% Newcombe interval One-sided task-stratified exact p
APPROACH − SPEC +3.03 pp −10.11 to +16.02 pp 0.252423469388 (1979/7840)
ACCEPT − SPEC +3.71 pp −9.45 to +16.68 pp 0.197455412923 (5309813/26891200)

The retained ITT contains 327 official grades and 69 ungraded ITT failures; four documented infrastructure exclusions were not replaced. Rejoining the refreshed 2026-10-07 official export by exact cell_id finds 150/327 grades (113 pass, 37 fail); the pinned export has SHA-256 4d639a3848b3661dcb5646630b6a4914e5053ff4b94ab029424d47d040e3c977. The prior 95/327 count applies only to the export snapshot at benchmark commit 58c9ca5ee40fb958c61c61e381191a922886f88f (2026-10-07T16:18:30+03:00; export SHA-256 cbf85effb6334c650fa0204da6e2777ed8a5a699f14ab854529d998b5177703a) and is superseded. The four-arm counts reproduce from the linked cohort CSV, but the row outcomes remain source-reported. The attempted configuration was kh-gpt, shell-only, Luna requested max clamped to effective xhigh, 64 turns/3600 seconds; per-cell model and effort are marked NOT-RUN where there was no effective run receipt. See R69 ITT data. No separate Kogen source commit is recorded.

R57 pooled shell-only versus default tools

Source-reported, not reproducible from the public export: PASS / scored valid cells was shell-only 114/155 and default tools 111/156; public grade matches were 155/155 and 156/156, respectively.

Method Difference One-sided 95% lower bound −10 pp rule
Published host × task weights; stratified Newcombe/MOVER +1.92 pp −4.70 pp Met
Equal-stratum sensitivity +0.00 pp −7.70 pp —

Source-reported, not reproducible from the public export: the counts and published interim bound reproduce in the linked cohort CSV. One planned Studio shell-only slot is missing; it has no recovered cell ID, grade, or terminal cause and is not counted as a failure. The source declaration specified the strata but not the estimator; these interval values reproduce the analyst-selected method documented in the pooled snapshot. Source cohort label: INTERIM. See R57 pooled data. Effective configuration for scored cells: kh-gpt, GPT-6 Luna, xhigh; shell-only exposes bash, default exposes read/write/edit/bash.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
reproduce/reconcile_l0.py
python3 reproduce/reconcile_l0.py
Inputs
Full execution replay
Unavailable
Scope
Recomputes selected reconciliation CSV summaries only; it does not replay historical model executions or complete the canonical official outcome join.

View the complete per-round reproduction inventory · Repository reproduction guide