Round l0-reconcile
L0 historical claim reconciliation
Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.
Round source README · rounds/l0-reconcile/README.md
Round date: HISTORICAL (before 2026-10-09) Publication badge: VALID; KEPT FOR AUDIT Recomputation status: NOT RECOMPUTABLE
Status
VALID — reconciliation only
Limits
- Restored to VALID 2026-10-08: the round executed its registered rule as registered and its numbers recompute; the points below are limits on public evidence, not contradictions in the round's own files.
- PRE_REGISTRATION_NOT_APPLICABLE: This retrospective reconciliation has no pre-registered result rule.
- EVIDENCE_INCOMPLETE: The full R74 scored cohort is not reconstructable from retained public records.
Lifecycle counts for this lane
This reconciliation added no benchmark cells; the historical cohort counts above belong to their source rounds.
| Lifecycle field | n | Basis |
|---|---|---|
| Planned | 0 | No new benchmark cells were planned in this reconciliation lane. |
| Started | 0 | No new benchmark cells were launched. |
| Finished | 0 | No new benchmark cells were run. |
| Officially graded | 0 | No new benchmark cells were graded. |
| ITT denominator | 0 | This lane generated no new benchmark outcomes. |
Required reproduction metadata
- Kogen commit: 260a73bc06be16e563cb4e56a168ffd078cec677 (the exact pin recorded in the cited round source, resolved against local Kogen history).
- Harness commit: not recorded in the cited round source as a Git commit; any adapter or runtime fingerprints are not Git commits.
- Historical execution command: not recorded in the cited round source.
- Raw records: the cited sources do not provide a complete public round-specific request and grade bundle; public delivery and outcome exports are linked above where applicable.
STATUS: VALID (reconciliation only)
Pre-registered: no new benchmark cells; this lane reconciles frozen historical receipts.
Label: DESCRIPTIVE
Question: Do the R72, R74, R69, and R57 historical claims match their recoverable cell level receipts?
n: R72 P1 168 cells and partial P3 24; R74 167 scored rows plus 9 smoke attempts; R69 400 assigned rows; R57 pooled 311 scored cells plus one unrecovered planned slot.
Headline: R72 P1 and R69 counts reproduce exactly; R57 counts and published interim bound reproduce; R74 has 167 internal scored grades but no public scored export, and its full retained cohort is not reconstructable.
Configuration: Historical configurations are retained per row in the cohort CSVs. R72 direct Codex CLI 0.160.0; R74 direct Codex CLI 0.160.0. R69 and R57 used kh-gpt; no separate harness version or Codex CLI version was recorded. Effective model and effort are listed per cell; NOT-RECORDED and NOT-RUN are explicit.
Limit: This reconciliation does not establish new performance estimates. R74 has 231 retained slots without recovered cell IDs or terminal statuses. R57 has one missing shell-only slot, excluded from the scored denominator. R57’s source declaration specified stratification but not an estimator; this page recomputes the analyst’s published estimator. The R74 cohort is withdrawn and mixed across time, venue, and configuration.
Sources: ROUNDS.md; LATEST.md; R72 round README and decision rule; R74 results/run-records/index.json, CODEX-POOL.md, amendment, status, and exclusion receipts; R69 itt.jsonl, decision rule, and verdict; R57-POOLED.md; official grades.final.jsonl and cells.jsonl; the probe grade ledger; and R2-EXPERIMENTS-LEVERS.md, GAPS.md, and MASTER-MAP.md. See reconstructed cohort data and the calculation script.
Reproduce
Run from the kogen-bench-sot repository root:
python3 reproduce/reconcile_l0.py
The standard-library-only script reads only the sanitized files below. Cell IDs, arms, tasks, reps, outcomes, stop causes, effective model/effort, and public-grade matches are in these files:
rounds/l0-reconcile/data/r72-p1.csvrounds/l0-reconcile/data/r72-p3-partial.csvrounds/l0-reconcile/data/r74-graded-subset.csvrounds/l0-reconcile/data/r74-smoke-attempts.csvrounds/l0-reconcile/data/r74-coverage.csvrounds/l0-reconcile/data/r69-itt.csvrounds/l0-reconcile/data/r57-pooled-itt.csv
Recorded Kogen commits:
- R72: 260a73bc06be16e563cb4e56a168ffd078cec677
- R74: cde7a380455e9793cb1fc8315791bc9beffb863b
- R57: 6747368333e55a694338d32312eb1f49ec2b4813
- R69: no Kogen source commit recorded.
Harness/CLI receipts: R72 and R74 direct Codex CLI 0.160.0; Kogen versions are pinned by the commits above. R69 and R57 kh-gpt; a separate version string was not recorded. The per-cell CSVs contain the observed effective model and effort, including role maps for Kogen cells.
Task IDs represented in the recovered data:
- R72 P1 (12):
elx-02-ingest-supervision,elx-04-queue-backpressure,elx-07-cli-stats,elx-12-retry-api-deprecation,elx-port-board-publish-unpublish-public-boundary,elx-port-erase-account,syn-01-live-ticket-filters,syn-06-migration-ticket-numbers,syn-15-validation-ticket-form,syn-20-email-invite-flow,syn-24-csv-import,syn-31-inbound-email-webhook. - R72 partial P3 (9):
elx-02-ingest-supervision,elx-04-queue-backpressure,elx-12-retry-api-deprecation,syn-01-live-ticket-filters,syn-06-migration-ticket-numbers,syn-15-validation-ticket-form,syn-20-email-invite-flow,syn-24-csv-import,syn-31-inbound-email-webhook. - R74 scored subset (12):
elx-02-ingest-supervision,elx-04-queue-backpressure,elx-07-cli-stats,elx-12-retry-api-deprecation,elx-port-board-publish-unpublish-public-boundary,elx-port-erase-account,syn-01-live-ticket-filters,syn-06-migration-ticket-numbers,syn-15-validation-ticket-form,syn-20-email-invite-flow,syn-24-csv-import,syn-31-inbound-email-webhook. This is the recovered scored subset; task IDs for the 231 ungraded retained slots are not recovered. - R69 (25):
elx-02-ingest-supervision,elx-04-queue-backpressure,rails-ac-throttle-search,rails-aj-enqueue-after-commit,rails-ar-archive-book-access,rails-ar-bulk-access-grants,rails-as-variant-processed-once,rails-ft-active-storage-tracking,rails-ft-activity-feed-api,rails-ft-board-publish-unpublish-public-boundary,rails-ft-cancelled-account-cleanup,rails-ft-entropy-sweep,rails-ft-i18n-support,rails-ft-mysql-fulltext-search-foundation,rails-ft-notification-bundle-window-overlap,rails-ft-saas-billing,rails-hw-scoped-broadcast,rails-sec-audit-sweep,rails-sup-legacy-conversions,syn-01-live-ticket-filters,syn-06-migration-ticket-numbers,syn-15-validation-ticket-form,syn-20-email-invite-flow,syn-24-csv-import,syn-31-inbound-email-webhook. - R57 pooled (11):
elx-05-cache-single-flight,elx-07-cli-stats,elx-12-retry-api-deprecation,rails-ac-throttle-search,rails-aj-enqueue-after-commit,rails-ar-bulk-access-grants,rails-as-variant-processed-once,rails-ft-mysql-fulltext-search-foundation,rails-hw-scoped-broadcast,syn-13-bug-empty-filter-crash,syn-14-bug-sla-business-hours.
Round results · rounds/l0-reconcile/RESULTS.md
Exact cell IDs, task IDs, reps, arm assignments, outcomes, stop causes, effective model/effort, and public grade matches are recorded in the linked sanitized CSVs.
R72: 64/84 versus 60/84; stopped early
| Recovered cohort | Cell outcomes | Public export | Reconciliation |
|---|---|---|---|
| P1 Kogen Luna ladder, n=84 | 64 PASS, 19 FAIL, 1 INVALID | 0 matching public scored rows | Reproduced exactly: 64/84. Invalid remains in denominator and is not a pass. |
| P1 Codex Luna max, n=84 | 60 PASS, 24 FAIL | 0 matching public scored rows | Reproduced exactly: 60/84. |
| P3 partial, n=24 | Kogen ladder 12/13; Codex Sol high 10/11 | 0 matching public scored rows | Partial follow-on only; not included in the P1 claim. |
The stopped-early description is consistent with the recovered record. One invalid Kogen P1 row has no effective model/effort receipt; its row is marked NOT-RECORDED in P1 data. The cell-level P1 and partial P3 rows are here and here. Source cohort label: INTERIM.
R74: stopped; no public scored export
| Recovered cohort | Count | Reconciliation |
|---|---|---|
| Amendment C retained specs | 398 | Receipt count reproduced. |
| Internal scored grades | 167 | Recovered in the internal grade ledger; 37/57 Codex Luna max, 11/11 Codex Sol high, 51/60 Kogen Luna ladder, 33/36 Kogen Sol high ladder, 3/3 Kogen Sol medium ladder. |
| Smoke attempt grade rows | 9 | 8 selected smoke rows and 1 withdrawn environment fault; excluded from the 167-cell scored subset. |
| Public scored export | 0 | Claim reproduced as a statement about public publication. |
| Retained slots without scored grade | 231 | Exact cell IDs and individual terminal statuses are not recovered. |
The claim is true for the public export, while internal scored grades exist. Full retained-cohort ITT is not reconstructable from the captured receipts. This source round was withdrawn. See graded subset, smoke attempts, and coverage record. Source cohort label: WITHDRAWN.
R69: SPEC / APPROACH / ACCEPT / NONE
Source-reported, not reproducible from the public export: PASS / retained ITT was SPEC 63/99, APPROACH 66/99, ACCEPT 66/98, and NONE 59/100. The listed intervals are source-reported in the linked cohort reconciliation.
| Pairwise comparison | Difference | 95% Newcombe interval | One-sided task-stratified exact p |
|---|---|---|---|
| APPROACH − SPEC | +3.03 pp | −10.11 to +16.02 pp | 0.252423469388 (1979/7840) |
| ACCEPT − SPEC | +3.71 pp | −9.45 to +16.68 pp | 0.197455412923 (5309813/26891200) |
The retained ITT contains 327 official grades and 69 ungraded ITT failures; four documented infrastructure exclusions were not replaced. Rejoining the refreshed 2026-10-07 official export by exact cell_id finds 150/327 grades (113 pass, 37 fail); the pinned export has SHA-256 4d639a3848b3661dcb5646630b6a4914e5053ff4b94ab029424d47d040e3c977. The prior 95/327 count applies only to the export snapshot at benchmark commit 58c9ca5ee40fb958c61c61e381191a922886f88f (2026-10-07T16:18:30+03:00; export SHA-256 cbf85effb6334c650fa0204da6e2777ed8a5a699f14ab854529d998b5177703a) and is superseded. The four-arm counts reproduce from the linked cohort CSV, but the row outcomes remain source-reported. The attempted configuration was kh-gpt, shell-only, Luna requested max clamped to effective xhigh, 64 turns/3600 seconds; per-cell model and effort are marked NOT-RUN where there was no effective run receipt. See R69 ITT data. No separate Kogen source commit is recorded.
R57 pooled shell-only versus default tools
Source-reported, not reproducible from the public export: PASS / scored valid cells was shell-only 114/155 and default tools 111/156; public grade matches were 155/155 and 156/156, respectively.
| Method | Difference | One-sided 95% lower bound | −10 pp rule |
|---|---|---|---|
| Published host × task weights; stratified Newcombe/MOVER | +1.92 pp | −4.70 pp | Met |
| Equal-stratum sensitivity | +0.00 pp | −7.70 pp | — |
Source-reported, not reproducible from the public export: the counts and published interim bound reproduce in the linked cohort CSV. One planned Studio shell-only slot is missing; it has no recovered cell ID, grade, or terminal cause and is not counted as a failure. The source declaration specified the strata but not the estimator; these interval values reproduce the analyst-selected method documented in the pooled snapshot. Source cohort label: INTERIM. See R57 pooled data. Effective configuration for scored cells: kh-gpt, GPT-6 Luna, xhigh; shell-only exposes bash, default exposes read/write/edit/bash.
Captured run records
Standard-schema records assigned to this round in the indexed snapshot.
No per-round file is listed. Check the run-record index for the complete partition list.
Reproduction inventory
Published analysis replay and full experiment replay are reported separately.
- Replay type
analysis-only- Script
- reproduce/reconcile_l0.py
python3 reproduce/reconcile_l0.py - Inputs
- Full execution replay
- Unavailable
- Scope
- Recomputes selected reconciliation CSV summaries only; it does not replay historical model executions or complete the canonical official outcome join.
View the complete per-round reproduction inventory · Repository reproduction guide