Round r57
r57
Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.
Round source README · rounds/r57/README.md
Round date: HISTORICAL (before 2026-10-09) Publication badge: INVALID; KEPT FOR AUDIT Why not VALID: The historical inventory retains INVALID because of ARM_MAPPING, EVIDENCE, NO_PREREG. See the round evidence and limitations below. Recomputation status: OBSERVED SOURCE ONLY
Status
INVALID
- NO_PREREG — No dated pre-registration is recorded in the round files.
- ARM_MAPPING — The control-default-tools pass count differs from official cells, and the label maps to multiple exported arms.
- EVIDENCE — The pooled r57–r57d cohort ledger and missing-cell identity are not linked.
Required reproduction metadata
- Kogen commit: not applicable; the r57 exported records identify the Harness arm as not using Kogen.
- Harness commit: not recorded in the cited round source as a Git commit; any adapter or runtime fingerprints are not Git commits.
- Model and effort: a complete per-cell mapping is not recorded in the cited round source; values present in the source summary remain as stated above.
- Raw records: the cited sources do not provide a complete public round-specific request and grade bundle; public delivery and outcome exports are linked above where applicable.
Terminology: public round glossary.
STATUS: INVALID
Data completeness: 49.9% (41 deliveries; PASS WITH DECLARED GAPS). Measurement contract; declared gaps.
Status reason: The control-default-tools pass count differs from official cells, and default-tools maps to multiple exported arms; the local table is removed. The pooled r57–r57d analysis remains unreproducible.
Pre-registered: no
Lifecycle: Historical capture. Snapshot 2026-10-05; earlier reports may use earlier cutoffs.
Question: Does shell-only reduce tokens without materially reducing accuracy?
Venue: r57 kogen-bench-us; r57b/r57d studio; r57c kogen-bench-eu.
Design: A pooled r57–r57d analysis was reported, but its cohort ledger and missing-cell identity are not linked; this page retains only the local r57 delivery record.
Decision rule: Lower shell median tokens in ≥80% strata; one-sided 95% accuracy lower bound ≥−10 pp.
Arms: default-tools and shell-only; Luna actual xhigh builder.
Planned tasks in the surviving round note:
- elx-05-cache-single-flight
- elx-07-cli-stats
- elx-12-retry-api-deprecation
- rails-ac-throttle-search
- rails-aj-enqueue-after-commit
- rails-ar-bulk-access-grants
- rails-as-variant-processed-once
- rails-ft-mysql-fulltext-search-foundation
- rails-hw-scoped-broadcast
- syn-13-bug-empty-filter-crash
- syn-14-bug-sla-business-hours
Verdict: Outcome interpretation is withheld because the official cells export and captured run-record export cannot be crosswalked for this round. No benchmark comparison is reported.
Public outcome reconciliation status
The prior delivery and outcome table is removed because its arm labels or outcome totals do not match the official cells export. The captured-delivery ledger remains available in r57.jsonl; no combined result is reported here.
Reproduction record and source reconciliation
This section documents exact replay limits and keeps the page lifecycle and earlier statuses unchanged.
- Kogen source commit:
not applicable (Harness does not use Kogen)in the per-cellsetup.kogen_shafield. - Harness source commit: not recorded. Per-cell
tools.harnessSHA-256 fingerprint(s):b5a8d85d4bdcb8fa2445e42690dabc90bdde5b912e7b26c7ef5ba69b4dee8abb. Only wrapper fingerprints are public; the underlying source commit remains unrecorded. - Models and effort (requested → effective):
kh-plan: model requested/effective gpt-6-luna → chatgpt/gpt-6-luna; effort requested/effective max → xhigh; runner requested low. - Observed task IDs:
elx-05-cache-single-flight,elx-07-cli-stats,elx-12-retry-api-deprecation,syn-13-bug-empty-filter-crash,syn-14-bug-sla-business-hours. - Original execution command: not recorded in the public snapshot; the benchmark cells cannot be replayed from this repository.
- Rebuild the public metadata from the repository root with
python3 reproduce/export_results.pyandpython3 reproduce/build_records.py; check this declaration withpython3 reproduce/validate_round.py --round r57. These commands regenerate or validate the archived public records. An original cell-launch command is not available. - Raw public records: r57.jsonl filtered by
audit_round/round_id, and cells.jsonl filtered byround.
Public snapshot rows for this tag: 41 captured deliveries (34 pass, 7 fail, 0 ungraded/unknown in results/run-records/r57.jsonl); official outcome export has 41 rows: 34 pass, 7 fail in cells.jsonl. These are separate source totals. The public files do not provide a complete joined intention-to-treat result.
Round-specific reconciliation: The L0 pooled CSV and python3 reproduce/reconcile_l0.py reproduce a separate recovered 311-row pooled population, with public outcome matches for those 311 rows. One planned Studio shell-only slot has no recovered cell ID or outcome, so the full planned denominator remains incomplete. This does not resolve the control-arm mapping conflict on this page: default-tools maps to multiple public exported arms, so this round's own arm-level comparison is not reconstructed. Do not pool this page with r57b/r57c/r57d or the separate r57e replacement.
Recomputed from raw records
`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 45 rows (fail 10, pass 35); overall pass rate is 77.8% (35/45) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.
Published-export outcome cross-check: 41/41 shared cell IDs match; 0/41 public-export rows have no mined source record. Exact IDs and results for unpaired rows are in recomputed.json.
Recovered-source discrepancies: 2 field mismatches across 1 cell IDs (patch.archive_member.483f7dfeaeda6af4162f696df1ed71c01450b09c7558588bdd1a55a055199375 1, patch.archive_member.cells/e6ccf777c8ffc919aad71d163d93fd192ff274e222a5eb9c981d026144723eed/attempt-1/patch.diff 1). Existing values were preserved; each cell, field, kept value, and recovered value is listed below.
kogen-rails-best__gpt-6-luna__max__default__rails-ar-archive-book-access__r1-kogen-rails-best-r57-t90-studio-smokefieldpatch.archive_member.483f7dfeaeda6af4162f696df1ed71c01450b09c7558588bdd1a55a055199375: kept{"archive_member": null, "sha256": "483f7dfeaeda6af4162f696df1ed71c01450b09c7558588bdd1a55a055199375", "size_bytes": 9408}, recoverednull.kogen-rails-best__gpt-6-luna__max__default__rails-ar-archive-book-access__r1-kogen-rails-best-r57-t90-studio-smokefieldpatch.archive_member.cells/e6ccf777c8ffc919aad71d163d93fd192ff274e222a5eb9c981d026144723eed/attempt-1/patch.diff: kept{"archive_member": "cells/e6ccf777c8ffc919aad71d163d93fd192ff274e222a5eb9c981d026144723eed/attempt-1/patch.diff", "sha256": "483f7dfeaeda6af4162f696df1ed71c01450b09c7558588bdd1a55a055199375", "size_bytes": 9408}, recoverednull. No per-cell outcome mismatches were found in the shared IDs. Token reconciliation under METHOD.md: 41/41 published metadata totals equal the manifestinput + cached input + outputsum; 0 differ (no mismatches). Reasoning remains a separate reported component and is not added to output. The recompute keeps both numbers visible. 0 unexplained single-response numeric errors were identified; multi-response rows retain the published full-cell total and the manifest counter separately because per-response usage records are excluded. This round also has 4 raw-only or non-public rows; they are kept separate from the published-export denominator.
Captured run records
Standard-schema records assigned to this round in the indexed snapshot.
Reproduction inventory
Published analysis replay and full experiment replay are reported separately.
- Replay type
analysis-only- Script
- reproduce/reconcile_l0.py
python3 reproduce/reconcile_l0.py - Inputs
- Full execution replay
- Unavailable
- Scope
- Recomputes selected reconciliation CSV summaries only; it does not replay historical model executions or complete the canonical official outcome join.
View the complete per-round reproduction inventory · Repository reproduction guide