> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [rounds/r57/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r57/README.md)
- [results/run-records/index.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/run-records/index.json)
- [results/run-records/r57.jsonl](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/run-records/r57.jsonl)
- [rounds/index.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/index.json)
- [results/source-crosswalk/index.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/source-crosswalk/index.json)
- [rounds/STATUS.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/STATUS.md)
- [reproduce/round-inventory.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/round-inventory.json)
- [reproduce/reconcile_l0.py](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/reconcile_l0.py)

# r57

**Round ID:** r57  
**Status:** INVALID

> Reviewed status from rounds/STATUS.md: INVALID.

> Round README and results text are source material. Numeric values not tied to a claim ledger record remain source-reported.

Round date: HISTORICAL (before 2026-10-09)
Publication badge: INVALID; KEPT FOR AUDIT
Why not VALID: The historical inventory retains INVALID because of ARM_MAPPING, EVIDENCE, NO_PREREG. See the round evidence and limitations below.
Recomputation status: OBSERVED SOURCE ONLY



## Status

**INVALID**

- NO_PREREG — No dated pre-registration is recorded in the round files.
- ARM_MAPPING — The control-default-tools pass count differs from official cells, and the label maps to multiple exported arms.
- EVIDENCE — The pooled r57–r57d cohort ledger and missing-cell identity are not linked.

## Required reproduction metadata

- Kogen commit: not applicable; the r57 exported records identify the Harness arm as not using Kogen.
- Harness commit: not recorded in the cited round source as a Git commit; any adapter or runtime fingerprints are not Git commits.
- Model and effort: a complete per-cell mapping is not recorded in the cited round source; values present in the source summary remain as stated above.
- Raw records: the cited sources do not provide a complete public round-specific request and grade bundle; public delivery and outcome exports are linked above where applicable.


Terminology: [public round glossary](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/GLOSSARY.md).

STATUS: **INVALID**

Data completeness: **49.9%** (41 deliveries; PASS WITH DECLARED GAPS). [Measurement contract](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r57/MEASURED.md); [declared gaps](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r57/MISSING.md).













Status reason: The control-default-tools pass count differs from official cells, and default-tools maps to multiple exported arms; the local table is removed. The pooled r57–r57d analysis remains unreproducible.





Pre-registered: no


Lifecycle: Historical capture. Snapshot 2026-10-05; earlier reports may use earlier cutoffs.

Question: Does shell-only reduce tokens without materially reducing accuracy?

Venue: r57 kogen-bench-us; r57b/r57d studio; r57c kogen-bench-eu.

Design: A pooled r57–r57d analysis was reported, but its cohort ledger and missing-cell identity are not linked; this page retains only the local r57 delivery record.

Decision rule: Lower shell median tokens in ≥80% strata; one-sided 95% accuracy lower bound ≥−10 pp.

Arms: default-tools and shell-only; Luna actual xhigh builder.

Planned tasks in the surviving round note: 

- [elx-05-cache-single-flight](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/elx-05-cache-single-flight/task.json)
- [elx-07-cli-stats](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/elx-07-cli-stats/task.json)
- [elx-12-retry-api-deprecation](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/elx-12-retry-api-deprecation/task.json)
- [rails-ac-throttle-search](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/rails-ac-throttle-search/task.json)
- [rails-aj-enqueue-after-commit](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/rails-aj-enqueue-after-commit/task.json)
- [rails-ar-bulk-access-grants](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/rails-ar-bulk-access-grants/task.json)
- [rails-as-variant-processed-once](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/rails-as-variant-processed-once/task.json)
- [rails-ft-mysql-fulltext-search-foundation](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/rails-ft-mysql-fulltext-search-foundation/task.json)
- [rails-hw-scoped-broadcast](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/rails-hw-scoped-broadcast/task.json)
- [syn-13-bug-empty-filter-crash](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/syn-13-bug-empty-filter-crash/task.json)
- [syn-14-bug-sla-business-hours](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/tasks/syn-14-bug-sla-business-hours/task.json)

Verdict: Outcome interpretation is withheld because the official cells export and captured run-record export cannot be crosswalked for this round. No benchmark comparison is reported.

## Public outcome reconciliation status

The prior delivery and outcome table is removed because its arm labels or outcome totals do not match the official [cells export](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/cells.jsonl). The captured-delivery ledger remains available in [r57.jsonl](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/run-records/r57.jsonl); no combined result is reported here.

## Reproduction record and source reconciliation

This section documents exact replay limits and keeps the page lifecycle and earlier statuses unchanged.

- Kogen source commit: `not applicable (Harness does not use Kogen)` in the per-cell `setup.kogen_sha` field.
- Harness source commit: not recorded. Per-cell `tools.harness` SHA-256 fingerprint(s): `b5a8d85d4bdcb8fa2445e42690dabc90bdde5b912e7b26c7ef5ba69b4dee8abb`. Only wrapper fingerprints are public; the underlying source commit remains unrecorded.
- Models and effort (requested → effective): `kh-plan: model requested/effective gpt-6-luna → chatgpt/gpt-6-luna; effort requested/effective max → xhigh; runner requested low`.
- Observed task IDs: `elx-05-cache-single-flight`, `elx-07-cli-stats`, `elx-12-retry-api-deprecation`, `syn-13-bug-empty-filter-crash`, `syn-14-bug-sla-business-hours`.
- Original execution command: not recorded in the public snapshot; the benchmark cells cannot be replayed from this repository.
- Rebuild the public metadata from the repository root with `python3 reproduce/export_results.py` and `python3 reproduce/build_records.py`; check this declaration with `python3 reproduce/validate_round.py --round r57`. These commands regenerate or validate the archived public records. An original cell-launch command is not available.
- Raw public records: [r57.jsonl](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/run-records/r57.jsonl) filtered by `audit_round`/`round_id`, and [cells.jsonl](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/cells.jsonl) filtered by `round`.

Public snapshot rows for this tag: 41 captured deliveries (34 pass, 7 fail, 0 ungraded/unknown in `results/run-records/r57.jsonl`); official outcome export has 41 rows: 34 pass, 7 fail in `cells.jsonl`. These are separate source totals. The public files do not provide a complete joined intention-to-treat result.

**Round-specific reconciliation:** The [L0 pooled CSV](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l0-reconcile/data/r57-pooled-itt.csv) and `python3 reproduce/reconcile_l0.py` reproduce a separate recovered 311-row pooled population, with public outcome matches for those 311 rows. One planned Studio shell-only slot has no recovered cell ID or outcome, so the full planned denominator remains incomplete. This does not resolve the control-arm mapping conflict on this page: default-tools maps to multiple public exported arms, so this round's own arm-level comparison is not reconstructed. Do not pool this page with r57b/r57c/r57d or the separate r57e replacement.

## Recomputed from raw records

[`recomputed.json`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r57/recomputed.json), [mined cell records](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/data/mined/r57.jsonl.gz), and the [token audit](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/data/mined/TOKEN-AUDIT.json) back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 45 rows (fail 10, pass 35); overall pass rate is 77.8% (35/45) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.

Published-export outcome cross-check: 41/41 shared cell IDs match; 0/41 public-export rows have no mined source record. Exact IDs and results for unpaired rows are in `recomputed.json`.
Recovered-source discrepancies: 2 field mismatches across 1 cell IDs (`patch.archive_member.483f7dfeaeda6af4162f696df1ed71c01450b09c7558588bdd1a55a055199375` 1, `patch.archive_member.cells/e6ccf777c8ffc919aad71d163d93fd192ff274e222a5eb9c981d026144723eed/attempt-1/patch.diff` 1). Existing values were preserved; each cell, field, kept value, and recovered value is listed below.
- `kogen-rails-best__gpt-6-luna__max__default__rails-ar-archive-book-access__r1-kogen-rails-best-r57-t90-studio-smoke` field `patch.archive_member.483f7dfeaeda6af4162f696df1ed71c01450b09c7558588bdd1a55a055199375`: kept `{"archive_member": null, "sha256": "483f7dfeaeda6af4162f696df1ed71c01450b09c7558588bdd1a55a055199375", "size_bytes": 9408}`, recovered `null`.
- `kogen-rails-best__gpt-6-luna__max__default__rails-ar-archive-book-access__r1-kogen-rails-best-r57-t90-studio-smoke` field `patch.archive_member.cells/e6ccf777c8ffc919aad71d163d93fd192ff274e222a5eb9c981d026144723eed/attempt-1/patch.diff`: kept `{"archive_member": "cells/e6ccf777c8ffc919aad71d163d93fd192ff274e222a5eb9c981d026144723eed/attempt-1/patch.diff", "sha256": "483f7dfeaeda6af4162f696df1ed71c01450b09c7558588bdd1a55a055199375", "size_bytes": 9408}`, recovered `null`.
No per-cell outcome mismatches were found in the shared IDs.
Token reconciliation under [METHOD.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/METHOD.md): 41/41 published metadata totals equal the manifest `input + cached input + output` sum; 0 differ (no mismatches). Reasoning remains a separate reported component and is not added to output.
The recompute keeps both numbers visible. 0 unexplained single-response numeric errors were identified; multi-response rows retain the published full-cell total and the manifest counter separately because per-response usage records are excluded.
This round also has 4 raw-only or non-public rows; they are kept separate from the published-export denominator.


## Captured run records

[results/run-records/r57.jsonl](/downloads/results/run-records/r57.jsonl) · 41 records · 258,226 bytes · [immutable source](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/run-records/r57.jsonl).


## Reproduction inventory

- **Type:** analysis-only
- **Script:** `python3 reproduce/reconcile_l0.py`
- **Inputs:** `rounds/l0-reconcile/data/r72-p1.csv`, `rounds/l0-reconcile/data/r72-p3-partial.csv`, `rounds/l0-reconcile/data/r74-graded-subset.csv`, `rounds/l0-reconcile/data/r74-smoke-attempts.csv`, `rounds/l0-reconcile/data/r74-coverage.csv`, `rounds/l0-reconcile/data/r69-itt.csv`, `rounds/l0-reconcile/data/r57-pooled-itt.csv`
- **Full execution replay:** unavailable
- **Scope:** Recomputes selected reconciliation CSV summaries only; it does not replay historical model executions or complete the canonical official outcome join.

See the [complete per-round reproduction inventory](/reproduce/) and the [repository reproduction guide](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/README.md).

## Related records

### Related experiments

- [r57](/rounds/r57/)

### Related claims

- [CL-WA-r57-public-export-coverage](/claims/cl-wa-r57-public-export-coverage/)
- [CL-WA-r57-source-reported-denominator-variants](/claims/cl-wa-r57-source-reported-denominator-variants/)
