Round r70-rve-rerun
Round 70 fixed-skeleton rerun
Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.
Round source README · rounds/r70-rve-rerun/README.md
Round date: HISTORICAL (before 2026-10-09) Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Why not VALID: The historical inventory retains DESCRIPTIVE because of SENSITIVITY_ONLY, SEPARATE_COHORTS, UNMATCHED_CONDITIONS. See the round evidence and limitations below. Recomputation status: OBSERVED SOURCE ONLY
Recomputation update: The mined raw records now provide per-cell outcome, manifest token-counter, and wall-time recomputation for the observed cohort. This does not reconstruct missing planned cells, a full intention-to-treat denominator, or model execution.
Status
DESCRIPTIVE
- SENSITIVITY_ONLY — The README describes this as a sensitivity check, not a standalone preregistered comparison.
- UNMATCHED_CONDITIONS — The rerun changes stdout encoding and front-end handling relative to the original cohort.
- SEPARATE_COHORTS — Original and rerun cells are explicitly separate cohorts.
Required reproduction metadata
- Kogen commit: not recorded in the cited round source.
- Harness commit: not recorded in the cited round source as a Git commit; any adapter or runtime fingerprints are not Git commits.
- Model and effort: a complete per-cell mapping is not recorded in the cited round source; values present in the source summary remain as stated above.
- Task IDs: r70-1-elixir-fe2, r70-4-elixir-fe2, r70-4-go-fe2, r70-4-rust-fe2, r70-4-ts-bun-fe2; the complete scored-cell mapping is not recorded where noted above.
- Historical execution command: not recorded in the cited round source.
- Raw records: the cited sources do not provide a complete public round-specific request and grade bundle; public delivery and outcome exports are linked above where applicable.
Terminology: public round glossary.
STATUS: DESCRIPTIVE
Data completeness (standard run-record capture): N/A (0 deliveries; NO DELIVERED DATA). Supplemental evidence: 15 official grades; 36 control receipts. Measurement contract; declared gaps.
Pre-registered: yes; descriptive sensitivity check. Label: DESCRIPTIVE Question: Are the task-1 Elixir results sensitive to fixed stdout encoding and task-4 results sensitive to fixed front-end handling? n: 15 official rerun grades: three for task-1 Elixir and three for each task-4 stack. Headline: The FE2 rerun recorded 11/15 full passes across the planned cells. Evidence note: The public test-count and control ledgers identify the scored cells, admission controls, and X-controls. Task-4 fixed-front-end controls did not pass for Rust, Elixir, and TypeScript/Bun; the reported counts are descriptive.
Design: 40 original RvE cells and 15 pre-registered FE2 rerun cells (task 1 Elixir; task 4 all four stacks); the outcomes are reported as separate cohorts.
Configuration: direct Codex, gpt-6-luna at max; n=3 for each rerun variant. Task-1 Elixir and task-4 Rust/Elixir ran on kogen-bench-eu; task-4 Go/TypeScript-Bun ran on kogen-bench-us. Wall comparisons are within-host only. The supplemental rows do not include a verified per-cell harness version. Original cohort context: 40-cell as-graded RvE comparison.
Limit: Task-4 admission controls failed for Rust, Elixir, and TypeScript/Bun, preventing causal front-end or stack conclusions.
Sources: decision rule; rerun outcomes; test-count ledger; cost/time ledger; controls ledger; frozen plan.
Recomputed from raw records
`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 30 rows (fail 8, invalid 6, pass 16); overall pass rate is 66.7% (16/24) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.
No shared cell IDs are present in the committed public grade export; the source-only cohort is checked against the page's reported figures below.
Recovered-source discrepancies: 20 field mismatches across 5 cell IDs (official_grade.classification 5, official_grade.outcome 5, official_grade.pass_fail 5, official_grade.test_counts.tests_passed 5). Existing values were preserved; each cell, field, kept value, and recovered value is listed below.
codex__gpt-6-luna__max__default__r70-1-elixir-fe2__r6fieldofficial_grade.classification: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-1-elixir-fe2__r6fieldofficial_grade.outcome: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-1-elixir-fe2__r6fieldofficial_grade.pass_fail: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-1-elixir-fe2__r6fieldofficial_grade.test_counts.tests_passed: kept25, recovered1.codex__gpt-6-luna__max__default__r70-4-go-fe2__r10fieldofficial_grade.classification: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r10fieldofficial_grade.outcome: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r10fieldofficial_grade.pass_fail: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r10fieldofficial_grade.test_counts.tests_passed: kept18, recovered0.codex__gpt-6-luna__max__default__r70-4-go-fe2__r13fieldofficial_grade.classification: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r13fieldofficial_grade.outcome: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r13fieldofficial_grade.pass_fail: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r13fieldofficial_grade.test_counts.tests_passed: kept18, recovered0.codex__gpt-6-luna__max__default__r70-4-go-fe2__r9fieldofficial_grade.classification: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r9fieldofficial_grade.outcome: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r9fieldofficial_grade.pass_fail: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r9fieldofficial_grade.test_counts.tests_passed: kept18, recovered0.codex__gpt-6-luna__max__default__r70-4-go-fe2__r919fieldofficial_grade.classification: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r919fieldofficial_grade.outcome: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r919fieldofficial_grade.pass_fail: kept"pass", recovered"fail".codex__gpt-6-luna__max__default__r70-4-go-fe2__r919fieldofficial_grade.test_counts.tests_passed: kept18, recovered0. No shared public-export cell IDs are available for a paired outcome comparison. No complete published metadata/manifest token pairs are available for this round (0 missing manifest usage; 30 missing published metadata). Missing counters remain unknown, not zero. This round also has 30 raw-only or non-public rows; they are kept separate from the published-export denominator.
Round results · rounds/r70-rve-rerun/RESULTS.md
STATUS: CONFOUNDED — DESCRIPTIVE
The 15 official FE2 rerun grades and the matching original cells are identified in the public test-count ledger. The latest official grade per exact cell ID is used; earlier invalidated rows are excluded. Test counts, runner status, and timestamps are reported per cell in that ledger.
Configuration and sample: direct Codex, gpt-6-luna at max; n=3 for task-1 Elixir and n=3 for each task-4 stack. Task-1 Elixir and task-4 Rust/Elixir ran on kogen-bench-eu; task-4 Go/TypeScript-Bun ran on kogen-bench-us. Wall comparisons are within-host only. This is one model and the task-4 controls failed for three stacks, so results remain descriptive. Original cohort context: 40-cell as-graded RvE comparison.
| Task | Language | Original outcome fraction | FE2 outcome fraction |
|---|---|---|---|
| Task 1 | Elixir | 0/3 | 3/3 |
| Task 4 | Rust | 1/3 | 3/3 |
| Task 4 | Elixir | 0/3 | 0/3 |
| Task 4 | Go | 1/3 | 3/3 |
| Task 4 | TypeScript/Bun | 0/3 | 2/3 |
| All rerun cells | — | — | 11/15 |
Per-repetition hidden-suite fractions for the original and FE2 cohorts:
| Task | Stack | Cohort | Rep | Hidden tests passed/total | Official result |
|---|---|---|---|---|---|
| 1 | Elixir | original | 6 | 24/25 | fail |
| 1 | Elixir | original | 12 | 24/25 | fail |
| 1 | Elixir | original | 15 | 24/25 | fail |
| 1 | Elixir | FE2 rerun | 6 | 25/25 | pass |
| 1 | Elixir | FE2 rerun | 12 | 25/25 | pass |
| 1 | Elixir | FE2 rerun | 15 | 25/25 | pass |
| 4 | Rust | original | 9 | 18/18 | pass |
| 4 | Rust | original | 10 | 16/18 | fail |
| 4 | Rust | original | 13 | 16/18 | fail |
| 4 | Rust | FE2 rerun | 9 | 18/18 | pass |
| 4 | Rust | FE2 rerun | 10 | 18/18 | pass |
| 4 | Rust | FE2 rerun | 13 | 18/18 | pass |
| 4 | Elixir | original | 9 | 15/18 | fail |
| 4 | Elixir | original | 10 | 15/18 | fail |
| 4 | Elixir | original | 13 | 17/18 | fail |
| 4 | Elixir | FE2 rerun | 9 | 17/18 | fail |
| 4 | Elixir | FE2 rerun | 10 | 17/18 | fail |
| 4 | Elixir | FE2 rerun | 13 | 17/18 | fail |
| 4 | Go | original | 19 | 18/18 | pass |
| 4 | Go | original | 20 | 17/18 | fail |
| 4 | Go | original | 23 | 16/18 | fail |
| 4 | Go | FE2 rerun | 9 | 18/18 | pass |
| 4 | Go | FE2 rerun | 10 | 18/18 | pass |
| 4 | Go | FE2 rerun | 13 | 18/18 | pass |
| 4 | TypeScript/Bun | original | 19 | 15/18 | fail |
| 4 | TypeScript/Bun | original | 20 | 16/18 | fail |
| 4 | TypeScript/Bun | original | 23 | 16/18 | fail |
| 4 | TypeScript/Bun | FE2 rerun | 9 | 17/18 | fail |
| 4 | TypeScript/Bun | FE2 rerun | 10 | 18/18 | pass |
| 4 | TypeScript/Bun | FE2 rerun | 13 | 18/18 | pass |
Task-level hidden-test totals (these are sums across three cells, not independent samples):
| Task | Stack | Original hidden-test counts | FE2 hidden-test counts |
|---|---|---|---|
| 1 | Elixir | each 24/25 | each 25/25 |
| 4 | Rust | 50/54 | 54/54 |
| 4 | Go | 51/54 | 54/54 |
| 4 | TypeScript/Bun | 47/54 | 53/54 |
| 4 | Elixir | 47/54 | 51/54 |
The earlier grades from a worker that failed to apply contestant patches were invalidated. The selected baseline rows record grader SHA-256 616b1d53fe6ee56376134f0cf6a9350e94b5ecb7ec5b84f46a93e3c677356416; the selected r70-4-rust row records grader SHA-256 5fe9ea3fa0cb748e3a60d4e9153e41c61ec735bd3b5d6590dc7708e7343ae91f. These are grader fingerprints, not worker identifiers. For task 4, Go used front end v1; Rust, Elixir, and TypeScript/Bun used front end v2. These are provenance labels: the observed task-4 results do not establish a causal front-end effect.
The controls ledger identifies each admission and X-control row. It records failed and invalid fixed-front-end admission controls for task 4; the official grade ledger also contains the task-4 contestant cells for those variants. The task-4 counts are observed outcomes; the control record does not identify a causal front-end effect. The task-1 Elixir result is descriptive: the three original cells returned 24/25 and the three FE2 cells returned 25/25, consistent with an encoding confound.
All controls, smokes, and X-controls are excluded from the scored denominator. Per-cell token counters and wall time for these 15 rerun cells are in the cost/time ledger, but no rerun resource comparison is made here. Cost and reasoning counters are unavailable for these rows and remain null in the ledger.
Captured run records
Standard-schema records assigned to this round in the indexed snapshot.
No per-round file is listed. Check the run-record index for the complete partition list.
Reproduction inventory
Published analysis replay and full experiment replay are reported separately.
- Replay type
unavailable- Script
- No per-round analysis script is published.
- Inputs
- No identified public input bundle.
- Full execution replay
- Unavailable
- Scope
- No public per-round analysis script with a complete, identified input bundle is published. See the round README, MEASURED.md, and MISSING.md for recorded scope and gaps; historical execution inputs/toolchains/grading are not bundled.
View the complete per-round reproduction inventory · Repository reproduction guide