Round r70-rve-ext
r70-RvE language extension
Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.
Round source README · rounds/r70-rve-ext/README.md
Round date: HISTORICAL (before 2026-10-09) Publication badge: VALID; KEPT FOR AUDIT Recomputation status: OBSERVED SOURCE ONLY
Recomputation update: The mined raw records now provide per-cell outcome, manifest token-counter, and wall-time recomputation for the observed cohort. This does not reconstruct missing planned cells, a full intention-to-treat denominator, or model execution.
Status
VALID — registered reps 31–32 only; rep 33 is post-hoc and reported separately
Limits
- Restored to VALID 2026-10-08: the round executed its registered rule as registered and its numbers recompute; the points below are limits on public evidence, not contradictions in the round's own files.
- PARTIAL_PREREG — Only reps 31–32 are stated as pre-registered; rep 33 is explicitly post-hoc.
- NO_PREREG_EVIDENCE — The files do not establish that the limited registration predates the first result.
- SCOPE — The reported cohort mixes registered and post-hoc cells.
Required reproduction metadata
- Kogen commit: not recorded in the cited round source.
- Harness commit: not recorded in the cited round source as a Git commit; any adapter or runtime fingerprints are not Git commits.
- Model and effort: a complete per-cell mapping is not recorded in the cited round source; values present in the source summary remain as stated above.
- Task IDs: r70-2-elixir, r70-2-go, r70-2-rust, r70-2-ts-bun, r70-5-elixir, r70-5-go, r70-5-rust, r70-5-ts-bun, r70-7-elixir, r70-7-go, r70-7-rust, r70-7-ts-bun; the complete scored-cell mapping is not recorded where noted above.
- Historical execution command: not recorded in the cited round source.
- Raw records: the cited sources do not provide a complete public round-specific request and grade bundle; public delivery and outcome exports are linked above where applicable.
Terminology: public round glossary.
STATUS: VALID — registered decision covers the 24 pre-registered rep 31–32 cells; the 5 post-hoc rep-33 cells are reported separately and are outside the registered rule.
Data completeness (standard run-record capture): N/A (0 deliveries; NO DELIVERED DATA). Supplemental evidence: 29 official grades; 60 control receipts. Measurement contract; declared gaps.
Pre-registered: yes for reps 31–32; design date 6 October 2026. Label: DESCRIPTIVE Question: What hidden-suite full-pass outcomes did Rust, Elixir, Go, and TypeScript/Bun record on tasks 2, 5, and 7, alongside the prior Round 70 language observations? n: 24 pre-registered cells at reps 31–32; 5 post-hoc rep-33 cells; 29 official grades total. Headline: Reps 31–32 per stack: Rust 5/6, Elixir 4/6, Go 4/6, and TypeScript/Bun 6/6. Evidence note: The public test-count and controls ledgers identify the rep-31 smokes, admission controls, and contestant-path X-controls. All smokes met Rule L; admission controls and X-controls had their expected outcomes. Configuration: direct Codex, gpt-6-luna at max; n=2 per task and stack for pre-registered reps 31–32, with selected post-hoc rep-33 additions. Rust and Elixir ran on kogen-bench-eu; Go and TypeScript/Bun ran on kogen-bench-us. Wall comparisons are within-host only. Per-cell token and wall medians are reported separately for pre-registered reps 31–32 and post-hoc rep 33 in the results; the cost/time ledger provides per-cell token and wall fields. Original cohort context: 40-cell as-graded RvE comparison. Limit: Descriptive, small per-task samples, one model, and author-selected task templates. The extension observations do not establish differences among stacks.
Sources: decision rule; results; test-count ledger; cost/time ledger; controls ledger; sanitized official grade rows; public cell export.
Recomputed from raw records
`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 41 rows (fail 6, pass 35); overall pass rate is 85.4% (35/41) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.
Published-export outcome cross-check: 29/29 shared cell IDs match; 0/29 public-export rows have no mined source record. Exact IDs and results for unpaired rows are in recomputed.json.
Arm-label mismatches between public and raw records: 29; exact labels and cell IDs are in recomputed.json.
No per-cell outcome mismatches were found in the shared IDs.
Token reconciliation under METHOD.md: 29/29 published metadata totals equal the manifest input + cached input + output sum; 0 differ (no mismatches). Reasoning remains a separate reported component and is not added to output.
The recompute keeps both numbers visible. 0 unexplained single-response numeric errors were identified; multi-response rows retain the published full-cell total and the manifest counter separately because per-response usage records are excluded.
This round also has 12 raw-only or non-public rows; they are kept separate from the published-export denominator.
Round results · rounds/r70-rve-ext/RESULTS.md
STATUS: VALID — DESCRIPTIVE
The primary outcome is official hidden-suite full pass. A timeout counts as a failure. The public cell-level test-count ledger identifies the 40 original RvE cells, 15 fixed-front-end rerun cells, and 29 extension cells. Earlier invalidated grade rows are excluded; the latest official row per exact cell ID is retained.
Configuration and sample: direct Codex, gpt-6-luna at max. The pre-registered extension used n=2 per task and stack at reps 31–32; rep 33 is a selected post-hoc addition. Rust and Elixir ran on kogen-bench-eu; Go and TypeScript/Bun ran on kogen-bench-us. Wall time is comparable only within a host. Effective model/effort and harness-version receipts are unavailable for these supplemental rows. Original cohort context: 40-cell as-graded RvE comparison.
Extension outcomes
Pre-registered reps 31–32 per stack: Rust 5/6, Elixir 4/6, Go 4/6, and TypeScript/Bun 6/6. Rep 33 is a separate post-hoc addition.
Per-task counts include the official extension grades at reps 31–33 where present. Test fractions and exact cell IDs are in test-counts.jsonl.
| Task ID | Exact arm label in export | Passes (all included reps) |
|---|---|---|
| r70-2-rust | rust | 2/2 |
| r70-2-elixir | elixir | 1/3 |
| r70-2-go | go | 2/3 |
| r70-2-ts-bun | ts-bun | 2/2 |
| r70-5-rust | rust | 2/2 |
| r70-5-elixir | elixir | 2/2 |
| r70-5-go | go | 2/3 |
| r70-5-ts-bun | ts-bun | 2/2 |
| r70-7-rust | rust | 2/3 |
| r70-7-elixir | elixir | 2/3 |
| r70-7-go | go | 2/2 |
| r70-7-ts-bun | ts-bun | 2/2 |
| Language | Reps 31–32 | Reps 31–33 (rep 33 post-hoc) |
|---|---|---|
| Rust | 5/6 | 6/7 |
| Elixir | 4/6 | 5/8 |
| Go | 4/6 | 6/8 |
| TypeScript/Bun | 6/6 | 6/6 |
These counts describe the observed cells. They do not establish differences among stacks. The timeout cell remains a failure even though its delivered state received a full test count.
Controls
The controls ledger contains the admission and X-control cell IDs, outcomes, test counts, timestamps, and available patch hashes. The 48 reference/no-op admission controls had their expected outcomes: every reference passed and every no-op failed. All 12 contestant-path X-controls passed. The admission controls and X-controls ran on kogen-bench-eu, while the Go and TypeScript/Bun cells ran on kogen-bench-us; the controls validate the task and the grader, not the host. The 12 rep-31 smoke cells met Rule L. Controls are not scored cells; the rep-31 smokes are scored and counted in the 24 pre-registered cells.
Seven-task descriptive counts
Task-4 admission controls failed for Rust, Elixir, and TypeScript/Bun. The pre-registered fixed-front-end rerun cells are shown in a separate column; original as-graded outcomes are kept. These totals do not establish a corrected stack comparison.
| Language | Original tasks 1/3/4/6 (as graded) | FE2 rerun cohort (15 cells) | Original + extension reps 31–32 | Original + extension reps 31–33 |
|---|---|---|---|---|
| Rust | 8/10 | 3/3 | 13/16 | 14/17 |
| TypeScript/Bun | 7/10 | 2/3 | 13/16 | 13/16 |
| Go | 7/10 | 3/3 | 11/16 | 13/18 |
| Elixir | 2/10 | 3/6 | 6/16 | 7/18 |
Original as-graded outcomes plus extension reps 31–32, excluding the separate FE2 rerun cells: Rust 13/16, TypeScript/Bun 13/16, Go 11/16, and Elixir 6/16.
Illustrative mixed cohort with FE2 substitution
This is an illustrative mixed cohort, not a corrected estimate. It substitutes the separate FE2 task-1 Elixir and task-4 results into the original tasks 1/3/4/6 cohort, then adds the extension. Task-4 admission controls failed or were invalid for three stacks, the cohorts differ, and rep 33 is post-hoc. Do not treat this table as corrected or pooled with the as-graded observations above.
| Stack | Mixed cohort + extension reps 31–32 | Same mix + reps 31–33 (rep 33 post-hoc) |
|---|---|---|
| Rust | 15/16 | 16/17 |
| TypeScript/Bun | 15/16 | 15/16 |
| Go | 13/16 | 15/18 |
| Elixir | 9/16 | 10/18 |
The public receipts report outcomes, test counts, cell IDs, timestamps, and available patch hashes. Per-cell token and wall observations are in the cost/time ledger.
| Stack | Host | Reps 31–32 median total tokens (headline) | Reps 31–32 median wall s (headline) | Reps 31–33 median total tokens (post-hoc) | Reps 31–33 median wall s (post-hoc) |
|---|---|---|---|---|---|
| Rust | kogen-bench-eu | 384,939 | 331 | 397,284 | 283 |
| Elixir | kogen-bench-eu | 991,440 | 891 | 991,440 | 891 |
| Go | kogen-bench-us | 415,341 | 326 | 417,204 | 326 |
| TypeScript/Bun | kogen-bench-us | 624,256 | 463 | 624,256 | 463 |
Wall time is comparable only within a host. Task 8 remains a separate not-run scored cohort; its smoke and control receipts are in task 8 results.
Publishability
- The extension is descriptive, with two or three cells per task and stack, one model, and author-selected task templates.
- Rep 33 was added under a rule recorded before those cells ran and remains labelled post-hoc.
- The combined counts retain the original, rerun, and extension cohorts as separate, traceable rows in the public ledger.
Captured run records
Standard-schema records assigned to this round in the indexed snapshot.
No per-round file is listed. Check the run-record index for the complete partition list.
Reproduction inventory
Published analysis replay and full experiment replay are reported separately.
- Replay type
unavailable- Script
- No per-round analysis script is published.
- Inputs
- No identified public input bundle.
- Full execution replay
- Unavailable
- Scope
- No public per-round analysis script with a complete, identified input bundle is published. See the round README, MEASURED.md, and MISSING.md for recorded scope and gaps; historical execution inputs/toolchains/grading are not bundled.
View the complete per-round reproduction inventory · Repository reproduction guide