Round lang-sol-replication

Sol-medium language replication

Status · DESCRIPTIVEReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/lang-sol-replication/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Why not VALID: The historical inventory retains DESCRIPTIVE because of DECISION_RULE_UNDEFINED, CONDITIONS_MISMATCH. See the round evidence and limitations below. Recomputation status: FULLY RECOMPUTABLE

Required reproduction metadata

  • Kogen commit: not applicable as a single round-level value; exact per-task SHAs are linked in the table below.
  • Harness commit: not recorded as a separate Git commit in the cited rows. Harness and CLI: direct Codex; CLI version recorded on every scoped cell is codex-cli 0.160.0. No official harness version label is recorded.
  • Model and effort: gpt-6.1-sol, MEDIUM.
  • Task IDs: r70-1-rust, r70-1-go, r70-1-ts-bun; r70-2-rust, r70-2-go, r70-2-ts-bun; r70-4-rust-fe2, r70-4-go-fe2, r70-4-ts-bun-fe2; reused r70-5, r70-6, and r70-7 variants for Rust, Go, and TS-Bun.
  • Historical execution command: not recorded in the cited design and official grade sources. The exact commands for reproducing this page are below.
  • Raw records: raw run captures are not retained; data/cells.csv preserves the exact cell IDs, official grades, token counts, wall times, grade-window times, Kogen base SHAs, and CLI version used here. DESIGN.md is a sanitized design presentation with operator-reported original source SHA-256 provenance.

STATUS: DESCRIPTIVE — raw rates are reported; the combined registered decision is UNRESOLVED.

Pre-registered: operator-reported at 10:01:53Z before the first grade window; the public bundle does not independently establish that timing. The round was added to this repository after the first grade window.

Label: DESCRIPTIVE — raw captures not retained; results verified against official grades

Question: Is the Round 70 language result specific to Luna max?

n: 27 new cells; 31 reused Sol-medium cells on t5–t7; 58 exact cell IDs in the comparison. The frozen design says 36 existing cells; the exact latest official IDs for the three named stacks yield 31 reusable rows (Rust 11, Go 11, TS-Bun 9). Results use all 31, with no reruns.

Population N
Planned new cells 27
Started new cells 27
Finished new cells 27
Officially graded new cells 27
ITT denominator for new cells 27

Headline: New passes were Rust 8/9, Go 7/9, and TS-Bun 6/9. Combined raw passes were Rust 18/20, Go 18/20, and TS-Bun 15/18. A post-hoc equal-task mean is Rust 90.3%, Go 88.9%, and TS-Bun 83.3%; the registered combined branch has no formula or scale for its four-pass threshold. The new-cell threshold was not triggered; the combined decision is UNRESOLVED.

Why not VALID

  • The registered combined four-pass branch lacks a formula and scale, so the full decision cannot be executed. The equal-task rates are post-hoc sensitivity values, not a substitute rule.
  • The available operator design bytes hash to 66f5d590…, while the published source hash is c9394487…. The available design's filesystem time follows the first new official grade, and the public rule was committed later. This does not prove a post-hoc edit, but independent pre-registration timing is unproven.
  • New Rust cells ran on EU, while Go and TS-Bun ran on US; stack and host cannot be separated in this comparison.
  • The 31 reused cells come from a different date window with unequal replication, including one INVALID Rust cell retained as a non-pass. The reused and new cohorts cannot support a single matched language effect.
  • No round ENVIRONMENT.md records start and end host conditions. New Standard rows omit effective model and effort, grader, host class, sandbox, and price; public reproducibility is incomplete.

Configuration: direct Codex, gpt-6.1-sol at MEDIUM, 3600-second cap, zero retries. New tasks were t1, t2, and t4-fe2 on fixed skeletons, crossed with Rust, Go, and TS-Bun at three reps each. Rep 1 was the rule-L smoke for each task-by-stack pair; reps 2–3 followed the host smoke gates. Reused t5–t7 cells were not rerun. New Rust cells ran on kogen-bench-eu; new Go and TS-Bun cells ran on kogen-bench-us. Wall times are comparable only within a host.

Release-gate evidence: all nine rep-1 cells received official grades through the sandbox before the bulk reps; both host smoke-gate receipts report passed=true. The TS-Bun t4-fe2 rep-1 cell scored 17/18 and remains a scored fail. The lane receipts say the pre-run gate accepted an emitter-gap exception and deferred full Standard-record validation until analysis; the referenced exception receipt is not in the public bundle. The exact design-freeze timing is operator-reported, not independently established.

Limit: small n, one model per arm, and a decision rule designed to detect only large reversals. The reused t5–t7 cohort has varying rep counts; the INVALID Rust t6 rep 3 row is retained as a non-pass in the denominator.

Sources: sanitized DESIGN.md, preserving the operator-reported original source SHA-256 c9394487c2a139ff57e98fe8c9d5c44d6f6b5e7b33d1a33abba87479fb575892; public smoke-gate receipt extracts; the cited cohort, host-assignment, and grade-window ledgers; and the sanitized per-cell rows in data/cells.csv. The original source bytes and strict-exception receipt are not included.

Decision and results

Reproduction

Run from the kogen-bench-sot repository root:

python3 rounds/lang-sol-replication/reproduce/lang_sol_replication.py
python3 reproduce/validate_repo.py

The round reproducer uses only Python's standard library. It validates the frozen design hash and exact cell set, recomputes every cell total, per-stack pass count, median, and verdict from data/cells.csv, and fails if RESULTS.md or the data-derived README claims differ.

Evidence links: H91 — matched language comparison; H114 — R70 language choice; F08 findings summary; claim CL-SOL-LANG-REPLICATION-POSTHOC-EQUAL-TASK (STATUS_ONLY). The equal-task formula is a post-hoc sensitivity analysis because the original formula was not recorded.

Kogen task commits

Recomputed from raw records

`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 18 rows (fail 5, pass 13); overall pass rate is 72.2% (13/18) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.

No shared cell IDs are present in the committed public grade export; the source-only cohort is checked against the page's reported figures below. No shared public-export cell IDs are available for a paired outcome comparison. No complete published metadata/manifest token pairs are available for this round (0 missing manifest usage; 18 missing published metadata). Missing counters remain unknown, not zero. This round also has 18 raw-only or non-public rows; they are kept separate from the published-export denominator.

Round results · rounds/lang-sol-replication/RESULTS.md

STATUS: DESCRIPTIVE — raw rates are reported; the combined registered decision is UNRESOLVED.

Release label: DESCRIPTIVE — raw captures not retained; results verified against official grades

The primary outcome is the latest official hidden-suite full-pass grade per exact cell ID. The public data contains 27 new cells and 31 reused t5–t7 cells; no reused cell was rerun. The frozen design text states 36 existing t5–t7 cells, while the exact eligible official IDs resolve to 31 (Rust 11, Go 11, TS-Bun 9). The results below use the exact-ID ledger rows.

Token accounting: uncached = usage.input; total tokens = uncached + cached + output. The grade-window timestamp is UTC. Wall times are comparable only within a host.

The two host smoke-gate receipts show all nine rep-1 task-by-stack cells were officially graded before bulk reps. Both host gates passed; the TS-Bun t4-fe2 rep-1 scored 17/18 and remains a FAIL outcome. The lane receipt says its full Standard-record strict check was deferred until analysis under an emitter-gap exception; the exception receipt itself is not in this public bundle.

New cells

Exact cell ID Task Stack Rep Host Outcome Hidden tests passed/total Uncached Cached Output Total tokens wall_s Grade window (UTC)
codex__gpt-6.1-sol__medium__default__r70-1-rust__r1 t1 Rust 1 kogen-bench-eu PASS 25/25 22,198 101,376 7,257 130,831 196.5 2026-10-07T10:36:53Z
codex__gpt-6.1-sol__medium__default__r70-1-rust__r2 t1 Rust 2 kogen-bench-eu PASS 25/25 37,887 123,520 8,593 170,000 241.28 2026-10-07T11:17:03Z
codex__gpt-6.1-sol__medium__default__r70-1-rust__r3 t1 Rust 3 kogen-bench-eu PASS 25/25 30,007 116,864 7,999 154,870 205.598 2026-10-07T11:17:03Z
codex__gpt-6.1-sol__medium__default__r70-2-rust__r1 t2 Rust 1 kogen-bench-eu PASS 25/25 16,295 101,504 5,770 123,569 165.03 2026-10-07T10:36:53Z
codex__gpt-6.1-sol__medium__default__r70-2-rust__r2 t2 Rust 2 kogen-bench-eu PASS 25/25 12,167 136,320 5,158 153,645 150.709 2026-10-07T11:17:03Z
codex__gpt-6.1-sol__medium__default__r70-2-rust__r3 t2 Rust 3 kogen-bench-eu PASS 25/25 17,990 97,024 5,066 120,080 148.939 2026-10-07T11:17:03Z
codex__gpt-6.1-sol__medium__default__r70-4-rust-fe2__r1 t4-fe2 Rust 1 kogen-bench-eu PASS 18/18 11,761 84,992 6,006 102,759 172.378 2026-10-07T10:36:53Z
codex__gpt-6.1-sol__medium__default__r70-4-rust-fe2__r2 t4-fe2 Rust 2 kogen-bench-eu FAIL 17/18 24,478 87,040 6,803 118,321 180.847 2026-10-07T11:17:03Z
codex__gpt-6.1-sol__medium__default__r70-4-rust-fe2__r3 t4-fe2 Rust 3 kogen-bench-eu PASS 18/18 21,425 113,152 7,302 141,879 199.254 2026-10-07T11:17:03Z
codex__gpt-6.1-sol__medium__default__r70-1-go__r1 t1 Go 1 kogen-bench-us PASS 25/25 38,456 258,048 12,853 309,357 389.49 2026-10-07T10:38:14Z
codex__gpt-6.1-sol__medium__default__r70-1-go__r2 t1 Go 2 kogen-bench-us PASS 25/25 13,635 129,920 7,304 150,859 190.92 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-1-go__r3 t1 Go 3 kogen-bench-us FAIL 24/25 39,736 219,904 13,901 273,541 390.246 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-2-go__r1 t2 Go 1 kogen-bench-us PASS 25/25 21,607 189,568 10,564 221,739 310.963 2026-10-07T10:38:14Z
codex__gpt-6.1-sol__medium__default__r70-2-go__r2 t2 Go 2 kogen-bench-us PASS 25/25 19,730 185,728 5,850 211,308 181.583 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-2-go__r3 t2 Go 3 kogen-bench-us PASS 25/25 34,328 181,632 7,174 223,134 207.929 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r1 t4-fe2 Go 1 kogen-bench-us PASS 18/18 17,992 135,040 7,413 160,445 205.942 2026-10-07T10:38:14Z
codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r2 t4-fe2 Go 2 kogen-bench-us FAIL 17/18 17,165 164,736 9,229 191,130 237.152 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r3 t4-fe2 Go 3 kogen-bench-us PASS 18/18 26,780 150,016 6,932 183,728 204.624 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-1-ts-bun__r1 t1 TS-Bun 1 kogen-bench-us PASS 25/25 23,860 178,688 9,807 212,355 261.012 2026-10-07T10:38:14Z
codex__gpt-6.1-sol__medium__default__r70-1-ts-bun__r2 t1 TS-Bun 2 kogen-bench-us PASS 25/25 17,022 132,352 9,195 158,569 231.869 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-1-ts-bun__r3 t1 TS-Bun 3 kogen-bench-us PASS 25/25 18,644 178,816 9,169 206,629 255.074 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-2-ts-bun__r1 t2 TS-Bun 1 kogen-bench-us PASS 25/25 21,372 227,712 7,119 256,203 209.011 2026-10-07T10:38:14Z
codex__gpt-6.1-sol__medium__default__r70-2-ts-bun__r2 t2 TS-Bun 2 kogen-bench-us PASS 25/25 25,986 254,336 6,802 287,124 215.12 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-2-ts-bun__r3 t2 TS-Bun 3 kogen-bench-us PASS 25/25 30,187 190,464 6,856 227,507 208.123 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-4-ts-bun-fe2__r1 t4-fe2 TS-Bun 1 kogen-bench-us FAIL 17/18 13,850 121,472 5,670 140,992 166.437 2026-10-07T10:38:14Z
codex__gpt-6.1-sol__medium__default__r70-4-ts-bun-fe2__r2 t4-fe2 TS-Bun 2 kogen-bench-us FAIL 17/18 24,844 137,472 6,878 169,194 189.982 2026-10-07T11:00:49Z
codex__gpt-6.1-sol__medium__default__r70-4-ts-bun-fe2__r3 t4-fe2 TS-Bun 3 kogen-bench-us FAIL 17/18 17,987 117,120 6,458 141,565 172.041 2026-10-07T11:00:49Z

Reused t5–t7 cells

These are the latest official Sol-medium rows for the named stacks. INVALID is retained in the denominator as a non-pass; no outcome has been inferred from test names or grader content.

Exact cell ID Task Stack Rep Outcome
codex__gpt-6.1-sol__medium__default__r70-5-rust__r1 t5 Rust 1 PASS
codex__gpt-6.1-sol__medium__default__r70-5-rust__r2 t5 Rust 2 PASS
codex__gpt-6.1-sol__medium__default__r70-5-rust__r3 t5 Rust 3 PASS
codex__gpt-6.1-sol__medium__default__r70-5-rust__r4 t5 Rust 4 PASS
codex__gpt-6.1-sol__medium__default__r70-6-rust__r1 t6 Rust 1 PASS
codex__gpt-6.1-sol__medium__default__r70-6-rust__r2 t6 Rust 2 PASS
codex__gpt-6.1-sol__medium__default__r70-6-rust__r3 t6 Rust 3 INVALID
codex__gpt-6.1-sol__medium__default__r70-6-rust__r4 t6 Rust 4 PASS
codex__gpt-6.1-sol__medium__default__r70-7-rust__r1 t7 Rust 1 PASS
codex__gpt-6.1-sol__medium__default__r70-7-rust__r2 t7 Rust 2 PASS
codex__gpt-6.1-sol__medium__default__r70-7-rust__r3 t7 Rust 3 PASS
codex__gpt-6.1-sol__medium__default__r70-5-go__r1 t5 Go 1 PASS
codex__gpt-6.1-sol__medium__default__r70-5-go__r2 t5 Go 2 PASS
codex__gpt-6.1-sol__medium__default__r70-5-go__r3 t5 Go 3 PASS
codex__gpt-6.1-sol__medium__default__r70-5-go__r4 t5 Go 4 PASS
codex__gpt-6.1-sol__medium__default__r70-6-go__r1 t6 Go 1 PASS
codex__gpt-6.1-sol__medium__default__r70-6-go__r2 t6 Go 2 PASS
codex__gpt-6.1-sol__medium__default__r70-6-go__r3 t6 Go 3 PASS
codex__gpt-6.1-sol__medium__default__r70-6-go__r4 t6 Go 4 PASS
codex__gpt-6.1-sol__medium__default__r70-7-go__r1 t7 Go 1 PASS
codex__gpt-6.1-sol__medium__default__r70-7-go__r2 t7 Go 2 PASS
codex__gpt-6.1-sol__medium__default__r70-7-go__r3 t7 Go 3 PASS
codex__gpt-6.1-sol__medium__default__r70-5-ts-bun__r1 t5 TS-Bun 1 PASS
codex__gpt-6.1-sol__medium__default__r70-5-ts-bun__r2 t5 TS-Bun 2 PASS
codex__gpt-6.1-sol__medium__default__r70-5-ts-bun__r3 t5 TS-Bun 3 PASS
codex__gpt-6.1-sol__medium__default__r70-6-ts-bun__r1 t6 TS-Bun 1 PASS
codex__gpt-6.1-sol__medium__default__r70-6-ts-bun__r2 t6 TS-Bun 2 PASS
codex__gpt-6.1-sol__medium__default__r70-6-ts-bun__r3 t6 TS-Bun 3 PASS
codex__gpt-6.1-sol__medium__default__r70-7-ts-bun__r1 t7 TS-Bun 1 PASS
codex__gpt-6.1-sol__medium__default__r70-7-ts-bun__r2 t7 TS-Bun 2 PASS
codex__gpt-6.1-sol__medium__default__r70-7-ts-bun__r3 t7 TS-Bun 3 PASS

Per-stack summary

Stack New passes Reused passes Combined raw passes Equal-task combined pass rate (post-hoc) Median total tokens/cell (new) Median wall_s (new)
Rust 8/9 10/11 18/20 90.3% 130,831 180.8
Go 7/9 11/11 18/20 88.9% 211,308 207.9
TS-Bun 6/9 9/9 15/18 83.3% 206,629 209.0

Verdict

Rust records 8/9 new passes; the best of Go and TS-Bun records 7/9. Combined raw official pass counts are Rust 18/20, Go 18/20, and TS-Bun 15/18. The equal-task means are Rust 90.3%, Go 88.9%, and TS-Bun 83.3%.

The new-cell threshold is not triggered: Rust is 8/9 and the best of Go and TS-Bun is 7/9, fewer than three passes apart. The combined branch says ‘equalized per task’ but records no formula or scale for its four-pass threshold, so it cannot be applied. The equal-task mean above is a post-hoc sensitivity summary. COMBINED DECISION: UNRESOLVED.

Limit: small n, one model per arm, and a decision rule designed to detect only large reversals.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
rounds/lang-sol-replication/reproduce/lang_sol_replication.py
python3 rounds/lang-sol-replication/reproduce/lang_sol_replication.py
Inputs
Full execution replay
Unavailable
Scope
Recomputes the round tables and post-hoc equal-task sensitivity from the 58 published cell rows; it does not replay model execution, official grading, host smoke gates, or the missing exception receipt.

View the complete per-round reproduction inventory · Repository reproduction guide