Round l5-sol-comparators
L5 — direct Sol-medium comparators
Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.
Round source README · rounds/l5-sol-comparators/README.md
Round date: HISTORICAL (before 2026-10-09) Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Why not VALID: The historical inventory retains DESCRIPTIVE because of CONDITIONS_MISMATCH, DESCRIPTIVE_ONLY_RULE, HISTORICAL_DATA_PRECEDES_REGISTRATION. See the round evidence and limitations below. Recomputation status: FULLY RECOMPUTABLE
Status
DESCRIPTIVE
Why not VALID
- DESCRIPTIVE_ONLY_RULE: The registered decision rule has no threshold, winner, stopping rule, or model-selection decision.
- HISTORICAL_DATA_PRECEDES_REGISTRATION: Twelve reused Sol observations predate the L5 registration.
- CONDITIONS_MISMATCH: Reused Sol observations were not randomized with new cells and include different host assignments.
- relabelled 2026-10-08 after scrutiny: Previously labelled VALID; the README limits the result to a descriptive comparison.
STATUS: DESCRIPTIVE
Pre-registered: yes for the 4 prospective new cells; design frozen 7 October 2026 before their first scored execution. The 12 reused official Sol observations are historical and predate the L5 design.
Label: DESCRIPTIVE (H45)
Question: Does direct Codex gpt-6.1-sol at medium show passes on selected hard task variants with official gpt-6-luna max failures, and at what token cost per pass?
n: 16 Sol observations (12 reused + 4 new); 21 Luna-max POOL-L5 observations across 6 task variants.
Headline: Sol recorded at least one pass on 5 of 6 variants; task-level rates and Luna failure IDs are in RESULTS.
Configuration: direct Codex, default prompt, Codex harness; Sol gpt-6.1-sol/medium, Luna gpt-6-luna/max; codex-cli 0.160.0. New-cell timeout 3,600 seconds, retries 0; Sol wrapper SHA-256 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300.
Design SHA-256: f45f183868a242065ee89b74e8e4cb28dfd749b2aea2e93f285801a358e9a75e.
Lifecycle counts
These lifecycle counts cover prospective new L5 cells; the reused Sol observations are historical and predate the L5 lane.
| Lifecycle measure | n |
|---|---|
| Planned cells | 4 |
| Identified scored starts | 4 |
| Finished cells | 4 |
| Officially graded cells | 4 |
| ITT denominator | 4 |
Limit: this is a selected task comparison with small per-task samples. The reused Sol cells came from an earlier round under the same configuration; they were not randomized with the prospective cells. Task-specific sample sizes differ, so the observations do not support a general model ranking or causal estimate.
Sources: frozen design, decision rule, results, Sol cell rows, Luna POOL-L5 cell rows, and new cell records.
Related rounds: L1 task and grader trust and direct Sol-medium replication.
Reproduce
Kogen commit: exact task and arm commits are linked below; first Sol task commit 470ba65c1d3c19191a2190a032cd9ebf24e71975.
Harness commit: unavailable; no harness Git commit is recorded in the cited source. Sol wrapper SHA-256: 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300.
Model and effort: direct Codex gpt-6.1-sol/medium; direct Codex gpt-6-luna/max.
Task IDs: r70-2-elixir, r70-2-go, r70-7-rust, r70-4-elixir-fe2, r70-4-go-fe2, r70-4-ts-bun-fe2.
Command: python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py.
Raw records: rounds/l5-sol-comparators/data/sol-cells.csv, rounds/l5-sol-comparators/data/luna-baseline.csv, and rounds/l5-sol-comparators/data/new-cells.csv.
CLI: codex-cli 0.160.0 on the official Sol and Luna rows.
Task IDs and Kogen commits by arm:
| Task ID | Sol-medium Kogen commit(s) | Luna-max POOL-L5 Kogen commit(s) |
|---|---|---|
r70-2-elixir |
470ba65c1d3c19191a2190a032cd9ebf24e71975 | 470ba65c1d3c19191a2190a032cd9ebf24e71975 |
r70-2-go |
13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82 | 13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82 |
r70-7-rust |
111a0c776ae6d93f6e7fccd3fc694d5f4fa26838 | 111a0c776ae6d93f6e7fccd3fc694d5f4fa26838 |
r70-4-elixir-fe2 |
5ccd0bf1e816e2b2f5b2694861f8596402409f52 | 5ccd0bf1e816e2b2f5b2694861f8596402409f52 |
r70-4-go-fe2 |
21d21a2f420a387c921f998e5b4d16efc8f80904 | 748ebe3a3bc03a112dab83e785159654e0709f9e |
r70-4-ts-bun-fe2 |
cc3a831a167020538c4ca2ad90185606a2e4ad6a | 880afb00474fe33e9cf38a6d70016ddcd4e38e81; cc3a831a167020538c4ca2ad90185606a2e4ad6a |
Data files: rounds/l5-sol-comparators/data/sol-cells.csv, rounds/l5-sol-comparators/data/luna-baseline.csv, and rounds/l5-sol-comparators/data/new-cells.csv.
Run from the SOT repository root:
python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py
python3 reproduce/validate_repo.py
Round results · rounds/l5-sol-comparators/RESULTS.md
STATUS: DESCRIPTIVE
Outcomes use the latest official grade row for each exact cell ID. Population: 16 Sol observations (12 reused + 4 new); 21 Luna-max POOL-L5 observations.
The arm labels below identify direct Codex gpt-6.1-sol at medium and direct Codex gpt-6-luna at max.
r70-2-elixir
- Sol-medium official full pass: 2/2 (100%); tokens per pass (uncached / cached / output / total): 52,816 / 529,472 / 15,767 / 598,055.
- Luna-max official full pass: 1/3 (33.3%); tokens per pass (uncached / cached / output / total): 696,484 / 21,394,176 / 295,879 / 22,386,539.
- Luna-max failure cell IDs:
codex__gpt-6-luna__max__default__r70-2-elixir__r31;codex__gpt-6-luna__max__default__r70-2-elixir__r33. - Sol-medium pass cell IDs on this task variant:
codex__gpt-6.1-sol__medium__default__r70-2-elixir__r901;codex__gpt-6.1-sol__medium__default__r70-2-elixir__r902.
r70-2-go
- Sol-medium official full pass: 3/3 (100%); tokens per pass (uncached / cached / output / total): 25,221.7 / 185,642.7 / 7,862.7 / 218,727.
- Luna-max official full pass: 2/3 (66.7%); tokens per pass (uncached / cached / output / total): 86,891 / 1,045,248 / 44,274 / 1,176,413.
- Luna-max failure cell IDs:
codex__gpt-6-luna__max__default__r70-2-go__r32. - Sol-medium pass cell IDs on this task variant:
codex__gpt-6.1-sol__medium__default__r70-2-go__r1;codex__gpt-6.1-sol__medium__default__r70-2-go__r2;codex__gpt-6.1-sol__medium__default__r70-2-go__r3.
r70-7-rust
- Sol-medium official full pass: 3/3 (100%); tokens per pass (uncached / cached / output / total): 15,679 / 88,064 / 7,032.7 / 110,775.7.
- Luna-max official full pass: 5/6 (83.3%); tokens per pass (uncached / cached / output / total): 63,173.8 / 473,292.8 / 20,395.8 / 556,862.4.
- Luna-max failure cell IDs:
codex__gpt-6-luna__max__default__r70-7-rust__r31. - Sol-medium pass cell IDs on this task variant:
codex__gpt-6.1-sol__medium__default__r70-7-rust__r1;codex__gpt-6.1-sol__medium__default__r70-7-rust__r2;codex__gpt-6.1-sol__medium__default__r70-7-rust__r3.
r70-4-elixir-fe2
- Sol-medium official full pass: 1/2 (50%); tokens per pass (uncached / cached / output / total): 78,874 / 731,264 / 18,831 / 828,969.
- Luna-max official full pass: 0/3 (0%); tokens per pass (uncached / cached / output / total): undefined (0 passes).
- Luna-max failure cell IDs:
codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10;codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13;codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9. - Sol-medium pass cell IDs on this task variant:
codex__gpt-6.1-sol__medium__default__r70-4-elixir-fe2__r902.
r70-4-go-fe2
- Sol-medium official full pass: 2/3 (66.7%); tokens per pass (uncached / cached / output / total): 30,968.5 / 224,896 / 11,787 / 267,651.5.
- Luna-max official full pass: 1/2 (50%); tokens per pass (uncached / cached / output / total): 74,166 / 662,272 / 35,900 / 772,338.
- Luna-max failure cell IDs:
codex__gpt-6-luna__max__default__r70-4-go-fe2__r820. - Sol-medium pass cell IDs on this task variant:
codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r1;codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r3.
r70-4-ts-bun-fe2
- Sol-medium official full pass: 0/3 (0%); tokens per pass (uncached / cached / output / total): undefined (0 passes).
- Luna-max official full pass: 2/4 (50%); tokens per pass (uncached / cached / output / total): 101,863 / 1,042,688 / 53,486.5 / 1,198,037.5.
- Luna-max failure cell IDs:
codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r819;codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9. - Sol-medium pass cell IDs on this task variant: none.
Token figures are per task and per official pass. Each cell total is uncached input + cached input + output; per-pass costs sum every attempt, including failures, then divide by passes. Values are rounded to one decimal when needed; separately rounded components can differ slightly from the displayed total. A zero-pass arm has undefined tokens per pass.
These are task-variant-level observations across separate cells; the Sol pass IDs are not paired to individual Luna failure IDs.
Limit: the reused Sol observations came from another round under the same configuration, and task-level sample sizes are small and unequal. The outcomes are descriptive; they do not establish a general model ranking or causal effect.
Source rows: Sol-medium official cells, Luna-max POOL-L5 official cells, and new Sol cells. The decision rule defines pass and token accounting.
Captured run records
Standard-schema records assigned to this round in the indexed snapshot.
No per-round file is listed. Check the run-record index for the complete partition list.
Reproduction inventory
Published analysis replay and full experiment replay are reported separately.
- Replay type
analysis-only- Script
- rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py
python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py - Inputs
- Full execution replay
- Unavailable
- Scope
- The public cell CSVs support analysis replay only; model execution and official grading are not replayed.
View the complete per-round reproduction inventory · Repository reproduction guide