Round l5-sol-comparators

L5 — direct Sol-medium comparators

Status · DESCRIPTIVEReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/l5-sol-comparators/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Why not VALID: The historical inventory retains DESCRIPTIVE because of CONDITIONS_MISMATCH, DESCRIPTIVE_ONLY_RULE, HISTORICAL_DATA_PRECEDES_REGISTRATION. See the round evidence and limitations below. Recomputation status: FULLY RECOMPUTABLE

Status

DESCRIPTIVE

Why not VALID

  • DESCRIPTIVE_ONLY_RULE: The registered decision rule has no threshold, winner, stopping rule, or model-selection decision.
  • HISTORICAL_DATA_PRECEDES_REGISTRATION: Twelve reused Sol observations predate the L5 registration.
  • CONDITIONS_MISMATCH: Reused Sol observations were not randomized with new cells and include different host assignments.
  • relabelled 2026-10-08 after scrutiny: Previously labelled VALID; the README limits the result to a descriptive comparison.

STATUS: DESCRIPTIVE

Pre-registered: yes for the 4 prospective new cells; design frozen 7 October 2026 before their first scored execution. The 12 reused official Sol observations are historical and predate the L5 design. Label: DESCRIPTIVE (H45) Question: Does direct Codex gpt-6.1-sol at medium show passes on selected hard task variants with official gpt-6-luna max failures, and at what token cost per pass? n: 16 Sol observations (12 reused + 4 new); 21 Luna-max POOL-L5 observations across 6 task variants. Headline: Sol recorded at least one pass on 5 of 6 variants; task-level rates and Luna failure IDs are in RESULTS. Configuration: direct Codex, default prompt, Codex harness; Sol gpt-6.1-sol/medium, Luna gpt-6-luna/max; codex-cli 0.160.0. New-cell timeout 3,600 seconds, retries 0; Sol wrapper SHA-256 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300. Design SHA-256: f45f183868a242065ee89b74e8e4cb28dfd749b2aea2e93f285801a358e9a75e.

Lifecycle counts

These lifecycle counts cover prospective new L5 cells; the reused Sol observations are historical and predate the L5 lane.

Lifecycle measure n
Planned cells 4
Identified scored starts 4
Finished cells 4
Officially graded cells 4
ITT denominator 4

Limit: this is a selected task comparison with small per-task samples. The reused Sol cells came from an earlier round under the same configuration; they were not randomized with the prospective cells. Task-specific sample sizes differ, so the observations do not support a general model ranking or causal estimate.

Sources: frozen design, decision rule, results, Sol cell rows, Luna POOL-L5 cell rows, and new cell records.

Related rounds: L1 task and grader trust and direct Sol-medium replication.

Reproduce

Kogen commit: exact task and arm commits are linked below; first Sol task commit 470ba65c1d3c19191a2190a032cd9ebf24e71975. Harness commit: unavailable; no harness Git commit is recorded in the cited source. Sol wrapper SHA-256: 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300. Model and effort: direct Codex gpt-6.1-sol/medium; direct Codex gpt-6-luna/max. Task IDs: r70-2-elixir, r70-2-go, r70-7-rust, r70-4-elixir-fe2, r70-4-go-fe2, r70-4-ts-bun-fe2. Command: python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py. Raw records: rounds/l5-sol-comparators/data/sol-cells.csv, rounds/l5-sol-comparators/data/luna-baseline.csv, and rounds/l5-sol-comparators/data/new-cells.csv. CLI: codex-cli 0.160.0 on the official Sol and Luna rows. Task IDs and Kogen commits by arm:

Data files: rounds/l5-sol-comparators/data/sol-cells.csv, rounds/l5-sol-comparators/data/luna-baseline.csv, and rounds/l5-sol-comparators/data/new-cells.csv.

Run from the SOT repository root:

python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py
python3 reproduce/validate_repo.py

Round results · rounds/l5-sol-comparators/RESULTS.md

STATUS: DESCRIPTIVE

Outcomes use the latest official grade row for each exact cell ID. Population: 16 Sol observations (12 reused + 4 new); 21 Luna-max POOL-L5 observations.

The arm labels below identify direct Codex gpt-6.1-sol at medium and direct Codex gpt-6-luna at max.

r70-2-elixir

  • Sol-medium official full pass: 2/2 (100%); tokens per pass (uncached / cached / output / total): 52,816 / 529,472 / 15,767 / 598,055.
  • Luna-max official full pass: 1/3 (33.3%); tokens per pass (uncached / cached / output / total): 696,484 / 21,394,176 / 295,879 / 22,386,539.
  • Luna-max failure cell IDs: codex__gpt-6-luna__max__default__r70-2-elixir__r31; codex__gpt-6-luna__max__default__r70-2-elixir__r33.
  • Sol-medium pass cell IDs on this task variant: codex__gpt-6.1-sol__medium__default__r70-2-elixir__r901; codex__gpt-6.1-sol__medium__default__r70-2-elixir__r902.

r70-2-go

  • Sol-medium official full pass: 3/3 (100%); tokens per pass (uncached / cached / output / total): 25,221.7 / 185,642.7 / 7,862.7 / 218,727.
  • Luna-max official full pass: 2/3 (66.7%); tokens per pass (uncached / cached / output / total): 86,891 / 1,045,248 / 44,274 / 1,176,413.
  • Luna-max failure cell IDs: codex__gpt-6-luna__max__default__r70-2-go__r32.
  • Sol-medium pass cell IDs on this task variant: codex__gpt-6.1-sol__medium__default__r70-2-go__r1; codex__gpt-6.1-sol__medium__default__r70-2-go__r2; codex__gpt-6.1-sol__medium__default__r70-2-go__r3.

r70-7-rust

  • Sol-medium official full pass: 3/3 (100%); tokens per pass (uncached / cached / output / total): 15,679 / 88,064 / 7,032.7 / 110,775.7.
  • Luna-max official full pass: 5/6 (83.3%); tokens per pass (uncached / cached / output / total): 63,173.8 / 473,292.8 / 20,395.8 / 556,862.4.
  • Luna-max failure cell IDs: codex__gpt-6-luna__max__default__r70-7-rust__r31.
  • Sol-medium pass cell IDs on this task variant: codex__gpt-6.1-sol__medium__default__r70-7-rust__r1; codex__gpt-6.1-sol__medium__default__r70-7-rust__r2; codex__gpt-6.1-sol__medium__default__r70-7-rust__r3.

r70-4-elixir-fe2

  • Sol-medium official full pass: 1/2 (50%); tokens per pass (uncached / cached / output / total): 78,874 / 731,264 / 18,831 / 828,969.
  • Luna-max official full pass: 0/3 (0%); tokens per pass (uncached / cached / output / total): undefined (0 passes).
  • Luna-max failure cell IDs: codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10; codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13; codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9.
  • Sol-medium pass cell IDs on this task variant: codex__gpt-6.1-sol__medium__default__r70-4-elixir-fe2__r902.

r70-4-go-fe2

  • Sol-medium official full pass: 2/3 (66.7%); tokens per pass (uncached / cached / output / total): 30,968.5 / 224,896 / 11,787 / 267,651.5.
  • Luna-max official full pass: 1/2 (50%); tokens per pass (uncached / cached / output / total): 74,166 / 662,272 / 35,900 / 772,338.
  • Luna-max failure cell IDs: codex__gpt-6-luna__max__default__r70-4-go-fe2__r820.
  • Sol-medium pass cell IDs on this task variant: codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r1; codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r3.

r70-4-ts-bun-fe2

  • Sol-medium official full pass: 0/3 (0%); tokens per pass (uncached / cached / output / total): undefined (0 passes).
  • Luna-max official full pass: 2/4 (50%); tokens per pass (uncached / cached / output / total): 101,863 / 1,042,688 / 53,486.5 / 1,198,037.5.
  • Luna-max failure cell IDs: codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r819; codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9.
  • Sol-medium pass cell IDs on this task variant: none.

Token figures are per task and per official pass. Each cell total is uncached input + cached input + output; per-pass costs sum every attempt, including failures, then divide by passes. Values are rounded to one decimal when needed; separately rounded components can differ slightly from the displayed total. A zero-pass arm has undefined tokens per pass.

These are task-variant-level observations across separate cells; the Sol pass IDs are not paired to individual Luna failure IDs.

Limit: the reused Sol observations came from another round under the same configuration, and task-level sample sizes are small and unequal. The outcomes are descriptive; they do not establish a general model ranking or causal effect.

Source rows: Sol-medium official cells, Luna-max POOL-L5 official cells, and new Sol cells. The decision rule defines pass and token accounting.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py
python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py
Inputs
Full execution replay
Unavailable
Scope
The public cell CSVs support analysis replay only; model execution and official grading are not replayed.

View the complete per-round reproduction inventory · Repository reproduction guide