> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [rounds/l5-sol-comparators/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/README.md)
- [rounds/l5-sol-comparators/RESULTS.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/RESULTS.md)
- [results/run-records/index.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/run-records/index.json)
- [rounds/index.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/index.json)
- [results/source-crosswalk/index.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/source-crosswalk/index.json)
- [rounds/STATUS.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/STATUS.md)
- [reproduce/round-inventory.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/round-inventory.json)
- [rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py)

# L5 — direct Sol-medium comparators

**Round ID:** l5-sol-comparators  
**Status:** DESCRIPTIVE

> Reviewed status from rounds/STATUS.md: DESCRIPTIVE.

> Round README and results text are source material. Numeric values not tied to a claim ledger record remain source-reported.

Round date: HISTORICAL (before 2026-10-09)
Publication badge: DESCRIPTIVE; KEPT FOR AUDIT
Why not VALID: The historical inventory retains DESCRIPTIVE because of CONDITIONS_MISMATCH, DESCRIPTIVE_ONLY_RULE, HISTORICAL_DATA_PRECEDES_REGISTRATION. See the round evidence and limitations below.
Recomputation status: FULLY RECOMPUTABLE


## Status

**DESCRIPTIVE**

### Why not VALID
- **DESCRIPTIVE_ONLY_RULE:** The registered decision rule has no threshold, winner, stopping rule, or model-selection decision.
- **HISTORICAL_DATA_PRECEDES_REGISTRATION:** Twelve reused Sol observations predate the L5 registration.
- **CONDITIONS_MISMATCH:** Reused Sol observations were not randomized with new cells and include different host assignments.
- **relabelled 2026-10-08 after scrutiny:** Previously labelled VALID; the README limits the result to a descriptive comparison.


STATUS: **DESCRIPTIVE**

<!-- L5-REGISTRATION:BEGIN -->
Pre-registered: yes for the 4 prospective new cells; design frozen 7 October 2026 before their first scored execution. The 12 reused official Sol observations are historical and predate the L5 design.
Label: **DESCRIPTIVE (H45)**
Question: Does direct Codex `gpt-6.1-sol` at `medium` show passes on selected hard task variants with official `gpt-6-luna` `max` failures, and at what token cost per pass?
n: 16 Sol observations (12 reused + 4 new); 21 Luna-max POOL-L5 observations across 6 task variants.
Headline: Sol recorded at least one pass on 5 of 6 variants; task-level rates and Luna failure IDs are in [RESULTS](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/RESULTS.md).
Configuration: direct Codex, default prompt, Codex harness; Sol `gpt-6.1-sol`/medium, Luna `gpt-6-luna`/max; `codex-cli 0.160.0`. New-cell timeout 3,600 seconds, retries 0; Sol wrapper SHA-256 `08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300`.
Design SHA-256: `f45f183868a242065ee89b74e8e4cb28dfd749b2aea2e93f285801a358e9a75e`.
<!-- L5-REGISTRATION:END -->

## Lifecycle counts

<!-- L5-LIFECYCLE:BEGIN -->
These lifecycle counts cover prospective new L5 cells; the reused Sol observations are historical and predate the L5 lane.

| Lifecycle measure | n |
|---|---:|
| Planned cells | 4 |
| Identified scored starts | 4 |
| Finished cells | 4 |
| Officially graded cells | 4 |
| ITT denominator | 4 |
<!-- L5-LIFECYCLE:END -->

Limit: this is a selected task comparison with small per-task samples. The reused Sol cells came from an earlier round under the same configuration; they were not randomized with the prospective cells. Task-specific sample sizes differ, so the observations do not support a general model ranking or causal estimate.

Sources: [frozen design](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/DESIGN.md), [decision rule](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/DECISION-RULE.md), [results](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/RESULTS.md), [Sol cell rows](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/data/sol-cells.csv), [Luna POOL-L5 cell rows](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/data/luna-baseline.csv), and [new cell records](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/data/new-cells.csv).

Related rounds: [L1 task and grader trust](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l1-task-grader-trust/README.md) and [direct Sol-medium replication](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/lang-sol-replication/README.md).

## Reproduce

<!-- L5-REPRODUCE:BEGIN -->
Kogen commit: exact task and arm commits are linked below; first Sol task commit `470ba65c1d3c19191a2190a032cd9ebf24e71975`.
Harness commit: unavailable; no harness Git commit is recorded in the cited source. Sol wrapper SHA-256: `08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300`.
Model and effort: direct Codex `gpt-6.1-sol`/`medium`; direct Codex `gpt-6-luna`/`max`.
Task IDs: `r70-2-elixir`, `r70-2-go`, `r70-7-rust`, `r70-4-elixir-fe2`, `r70-4-go-fe2`, `r70-4-ts-bun-fe2`.
Command: `python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py`.
Raw records: `rounds/l5-sol-comparators/data/sol-cells.csv`, `rounds/l5-sol-comparators/data/luna-baseline.csv`, and `rounds/l5-sol-comparators/data/new-cells.csv`.
CLI: `codex-cli 0.160.0` on the official Sol and Luna rows.
Task IDs and Kogen commits by arm:

| Task ID | Sol-medium Kogen commit(s) | Luna-max POOL-L5 Kogen commit(s) |
|---|---|---|
| `r70-2-elixir` | [470ba65c1d3c19191a2190a032cd9ebf24e71975](https://github.com/KogenAI/kogen-ex/commit/470ba65c1d3c19191a2190a032cd9ebf24e71975) | [470ba65c1d3c19191a2190a032cd9ebf24e71975](https://github.com/KogenAI/kogen-ex/commit/470ba65c1d3c19191a2190a032cd9ebf24e71975) |
| `r70-2-go` | [13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82](https://github.com/KogenAI/kogen-ex/commit/13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82) | [13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82](https://github.com/KogenAI/kogen-ex/commit/13a08875ea5e1fd0a3debc6f0b8c6b98a4b37a82) |
| `r70-7-rust` | [111a0c776ae6d93f6e7fccd3fc694d5f4fa26838](https://github.com/KogenAI/kogen-ex/commit/111a0c776ae6d93f6e7fccd3fc694d5f4fa26838) | [111a0c776ae6d93f6e7fccd3fc694d5f4fa26838](https://github.com/KogenAI/kogen-ex/commit/111a0c776ae6d93f6e7fccd3fc694d5f4fa26838) |
| `r70-4-elixir-fe2` | [5ccd0bf1e816e2b2f5b2694861f8596402409f52](https://github.com/KogenAI/kogen-ex/commit/5ccd0bf1e816e2b2f5b2694861f8596402409f52) | [5ccd0bf1e816e2b2f5b2694861f8596402409f52](https://github.com/KogenAI/kogen-ex/commit/5ccd0bf1e816e2b2f5b2694861f8596402409f52) |
| `r70-4-go-fe2` | [21d21a2f420a387c921f998e5b4d16efc8f80904](https://github.com/KogenAI/kogen-ex/commit/21d21a2f420a387c921f998e5b4d16efc8f80904) | [748ebe3a3bc03a112dab83e785159654e0709f9e](https://github.com/KogenAI/kogen-ex/commit/748ebe3a3bc03a112dab83e785159654e0709f9e) |
| `r70-4-ts-bun-fe2` | [cc3a831a167020538c4ca2ad90185606a2e4ad6a](https://github.com/KogenAI/kogen-ex/commit/cc3a831a167020538c4ca2ad90185606a2e4ad6a) | [880afb00474fe33e9cf38a6d70016ddcd4e38e81](https://github.com/KogenAI/kogen-ex/commit/880afb00474fe33e9cf38a6d70016ddcd4e38e81); [cc3a831a167020538c4ca2ad90185606a2e4ad6a](https://github.com/KogenAI/kogen-ex/commit/cc3a831a167020538c4ca2ad90185606a2e4ad6a) |

Data files: `rounds/l5-sol-comparators/data/sol-cells.csv`, `rounds/l5-sol-comparators/data/luna-baseline.csv`, and `rounds/l5-sol-comparators/data/new-cells.csv`.

Run from the SOT repository root:

```sh
python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py
python3 reproduce/validate_repo.py
```
<!-- L5-REPRODUCE:END -->


## Results source

# L5 results — final descriptive record

STATUS: **DESCRIPTIVE**

<!-- L5-RESULTS:BEGIN -->
Outcomes use the latest official grade row for each exact cell ID.
Population: 16 Sol observations (12 reused + 4 new); 21 Luna-max POOL-L5 observations.

The arm labels below identify direct Codex `gpt-6.1-sol` at medium and direct Codex `gpt-6-luna` at max.

### `r70-2-elixir`

- Sol-medium official full pass: **2/2 (100%)**; tokens per pass (uncached / cached / output / total): 52,816 / 529,472 / 15,767 / 598,055.
- Luna-max official full pass: **1/3 (33.3%)**; tokens per pass (uncached / cached / output / total): 696,484 / 21,394,176 / 295,879 / 22,386,539.
- Luna-max failure cell IDs: `codex__gpt-6-luna__max__default__r70-2-elixir__r31`; `codex__gpt-6-luna__max__default__r70-2-elixir__r33`.
- Sol-medium pass cell IDs on this task variant: `codex__gpt-6.1-sol__medium__default__r70-2-elixir__r901`; `codex__gpt-6.1-sol__medium__default__r70-2-elixir__r902`.

### `r70-2-go`

- Sol-medium official full pass: **3/3 (100%)**; tokens per pass (uncached / cached / output / total): 25,221.7 / 185,642.7 / 7,862.7 / 218,727.
- Luna-max official full pass: **2/3 (66.7%)**; tokens per pass (uncached / cached / output / total): 86,891 / 1,045,248 / 44,274 / 1,176,413.
- Luna-max failure cell IDs: `codex__gpt-6-luna__max__default__r70-2-go__r32`.
- Sol-medium pass cell IDs on this task variant: `codex__gpt-6.1-sol__medium__default__r70-2-go__r1`; `codex__gpt-6.1-sol__medium__default__r70-2-go__r2`; `codex__gpt-6.1-sol__medium__default__r70-2-go__r3`.

### `r70-7-rust`

- Sol-medium official full pass: **3/3 (100%)**; tokens per pass (uncached / cached / output / total): 15,679 / 88,064 / 7,032.7 / 110,775.7.
- Luna-max official full pass: **5/6 (83.3%)**; tokens per pass (uncached / cached / output / total): 63,173.8 / 473,292.8 / 20,395.8 / 556,862.4.
- Luna-max failure cell IDs: `codex__gpt-6-luna__max__default__r70-7-rust__r31`.
- Sol-medium pass cell IDs on this task variant: `codex__gpt-6.1-sol__medium__default__r70-7-rust__r1`; `codex__gpt-6.1-sol__medium__default__r70-7-rust__r2`; `codex__gpt-6.1-sol__medium__default__r70-7-rust__r3`.

### `r70-4-elixir-fe2`

- Sol-medium official full pass: **1/2 (50%)**; tokens per pass (uncached / cached / output / total): 78,874 / 731,264 / 18,831 / 828,969.
- Luna-max official full pass: **0/3 (0%)**; tokens per pass (uncached / cached / output / total): undefined (0 passes).
- Luna-max failure cell IDs: `codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r10`; `codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r13`; `codex__gpt-6-luna__max__default__r70-4-elixir-fe2__r9`.
- Sol-medium pass cell IDs on this task variant: `codex__gpt-6.1-sol__medium__default__r70-4-elixir-fe2__r902`.

### `r70-4-go-fe2`

- Sol-medium official full pass: **2/3 (66.7%)**; tokens per pass (uncached / cached / output / total): 30,968.5 / 224,896 / 11,787 / 267,651.5.
- Luna-max official full pass: **1/2 (50%)**; tokens per pass (uncached / cached / output / total): 74,166 / 662,272 / 35,900 / 772,338.
- Luna-max failure cell IDs: `codex__gpt-6-luna__max__default__r70-4-go-fe2__r820`.
- Sol-medium pass cell IDs on this task variant: `codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r1`; `codex__gpt-6.1-sol__medium__default__r70-4-go-fe2__r3`.

### `r70-4-ts-bun-fe2`

- Sol-medium official full pass: **0/3 (0%)**; tokens per pass (uncached / cached / output / total): undefined (0 passes).
- Luna-max official full pass: **2/4 (50%)**; tokens per pass (uncached / cached / output / total): 101,863 / 1,042,688 / 53,486.5 / 1,198,037.5.
- Luna-max failure cell IDs: `codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r819`; `codex__gpt-6-luna__max__default__r70-4-ts-bun-fe2__r9`.
- Sol-medium pass cell IDs on this task variant: none.

Token figures are per task and per official pass. Each cell total is uncached input + cached input + output; per-pass costs sum every attempt, including failures, then divide by passes. Values are rounded to one decimal when needed; separately rounded components can differ slightly from the displayed total. A zero-pass arm has undefined tokens per pass.

These are task-variant-level observations across separate cells; the Sol pass IDs are not paired to individual Luna failure IDs.
<!-- L5-RESULTS:END -->

Limit: the reused Sol observations came from another round under the same configuration, and task-level sample sizes are small and unequal. The outcomes are descriptive; they do not establish a general model ranking or causal effect.

Source rows: [Sol-medium official cells](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/data/sol-cells.csv), [Luna-max POOL-L5 official cells](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/data/luna-baseline.csv), and [new Sol cells](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/data/new-cells.csv). The [decision rule](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l5-sol-comparators/DECISION-RULE.md) defines pass and token accounting.


## Captured run records

No records for this round are listed in the [run-record index](/downloads/results/run-records/index.json).


## Reproduction inventory

- **Type:** analysis-only
- **Script:** `python3 rounds/l5-sol-comparators/reproduce/l5_sol_comparators.py`
- **Inputs:** `rounds/l5-sol-comparators/data/sol-cells.csv`, `rounds/l5-sol-comparators/data/luna-baseline.csv`, `rounds/l5-sol-comparators/data/new-cells.csv`
- **Full execution replay:** unavailable
- **Scope:** The public cell CSVs support analysis replay only; model execution and official grading are not replayed.

See the [complete per-round reproduction inventory](/reproduce/) and the [repository reproduction guide](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/README.md).

## Related records

### Related experiments

- [l5-sol-comparators](/rounds/l5-sol-comparators/)
