> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [reproduce/morning-report-20261030/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/morning-report-20261030/README.md)
- [categories/head-to-head.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/head-to-head.md)
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)

# Report on whether kogen-loop is cheaper, as reliable and faster than Codex: goal not met

- **Finding id:** `morning-report-20261030`
- **Verdict:** Unresolved
- **Finished:** 30 October 2026
- **Comparison:** [kogen-loop vs Codex](/comparisons/loop-vs-codex/)
- **Reproduction pack:** [reproduce/morning-report-20261030/](https://github.com/KogenAI/kogen-bench/tree/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/morning-report-20261030) (data, analysis script and rerun steps)
- **Category page:** [categories/head-to-head.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/head-to-head.md)

## In plain terms

A summary of the first night of kogen-loop vs Codex testing on the same model and effort. On the clean API route at high effort loop was slower (median 139.76 s against 117.40 s) at equal pass rate and cost; at medium effort it cost 20.5% less per passed task, with a 4% lower median time but a worse slowest-tenth. On the subscription route loop's cost was 50-58% lower at tied pass rates, but the time gaps were mixed up with a speed change on the provider side and are not read as differences between the tools. The study's goal (cheaper, as reliable and faster) was therefore not met on that night's evidence.

Verdict key: **Yes**, The result supports the change or claim that was tested. **No**, The result does not support it, so nothing changes. **Unresolved**, The evidence is not strong enough either way. **Superseded**, A later test replaced this result.

## Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

The morning report of 30 October 2026 for the loop-efficiency study: is loop cheaper, at least as reliable and faster than Codex at the same model and effort? Verdict: **goal not met** (unresolved). On the clean API route at high effort, loop is slower (median 139.76 s against 117.40 s) at equal pass and cost; at medium effort loop costs 20.5% less per pass and has a 4% lower median wall, but a worse p90. On the subscription route loop's cost is 50–58% lower at tied pass, and the wall gaps measured that night are confounded by a route/provider throughput shift (see [`reproduce/loop-sub-route-throughput/`](/findings/loop-sub-route-throughput/), itself later superseded) and so are not read as tool differences; the rule that came out of this is in [`reproduce/protocol-split-20261030/`](/findings/protocol-split-20261030/).

This pack reproduces the numbers that can be recomputed from sanitized per-run rows, and says for every other number what its source is and how it was computed.

## What is reproduced from rows

`runs.csv` has 80 rows from three experiments on the clean API route (both tools on the same OpenAI API key, `gpt-6.1-terra`): the high-effort confirmation (loop against Codex), the medium-effort confirmation (loop, Codex and `speed2`), and the high-effort patch screen (loop and six variants). Columns: experiment, effort, stratum (`counted` or `dry`), task, tool, rep, outcome, passed, uncached/cached/cache-write/output tokens, requests, list-price cost in USD and agent wall in seconds. Reasoning tokens are not in the source rows (blank). `stream-rates.csv` has 355 loop-type subscription runs (output tokens and body-stream seconds) for the throughput-regime paragraph.

`analyze.py` prints, and `expected-output.txt` holds, these report numbers, all matching the report:

- **API high** (4 tasks × 2 repetitions): loop 8/8, median/p90 139.76/257.87 s, $0.091294/pass; Codex 8/8, 117.40/154.72 s, $0.091736/pass; mean-wall ratio 1.334, cost ratio 0.995; loop faster on 0/4 task means and cheaper on 3/4; including dry attempts $0.128629 against $0.128552 per counted pass.
- **API medium**: loop 8/8, 94.03/138.53 s, $0.056310/pass; Codex 8/8, 98.36/113.24 s, $0.070846/pass; mean-wall ratio 1.019, cost ratio 0.795; faster 3/4, cheaper 3/4. `speed2`: 8/8, 94.16/177.64 s, $0.059722/pass.
- **Noise floor** (|wall₁ − wall₂| / mean over same-tool same-task repetitions): median 8.35% at high (range 0.34–14.72%) and 7.23% at medium (2.52–12.66%), eight task-tool pairs each. This is empirical scatter, not a confidence interval.
- **API patch screen** (high, one run per task): loop 145.55/304.08 s, $0.096322; `speed2` 130.45/381.14 s, $0.121729, mean wall +7.1%; `api-replay` 176.77/226.34 s, $0.101571, +0.0%; `api-batch` 129.68/274.10 s, $0.094654, −6.8%; `api-answer-end` 151.40/295.19 s, $0.101195, +2.6%; `wall-http2` 135.17/251.05 s, $0.090700, −8.6%; `wall-h1reuse` 130.44/278.31 s, $0.086608, −7.5%; `api-batch` with its dry attempt $0.116635 per counted pass. All four-task cells pass.
- **Loop output rate by window** (output tokens per body-stream second, from the throughput pack's rows): 30 October 04–05 UTC per-round medians 46.3, 47.8, 47.0 and 47.5 tokens/s across four rounds on both Macs.

Definitions: wall is agent-process time (tools and transport, not setup or grading); p90 is nearest rank over all counted runs with a wall; cost per pass is all counted USD over counted passes, so failures are in the numerator; USD is the frozen list-price estimate, not an invoice.

## Two numbers that do not reproduce exactly

1. The report says loop's output rate on 29 October 18–21 UTC had "medians 100–125 tokens/s across five rounds on both machines". The rows give 105.8, 111.1, 111.4 and 105.9 tokens/s for four rounds (h2h-21, -24, -25, -26) and 85.3 for the fifth (h2h-22, the hard-task cohort, whose own median the report gives as 85.35 elsewhere). The finding's direction and size hold (about 106–112 before, about 46–48 after), but "100–125" is a wider band than the data and h2h-22 is outside it.
2. The patch screen's `wall-h1reuse` p90 is 278.32 s in the report; the rows give 278.31 (the underlying value is 278.315, which the report rounded up and Python's formatter rounds down).

## Numbers that are NOT reproduced here, and where they come from

These come from live scorer output on the benchmark hosts, taken 05:00–05:06 UTC on 30 October while rounds were still running, and per-run rows for them are not part of this pack:

- **Subscription tables** (high and medium, the fresh eight-task rounds h2h-12f, the 24-task rounds h2h-21 and h2h-24, the hard-task round h2h-22, the Sonnet round h2h-11b, and the provisional rounds h2h-25/26/28/29 including the four-tool screen, the subscription patch screen and the five/eleven matched-task changes): computed by the study's aggregator and pairing scripts (`pubagg.py --exploratory`, `paired.py --exploratory`) over each round's per-run result files. "Equivalent tokens" are the study proxy **uncached input + 0.1 × cached input + 4 × output**, not measured subscription dollars. Paired signs are loop wins/losses/ties across task clusters with matched repetitions reduced within task and a one-sided sign-test p; they test pass differences, not joint cost and speed.
- **Where the time is** (65.4% of loop's high-effort excess wall in `readtime-go`; 88.7% of wall inside provider calls; 108 against 90 requests; 18,401 extra output tokens; hard-task tails about 74.5% shell time): from the API and subscription timing diagnoses, which are separate findings.
- **Infrastructure and overlap accounting** (the 22:08 UTC reboot of 29 October that lost two charged results; the Mac laptop reboot that cut rounds h2h-25/26; the one transcript scanner flag, `.lab-eval` in a Codex run; the delivery overlap windows 29 Oct 19:31–19:34, 19:48–20:12, from 20:12 for about 20 minutes, 21:26–21:28, and 30 Oct 04:28–04:31 UTC): from the run logs and the owner's night notes. Cells cut by a reboot were finalized as ungraded agent errors and stay unknown-grade; unknown grades and incomplete costs are never counted as zero.
- **Wider clean-route confirmation** (4 of 32 runs complete, $0.451444 counted spend at report time): a later, finished version of this experiment is in [`reproduce/confirm5-clean-route-8tasks/`](/findings/confirm5-clean-route-8tasks/).

## Tools

| Tool | Version | Model | Effort | Route |
|---|---|---|---|---|
| loop (`loop`, `loop-default`, `loop-b8a4046`) | build b8a4046 | gpt-6.1-terra | high and medium | API key (clean route) and ChatGPT login (subscription) |
| loop variants (`speed2`, `api-replay`, `api-batch`, `api-answer-end`, `wall-http2`, `wall-h1reuse`) | patch files in `lanes/` and in [`reproduce/api-patch-screen-2nd-host/patches/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/api-patch-screen-2nd-host/patches/) | gpt-6.1-terra | high | API key |
| Codex CLI | 0.165.0 | gpt-6.1-terra | high and medium | same API key as loop; ChatGPT login for the subscription rounds |
| Pi / OpenCode | 0.73.1 / 1.19.5 | gpt-6.1-terra | high | ChatGPT login (descriptive screens only) |
| Claude Code | version recorded in that round's manifest (not in these rows) | Claude Sonnet 5.5 | medium | subscription (descriptive Sonnet round only; not reproduced here) |

Tasks: [tasks.txt](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/morning-report-20261030/tasks.txt) (the four counted API tasks plus `r70-1-go`, which only appears in dry runs), directories under `tasks/`.

## Rerun

The API rounds in this report were run with two published drivers: [`lanes/confirm-eu-20261029/run_confirmation.py`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/confirm-eu-20261029/run_confirmation.py) (high effort, loop and Codex, `preflight | dry-loop | dry-codex | counted`) and [`lanes/confirm-eu-20261029/medium/run_confirmation.py`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/confirm-eu-20261029/medium/run_confirmation.py) (medium effort, `preflight | dry-speed2 | counted`), each with its gateway extension next to it. Host paths come from `LANES_CONFIRM_DIR`, `CODEX_BIN_DIR`, `LANES_GOROOT` and `LANES_NODEROOT`; the harness is `lanes/bare/` and the release hashes are in [`lanes/RELEASES.md`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/RELEASES.md). Run from a round directory laid out as in [`lanes/confirm-eu/README.md`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/confirm-eu/README.md):

```sh
LANES_CONFIRM_DIR=/path/to/round python3 -I lanes/confirm-eu-20261029/run_confirmation.py counted
```

Checked 31 October 2026, with a stand-in round directory and no model call, that both drivers print their modes with `--help`. A `counted` run was not tried: it needs the real task kits, builds and key. There is no single command for the whole report; each round it cites has its own driver, all in this repository: the two above for the first high and medium same-key rounds, [`lanes/apiscreen-eu-high-20261029/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/apiscreen-eu-high-20261029/) for the high-effort patch screen (its patch files are in [`lanes/patches/loop/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/patches/loop/) and [`reproduce/api-patch-screen-2nd-host/patches/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/api-patch-screen-2nd-host/patches/)), [`lanes/confirm-eu-wide-20261029/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/confirm-eu-wide-20261029/) for the wider clean-route round (see [`reproduce/confirm5-clean-route-8tasks/`](/findings/confirm5-clean-route-8tasks/)), [`lanes/apiscreen-patches-high-20261030/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/apiscreen-patches-high-20261030/) for the second-host patch screen, and for the subscription rounds the batch files: h2h-11b, h2h-12f, h2h-25, h2h-26 and h2h-28 (with the launch scripts `run-h2h-25.sh`, `-26` and `-28`, which run a dry cell per arm first) in [`experiments/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/morning-report-20261030/experiments/), and h2h-21, h2h-22, h2h-24 and h2h-29 in [`reproduce/loop-shell-tail/experiments/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/loop-shell-tail/experiments/). Run a batch file with `python3 lanes/bare/lanes_bare.py batch FILE --tasks "$TASKS" --run-dir "$RUN" --slots 1`; the home, binary and task paths in them are placeholders (`~/bin/kogen-loop` is loop `b8a4046`, `~/bin/kogen-loop-<variant>` is `b8a4046` plus `speed2` and that variant's patch, `~/bin/codex` is Codex 0.165.0, `~/bin/pi` and `~/bin/opencode` are the Pi and OpenCode builds named in the Tools table). The `stripped` loop instructions of `h2h-12f` are `STRIPPED_POLICY` in `lanes/bare/lanes_bare.py`. The `h2h-11b` Claude Code arm needs your own Claude Code sign-in under the `claude_token_home` placeholder.

The frozen protocols of the rounds are in [`lanes/confirm-eu-20261029/protocol.json`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/confirm-eu-20261029/protocol.json), [`lanes/confirm-eu-20261029/medium/protocol.json`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/confirm-eu-20261029/medium/protocol.json) and the other round directories named above. The task kits with their hidden graders stay private until the Kogen hill-climb ends; no date is set. A rerun also needs Linux with Bubblewrap, Go 1.27.2 and your own OpenAI API key (the same key for every tool). The API rounds cost about $0.73 per tool at high effort, $0.45 (loop) and $0.57 (Codex) at medium, plus dry runs, at list price. Wall times depend on the host and the provider on the day, so fresh values will differ.

## Reproduce the numbers

```sh
python3 -I reproduce/morning-report-20261030/analyze.py | diff - reproduce/morning-report-20261030/expected-output.txt
```

Verified 30 October 2026: it reproduces every figure listed above exactly except the two in "do not reproduce exactly".

## Limits

Four task clusters cannot support a general winner: even four favourable non-tied signs give one-sided p = 0.0625. The medium tasks were already exposed in the high run. The "at most one net task worse per 13" guard allows about 7.7 points of descriptive loss and does not establish non-inferiority. Rows here are API-route only; the subscription evidence is a provisional snapshot.
