> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)
- [site/content/comparison-loop-vs-codex.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/comparison-loop-vs-codex.md)
- [reproduce/shell-yield-eu2/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/shell-yield-eu2/README.md)
- [reproduce/apply-patch-edit5/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/apply-patch-edit5/README.md)
- [reproduce/verbosity-low-eu6/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/verbosity-low-eu6/README.md)
- [reproduce/route-same-host-throughput/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/route-same-host-throughput/README.md)
- [reproduce/replayconf-14task-high/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/replayconf-14task-high/README.md)
- [reproduce/morning-report-20261030/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/morning-report-20261030/README.md)
- [reproduce/loop-sub-route-throughput/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/loop-sub-route-throughput/README.md)
- [reproduce/loop-shell-tail/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/loop-shell-tail/README.md)
- [reproduce/loop-high-reasoning-api-replay13/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/loop-high-reasoning-api-replay13/README.md)
- [reproduce/loop-high-reasoning-api/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/loop-high-reasoning-api/README.md)
- [reproduce/h2h30-sub-route-tokens/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/h2h30-sub-route-tokens/README.md)
- [reproduce/confirm5-clean-route-8tasks/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/confirm5-clean-route-8tasks/README.md)
- [reproduce/batch-tool-short-test/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/batch-tool-short-test/README.md)
- [reproduce/batch-tool-confirm-12task/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/batch-tool-confirm-12task/README.md)

# kogen-loop vs Codex: what the published evidence shows

Scope: every published comparison between kogen-loop (Kogen's small, stand-alone coding agent, "loop" below) and Codex CLI. Source: the public kogen-bench repository and its per-result reproduction packs. Each row links to that result's page on this site, which carries the pack text and a link to the pack. All numbers are copied from those pages. No new runs. Written 31 October 2026.

Same-pair rule: in every row below, loop and Codex ran the **same model (gpt-6.1-terra) at the same effort** in the comparison itself. Where a row is not a loop-vs-Codex task comparison (route speed, analysis of old runs) it says so. "Effort" is the model's reasoning-effort setting (medium or high).

Common facts, unless a row says otherwise: Codex CLI 0.165.0; kogen-loop commit `b8a4046` (default build); one run per task and tool; hidden tests grade each final result; "API route" means both tools used the same API key through one gateway, so cost is at list prices from per-request receipts; hosts are cloud Linux servers, one run at a time. Core counts are published only for the apply_patch row.

## Table

| # | Finding (page) | Arms compared | Model / effort | Task set, n | Pass | Cost | Wall time | Host class | Verdict as published |
|---|---|---|---|---|---|---|---|---|---|
| 1 | replayconf-14task-high ([page](/findings/replayconf-14task-high/)) | loop default; loop + reasoning replay (patched); Codex | gpt-6.1-terra / high, all arms | 14 fresh Go tasks, 14 runs per arm (42) | 14/14 each | Per pass: loop $0.0876, loop+replay $0.0874, Codex $0.1053. Loop cheaper than Codex on 13 of 14 tasks (p=0.0018). Replay vs loop default: cheaper on 4 of 14 (p=0.18). | Median per run: loop 115.2 s, replay 131.4 s, Codex 112.0 s. Loop vs Codex: faster on 5 of 14 (p=0.42): tied | Cloud Linux server | **Yes**: loop cheaper than Codex at high effort. Replay does not advance |
| 2 | batch-tool-confirm-12task ([page](/findings/batch-tool-confirm-12task/)) | loop default; loop + batching sentence; loop + `batch` tool (several operations per request); Codex | gpt-6.1-terra / high | 12 fresh Go tasks, 12 runs per arm (48) | loop default 12/12; sentence 10/12; batch 12/12; Codex 12/12 | Per pass: loop default $0.1101, sentence $0.1303, batch $0.1085, Codex $0.1420. Loop default and batch cheaper than Codex on 11 of 12 (p=0.0063); sentence 9 of 12 (p=0.1460) | Median: 163.7 / 177.2 / 166.7 / 176.7 s | Cloud Linux server | **No** for the batch tool (no change in cost or requests). Loop cheaper than Codex here too |
| 3 | batch-tool-short-test ([page](/findings/batch-tool-short-test/)) | loop default; `batch`; `exec` (script tool); Codex | gpt-6.1-terra / high | 4 fresh Go tasks, 1 run each | loop 4/4; batch 4/4; exec 2/4; Codex 4/4 | Per pass: loop $0.1813, batch $0.1846, exec $1.1293, Codex $0.2388 | Median: 274.3 / 258.9 / 1043.3 / 276.3 s | Cloud Linux server | **Unresolved** for batch (n=4, smallest possible p is 0.125); exec does not advance. Superseded by row 2 |
| 4 | verbosity-low-eu6 ([page](/findings/verbosity-low-eu6/)) | loop + reasoning replay; same + low text verbosity; Codex | gpt-6.1-terra / high | 3 fresh Go tasks x 6 repetitions = 18 blocks, 54 counted runs | 12/18, 12/18, Codex 13/18 | Per pass: $0.1398, $0.1305, Codex $0.1868. Both loop arms cost 0.75x and 0.70x of Codex per pass (p=0.0034, p=0.023) | Median: 125.1 / 128.6 / 121.1 s (loop vs Codex not different, p=0.4807) | Cloud Linux server | **Does not advance** (verbosity setting does nothing measurable). Loop cost lower than Codex, but see caveats: one task failed in every loop run |
| 5 | apply-patch-edit5 ([page](/findings/apply-patch-edit5/)) | loop default; loop + `apply_patch` edit tool; Codex | gpt-6.1-terra / high | 5 Go tasks x 2 repetitions = 10 blocks, 30 counted runs (tasks reused from earlier rounds, not fresh) | 9/10, 10/10, Codex 10/10 | Totals per pass: loop default $0.220, apply_patch $0.164, Codex $0.194. Loop default vs Codex per pass: loop cheaper in 9 blocks, dearer in 1 (p=0.02, arm ratio 1.14 in Codex's favour as printed on the page) | Median: 190 / 175 / 198 s | 2-core cloud Linux server | **No** for apply_patch (edit output not reduced; its cost per pass vs loop default unresolved, 4 vs 6 blocks, p=0.75). Loop default vs Codex: cheaper in 9 of 10 blocks (p=0.02), dearer in arm totals because of one runaway run |
| 6 | shell-yield-eu2 ([page](/findings/shell-yield-eu2/)) | loop default shell; loop + yielding shell (patched); Codex | gpt-6.1-terra / high | 3 hard tasks; 16 of 18 planned cells ran; 5 complete pairs | loop default 4/5; yielding 4/5; Codex 6/6 | Per pass: $0.545, $0.561, Codex $1.350 | Mean agent wall: 1,242 / 615 / 950 s; yielding shell 0.49x of default wall (3 vs 2 pairs, p=1.0). Default hit the 3,600 s limit once | Cloud Linux server | **Unresolved** (5 pairs, nothing significant) |
| 7 | loop-high-reasoning-api-replay13 ([page](/findings/loop-high-reasoning-api-replay13/)) | loop default; loop + reasoning replay; Codex | gpt-6.1-terra / high | 8 tasks, 8 runs per arm (24), fixed order within a task | 8/8 each | Per pass: loop $0.2254, replay $0.1751, Codex $0.2358 (sign tests unresolved) | Per pass: 296 / 268 / 273 s | Cloud Linux server | **Unresolved** (superseded in practice by row 1) |
| 8 | loop-high-reasoning-api ([page](/findings/loop-high-reasoning-api/)) | loop; Codex | gpt-6.1-terra / high (also medium) | 4 Go tasks x 2 repetitions = 8 runs per tool | 8/8 each at high and at medium | High: loop $0.730 vs Codex $0.734 total; 108 vs 90 requests; 27,769 vs 14,106 reasoning tokens. Medium: $0.450 vs $0.567; 84 vs 85 requests | Not published on this page | Cloud Linux server | **Unresolved** (loop used about twice the reasoning; cause not settled) |
| 9 | h2h30-sub-route-tokens ([page](/findings/h2h30-sub-route-tokens/)) | loop; Codex, at each effort | gpt-6.1-terra / medium and high (each compared with the same effort only) | 24 Go tasks, 24 runs per arm and effort (96) | 24/24 in all four | No dollar cost (this route is not priced). "Equivalent tokens" per pass (input + output weighted): loop 38.9% of Codex at medium (32,697 vs 84,035), 42.2% at high (40,747 vs 96,476); loop lower on 24 of 24 tasks at both efforts | Median: medium loop 81.16 s vs Codex 66.91 s; high loop 101.37 s vs 77.79 s. Loop 25-30% slower, **route-confounded**, not a tool comparison | One cloud Linux server | **Yes** for the token result only. Wall is not a verdict |
| 10 | loop-shell-tail ([page](/findings/loop-shell-tail/)) | loop vs Codex: shell activity in 104 old both-pass pairs (read-only analysis) | gpt-6.1-terra / medium and high, same effort per pair | 104 pairs across 6 cohorts (2 on the API route, 4 on the ChatGPT subscription route); 12 failed-both pairs kept apart | Only both-pass pairs used | Not applicable | Not a timing test | Mixed (Linux server, Macs) | **No** (no change follows). Loop ran fewer shell commands than Codex in all 7 cohorts and read back fewer bytes in every subscription cohort |
| 11 | route-same-host-throughput ([page](/findings/route-same-host-throughput/)) | Streaming speed (output tokens/s) of loop's route vs Codex's route (a third client also tested) | gpt-6.1-terra / medium | Short fixed-prompt probes: 20 on one Linux server, 10 on another, 10 on a Mac, per tool | Not applicable | Not applicable | Median tokens/s: loop 40.6 vs Codex 41.3 (Linux A), 40.2 vs 41.0 (Linux B), 40.7 vs 44.0 (Mac, p=0.11) | Two cloud Linux servers and one desktop Mac | **No route gap** (all p >= 0.11) |
| 12 | loop-sub-route-throughput ([page](/findings/loop-sub-route-throughput/)) | Same question as row 11, earlier data: 496 historical runs and tiny probes | gpt-6.1-terra / medium and high | 496 old runs on two Macs; 6 + 40 probes | Not applicable | Not applicable | Loop's rate fell from about 110 to about 48 tokens/s early on 30 Oct; direct probe pair 44.97 vs 80.88 pooled (medians 45.33 vs 105.57) | Two Macs; probes on Linux | **Superseded** by row 11; the probe gap did not reproduce and the cause of the earlier drop was not found |

- **Sample sizes are small.** Most rounds are 3 to 24 tasks with one run each, so sign tests can only detect large effects. Row 3 has n=4 (the smallest possible p is 0.125); row 6 has 5 pairs; row 8 has 8 runs per tool. Pages state test power where known: row 2 about 0.75 for a 20% effect, row 4 about 0.75 for a 10% effect and under 0.05 for a 1-3% cost effect, row 5 about 0.79 for a 30% effect only, row 1 about 0.3 for a 20% cost cut in the replay test. "No effect seen" is not "equal".
- **Confidence intervals** are not published for these rows; the pages give sign-test p-values and counts instead.
- **Repetitions are not independent tasks.** Row 4 uses 3 tasks x 6 repetitions, so the effective sample is 3 tasks and the page says the p-values overstate the evidence. One of its tasks (`revive-baseline-go`) failed in all 12 loop runs and passed in 1 of 6 Codex runs (p=0.33); the failing hidden test was never diagnosed; all of the 6.6% lower cost per pass of the verbosity arm comes from that task. Row 5 reused tasks from earlier rounds. Row 2 and row 7 used a rotated or fixed order within a task, not a random one, so order or caching effects are possible.
- **Task difficulty.** Rows 1 to 5 and 7 to 9 use short Go tasks; row 5 notes one task costs several times the others and dominates totals. Only row 6 uses hard tasks. Most task kits are withheld, so outside readers cannot rerun every row. Only 5 of the 24 tasks of row 9 are published.
- **One host per round, one run at a time.** Wall times depend on that host. Cost is at list prices from the gateway receipts, not an invoice.
- **Timing-host rules.** The published rule ([protocol-split-20261030](/findings/protocol-split-20261030/)) says loop-vs-Codex wall time and cost are claimed only from the Linux servers, interleaved per task; the Macs make no wall-time claim. Row 9 (subscription route, one Linux server) and row 12 (two Macs) predate that rule and carry route confounds: loop and Codex reached the model by different routes, and a provider-side speed change early on 30 October was not explained.
- **Two arms are patched experiments, not releases:** loop + reasoning replay, `batch`, `exec`, verbosity low, `apply_patch`, yielding shell. Comparisons of loop default vs Codex use the unpatched build.
- **Row 5**: the default arm's total includes one runaway run (hit the 128,000-token output cap and failed); without that block `apply_patch` cost 13% more (not registered in advance).
- **Row 6**: one block was cut by the spend guard, on the task that shows the tail, so the yield effect may be understated. Two default-shell runs hit the 3,600 s limit (counted as outcomes, wall censored at 3,600 s). 4 of 6 yielding cells ended on loop's 512 KB request limit.
- **Row 2 and row 3**: the loop binaries are rebuilds identified by hash, and the tool and its added instruction sentence are not separated.
- **Not included.** Rows where Kogen (the full product driving Codex) is compared with Codex alone on hard tasks, and the rule-only finding ([protocol-split-20261030](/findings/protocol-split-20261030/)), are outside "loop vs Codex" and are left out of the table. [loop-sub-route-honest-probe](/findings/loop-sub-route-honest-probe/) tests loop's own routes only (no Codex arm), so it is also left out.

## Other published results in this comparison

These results are not rows of the table above (it was written from the first twelve). Each has its own page.

- [Report on whether kogen-loop is cheaper, as reliable and faster than Codex: goal not met](/findings/morning-report-20261030/) (Unresolved, 30 October 2026)
- [kogen-loop vs Codex on 8 Go tasks: cheaper at medium effort, no clear gap at high](/findings/confirm5-clean-route-8tasks/) (Yes, 30 October 2026)

## All results in this comparison

| Result | Verdict | Finished |
| --- | --- | --- |
| [A shell that hands control back during long waits: no clear effect on hard tasks](/findings/shell-yield-eu2/) | Unresolved | 31 October 2026 |
| [Adding an 'apply_patch' editing tool to kogen-loop did not reduce its output](/findings/apply-patch-edit5/) | No | 31 October 2026 |
| [Asking the model for terser text does not change kogen-loop's cost](/findings/verbosity-low-eu6/) | No | 30 October 2026 |
| [kogen-loop's connection to the model streams as fast as Codex's](/findings/route-same-host-throughput/) | No | 30 October 2026 |
| [kogen-loop costs less than Codex at high effort, on 14 Go tasks](/findings/replayconf-14task-high/) | Yes | 30 October 2026 |
| [Report on whether kogen-loop is cheaper, as reliable and faster than Codex: goal not met](/findings/morning-report-20261030/) | Unresolved | 30 October 2026 |
| [Did kogen-loop's connection slow down on 30 October? Earlier data, now superseded](/findings/loop-sub-route-throughput/) | Superseded | 30 October 2026 |
| [kogen-loop runs fewer shell commands than Codex in old runs](/findings/loop-shell-tail/) | No | 30 October 2026 |
| [Sending reasoning back to the model: promising cost, but unresolved on 8 tasks](/findings/loop-high-reasoning-api-replay13/) | Unresolved | 30 October 2026 |
| [kogen-loop used about twice Codex's reasoning, for the same total cost](/findings/loop-high-reasoning-api/) | Unresolved | 30 October 2026 |
| [On the subscription route kogen-loop used about 40% of Codex's tokens](/findings/h2h30-sub-route-tokens/) | Yes | 30 October 2026 |
| [kogen-loop vs Codex on 8 Go tasks: cheaper at medium effort, no clear gap at high](/findings/confirm5-clean-route-8tasks/) | Yes | 30 October 2026 |
| [Short test of two kogen-loop 'several steps per request' tools (batch and exec)](/findings/batch-tool-short-test/) | Unresolved | 30 October 2026 |
| [A 'batch' tool for kogen-loop packs more work per request but does not cut cost](/findings/batch-tool-confirm-12task/) | No | 30 October 2026 |
