Comparison
kogen-loop vs Codex
Head-to-head tests of kogen-loop, Kogen's small stand-alone coding agent, against OpenAI's Codex command-line agent, using the same AI model at the same effort setting.
Comparison · site/content/finding-pages.json
Scope: every published comparison between kogen-loop (Kogen's small, stand-alone coding agent, "loop" below) and Codex CLI. Source: the public kogen-bench repository and its per-result reproduction packs. Each row links to that result's page on this site, which carries the pack text and a link to the pack. All numbers are copied from those pages. No new runs. Written 31 October 2026.
Same-pair rule: in every row below, loop and Codex ran the same model (gpt-6.1-terra) at the same effort in the comparison itself. Where a row is not a loop-vs-Codex task comparison (route speed, analysis of old runs) it says so. "Effort" is the model's reasoning-effort setting (medium or high).
Common facts, unless a row says otherwise: Codex CLI 0.165.0; kogen-loop commit b8a4046 (default build); one run per task and tool; hidden tests grade each final result; "API route" means both tools used the same API key through one gateway, so cost is at list prices from per-request receipts; hosts are cloud Linux servers, one run at a time. Core counts are published only for the apply_patch row.
Table
| # | Finding (page) | Arms compared | Model / effort | Task set, n | Pass | Cost | Wall time | Host class | Verdict as published |
|---|---|---|---|---|---|---|---|---|---|
| 1 | replayconf-14task-high (page) | loop default; loop + reasoning replay (patched); Codex | gpt-6.1-terra / high, all arms | 14 fresh Go tasks, 14 runs per arm (42) | 14/14 each | Per pass: loop $0.0876, loop+replay $0.0874, Codex $0.1053. Loop cheaper than Codex on 13 of 14 tasks (p=0.0018). Replay vs loop default: cheaper on 4 of 14 (p=0.18). | Median per run: loop 115.2 s, replay 131.4 s, Codex 112.0 s. Loop vs Codex: faster on 5 of 14 (p=0.42): tied | Cloud Linux server | Yes: loop cheaper than Codex at high effort. Replay does not advance |
| 2 | batch-tool-confirm-12task (page) | loop default; loop + batching sentence; loop + batch tool (several operations per request); Codex |
gpt-6.1-terra / high | 12 fresh Go tasks, 12 runs per arm (48) | loop default 12/12; sentence 10/12; batch 12/12; Codex 12/12 | Per pass: loop default $0.1101, sentence $0.1303, batch $0.1085, Codex $0.1420. Loop default and batch cheaper than Codex on 11 of 12 (p=0.0063); sentence 9 of 12 (p=0.1460) | Median: 163.7 / 177.2 / 166.7 / 176.7 s | Cloud Linux server | No for the batch tool (no change in cost or requests). Loop cheaper than Codex here too |
| 3 | batch-tool-short-test (page) | loop default; batch; exec (script tool); Codex |
gpt-6.1-terra / high | 4 fresh Go tasks, 1 run each | loop 4/4; batch 4/4; exec 2/4; Codex 4/4 | Per pass: loop $0.1813, batch $0.1846, exec $1.1293, Codex $0.2388 | Median: 274.3 / 258.9 / 1043.3 / 276.3 s | Cloud Linux server | Unresolved for batch (n=4, smallest possible p is 0.125); exec does not advance. Superseded by row 2 |
| 4 | verbosity-low-eu6 (page) | loop + reasoning replay; same + low text verbosity; Codex | gpt-6.1-terra / high | 3 fresh Go tasks x 6 repetitions = 18 blocks, 54 counted runs | 12/18, 12/18, Codex 13/18 | Per pass: $0.1398, $0.1305, Codex $0.1868. Both loop arms cost 0.75x and 0.70x of Codex per pass (p=0.0034, p=0.023) | Median: 125.1 / 128.6 / 121.1 s (loop vs Codex not different, p=0.4807) | Cloud Linux server | Does not advance (verbosity setting does nothing measurable). Loop cost lower than Codex, but see caveats: one task failed in every loop run |
| 5 | apply-patch-edit5 (page) | loop default; loop + apply_patch edit tool; Codex |
gpt-6.1-terra / high | 5 Go tasks x 2 repetitions = 10 blocks, 30 counted runs (tasks reused from earlier rounds, not fresh) | 9/10, 10/10, Codex 10/10 | Totals per pass: loop default $0.220, apply_patch $0.164, Codex $0.194. Loop default vs Codex per pass: loop cheaper in 9 blocks, dearer in 1 (p=0.02, arm ratio 1.14 in Codex's favour as printed on the page) | Median: 190 / 175 / 198 s | 2-core cloud Linux server | No for apply_patch (edit output not reduced; its cost per pass vs loop default unresolved, 4 vs 6 blocks, p=0.75). Loop default vs Codex: cheaper in 9 of 10 blocks (p=0.02), dearer in arm totals because of one runaway run |
| 6 | shell-yield-eu2 (page) | loop default shell; loop + yielding shell (patched); Codex | gpt-6.1-terra / high | 3 hard tasks; 16 of 18 planned cells ran; 5 complete pairs | loop default 4/5; yielding 4/5; Codex 6/6 | Per pass: $0.545, $0.561, Codex $1.350 | Mean agent wall: 1,242 / 615 / 950 s; yielding shell 0.49x of default wall (3 vs 2 pairs, p=1.0). Default hit the 3,600 s limit once | Cloud Linux server | Unresolved (5 pairs, nothing significant) |
| 7 | loop-high-reasoning-api-replay13 (page) | loop default; loop + reasoning replay; Codex | gpt-6.1-terra / high | 8 tasks, 8 runs per arm (24), fixed order within a task | 8/8 each | Per pass: loop $0.2254, replay $0.1751, Codex $0.2358 (sign tests unresolved) | Per pass: 296 / 268 / 273 s | Cloud Linux server | Unresolved (superseded in practice by row 1) |
| 8 | loop-high-reasoning-api (page) | loop; Codex | gpt-6.1-terra / high (also medium) | 4 Go tasks x 2 repetitions = 8 runs per tool | 8/8 each at high and at medium | High: loop $0.730 vs Codex $0.734 total; 108 vs 90 requests; 27,769 vs 14,106 reasoning tokens. Medium: $0.450 vs $0.567; 84 vs 85 requests | Not published on this page | Cloud Linux server | Unresolved (loop used about twice the reasoning; cause not settled) |
| 9 | h2h30-sub-route-tokens (page) | loop; Codex, at each effort | gpt-6.1-terra / medium and high (each compared with the same effort only) | 24 Go tasks, 24 runs per arm and effort (96) | 24/24 in all four | No dollar cost (this route is not priced). "Equivalent tokens" per pass (input + output weighted): loop 38.9% of Codex at medium (32,697 vs 84,035), 42.2% at high (40,747 vs 96,476); loop lower on 24 of 24 tasks at both efforts | Median: medium loop 81.16 s vs Codex 66.91 s; high loop 101.37 s vs 77.79 s. Loop 25-30% slower, route-confounded, not a tool comparison | One cloud Linux server | Yes for the token result only. Wall is not a verdict |
| 10 | loop-shell-tail (page) | loop vs Codex: shell activity in 104 old both-pass pairs (read-only analysis) | gpt-6.1-terra / medium and high, same effort per pair | 104 pairs across 6 cohorts (2 on the API route, 4 on the ChatGPT subscription route); 12 failed-both pairs kept apart | Only both-pass pairs used | Not applicable | Not a timing test | Mixed (Linux server, Macs) | No (no change follows). Loop ran fewer shell commands than Codex in all 7 cohorts and read back fewer bytes in every subscription cohort |
| 11 | route-same-host-throughput (page) | Streaming speed (output tokens/s) of loop's route vs Codex's route (a third client also tested) | gpt-6.1-terra / medium | Short fixed-prompt probes: 20 on one Linux server, 10 on another, 10 on a Mac, per tool | Not applicable | Not applicable | Median tokens/s: loop 40.6 vs Codex 41.3 (Linux A), 40.2 vs 41.0 (Linux B), 40.7 vs 44.0 (Mac, p=0.11) | Two cloud Linux servers and one desktop Mac | No route gap (all p >= 0.11) |
| 12 | loop-sub-route-throughput (page) | Same question as row 11, earlier data: 496 historical runs and tiny probes | gpt-6.1-terra / medium and high | 496 old runs on two Macs; 6 + 40 probes | Not applicable | Not applicable | Loop's rate fell from about 110 to about 48 tokens/s early on 30 Oct; direct probe pair 44.97 vs 80.88 pooled (medians 45.33 vs 105.57) | Two Macs; probes on Linux | Superseded by row 11; the probe gap did not reproduce and the cause of the earlier drop was not found |
- Sample sizes are small. Most rounds are 3 to 24 tasks with one run each, so sign tests can only detect large effects. Row 3 has n=4 (the smallest possible p is 0.125); row 6 has 5 pairs; row 8 has 8 runs per tool. Pages state test power where known: row 2 about 0.75 for a 20% effect, row 4 about 0.75 for a 10% effect and under 0.05 for a 1-3% cost effect, row 5 about 0.79 for a 30% effect only, row 1 about 0.3 for a 20% cost cut in the replay test. "No effect seen" is not "equal".
- Confidence intervals are not published for these rows; the pages give sign-test p-values and counts instead.
- Repetitions are not independent tasks. Row 4 uses 3 tasks x 6 repetitions, so the effective sample is 3 tasks and the page says the p-values overstate the evidence. One of its tasks (
revive-baseline-go) failed in all 12 loop runs and passed in 1 of 6 Codex runs (p=0.33); the failing hidden test was never diagnosed; all of the 6.6% lower cost per pass of the verbosity arm comes from that task. Row 5 reused tasks from earlier rounds. Row 2 and row 7 used a rotated or fixed order within a task, not a random one, so order or caching effects are possible. - Task difficulty. Rows 1 to 5 and 7 to 9 use short Go tasks; row 5 notes one task costs several times the others and dominates totals. Only row 6 uses hard tasks. Most task kits are withheld, so outside readers cannot rerun every row. Only 5 of the 24 tasks of row 9 are published.
- One host per round, one run at a time. Wall times depend on that host. Cost is at list prices from the gateway receipts, not an invoice.
- Timing-host rules. The published rule (protocol-split-20261030) says loop-vs-Codex wall time and cost are claimed only from the Linux servers, interleaved per task; the Macs make no wall-time claim. Row 9 (subscription route, one Linux server) and row 12 (two Macs) predate that rule and carry route confounds: loop and Codex reached the model by different routes, and a provider-side speed change early on 30 October was not explained.
- Two arms are patched experiments, not releases: loop + reasoning replay,
batch,exec, verbosity low,apply_patch, yielding shell. Comparisons of loop default vs Codex use the unpatched build. - Row 5: the default arm's total includes one runaway run (hit the 128,000-token output cap and failed); without that block
apply_patchcost 13% more (not registered in advance). - Row 6: one block was cut by the spend guard, on the task that shows the tail, so the yield effect may be understated. Two default-shell runs hit the 3,600 s limit (counted as outcomes, wall censored at 3,600 s). 4 of 6 yielding cells ended on loop's 512 KB request limit.
- Row 2 and row 3: the loop binaries are rebuilds identified by hash, and the tool and its added instruction sentence are not separated.
- Not included. Rows where Kogen (the full product driving Codex) is compared with Codex alone on hard tasks, and the rule-only finding (protocol-split-20261030), are outside "loop vs Codex" and are left out of the table. loop-sub-route-honest-probe tests loop's own routes only (no Codex arm), so it is also left out.
Other published results in this comparison
These results are not rows of the table above (it was written from the first twelve). Each has its own page.
- Report on whether kogen-loop is cheaper, as reliable and faster than Codex: goal not met (Unresolved, 30 October 2026)
- kogen-loop vs Codex on 8 Go tasks: cheaper at medium effort, no clear gap at high (Yes, 30 October 2026)