Comparison

kogen-loop vs Codex

Head-to-head tests of kogen-loop, Kogen's small stand-alone coding agent, against OpenAI's Codex command-line agent, using the same AI model at the same effort setting.

Comparison · site/content/finding-pages.json

Scope: every published comparison between kogen-loop (Kogen's small, stand-alone coding agent, "loop" below) and Codex CLI. Source: the public kogen-bench repository and its per-result reproduction packs. Each row links to that result's page on this site, which carries the pack text and a link to the pack. All numbers are copied from those pages. No new runs. Written 31 October 2026.

Same-pair rule: in every row below, loop and Codex ran the same model (gpt-6.1-terra) at the same effort in the comparison itself. Where a row is not a loop-vs-Codex task comparison (route speed, analysis of old runs) it says so. "Effort" is the model's reasoning-effort setting (medium or high).

Common facts, unless a row says otherwise: Codex CLI 0.165.0; kogen-loop commit b8a4046 (default build); one run per task and tool; hidden tests grade each final result; "API route" means both tools used the same API key through one gateway, so cost is at list prices from per-request receipts; hosts are cloud Linux servers, one run at a time. Core counts are published only for the apply_patch row.

Table

# Finding (page) Arms compared Model / effort Task set, n Pass Cost Wall time Host class Verdict as published
1 replayconf-14task-high (page) loop default; loop + reasoning replay (patched); Codex gpt-6.1-terra / high, all arms 14 fresh Go tasks, 14 runs per arm (42) 14/14 each Per pass: loop $0.0876, loop+replay $0.0874, Codex $0.1053. Loop cheaper than Codex on 13 of 14 tasks (p=0.0018). Replay vs loop default: cheaper on 4 of 14 (p=0.18). Median per run: loop 115.2 s, replay 131.4 s, Codex 112.0 s. Loop vs Codex: faster on 5 of 14 (p=0.42): tied Cloud Linux server Yes: loop cheaper than Codex at high effort. Replay does not advance
2 batch-tool-confirm-12task (page) loop default; loop + batching sentence; loop + batch tool (several operations per request); Codex gpt-6.1-terra / high 12 fresh Go tasks, 12 runs per arm (48) loop default 12/12; sentence 10/12; batch 12/12; Codex 12/12 Per pass: loop default $0.1101, sentence $0.1303, batch $0.1085, Codex $0.1420. Loop default and batch cheaper than Codex on 11 of 12 (p=0.0063); sentence 9 of 12 (p=0.1460) Median: 163.7 / 177.2 / 166.7 / 176.7 s Cloud Linux server No for the batch tool (no change in cost or requests). Loop cheaper than Codex here too
3 batch-tool-short-test (page) loop default; batch; exec (script tool); Codex gpt-6.1-terra / high 4 fresh Go tasks, 1 run each loop 4/4; batch 4/4; exec 2/4; Codex 4/4 Per pass: loop $0.1813, batch $0.1846, exec $1.1293, Codex $0.2388 Median: 274.3 / 258.9 / 1043.3 / 276.3 s Cloud Linux server Unresolved for batch (n=4, smallest possible p is 0.125); exec does not advance. Superseded by row 2
4 verbosity-low-eu6 (page) loop + reasoning replay; same + low text verbosity; Codex gpt-6.1-terra / high 3 fresh Go tasks x 6 repetitions = 18 blocks, 54 counted runs 12/18, 12/18, Codex 13/18 Per pass: $0.1398, $0.1305, Codex $0.1868. Both loop arms cost 0.75x and 0.70x of Codex per pass (p=0.0034, p=0.023) Median: 125.1 / 128.6 / 121.1 s (loop vs Codex not different, p=0.4807) Cloud Linux server Does not advance (verbosity setting does nothing measurable). Loop cost lower than Codex, but see caveats: one task failed in every loop run
5 apply-patch-edit5 (page) loop default; loop + apply_patch edit tool; Codex gpt-6.1-terra / high 5 Go tasks x 2 repetitions = 10 blocks, 30 counted runs (tasks reused from earlier rounds, not fresh) 9/10, 10/10, Codex 10/10 Totals per pass: loop default $0.220, apply_patch $0.164, Codex $0.194. Loop default vs Codex per pass: loop cheaper in 9 blocks, dearer in 1 (p=0.02, arm ratio 1.14 in Codex's favour as printed on the page) Median: 190 / 175 / 198 s 2-core cloud Linux server No for apply_patch (edit output not reduced; its cost per pass vs loop default unresolved, 4 vs 6 blocks, p=0.75). Loop default vs Codex: cheaper in 9 of 10 blocks (p=0.02), dearer in arm totals because of one runaway run
6 shell-yield-eu2 (page) loop default shell; loop + yielding shell (patched); Codex gpt-6.1-terra / high 3 hard tasks; 16 of 18 planned cells ran; 5 complete pairs loop default 4/5; yielding 4/5; Codex 6/6 Per pass: $0.545, $0.561, Codex $1.350 Mean agent wall: 1,242 / 615 / 950 s; yielding shell 0.49x of default wall (3 vs 2 pairs, p=1.0). Default hit the 3,600 s limit once Cloud Linux server Unresolved (5 pairs, nothing significant)
7 loop-high-reasoning-api-replay13 (page) loop default; loop + reasoning replay; Codex gpt-6.1-terra / high 8 tasks, 8 runs per arm (24), fixed order within a task 8/8 each Per pass: loop $0.2254, replay $0.1751, Codex $0.2358 (sign tests unresolved) Per pass: 296 / 268 / 273 s Cloud Linux server Unresolved (superseded in practice by row 1)
8 loop-high-reasoning-api (page) loop; Codex gpt-6.1-terra / high (also medium) 4 Go tasks x 2 repetitions = 8 runs per tool 8/8 each at high and at medium High: loop $0.730 vs Codex $0.734 total; 108 vs 90 requests; 27,769 vs 14,106 reasoning tokens. Medium: $0.450 vs $0.567; 84 vs 85 requests Not published on this page Cloud Linux server Unresolved (loop used about twice the reasoning; cause not settled)
9 h2h30-sub-route-tokens (page) loop; Codex, at each effort gpt-6.1-terra / medium and high (each compared with the same effort only) 24 Go tasks, 24 runs per arm and effort (96) 24/24 in all four No dollar cost (this route is not priced). "Equivalent tokens" per pass (input + output weighted): loop 38.9% of Codex at medium (32,697 vs 84,035), 42.2% at high (40,747 vs 96,476); loop lower on 24 of 24 tasks at both efforts Median: medium loop 81.16 s vs Codex 66.91 s; high loop 101.37 s vs 77.79 s. Loop 25-30% slower, route-confounded, not a tool comparison One cloud Linux server Yes for the token result only. Wall is not a verdict
10 loop-shell-tail (page) loop vs Codex: shell activity in 104 old both-pass pairs (read-only analysis) gpt-6.1-terra / medium and high, same effort per pair 104 pairs across 6 cohorts (2 on the API route, 4 on the ChatGPT subscription route); 12 failed-both pairs kept apart Only both-pass pairs used Not applicable Not a timing test Mixed (Linux server, Macs) No (no change follows). Loop ran fewer shell commands than Codex in all 7 cohorts and read back fewer bytes in every subscription cohort
11 route-same-host-throughput (page) Streaming speed (output tokens/s) of loop's route vs Codex's route (a third client also tested) gpt-6.1-terra / medium Short fixed-prompt probes: 20 on one Linux server, 10 on another, 10 on a Mac, per tool Not applicable Not applicable Median tokens/s: loop 40.6 vs Codex 41.3 (Linux A), 40.2 vs 41.0 (Linux B), 40.7 vs 44.0 (Mac, p=0.11) Two cloud Linux servers and one desktop Mac No route gap (all p >= 0.11)
12 loop-sub-route-throughput (page) Same question as row 11, earlier data: 496 historical runs and tiny probes gpt-6.1-terra / medium and high 496 old runs on two Macs; 6 + 40 probes Not applicable Not applicable Loop's rate fell from about 110 to about 48 tokens/s early on 30 Oct; direct probe pair 44.97 vs 80.88 pooled (medians 45.33 vs 105.57) Two Macs; probes on Linux Superseded by row 11; the probe gap did not reproduce and the cause of the earlier drop was not found
  • Sample sizes are small. Most rounds are 3 to 24 tasks with one run each, so sign tests can only detect large effects. Row 3 has n=4 (the smallest possible p is 0.125); row 6 has 5 pairs; row 8 has 8 runs per tool. Pages state test power where known: row 2 about 0.75 for a 20% effect, row 4 about 0.75 for a 10% effect and under 0.05 for a 1-3% cost effect, row 5 about 0.79 for a 30% effect only, row 1 about 0.3 for a 20% cost cut in the replay test. "No effect seen" is not "equal".
  • Confidence intervals are not published for these rows; the pages give sign-test p-values and counts instead.
  • Repetitions are not independent tasks. Row 4 uses 3 tasks x 6 repetitions, so the effective sample is 3 tasks and the page says the p-values overstate the evidence. One of its tasks (revive-baseline-go) failed in all 12 loop runs and passed in 1 of 6 Codex runs (p=0.33); the failing hidden test was never diagnosed; all of the 6.6% lower cost per pass of the verbosity arm comes from that task. Row 5 reused tasks from earlier rounds. Row 2 and row 7 used a rotated or fixed order within a task, not a random one, so order or caching effects are possible.
  • Task difficulty. Rows 1 to 5 and 7 to 9 use short Go tasks; row 5 notes one task costs several times the others and dominates totals. Only row 6 uses hard tasks. Most task kits are withheld, so outside readers cannot rerun every row. Only 5 of the 24 tasks of row 9 are published.
  • One host per round, one run at a time. Wall times depend on that host. Cost is at list prices from the gateway receipts, not an invoice.
  • Timing-host rules. The published rule (protocol-split-20261030) says loop-vs-Codex wall time and cost are claimed only from the Linux servers, interleaved per task; the Macs make no wall-time claim. Row 9 (subscription route, one Linux server) and row 12 (two Macs) predate that rule and carry route confounds: loop and Codex reached the model by different routes, and a provider-side speed change early on 30 October was not explained.
  • Two arms are patched experiments, not releases: loop + reasoning replay, batch, exec, verbosity low, apply_patch, yielding shell. Comparisons of loop default vs Codex use the unpatched build.
  • Row 5: the default arm's total includes one runaway run (hit the 128,000-token output cap and failed); without that block apply_patch cost 13% more (not registered in advance).
  • Row 6: one block was cut by the spend guard, on the task that shows the tail, so the yield effect may be understated. Two default-shell runs hit the 3,600 s limit (counted as outcomes, wall censored at 3,600 s). 4 of 6 yielding cells ended on loop's 512 KB request limit.
  • Row 2 and row 3: the loop binaries are rebuilds identified by hash, and the tool and its added instruction sentence are not separated.
  • Not included. Rows where Kogen (the full product driving Codex) is compared with Codex alone on hard tasks, and the rule-only finding (protocol-split-20261030), are outside "loop vs Codex" and are left out of the table. loop-sub-route-honest-probe tests loop's own routes only (no Codex arm), so it is also left out.

Other published results in this comparison

These results are not rows of the table above (it was written from the first twelve). Each has its own page.

All results in this comparison