> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [reproduce/lab-cc-layers-b-20261105/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/lab-cc-layers-b-20261105/README.md)
- [categories/head-to-head.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/head-to-head.md)
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)

# kogen-loop is faster and lighter than Claude Code on the same API key

- **Finding id:** `lab-cc-layers-b-20261105`
- **Verdict:** Yes
- **Finished:** 5 November 2026
- **Comparison:** [kogen-loop speed, patches and connection routes](/comparisons/loop-speed-and-routes/)
- **Reproduction pack:** [reproduce/lab-cc-layers-b-20261105/](https://github.com/KogenAI/kogen-bench/tree/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/lab-cc-layers-b-20261105) (data, analysis script and rerun steps)
- **Category page:** [categories/head-to-head.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/head-to-head.md)

## In plain terms

Same model (claude-sonnet-5-5, medium effort), prompt, server and API key, 12 Go tasks x 2 repetitions. kogen-loop took 0.80x the agent wall time of Claude Code configured as Kogen runs it and 0.88x that of a lean Claude Code, faster in 21 of 23 paired blocks each, with 0.64x and 0.85x the tokens and 2-3 fewer requests per task. Passes: loop 21/24, Kogen-style Claude Code 20/23, lean 22/23. Kogen's executor arguments cost Claude Code about 9% wall and 33% tokens; running Claude Code with kogen-loop's instructions as its system prompt was no slower and passed 24/24. Descriptive; the last block's two Claude Code cells did not run (spend cap).

Verdict key: **Yes**, The result supports the change or claim that was tested. **No**, The result does not support it, so nothing changes. **Unresolved**, The evidence is not strong enough either way. **Superseded**, A later test replaced this result.

## Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

On the same model, effort, prompt, server and API key, is kogen-loop faster and lighter than Claude Code, and how much of the gap is Claude Code's own layers? **kogen-loop finished faster than every Claude Code setup: 0.80x the agent wall time of Claude Code configured exactly as Kogen runs it (97.5% CI 0.76-0.85, faster in 21 of 23 paired blocks) and 0.88x the wall of a lean Claude Code (0.84-0.92, 21 of 23).** It used 0.64x and 0.85x the uncached-equivalent tokens and list cost, and about 2-3 fewer model requests per task. It passed 21 of 24 tasks, against 20 of 23 and 22 of 23. Its three failures are two csvq-go repetitions and one splitter-go repetition. Stage B of Lab cc-layers-1 is descriptive: no product change follows from it directly.

## Setup

- **Tasks:** 12 Go tasks (`tasks.txt`; each has its task directory in `tasks/`), 2 repetitions each, giving 24 blocks.
- **Block order:** one block holds the four arms of one (task, repetition). Cells ran one at a time on one 4-core Linux server, with the arm order rotated per block. Every arm got the same prompt (the bare harness's `build_prompt`: request, acceptance scenarios, checks and rules).
- **Model and route:** claude-sonnet-5-5 at medium effort in every cell. Every cell used the same Anthropic API key.
- **Deadline:** 3600 s.
- **Grading:** cells were graded by the harness's own checks.
- **Server load:** no delivery job overlapped any counted cell; the runner sampled every 10 s.

| Arm | Client | What differs |
| --- | --- | --- |
| `loop` | kogen-loop `e5054c8` (the shipped default of the earlier head-to-head rounds), provider anthropic | its shipped default instructions (`patches/loop-instructions.txt`) |
| `claude-kexec` | Claude Code 2.1.312 | the exact arguments Kogen's executor passes for a build round: appended executor constraints (`patches/constraints.txt`), the build submission schema, the executor's tool settings, a fresh config dir, background tasks, auto-memory and updates off, Kogen's own pre/post tool hook |
| `claude-lean` | Claude Code 2.1.312 | same, but the constraints replace the system prompt (`--system-prompt`), tools limited to Bash, Read, Write, Edit, Glob, Grep, git instructions off |
| `claude-c2` | Claude Code 2.1.312 | `claude-lean` with kogen-loop's instructions as the system prompt instead of the constraints |

The arms are defined in `patches/claude-arms.diff`, which applies to `lanes/bare/lanes_bare.py`. Checksums are in `patches/SHA256SUMS`. Credentials reach Claude Code only on an inherited file descriptor, never through arguments, environment values or files.

## Results

| Arm | Cells | Passed | Checks passed | Median agent wall | Mean requests | List cost | Cost per pass |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `loop` | 24 | 21 | 2075/2098 | 36.8 s | 5.7 | $2.32 | $0.110 |
| `claude-kexec` | 23 | 20 | 1335/2091 | 45.4 s | 8.4 | $3.27 | $0.163 |
| `claude-lean` | 23 | 22 | 2084/2091 | 42.6 s | 8.0 | $2.55 | $0.116 |
| `claude-c2` | 24 | 24 | 2098/2098 | 44.5 s | 8.6 | $2.87 | $0.120 |

Paired over the blocks that hold both arms (geometric means; normal CI on per-block log ratios; exact two-sided sign test):

| Pair | Kind | Agent wall | Faster in | Sign p | Tokens | Requests per task |
| --- | --- | --- | --- | --- | --- | --- |
| loop / claude-kexec | primary, 97.5% CI | 0.804 [0.758, 0.853] | 21/23 | 0.0001 | 0.644 [0.606, 0.684] | -2.7 |
| loop / claude-lean | primary, 97.5% CI | 0.877 [0.836, 0.920] | 21/23 | 0.0001 | 0.853 [0.819, 0.889] | -2.2 |
| claude-kexec / claude-lean | secondary, 95% CI | 1.090 [1.047, 1.135] | 5/23 | 0.011 | 1.326 [1.271, 1.384] | +0.4 |
| claude-c2 / claude-lean | secondary, 95% CI | 1.016 [0.964, 1.071] | 9/23 | 0.40 | 1.027 [0.962, 1.097] | +0.7 |
| loop / claude-c2 | secondary, 95% CI | 0.859 [0.807, 0.914] | 20/24 | 0.0015 | 0.826 [0.774, 0.881] | -2.9 |

- **Tokens:** "tokens" means uncached-equivalent tokens: uncached input (including cache writes), plus 0.1 x cache reads, plus 5 x output. With one model on one key, this tracks list cost one to one.
- **Multiplicity:** both primary sign tests stay below 0.001 after the Holm adjustment over the two primaries.
- **Claude Code's own layers:** on this route, Kogen's executor arguments cost Claude Code about 9% wall and 33% tokens against the lean setup (`claude-kexec` / `claude-lean`).
- **Gap left:** kogen-loop is still 12% faster and 15% lighter than the lean setup. So most of the gap to Claude Code as Kogen runs it is not those layers alone.
- **One system prompt:** with kogen-loop's instructions, Claude Code (`claude-c2`) was no slower and no heavier than with the executor constraints. It passed 23 of the 23 shared blocks, against `claude-lean`'s 22.

Failed cells:
- csvq-go r1 failed in `loop`, `claude-kexec` and `claude-lean`, at 0 of 7 checks.
- csvq-go r2 failed in `loop` only, at 0 of 7.
- splitter-go r2 failed in `loop`, at 0 of 9.
- patchapply-go r2 failed in `claude-kexec`, at 0 of 9.
- task-tags-go r2 failed in `claude-kexec`, at 160 of 900.

## Limits

- **Missing block:** the run stopped at its own registered spend cap for the API key ($11.50; $11.0 spent) before the last block's `claude-kexec` and `claude-lean` cells. That leaves 94 of 96 cells and 23 paired blocks for the pairs involving those arms. The registered sizing needed 22.
- **Scope:**
  - One model at one effort.
  - One server.
  - Twelve small Go tasks.
  - Claude Code 2.1.312 only.
- **Pass counts:** too few failures to compare pass rates. kogen-loop's two csvq-go failures are a task-level weakness worth a look, not a measured difference.
- **What was measured:** agent wall time, from agent start to exit, excluding setup and grading. Not end-to-end Kogen Builds.
- **Missing comparisons:** Stage A (the same Claude Code arms on subscription seats) and the Codex leg have not finished and are not part of this finding.
- **Raw files:** the raw per-cell logs stay in the private round directory. `data/cells.tsv` holds every number used here.

## Rerun

```
python3 reproduce/lab-cc-layers-b-20261105/report.py
```

`report.py` rebuilds every table above from `data/cells.tsv`; the expected output is in `expected-output.txt`.

To repeat the runs:
1. Apply `patches/claude-arms.diff` to `lanes/bare/lanes_bare.py`, and put `constraints.txt` and `loop-instructions.txt` next to it.
2. Run one cell per (task, repetition, arm) with these flags:
   - all arms: `lanes_bare.py cell --tasks <tasks dir> --task <id> --arm <arm> --rep <r> --model claude-sonnet-5-5 --effort medium`;
   - the `loop` arm adds `--loop-bin <kogen-loop e5054c8> --provider anthropic`;
   - the Claude Code arms add `--claude-bin <Claude Code 2.1.312> --kogen-hook-bin <Kogen release> --claude-route api`.
3. Run the four arms of a block one after the other, with the order rotated per block.
