Published result
Three speed patches for kogen-loop: none faster or cheaper beyond noise
A plain-language summary comes first, followed by the full text of the result's reproduction pack.
Result summary and pack text · reproduce/api-patch-screen-2nd-host/README.md
- Finding id:
api-patch-screen-2nd-host - Verdict: No
- Finished: 30 October 2026
- Comparison: kogen-loop speed, patches and connection routes
- Reproduction pack: reproduce/api-patch-screen-2nd-host/ (data, analysis script and rerun steps)
- Category page: categories/head-to-head.md
In plain terms
A short screening test of three changes to kogen-loop against its faster baseline build, run on a second computer with both sides on the same API key. All 16 runs passed, and no patch was faster or cheaper by more than run-to-run noise, so none was adopted. Total spend was $1.76 against a $3.00 limit.
Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.
Full text of the reproduction pack
The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).
A short test of three loop patches against the speed2 loop baseline on a second host, using the clean API route (both sides on one OpenAI API key, no subscription). All 16 runs passed and no patch was faster or cheaper beyond noise. Verdict: no change (nothing promoted), 30 October 2026. Total spend $1.76 of a $3.00 cap.
What was measured
Four tasks, one run per tool per task, 16 counted runs, one run at a time on one Linux laptop, in a random order fixed in advance (seed 20261030; the task order and the tool order within each task were shuffled from that seed). Model gpt-6.1-terra, effort high, shipped loop instructions, the common Responses endpoint through a gateway with zero upstream retries and the default service tier, an offline Go standard-library cache restored before every run. A run counts only if the pass/fail grade exists, every response reports usage, and every response's returned model is gpt-6.1-terra. For each patch, the first run also had to show a witness receipt that the patch's code path was active (for example, that the finish tool was omitted from the default terminal tools for api-answer-end); all 12 witnesses verified.
Tools
| Tool | What it is | Model | Effort | Patch |
|---|---|---|---|---|
speed2 |
loop build b8a4046 plus the speed2 variant (baseline for this test) | gpt-6.1-terra | high | patches/speed2.diff |
api-replay |
baseline plus native Responses encrypted-reasoning replay | gpt-6.1-terra | high | patches/api-replay.diff |
api-batch |
baseline plus a batched-independent-I/O instruction | gpt-6.1-terra | high | patches/api-batch.diff |
api-answer-end |
baseline plus finish tool omitted from the default terminal tools | gpt-6.1-terra | high | patches/api-answer-end.diff |
Each .diff.sha256 file holds the checksum of the patch as run; the binaries were attested by hash in the protocol file before the first run. Harness: the lanes/ bare-agent release (lanes_bare.py sha256 9744637d…) with the round driver in `lanes/apiscreen-patches-high-20261030/`; task ids in tasks.txt are directories under tasks/.
Rerun
The driver that ran the four patched tools and its frozen protocol are in `lanes/apiscreen-patches-high-20261030/` (run_confirmation.py, confirmation_ext.py, protocol-frozen.json, README with the setup). Build the four tools from kogen-loop b8a4046 with patches/speed2.diff (the baseline) and, for each candidate, its own diff on top; then, with your own task kits (tasks.txt), an OpenAI API key and the variables the README names:
python3 -I lanes/apiscreen-patches-high-20261030/run_confirmation.py prepare
python3 -I lanes/apiscreen-patches-high-20261030/run_confirmation.py counted 0
prepare freezes the 16-cell order from seed 20261030, and counted N runs cell N (0 to 15, in order) and grades it. The task kits (start trees, hidden tests and scenarios) stay private until the Kogen hill-climb ends; no date is set. A rerun also needs Linux with Bubblewrap, Go 1.27.2 and the host lease file the driver checks. Checked 31 October 2026 with a stand-in round directory and no model call: the files compile and --help prints the modes; prepare and the cells were not run. The original round cost about $1.5 at list price; wall times depend on the host and the provider on the day, so fresh values will differ.
Reproduce the numbers
python3 -I reproduce/api-patch-screen-2nd-host/analyze.py | diff - reproduce/api-patch-screen-2nd-host/expected-output.txt
Verified 30 October 2026: this reproduces every published number exactly (arm medians, all twelve paired deltas, counted cost $1.52016145 and total $1.76347695).
Files
runs.csv: the 16 counted runs: task, tool, rep, outcome, passed, input tokens (uncached + cached), cached tokens, output tokens, requests, list-price cost in USD, agent wall in seconds, the returned model, whether all responses matched it, whether usage was complete, and whether the patch witness verified (n/afor the baseline). Reasoning tokens are not recorded separately in this source and are blank. Wall is agent-process time.excluded.csv: the charged attempts that are not in the counted rows (zero-call setup error, a witness-less firstapi-batchrun, and two runs lost in a host reboot) so the cost denominator is complete.patches/: the four loop variants and their checksums.analyze.py,expected-output.txt.
Expected output
| Tool | Passes | Median wall (s) | Median USD per pass |
|---|---|---|---|
| speed2 | 4/4 | 142.442 | $0.066458 |
| api-replay | 4/4 | 166.319 | $0.082623 |
| api-batch | 4/4 | 143.939 | $0.065811 |
| api-answer-end | 4/4 | 173.418 | $0.080588 |
Task-paired deltas are candidate minus speed2; negative means faster or cheaper:
| Task | api-replay wall / USD | api-batch wall / USD | api-answer-end wall / USD |
|---|---|---|---|
| cron-go | +29.317 s / +$0.021549 | +1.335 s / −$0.007428 | +2.679 s / −$0.000971 |
| vm-go | +18.437 s / +$0.010782 | +1.658 s / +$0.006135 | +59.272 s / +$0.029232 |
| artifact-go | −5.411 s / −$0.001504 | −15.056 s / −$0.003174 | −2.384 s / −$0.001071 |
| readtime-go | −79.898 s / −$0.085844 | −48.850 s / −$0.038080 | −38.259 s / −$0.016653 |
Cost
Counted runs $1.52016145; the charged-but-excluded attempts add $0.24331550, of which $0.17033925 is the two reboot-lost runs (not itemised in runs.csv; the figure is the sum of two spend-ledger rows), for a total of $1.76347695.
Limits
One run per tool per task: four task clusters and four passing runs per tool cannot show a winner,. The deltas are dominated by readtime-go (large tails) and by one vm-go run. The reboot-lost charges are reported from the spend ledger, not from per-run rows. An earlier 16-run screen on this host (loop against speed2, 4 tasks, two repetitions) is described in the finding's source note but its per-run wall times were not available when this pack was built (only its token and cost rows are in the spend ledger), so it is not part of this pack. No Codex arm was run.