Published result
Message Batches review of landed commits
A plain-language summary comes first, followed by the full text of the result's reproduction pack.
Result summary and pack text · reproduce/api-batch-review-landed-commits-20261104/README.md
- Finding id:
api-batch-review-landed-commits-20261104 - Verdict: Unresolved
- Finished: 4 November 2026
- Comparison: kogen-loop speed, patches and connection routes
- Reproduction pack: reproduce/api-batch-review-landed-commits-20261104/ (data, analysis script and rerun steps)
- Category page: categories/models-and-effort.md
In plain terms
The Message Batches API reviewed 93 landed commits for half the synchronous price, but a first pass cut 37 reviews short and the comparison with the seat-run review is weak, so the result is unresolved.
Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.
Full text of the reproduction pack
The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).
Verdict: unresolved. The Message Batches API reviewed 93 landed Kogen commits in 184 s (first pass) on claude-sonnet-5-5. The first pass was flawed: default adaptive thinking used up the 4,000-token output limit on 37 of 93 requests (33 returned no text). Those 37 were rerun at 16,000 tokens (246 s); 1 of the 93 final reviews is still empty. The comparison with the seat-run review is weak: the seat reviews are free-form prose with few severity markers, and the agreement figures below come from a lexical clean/not-clean test.
What was measured
Each request carries the same review prompt as the seat-run review of the same commit and the same patch text (git show --stat --patch <sha>, cut at 200,000 bytes). Model claude-sonnet-5-5, API version 2023-06-01, default thinking, one batch per pass, no retries beyond the single 16,000-token rerun of the reviews that stopped at the output limit.
| Measure | Value |
|---|---|
| Requests / completed | 93 / 93 (1 final review empty) |
| Input / output tokens (both passes, thinking included) | 3,503,074 / 529,318 |
| Cost at Batches price (50% of list) | USD 6.15 (pass 1 USD 3.68, rerun USD 2.47) |
| Same tokens at synchronous list price | USD 12.30 |
| Clean/not-clean agreement with the seat review | 60/93 (64.5%), lexical test; batch said clean 32, seat text clean 13 |
| File-name overlap (Jaccard) where both flag something (n=54) | 0.32 |
| Wall | 184 s + 246 s rerun, including the flawed first pass |
Cost per pass stayed under the USD 5 cap per batch. Agreement is not a correctness measure: neither review was checked against ground truth.
Files
data/commits.tsv: the 93 commits (repository, Kogen Build slug, commit sha). 91 are in public KogenAI repositories; the other 2 are from a private repository and appear asprivate-1andprivate-2.data/seat_reviews.json: for each public commit, what the lexical test found in the seat-run review: clean, P1-P3 marker count, file names mentioned, length.data/rows.json: the per-commit comparison table for all 93 commits;data/summary.json: the figures above.data/first_pass_status.json,data/rerun_status.json: batch counts and wall times.
Rerun
Clone the public repositories named in data/commits.tsv from github.com/KogenAI into one folder, then:
python3 -I build_requests.py data/commits.tsv CLONES requests.json
ANTHROPIC_API_KEY=... python3 -I submit_batch.py requests.json results.jsonl status.json
python3 -I build_requests.py data/commits.tsv CLONES rerun_requests.json --rerun-truncated results.jsonl --max-tokens 16000
ANTHROPIC_API_KEY=... python3 -I submit_batch.py rerun_requests.json rerun_results.jsonl rerun_status.json
python3 -I report.py results.jsonl status.json rerun_results.jsonl rerun_status.json data/seat_reviews.json rows.json
submit_batch.py refuses a batch estimated over USD 4.50. The rerun covers the 91 public commits. The model's review text varies between runs, so the figures will move; the request text is deterministic. We checked that these scripts rebuild our run's request text exactly for the 91 public commits (both passes), and that report.py on our raw outputs gives data/rows.json and data/summary.json exactly. The raw outputs themselves are left out of this pack because they quote review text.