> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [reproduce/kogen-aor-claude-ceiling-514dab4-r2/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/kogen-aor-claude-ceiling-514dab4-r2/README.md)
- [categories/harness-levers.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/harness-levers.md)
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)

# A second block of Kogen on Claude Opus 5.5 passed every hidden check in 4 of 12 runs on six Rails tasks, and 11 of 12 wrote a valid intent

- **Finding id:** `kogen-aor-claude-ceiling-514dab4-r2`
- **Verdict:** Unresolved
- **Finished:** 2 November 2026
- **Comparison:** [Kogen shaping and delivery-build speed](/comparisons/kogen-shaping-and-builds/)
- **Reproduction pack:** [reproduce/kogen-aor-claude-ceiling-514dab4-r2/](https://github.com/KogenAI/kogen-bench/tree/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/kogen-aor-claude-ceiling-514dab4-r2) (data, analysis script and rerun steps)
- **Category page:** [categories/harness-levers.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/harness-levers.md)

## In plain terms

We repeated the Claude Opus 5.5 check on Kogen as it is today (514dab4, no patch): the same six Rails tasks, given only the request text, one run per task on each of two servers, with Claude Opus 5.5 at high effort through Claude Code for every step. Eleven of the 12 runs wrote a valid intent and 10 landed a build; four passed every hidden check, both runs of two tasks. Of the other two, one was refused at shaping and one ran out of its 3600 second limit in the build. Review approved all 10 landed builds. Together with the first block of 12 this makes 24 runs: 15 valid intents, 14 landed builds and 5 full hidden passes, all on two of the six tasks. The first block lost 8 runs to shaping refusals caused by a Kogen test-run setup problem; this block lost 1, and we do not know why the rate fell. With two runs per task per block and an 18 to 27 % chance that a rerun flips a result, this reports successes only and claims no model is better.

Verdict key: **Yes**, The result supports the change or claim that was tested. **No**, The result does not support it, so nothing changes. **Unresolved**, The evidence is not strong enough either way. **Superseded**, A later test replaced this result.

## Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

A second block of the Claude Opus 5.5 ceiling check on current Kogen (`514dab4`, no patch), with the same six Rails AoR tasks and the same settings as `kogen-aor-claude-ceiling-514dab4`. It tightens that first block's 12 runs to 24. **Result: 4 of 12 runs in this block passed every hidden check (T1 and T5 on both servers). 11 of 12 shaped a valid Intent and 10 landed a build. Pooled with the first block: 5 of 24 passed every hidden check, 15 of 24 shaped a valid Intent and 14 landed.**

## What was run

Everything is as in the first block, so only the differences are listed.

- **Same:** Kogen `514dab4be3c737c63bd64a418f1bfe707f18882f`, artifact sha256 `e8be7cb6e56e65840069f56aa011355929f6e3db4063bb130b016688c4c8b3aa`, lanes `7503c273`, Claude Code 2.1.312 with claude-opus-5-5 at high effort for shape, build and review, request-only, no answerer, deadline 3600 s, no timeout raised, the six tasks labelled `T1` to `T6` as before (names and hidden suites withheld), the same two Linux hosts `H1` (3 parallel runs) and `H2` (2 parallel runs).
- **New:** one more run per task per host (12 runs) under new run ids, started about 45 minutes after the first block's last run. The first run on each host was checked to use the Claude subscription sign-in through Kogen's own managed Claude Code config, with no stored token.
- **Voids and stop rule:** no run was voided or rerun. The stop rule on subscription use (new runs stop at 97 % of the weekly limit) never fired.
- **Not in this table:** a separate third block on a third server (`kogen-aor-claude-ceiling-514dab4-eu`).

## Result of this block

| Task | Runs | Valid Intent | Reached build | Landed | Hidden all-pass | Requested checks all pass |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| T1 | 2 | 2 | 2 | 2 | 2 | 2 |
| T2 | 2 | 2 | 2 | 2 | 0 | 0 |
| T3 | 2 | 2 | 2 | 2 | 0 | 1 |
| T4 | 2 | 2 | 2 | 2 | 0 | 0 |
| T5 | 2 | 2 | 2 | 2 | 2 | 2 |
| T6 | 2 | 1 | 1 | 0 | 0 | 0 |
| All | 12 | 11 | 11 | 10 | 4 | 5 |

- **Outcomes:** 10 landed, 1 refused at shaping, 1 timed out in build.
- **Landed builds:** review approved all 10 on its first pass. Two passed every hidden check on each of T1 and T5. The other six missed requested checks (T2 twice, T3 once, T4 twice) or, for T3 on H2, passed every requested check and missed one regression check.
- **The two T6 runs:** on H2 shaping was refused after three submissions; the last refusal was Kogen's step self-check re-running an observer (`billing-driver`) whose good control exited 1 where the shaper had recorded exit 0. On H1 shaping took 3018 s and the build hit the 3600 s deadline with 1 of 8 requested checks passing.
- **First mistake per run:** listed in `data/aor_claude_cells.tsv`.
- **Cost:** 77,206,524 input tokens (73,662,178 cached) and 1,245,408 output over the 11 runs that recorded totals; the timed-out run recorded none. It ran on a subscription, so there are no API dollars. The subscription's weekly usage rose from 54 % to 59 % over the block. Wall time per run was 774 to 3601 s.

## Pooled with the first block (descriptive)

| Task | Runs | Valid Intent | Reached build | Landed | Hidden all-pass |
| --- | ---: | ---: | ---: | ---: | ---: |
| T1 | 4 | 3 | 3 | 3 | 2 |
| T2 | 4 | 3 | 3 | 3 | 0 |
| T3 | 4 | 3 | 3 | 3 | 0 |
| T4 | 4 | 2 | 2 | 2 | 0 |
| T5 | 4 | 3 | 3 | 3 | 3 |
| T6 | 4 | 1 | 1 | 0 | 0 |
| All | 24 | 15 | 15 | 14 | 5 |

Only T1 and T5 ever passed every hidden check, and T1 did so in 2 of 4 runs, T5 in 3 of 4.

## Shaping refusals

The first block saw 8 refusals in 12 runs. This block saw 1 in 12 on the same Kogen build, tasks, model and settings. The first block's refusals were traced separately to mise refusing the project's `.mise.toml` inside Kogen's temporary probe copy, so the shaper's Ruby observer controls exited 1 on replay (Kogen issue `bcf8cfc2`; see the first block's note and that issue). This block does not establish why the rate fell: this study changed no Kogen build or setting (it only restored a pruned artifact index file on H1, with the same content as on H2), and one block of 12 cannot separate a change in the hosts from the large run-to-run spread, which shaping outcomes already show. Refusal counts are therefore reported as they happened, not as a model property.

## Grading

- **Hidden tests:** run through the lanes grader on each host as part of each run, with the real tool paths first instead of the mise shims. Every grade's setup commands exited 0 and none timed out. No run was regraded.
- **Intent-quality grade:** needs a reference tree that the private task copy does not hold, so it was not obtained and is not reported, as in the first block.

## Limits

- Two runs per task per host block. Per-task rates remain very uncertain; a rerun flips a result 18 to 27 % of the time per pair.
- The two hosts differ in speed and parallel slots, not route.
- A ceiling check on one model: it tests no change to Kogen, and no cost or speed claim is made against other models.

## Rerun

```sh
python3 -I reproduce/kogen-aor-claude-ceiling-514dab4-r2/report.py reproduce/kogen-aor-claude-ceiling-514dab4-r2/data/aor_claude_cells.tsv reproduce/kogen-aor-claude-ceiling-514dab4-r2/data/dd593dd3_cells.tsv | diff - reproduce/kogen-aor-claude-ceiling-514dab4-r2/expected-output.txt
```

`report.py` uses only the standard library. It rebuilds every count above from `data/aor_claude_cells.tsv` (this block) and `data/dd593dd3_cells.tsv` (the first block's table, copied unchanged).

To run the cells again you need Kogen `514dab4` with the artifact sha256 above, lanes `7503c273`, the six AoR tasks, and Claude Code 2.1.312 signed in to a Claude subscription through Kogen's managed config folder with no stored Claude Code token.
