> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [reproduce/kogen-aor-control-514dab4-st2/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/kogen-aor-control-514dab4-st2/README.md)
- [categories/harness-levers.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/harness-levers.md)
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)

# Four more runs per task on one machine: current Kogen shaped a valid intent in 3 of 14 counted runs and passed every hidden check in none

- **Finding id:** `kogen-aor-control-514dab4-st2`
- **Verdict:** Unresolved
- **Finished:** 2 November 2026
- **Comparison:** [Kogen shaping and delivery-build speed](/comparisons/kogen-shaping-and-builds/)
- **Reproduction pack:** [reproduce/kogen-aor-control-514dab4-st2/](https://github.com/KogenAI/kogen-bench/tree/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/kogen-aor-control-514dab4-st2) (data, analysis script and rerun steps)
- **Category page:** [categories/harness-levers.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/harness-levers.md)

## In plain terms

This adds more runs to our baseline of Kogen as it is today (514dab4, no patch) on six Rails tasks, given only the request text, with gpt-6.1-terra at medium effort for every step, all on one Mac. We planned three more runs per task; 14 of the 18 gave a result. Three wrote a valid intent, all three reached a build, none landed, and none passed every hidden check; seven stopped to ask questions nobody answered and four were refused at shaping. Eight runs were machine faults, not results: five were refused because a fixed 120-second acceptance check timed out while the machine was heavily loaded, and three lost their process. The four timeouts of the first batch were rerun once with mixed luck, and two late reruns had not finished at the cut-off. With the earlier runs on the same machine, the six tasks have 20 results: 6 valid intents, 2 landed builds, no full pass. With four runs per task and an 18 to 27 % chance that a rerun flips a result, a per-task rate means little yet; this is the comparison base for later shaping changes on the same machine.

Verdict key: **Yes**, The result supports the change or claim that was tested. **No**, The result does not support it, so nothing changes. **Unresolved**, The evidence is not strong enough either way. **Superseded**, A later test replaced this result.

## Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

Follow-up to [kogen-aor-control-514dab4](/findings/kogen-aor-control-514dab4/): three more Studio repetitions per task of the current-Kogen control (`514dab4`, no patch, request text only), so each task has more than one Studio run. **Of 22 follow-up runs, 14 gave a result (8 were instrument voids). Of the 14, 3 shaped a valid Intent (all of them reached a build, none landed) and 0 passed every hidden check. With the first Studio runs, the Studio block holds 20 results for six tasks: 6 valid Intents, 2 landed builds, 0 all-pass.**

## What was run

Same design and artifacts as the first pack: Kogen `514dab4be3c737c63bd64a418f1bfe707f18882f` (vendored loop pin `c6e23602`), Darwin arm64 artifact sha256 `f788471f5dc68ce8f0e1e72fdca46058b52dee5c498ea4627d1abb2cdbaf6f4f`, `gpt-6.1-terra` at medium effort for shape, build and review, request-only tasks `T1` to `T6` (same labels as the first pack), full pipeline (shape, approve, build, review), no answerer (a shaper that asks questions ends the run), 3600 s deadline, nothing raised. Lanes `7503c273` ran the cells; the Studio runner account with the AoR support environment, the same route as the first pack's Studio runs. Planned: 3 more runs of each task (18 cells).

Voids, all instrument, none model results: the acceptance gate has a fixed 120 s limit and the Studio load was 16 to 80 during these runs (other jobs shared it), so 5 cells were refused at approval with "acceptance timed out" after the Intent was written (3 first runs of `T1`, 1 of `T4`, and 1 rerun of `T1`), and 3 cells had their process disappear without writing a result (host fault). The four first-run gate timeouts were rerun once on the same route; the rerun of `T1` produced one usable run, one more gate timeout and one host fault, and the rerun of `T4` produced one usable run. The two host faults in the first runs (`T4` run 3, `T5` run 2) were not rerun in time: the rerun rule needs the Studio load below 10, it was above 19 at every check until about 20:05 UTC, and the reruns queued then (one `T4`, one `T5`) had not finished at the 20:30 UTC cut-off, so they are not counted. No void is counted in any rate; the table shows them.

## Result (counted runs only)

| Task | Follow-up runs counted / planned | Valid Intent | Reached build | Landed | Hidden all-pass | Studio block incl. first run: counted, valid Intent, landed |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| T1 | 1 / 3 | 1 | 1 | 0 | 0 | 2, 2, 0 |
| T2 | 3 / 3 | 0 | 0 | 0 | 0 | 4, 1, 1 |
| T3 | 3 / 3 | 0 | 0 | 0 | 0 | 4, 0, 0 |
| T4 | 2 / 3 | 0 | 0 | 0 | 0 | 3, 1, 1 |
| T5 | 2 / 3 | 2 | 2 | 0 | 0 | 3, 2, 0 |
| T6 | 3 / 3 | 0 | 0 | 0 | 0 | 4, 0, 0 |
| All | 14 / 18 | 3 | 3 | 0 | 0 | 20, 6, 2 |

- **Outcomes of the 14 counted follow-up runs:** 7 stopped at questions pending, 4 were refused at shaping (2 `T2` runs: the shaper answered without calling a tool three times; 2 `T3` runs: its output was not a valid Intent after three attempts), 1 build was blocked, 2 builds ran into the 3600 s deadline without landing. Nothing landed in the follow-up; the two landed Studio builds are from the first runs (`T2`, `T4`).
- **First mistake per run** is in `data/aor_control_st2_cells.tsv` (stage and exact refusal class, no hidden test content). Shape-stage stops: 11 of 14.
- **Hidden counts:** every run that did not land has the untouched start tree (the same tree id for all runs of a task), so its hidden count is the start count; the graded ones agree (`T5`: 2 builds, 4 of 11, requested 1 of 8). The kept build of the `T1` rerun was graded from a saved copy of its tree and is also at the start count. The first Studio runs of the first pack are included unchanged.
- **Per-task rates:** four runs per task cannot separate a task's rate from the 18 to 27 % chance that a rerun flips a pair; the Studio block is the comparison base for later shaping changes on the same route, not a rate.
- **Cost (tokens, dollars not priced for this model):** the 18 follow-up rows that reported usage used 3,417,728 input and 37,618 output tokens; 4 rows (3 host faults and one `T5` deadline run) report none. The heaviest counted follow-up run was the `T1` rerun that hit the deadline (428k input).

## Limits

The cut-off was reached with 4 of 18 planned follow-up runs missing, and the four missing runs belong to `T1` (2), `T4` and `T5`, so the tasks that reach a build carry the least data. The Intent-quality grade (acceptance that separates correct from incorrect implementations) was not obtained: the private task copies lack the reference directory the Intent grade needs. The first-pack Linux blocks (eu2, laptop) are not part of this table; Studio and Linux differ in route.

## Rerun

```sh
python3 -I reproduce/kogen-aor-control-514dab4-st2/report.py reproduce/kogen-aor-control-514dab4-st2/data/aor_control_st2_cells.tsv | diff - reproduce/kogen-aor-control-514dab4-st2/expected-output.txt
```

`report.py` uses only the standard library and rebuilds every count above from `data/aor_control_st2_cells.tsv` (one row per cell, the six first Studio runs included, voids listed). To run the cells again you need Kogen `514dab4` with the artifact sha256 above, lanes `7503c273`, the six AoR tasks and a gpt-6.1-terra seat.
