> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [reproduce/lab-shaping-aor-diagnosis-20261107/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/lab-shaping-aor-diagnosis-20261107/README.md)
- [categories/harness-levers.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/harness-levers.md)
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)

# Naming the decision line and controls.json gets Rails shapes through shaping

- **Finding id:** `lab-shaping-aor-diagnosis-20261107`
- **Verdict:** Yes
- **Finished:** 7 November 2026
- **Comparison:** [Kogen shaping component](/comparisons/kogen-shaping/)
- **Reproduction pack:** [reproduce/lab-shaping-aor-diagnosis-20261107/](https://github.com/KogenAI/kogen-bench/tree/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/lab-shaping-aor-diagnosis-20261107) (data, analysis script and rerun steps)
- **Category page:** [categories/harness-levers.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/harness-levers.md)

## In plain terms

Stage 3's small-request fix reached readiness on 0 of 5 request-only Rails tasks the release is refused on. Reading those cells' own events and replaying their submissions through the gate functions showed three refusals the shaper had no named way to avoid: decisions recorded and checked in the same submission against an exact line no prompt shows, the shaper's own command runs refused as decision evidence although the guidance names them, and a wrong-implementation file (controls.json, with keyword-exact step lines) no prompt names. Four text changes and no gate change: the same five tasks reached readiness on 3 of 5 (before 0 of 5), and five held-out small requests stayed at 10 of 10. The two misses show the next limits: writing the correct implementation up front can outlast the 30-minute deadline, and eight acceptance runs can exceed the check's 300-second cap.

Verdict key: **Yes**, The result supports the change or claim that was tested. **No**, The result does not support it, so nothing changes. **Unresolved**, The evidence is not strong enough either way. **Superseded**, A later test replaced this result.

## Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

**Finding.** Stage 3's candidate C6 got small request-only shapes to readiness but its kogen-shaping changes reached readiness on 0 of 5 of the request-only Rails tasks the release is refused on. A model-free reading of those five cells' own events, and a replay of their recorded submissions through the gate functions, show that none of the three later refusals comes from C6's text and none from a wrong gate: (1) the carried-decisions gate checks, in the same submission that records the shaper's decisions, that the Intent already holds each one as an exact ledger line (`(decision: D1, round 1) ...`) that no prompt names; (2) the decision record refuses every command an external coding-agent session ran, while the tool description and the Intent's own evidence rule name `ran` commands as evidence; (3) the last check needs a `controls.json` (a correct implementation and wrong ones) that no prompt names, and its step lines must carry their Gherkin keyword, which neither the form nor the refusal says. C7 adds four text changes to C6 (the decision line and decision evidence named in the shaper prompt and in the external session's note, `controls.json` named as item 4 of the package, and a keyword hint in one refusal line) and changes no accept or refuse decision. On the same five Rails tasks C7 reached readiness on **3 of 5** (C6 0 of 5, the release 0 of 5); on stage 3's held-out small requests it stayed at **10 of 10** (7 at the first attempt). The registered rule (at least 2 of 5, at least 9 of 10, no new refusal class) holds. **Verdict: yes.** The two Rails cells that did not reach readiness show the next walls: writing the correct implementation up front made one first shaper round outlast the 30-minute deadline, and on another the check's eight acceptance runs did not fit its 300-second cap.
## Question

Stage 3 of this climb got small, ordinary request-only shapes to readiness (candidate C6: 20 of 20 small cells) but its kogen-shaping changes reached readiness on 0 of 5 of the request-only Rails tasks the release is refused on: the shapes stopped at three later gates (the decision record, carried decisions, and the wrong-implementation file `controls.json`), two at the 30-minute deadline. Why do these gates refuse, does C6's own prompt text cause any of it, and what is the smallest general change that gets the Rails shapes through without weakening a gate?

## Arms

- **Pin:** kogen-shaping [`c3f9c5e`](https://github.com/KogenAI/kogen-shaping/tree/c3f9c5e099e73cdbdb73ecb6044712379a2f79f3), the pin of release v0.1.0-dev.36075527 (Kogen [`3607552`](https://github.com/KogenAI/kogen/commit/360755276001531e5f9182d74fb72717f89ccfb4)), the release the Rails refusals were measured on.
- **C6 (stage 3's candidate):** the pin plus stage 3's changes (B1, B2, B3, the shaper-prompt part of P, Q3, Q5, Q6 and the decision-check sentence R1; described in stage 3's finding, `lab-shaping-small-requests-20261107`). Its Rails cells (one rep, 1800 s) are the stored comparator, together with the release's stored Rails cells of the same five tasks.
- **C7 (this stage's candidate):** C6 plus four text changes and no change to what any gate accepts or refuses (texts verbatim in `data/gate-texts.json`):
  - **D** (shaper prompt): when a submission carries `decisions`, Kogen records each as `D<n>` of the current round and the same submission's Intent must already keep it in `## Decisions` as exactly its line `- <decision> (decision: D<n>, round <r>)` followed by its constraints, evidence, assumption and revision parts and `(source: request)`; decision evidence is files tracked at HEAD that this shaping read; a file the package adds is no evidence.
  - **E** (the note Kogen adds for an external coding-agent session): a command the session ran is no evidence of a submitted decision; cite the files read and put what the command showed in the uncertainty.
  - **K** (shaper prompt, a new item 4): names `controls.json`: when it is needed, that it goes with the first package, its form (a correct implementation and wrong ones, each naming a scenario and the step line that must reject it, keyword included), its limits, that Kogen runs it and the shaper does not, and that it is not Feasibility's observer controls.
  - **G** (gate text only): when a wrong implementation's step is the text of a step of its scenario without the Given/When/Then/And keyword, the refusal line adds "write the step line as acceptance.feature writes it, keyword included: "<keyword> <text>"". Same verdict.
  - C7 is built twice from the same change: on the pin (Kogen 3607552) for the Rails cells, and on kogen-shaping [`0484edf`](https://github.com/KogenAI/kogen-shaping/tree/0484edf817f6dc74f691f44d5234cf7f6ae0435e) (the pin of the adopted release, Kogen [`47236bbb`](https://github.com/KogenAI/kogen/commit/47236bbb93385bc960fd2ac353c9b000bd6ed571)) for the small held-out cells, as in stage 3. The changed files are byte-identical in the two builds.

All cells: Claude Code 2.1.312 with claude-sonnet-5-5 at high effort for shaping, one Mac, shape only, request only, no answers, the login of the subscription seat admitted for that cell.

## Method

1. **Model-free diagnosis.** From the five C6 Rails cells' own shaping events: per attempt the refusing gate, its problem lines, the shaper's round time and tool calls, and what the shaper changed next; each refusal classed (a) the shaper erred and the gate names the form, (b) a prompt or guidance text asks for what the gate refuses, (c) no named way to pass before the refusal. Compared with the release's five stored Rails cells and stage 3's 105 small cells.
2. **Model-free replay.** The recorded submissions of the decision-record, carried-decisions and controls refusals run through the gate functions (a test file in scratch copies of kogen-shaping; no model call) in three trees: the pin, C6 and C7; then with each text change's effect applied mechanically (the decisions with non-file evidence moved into the uncertainty, the Intent lines written from the D sentence's description alone, the controls steps written with their keyword), and with deliberately broken inputs to show the gates still refuse.
3. **Package checks.** kogen-shaping's suite (`go test -count=1 -v ./...`) on C7 against the pin, the prompt goldens regenerated with the package's own `go test -update`.
4. **Cells (registered rule).** C7 on the five Rails tasks (one rep, 1800 s, shape only) and on stage 3's five held-out small requests twice (900 s). C7 passes if it reaches readiness on at least 2 of the 5 Rails tasks (a cell at the deadline counts as not ready), on at least 9 of the 10 held-out small cells, and shows no new refusal class (no refusal from a path C7 changed other than that gate refusing as before, none caused by following a C7 instruction).
## Results

**Model-free diagnosis of C6's Rails cells** (`data/c6-aor-attempts.json`; a, b, c as in Method):

| task | C6 outcome | refusals (attempt: gate, problem lines, class) | wall s | cost USD |
|---|---|---|---|---|
| task 1 | refused | 1: decision record 4 (b); 2: decision record 1 (a); 3: carried decisions 4 (c) | 1399 | 3.64 |
| task 2 | refused | 1: decision record 4 (b); 2: carried decisions 3 (c); 3: wrong-implementation check, controls.json missing 1 (c) | 1115 | 3.18 |
| task 3 | deadline | 1: controls.json missing 1 (c); the next round wrote and ran a correct implementation until the deadline | 1801 | 2.47 |
| task 4 | deadline | 1: controls.json missing 1 (c); same | 1802 | 1.54 |
| task 5 | refused | 1: controls.json missing 1 (c); 2 and 3: controls.json step without its keyword 7 + 7 (c) | 1468 | 3.91 |

(Cost here is Claude Code's reported cost per cell; stage 3's table summed each round's running total and so overstated it.) Four of the five C6 cells passed every gate up to the last one. The release's stored cells of the same tasks stopped earlier (15 refusals: decision check 8, evidence 3, Intent lint 2, validation 2; none reached the contract review) and never submitted a decisions list; two C6 cells did, at their first attempt. Stage 3's 105 small cells met no decision-record or carried-decisions refusal and three controls.json refusals (each passed at the next attempt).

- **Carried decisions (c).** kogen-shaping runs the decision record before the package checks (`shape/shape.go:1499`): it records the submitted decisions as `D1..Dn` of the current round (`shape/ledger.go:601-625`). The package check of the same submission then runs the carried-decisions gate (`shape/shape.go:1683`, `shape/ledger.go:634-660`), which requires each recorded decision in `## Decisions` as exactly its ledger line (`intentLine`, `shape/ledger.go:209-231`). That line is shown to the shaper only in a later round's context or in this refusal, so the first package that records decisions is refused whatever the shaper writes, unless it guesses the line. No C6 text tells the shaper to drop a decision or mentions the decisions list.
- **Decision record (b).** The decisions tool description (`shape/policy.go:151`) and C6's prompt sentence P name `ran` commands as evidence; the Intent's evidence gate accepts an external session's `ran` command (`shape/evidencemark.go:148`), but the decision record refuses all of them (`shape/ledger.go:470-472`). The shapers cited their own probe runs (as prose or backticked commands) and were refused as "not a file tracked at HEAD". One line (a file the package adds) is (a).
- **controls.json (c).** The wrong-implementation check (`shape/controls.go:757-860`) runs after readiness and a clean contract review and refuses a package without `.kogen/intents/<slug>/controls.json` (`:811`). No prompt names the file (the Feasibility guidance's "observer controls" are a different mechanism), and the external session's note says not to prototype the implementation, which a correct implementation is. Its step match takes the feature line with its keyword (`:171-184`); the form says "<step line>" and the refusal names no fix.

**Replay** (`data/replay.json`; go test of a replay file in scratch copies of kogen-shaping, no model call). As recorded, the submissions give exactly the recorded refusals in the pin and in C6 (16 of 16 decision-record and carried-decisions lines, 7 of 7 controls lines): the refusals are pin behaviour, the same with and without C6. Applying the C7 texts mechanically: the two tasks' decisions with their own runs moved into the uncertainty pass the decision record, and the Intent with each decision's line written from the C7 sentence alone (not from the gate's own renderer) passes the carried-decisions gate in the same submission; the same Intent without the lines (4 and 3 lines) or with one constraint dropped (1 line) is still refused. Task 5's recorded `controls.json` with each step written with its keyword parses clean; a step of another scenario stays refused. C7 gives the same verdicts as the pin everywhere; only the keyword-less step line gains its hint.

**C7 on the Rails tasks** (`data/c7-aor-cells.json`; one rep, 1800 s):

| task | C7 outcome | refusals (attempt: gate, problem lines) | first package | first shaper round s (C6) | wall s | cost USD |
|---|---|---|---|---|---|---|
| task 1 | deadline | none: the first round did not end | not submitted | more than 1800 (1108) | 1801 | not reported |
| task 2 | **ready** | none | with controls.json; wrong-implementation check clean, 8 runs in 73 s | 1402 (785) | 1642 | 4.29 |
| task 3 | **ready** | 1: decision check 2 | with controls.json; check clean, 8 runs in 195 s | 804 (763) | 1184 | 2.89 |
| task 4 | **ready** | 1: decision check 2 | with controls.json; the contract review named no checked property, so no run | 497 (425) | 693 | 2.03 |
| task 5 | deadline | 1: decision check 2; 2: wrong-implementation check, time cap (6 of 8 runs in 300 s) | with controls.json; steps with their keyword parsed clean | 940 (474) | 1801 | 2.92 |

3 of 5 reached readiness. No C7 cell submitted a decisions list, so the decision-record and carried-decisions gates were not met; no cell met the keyword refusal. Task 1's cost is not reported because its only shaper round never ended.

**C7 on the held-out small requests** (`data/c7-heldout-cells.json`; stage 3's five held-out requests, two reps, 900 s): 10 of 10 reached readiness, 7 at the first attempt, mean 100 s, 3.53 USD for the ten. Refusals: decision check in two cells (h2, both reps; kinds the release shows), Intent lint in one cell (a malformed optional `## Stages` section). Stage 3's stored cells: C6 10 of 10 (8 at the first attempt), the release 2 of 10 (0).

**Package checks.** kogen-shaping's suite on C7 equals the pin (895 pass lines, 3 fail lines from two tests that need a model provider and Quint, 2 skip; same failing tests) with the three prompt goldens regenerated.
## Verdict: yes

Registered before any measurement (first revision of the Lab record): C7 passes if it reaches readiness on at least 2 of the 5 Rails tasks (one rep; a cell at the 1800 s deadline counts as not ready), on at least 9 of 10 held-out small cells, and shows no new refusal class (no refusal from a path C7 changed other than that gate refusing as before, none caused by following a C7 instruction).

- **Rails tasks: holds.** 3 of 5 (needs 2); C6 and the release 0 of 5.
- **Held-out small requests: holds.** 10 of 10 (needs 9), 7 at the first attempt.
- **No new refusal class: holds.** C7's refusals are decision-check lines (a kind the release shows), one Intent-lint line about a Stages section no C7 text mentions, and one wrong-implementation time cap: a pin gate on a path C7 does not change that refuses the same controls the same way in the pin, met only because C7's package now reaches the runs. The one problem line C7 changed was never met.

What each change rests on: K (`controls.json` named) on the cells, every submitted first package carried the file and two passed the check at once where C6 was refused for its absence; D, E and G on the model-free replay only, because no C7 cell used a decisions list or met the keyword refusal.

## Deviations and limits (plain)

1. **One rep per task.** Shaping is stochastic (earlier screens measured 18 to 27 percent of paired cells flipping with identical arms); counts, no significance claim. On one task the contract review named no checked property in the C7 cell, so that task's readiness does not test K.
2. **The cost of naming controls.json.** The first shaper round grew on every Rails task (from C6 to C7: 425 to 497 s, 763 to 804 s, 785 to 1402 s, 474 to 940 s, and 1108 s to more than 1800 s), because the shaper writes a correct implementation, and runs it, before the first submission, although the text says Kogen runs it. One task ran out of time inside that round.
3. **The 300 s cap.** The wrong-implementation check needs at least one wrong implementation per checked property and runs the whole acceptance once per implementation; with seven properties on a slow Rails suite, eight runs did not fit in 300 s. Not changed here.
4. **No ablation run.** The replay names no C6 piece as the cause of a refusal, so the allowed model run for a piece's ablation was not spent; whether C6's text makes shapers use the decisions list more (2 of 5 C6 Rails cells, 0 of 5 release Rails cells, none of the 14 C7 cells that submitted) is not separated.
5. **Small repositories rebuilt.** Stage 3's scratch repositories were deleted with its run trees; they were rebuilt from its published description and creation commands, with its check strings (the Node check `node --test test/` fails under Node 24, as it did there; shaping does not run it).
6. **Two builds of one change.** The Rails cells ran C7 on release 3607552 (where the refusals were measured), the small cells on the adopted release 47236bbb, as stage 3 did; the changed files are byte-identical. Rebuilding C6 here gave stage 3's binary except for build-path identifiers.
7. **Cost** is the coding agent's reported list-equivalent figure on subscription seats (15.66 USD reported in all, plus one unreported first round).

## Rerun

1. Get kogen-shaping [`c3f9c5e`](https://github.com/KogenAI/kogen-shaping/tree/c3f9c5e099e73cdbdb73ecb6044712379a2f79f3) and [`0484edf`](https://github.com/KogenAI/kogen-shaping/tree/0484edf817f6dc74f691f44d5234cf7f6ae0435e). Make C6 as stage 3's finding (`lab-shaping-small-requests-20261107`) describes, then C7 by adding the four texts in `data/gate-texts.json` (`c7_text`): D after the paragraph of `shape/prompts/shaper.md` that ends "nothing else goes inside the tag's parentheses.", K after the line "List every step file the feature relies on in `Acceptance:`, right after the feature." (a blank line, then the item), E as one more line at the end of the external session note in `shape/external.go`, and G appended to the step problem line in `shape/controls.go` when the step equals the text of a step of its scenario without the keyword. Regenerate the prompt goldens with `go test ./shape/ -update`; `go test -count=1 -v ./...` should give the pin's counts.
2. Model-free replay: from a cell's `.kogen/local/shaping/<slug>/events.ndjson`, take each shaper submission's `decisions` and the submitted Intent, and call the package's own `decisionRecordGate` then `carriedDecisionsGate` (external session, empty ledger, round 1, the repository's tracked files), and `parseControls` on the controls file and feature; compare with the `submission_refused` events. `data/replay.json` lists every case and verdict.
3. Cells: build Kogen [`3607552`](https://github.com/KogenAI/kogen/commit/360755276001531e5f9182d74fb72717f89ccfb4) and [`47236bbb`](https://github.com/KogenAI/kogen/commit/47236bbb93385bc960fd2ac353c9b000bd6ed571) in `cli/` with `go mod edit -replace=github.com/KogenAI/kogen-shaping=<the C7 tree>` and `go build -trimpath -buildvcs=false` (Go 1.27.2), swap the release's `bin/kogen`, and run `kogen intent shape <slug> <request file>` once per task or request, shape only, request only, with claude-sonnet-5-5 at high effort through Claude Code 2.1.312; read `submission_refused` events and the readiness receipt. The small requests are in `data/heldout-requests.json`.
4. `python3 -I report.py .` prints every table above from `data/`.
