Published result

The second shaper text changes leave all six cells at the 30-minute deadline, with the first draft using most of it

A plain-language summary comes first, followed by the full text of the result's reproduction pack.

Result summary and pack text · reproduce/lab-aor-shaping-prompt-screen2-20261106/README.md

In plain terms

Stage 1 reached readiness on 1 of 5 refused tasks, and its refusals were classified model-free. A second set of general text changes (P2, then P2b) was screened on a model (Claude Code, Sonnet 5.5, high effort, shape only, request only, no answerer). Only 6 of 24 planned cells ran, and none reached readiness: all six ended at the 1800 s shape deadline, which is Kogen's own shape budget. The first shaper round alone took 1209 to 1800 s against 410 to 1019 s for stage 1's candidate cells, so the cells ran out of time before they ran out of attempts. The text changes did move the refusals down the gate chain: the decision check refused once in 6 cells (stage 1: 8 refusals in 5 cells), two cells reached the last gate before readiness, and one was past four gates with no refusal when time ran out. P2 told the shaper to list every observer file under Acceptance, which the step-ownership gate refuses, so it sent the shaper into a new refusal class; P2b corrected it and none of its three cells met that class. By the registered rule the verdict is unresolved (6 of the 20 cells needed), not no: three failures leave the 7 of 10 requirement possible. Held-out tasks and the release control were not run. Stage 3 should use P2b and report first-round time next to readiness.

Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.

Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

Finding. Stage 1 of this climb (`lab-aor-shaping-prompt-screen-20261106`, verdict no) reached readiness on 1 of 5 refused tasks; its refusals were classified model-free (10 of 14 shaper errors that a gate names, 4 where the prompt asks for something a gate cannot pass) and a second set of general text changes was written against them: P2, then P2b. Only 6 of the 24 planned cells ran, and none reached readiness: all six ended at the 1800 s shape deadline, still shaping. The first shaper round alone took 1209 to 1800 s (67 to 100 percent of the deadline) against 410 to 1019 s for stage 1's candidate cells on the same tasks, so the cells ran out of time before they ran out of attempts. The cells nevertheless show the text changes working at the gates they target: the decision check refused once in 6 cells (stage 1: 8 refusals in 5 cells), two cells reached the last gate before readiness (the wrong-implementation check) and a third was past four gates with no refusal when the deadline hit, just before it. P2 contained a contradiction between two gates that sent the shaper into a new refusal class (observer files must be listed under Acceptance, but Acceptance may only hold step artifacts); P2b corrected it, and none of the three P2b cells met that class. Verdict by the registered rule: unresolved (6 valid cells of 20 needed; check 1 is not out of reach after 3 failures; check 2 not evaluated; check 3 not violated for the rule's arm B2b, violated by B2 as built).

Question

Does Kogen release v0.1.0-dev.36075527 with kogen-shaping changed by stage 1's patches (B1, B2, B3, P) plus a second set of general text changes get through shaping to readiness on the five request-only Rails tasks the release was refused on, without doing worse on five other qualified tasks? Shape only, request only, no answerer, Claude Code with claude-sonnet-5-5 at high effort, 1800 s per cell.

Arms

All arms: Kogen `3607552` for darwin-arm64, Go 1.27.2, claude-code route, Claude Code 2.1.312, the same Mac for every cell, deadline 1800 s per cell (this equals Kogen's own 30-minute shape response budget, recorded as the response_budget event of every cell).

  • A (release, as installed): registered as the live control for the held-out tasks and four refused tasks. Not run (time).
  • B2 (b2-shaping-p2): the same commit built with a replace of kogen-shaping by a scratch copy of `c3f9c5e` plus stage 1's B1, B2, B3 and P (5 files, 103 insertions, 3 deletions against the pin) plus P2 (4 files, 24 insertions, 2 deletions; patch sha256 35316b5ee600c75bd5eb51e687534c1ea2d96cd7ca894d21a655d6ada54d4c85).
  • B2b (b2b-shaping-p2b): the same with P2b instead of P2 (same 4 files, 24 insertions, 2 deletions; patch sha256 f75433cedb30cb5479ad2f057d3a6394adace163a47630387122fab35e414455). P2b differs from P2 only in the observer wording (below). B2b is the arm of the registered rule after the recorded addendum.
  • What P2 and P2b change (text only, no task wording, no gate logic). The shaper prompt: the decision check reads the whole intent.md (text before the first heading, prose, examples, Tests lines, exclusions), so read the Intent once as that check would before each submit and mark every statement the request does not state; the source tag is exactly (source: request) or (source: answer <n>); submit every file the Intent names in one call; Tests lines have the rule-colon form; write controls.json (a correct implementation and wrong ones as file overlays, with the form and limits stated) with the first package. The feasibility guidance: P2 says every observer path must also be listed under Acceptance:; P2b says every observer path must be a step artifact of the Intent (features/steps/<slug>__<suffix>.js or a file in a features/steps/<slug>__<suffix>/ folder) listed under Acceptance:, and that an Intent-folder file cannot be an observer (it holds fixtures and inputs). Gate text only: the readiness line "invalid evidence checks or observers" gets the rules appended on the same line, and the decision-check refusal head says the listed statements are those found in the whole intent.md. No gate accepts or refuses anything it did not.
  • Binaries: B2 bin/kogen sha256 ee44d3e422898da32761ed98fa1ec81bcc851638a40adb05f1d4554ba13570b9; B2b bin/kogen sha256 f141c2037ea913cefe07de6579c015118f4d890319ac070ad13bd3474291b50e. A plain rebuild of the commit is byte-identical to the installed release (same toolchain, go build -trimpath -buildvcs=false), and each arm's module list differs from it only on the kogen-shaping replace line.

Method

Shape only: the cell runs the shaper until readiness or refusal after at most 3 submissions, or until the deadline, and stops (no approval, no build, no review). Two cells at a time, two subscription accounts, admission before every cell. Per cell: outcome, submissions, refusals by gate and problem-line count with a model-free cause class (a: the shaper erred and the gate names the form; b: the prompt or guidance asks for something a gate refuses or cannot pass), first shaper round, gate-check time, tool calls, wall, Claude Code's reported list-equivalent cost. scripts/timeline.py rebuilds the time split of a cell from its shaping-events.ndjson: each executor round has started_ms, wall_ms and model_ms, and the shape_time event splits the cell's elapsed time into shaper time and gate-check time (the two sum to the elapsed time within a second in all six cells).

Registered rule (stage 2): YES if B reaches readiness on at least 7 of the 10 refused-set cells, B is not worse than A on the 5 held-out tasks, and no refusal comes from a path a change touched other than the existing gate refusing as it did; fewer than 20 valid cells is too few. After the first cell the rule's arm became B2b (recorded addendum before any B2b cell), with a NO available only if check 1 is out of reach (4 or more B2b refused-set cells that ran fail) or check 3 is violated.

Results

task arm rep outcome submissions refusals (attempt: gate lines, class) first shaper round s (share of 1800 s) gate checks s tool calls wall s cost USD
task 1 B2 1 deadline 3 1: step_ownership 1 (b), 2: decision_check 4 (a) 1694 (94%) 51 72 1801 3.98
task 1 B2 2 deadline 0 none 1800 (100%) 1 58 1801 not recorded
task 2 B2 1 deadline 2 1: contract_review 4 (a), 2: contract_review 1 (a) 1209 (67%) 382 97 1801 4.33
task 1 B2b 1 deadline 1 1: literal_check 3 (a) 1746 (97%) 53 74 1801 4.40
task 1 B2b 2 deadline 1 none 1541 (86%) 259 63 1801 4.26
task 2 B2b 1 deadline 1 1: contract_review 7 (a) 1594 (89%) 166 68 1801 4.01

Cost of the five cells with a record: 20.98 USD list-equivalent (cap 60; the sixth cell's usage was lost with its interrupted round). Valid cells: 6 (each ran the shape stage and has a complete result). Not run: B2 cells 4 to 10 (stopped on purpose after the P2 defect), B2b cells 4 to 10, the 10 held-out cells and A's 4 live-control cells (launches stopped at the registered stop time; two cells at a time at about 30 minutes each).

What each cell was doing (from the events; data/cells.json has the same facts as data):

  • Task 1, B2, rep 1. First shaper round 1694 s (69 tool calls). Submission 1 refused at step_ownership (the driver file in the Intent folder that the shaper listed under Acceptance, following P2, is not a step artifact). Resume round 27 s. Submission 2 refused at the decision check (one head line and three statements without an assumption marker, class a), after a 47 s checker session. Resume round 27 s, submission 3 at about 1798 s; its first check was cut at the deadline.
  • Task 1, B2, rep 2. One shaper round of 1800 s, 58 tool calls, no submission. Tool calls 35 to 57 restart, inspect (ps, lsof, sample) and finally kill a probe process that did not finish, so the last 23 calls made no progress on the package.
  • Task 2, B2, rep 1. First round 1209 s (58 tool calls). Attempt 1: contract review confirmed 4 findings (the driver reads data that includes the export's own artifact, so three expected values cannot hold). Resume round 66 s. Attempt 2: decision check, static checks and contract review passed (0 findings); the wrong-implementation check refused: the controls file does not exist. Third shaper round started at 1657 s and was cut at the deadline 143 s in; its last two tool calls name the controls file. Gate checks took 382 s over two chains (about 190 s each).
  • Task 1, B2b, rep 1. First round 1746 s (72 tool calls), one submission with a 24 KB controls file. Refused at the literal check (3 request literals not found in the package, class a); the decision check had passed. Second round started at 1800 s.
  • Task 1, B2b, rep 2. First round 1541 s (55 tool calls), one submission without a controls file. Decision check, literal check, static checks and contract review (45 s, ended at 1776 s) all ran with no refusal; the deadline cut the cell before the property check, the wrong-implementation check and any result. This is the cell closest to a pass.
  • Task 2, B2b, rep 1. First round 1594 s (56 tool calls), one submission with a 94 KB controls file. Decision check, literal check, static checks, contract review (0 findings) and property check all passed; the wrong-implementation check refused 7 lines (class a): in each, a wrong implementation names a step that is not a step of the scenario it names. Second shaper round started at 1761 s and was cut 39 s in.

Diagnosis of the timeouts

  1. The time went into the first shaper round, not into repeated attempts. First round: 1209, 1541, 1594, 1694, 1746 and 1800 s (median 1644 s); stage 1's candidate cells on the same tasks: task 1 1019 s, task 2 596 s (all five stage 1 candidate cells: 410 to 1019 s, median 630 s; the release on task 1: 1309 s). First-round tool calls: 55 to 72 in stage 2 against 25 to 51 in stage 1's candidate cells. After the first draft, a resumed shaper round took 27 to 66 s and a gate-check chain that runs to its end 165 to 233 s (a decision-check refusal stops the chain after 47 s), so an attempt costs about 3.5 to 5 minutes: three attempts after a first draft of 1209 s (the shortest here) need about 1900 s, more than the 1800 s deadline even if no later round took longer (estimate from the task 2, B2 cell: 1209 + 190 + 66 + 190 + about 60 + about 190).
  2. The shaper was making progress, not looping, except in one cell. No gate refused the same thing twice; the refusal moved down the gate chain (step ownership, decision check, contract review, wrong-implementation check, literal check). Contract review ran in 3 of 6 cells. The one stalled cell is task 1, B2, rep 2: 23 of its 58 tool calls fight a probe process that never finished.
  3. What the first round contained. In two of the three cells read command by command (task 2, B2; task 1, B2b, rep 2) the shaper copied the workspace to a scratch tree, wrote an implementation of the requested feature in it (a migration, an initializer, a route edit) and ran the Rails driver or the acceptance run against it many times before it submitted (task 2, B2: tool calls 15 to 44, about 30 driver or test runs in the cell by command-text match; stage 1's same-task cell ran 3 probe tests of the existing code and submitted at call 24). In the third (task 1, B2, rep 2) it ran a copy of the workspace as a probe for its driver. Writing the controls file with the first package adds a correct implementation as file overlays and wrong ones (24 to 94 KB): tool calls that name the controls file appear in 4 of 6 stage 2 cells and in 0 of 5 stage 1 candidate cells; two of the three B2b cells wrote it in round 1 (and the one that had it at 94 KB reached the wrong-implementation check on its first submission). The P2 sentence "which you write with the first package" (kept in P2b) is the candidate cause; the screen has no arm without it, so it is not shown.
  4. The deadline is the product's own. The 1800 s is Kogen's shape response budget (event response_budget, "30 minutes"), not only a harness limit, so a request whose first draft needs 25 to 30 minutes stops the same way in use. The release itself used 1453 s on task 1 in stage 1 (first round 1309 s).
  5. Host load was not controlled. Cells ran two at a time on a shared Mac; the load average was 15.7 just before the first cells launched (a test suite of this study) and 4.4 four minutes later, and later load was not recorded. Tool time (wall minus model time of the first round, as the harness records it) was 23 to 35 percent in stage 2 and 2 to 6 percent for stage 1's task 2 to 5 candidate cells, 20 to 31 percent for task 1; whether load inflated it is not determined here.

What P2 and P2b showed

  • P2 sent the shaper into a new refusal class. The first B2 cell was refused on its first submission at the step-ownership gate: every non-feature Acceptance entry must be a step artifact of the Intent (features/steps/<slug>__<suffix>.js or inside such a folder), but P2's sentence told the shaper to list every observer path under Acceptance, and stage 1's sentence (P, B3) told it to put observer files in the Intent folder. Read together, the readiness rule (observers are Acceptance paths) and the ownership rule (Acceptance paths are step artifacts) mean an observer must be a step artifact; the Intent folder can hold inputs and fixtures only. This is the third appearance of the same contradiction (stage 1's readiness refusals on the same line were its first two). Registered check 3 fails for B2 as built.
  • P2b closed it for the cells that ran. None of the three B2b cells was refused at step ownership or at the readiness rule line; their only refusals were at the literal check and the wrong-implementation check, both later and unchanged gates (reported by cause, not counted as introduced).
  • The decision check stopped being the wall. Stage 1's candidate cells: 8 decision-check refusals among 14 refusals in 5 cells (3 of 5 cells refused there at their first submission). Stage 2: 1 decision-check refusal among 6 refusals in 6 cells (B2, task 1, attempt 2); the decision check passed on the first submission in every B2b cell and in B2's task 2 cell.
  • The last gate before readiness was met, and its form is still unnamed. Two cells reached the wrong-implementation check (B2 task 2 attempt 2, B2b task 2 attempt 1); B2b task 1 rep 2 was cut just before it. The two refusals there: the file missing (the shaper had built one in scratch but not submitted it), and 7 wrong implementations naming a step that is not a step of the named scenario (the gate message names the rule, so class a, but the form "the step line must be copied from the scenario" is not in the prompt).
  • Golden tests need updating before any delivery request. The kogen-shaping suite on B2 has 8 failing lines more than the release (release 881 pass lines and 3 fail lines; B2 873 and 11): 7 also fail on stage 1's tree (prompt byte-equality for shaper, refine and feasibility guidance, the shaper policy golden, a policy-text test, a force-wire test) and 1 is new with P2's gate text (TestDecisionOutcomeRules asserts the old refusal head). The suite was not run for B2b.

Model-free replay (before the cells)

Replay of stage 1's recorded gate inputs through the release (A), stage 1's tree and B2: 15 decision-check events and 5 evidence declarations (the final Intents of stage 1's cells). The decision check gives the same count on all three arms (41 problem lines against 34 in the recorded refusal lists; the gap is the replay's reconstruction of the Intent text, identical across arms), as expected for a text-only change. Evidence declarations: 2 of 5 are invalid as written (the line "invalid evidence checks or observers") on A and B2 alike; on B2 the line gains the rule text; with every observer path added under Acceptance all 5 are valid on both arms. The replay covers the readiness rule only: it did not run the declaration through the step-ownership gate, and that gap is exactly where P2's contradiction sat. Data: data/model-free-replay.json; harness in harness/.

What stage 3 should carry (stage 3 runs small requests; not repeated here)

  1. Use P2b, not P2: observers are step artifacts; the Intent folder holds fixtures and inputs.
  2. Measure the first shaper round, the gate-check chain and tool calls per cell (scripts/timeline.py), next to readiness: on large requests the first draft alone uses 67 to 100 percent of the deadline, so a readiness count there measures time, not text. On small requests the first draft is short, which is the condition under which P2b's effect on the gates can show.
  3. Report the controls-file sentence as its own question: it is the one change that adds a full implementation and 24 to 94 KB of overlays to the first round, and it moved the refusals to the last gate; the wrong-implementation form (step lines copied verbatim from the named scenario) is not stated in the prompt.
  4. Replay any change to observers, Acceptance or readiness through the whole gate chain (step ownership, readiness, controls), not through the one gate it edits.
  5. Not run here and still owed: the held-out five, the release as live control, and the fresh-home onboarding regression row (a trivial request refused on 2 of 3 attempts by the decision and evidence gates in a recorded session).

Deviations and limits

  1. 6 of 24 planned cells (3 B2, 3 B2b); minimum for a reading is 20. B2's 3 cells are a separate arm and are not counted toward B2b.
  2. All six cells ended at the deadline; "did not reach readiness" here means "did not finish in 30 minutes", and one cell (task 1, B2b, rep 2) was cut with no refusal outstanding.
  3. One cell has no cost record (its only round was interrupted before usage was logged). Cost is Claude Code's list-equivalent, not a bill.
  4. Shaping is stochastic; earlier screens measured 18 to 27 percent of identical-arm pairs flipping. n is 1 to 2 per task and arm: counts, no significance claim.
  5. The first-round cause analysis (point 3 above) reads tool-call command heads in three cells and counts tool calls that name the controls file; it does not separate P2 or P2b from the model's own variation, and no arm lacks the controls sentence.
  6. The cell deadline and the product's shape budget are the same 1800 s, so the screen could not tell a slow shaper from a stuck one by the deadline alone; the event split above does.

Rerun

  1. Build the arms as above: Kogen 3607552, go mod edit -replace=github.com/KogenAI/kogen-shaping=<scratch copy of kogen-shaping c3f9c5e with B1, B2, B3, P and P2 (arm B2) or P2b (arm B2b)>, go build -trimpath -buildvcs=false in cli/, Go 1.27.2; check that a build of the plain commit has the release's sha256 (14cc356a4955a15d277d4a26b5eb536590e1113c5ccea219750824473b2028ac).
  2. Stage a shape-only run (phase shape, claude-code route, claude-sonnet-5-5 high, request-only tasks, deadline 1800 s), two cells at a time.
  3. Run python3 -I scripts/timeline.py <run dir> on the finished cell directories (any directory that holds shaping-events.ndjson and result.json); it prints rounds, gate-check sessions, refusals and the shaper/gate-check split per cell.
  4. python3 -I report.py . prints the tables of this README from data/cells.json and data/stage1-comparators.json.
  5. The model-free replay: harness/build_cases.py builds the cases from stage 1's recorded events, and harness/aor2_replay_test.go with its three small files runs them inside kogen-shaping's shape package (AOR2_CASES, AOR2_OUT, AOR2_ARM); its counts are in data/model-free-replay.json.

Files: report.py, data/cells.json, data/stage1-comparators.json, data/model-free-replay.json, scripts/timeline.py, harness/. Tasks are labelled task 1 to 5 as in stage 1; private task text stays out.