Published result

Deciding instead of asking shortens shaping's question loop but does not produce Intents

A plain-language summary comes first, followed by the full text of the result's reproduction pack.

Result summary and pack text · reproduce/lab-shaping-converge-20261107/README.md

In plain terms

On a ChatGPT shaping lineup, eight ordinary feature requests ended without an Intent because every shaping session asked one new question, often about the option just chosen, and unattended Builds answer only three rounds. Every question carried a recommended option, which Kogen requires, so each was the shaper's to decide. Telling the shaper to decide such choices and record them as assumptions cut the questions from 25 to 7 and removed every follow-up; a question gate on top cut them to 4. No request reached a ready Intent in any arm: without questions the shaper ran out of submissions on other package checks. Five small held-out requests asked no question in any arm (ready 10 of 10 for the release and for B and 9 of 10 for C on the ChatGPT lineup; the Claude-lineup held-out cells could not run because the subscription seat was over its weekly pace). Verdict no: the change was meant to give ready Intents and zero questions. The data also shows the decision check never sees the user's answers and refuses answer-sourced decisions.

Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.

Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

Finding. On a ChatGPT shaping lineup (gpt-6.1-sol, low effort) eight ordinary feature requests ended without an Intent because every shaping session asked exactly one new question, often a follow-up of the option just chosen, and the unattended Build answers only three rounds. All 30 questions the Builds saw carried a recommended option, which the release requires, so by the rule under test every one was the shaper's to decide. Telling the shaper to decide such choices and record them as assumptions (B, prompt only) cut the questions on the same eight requests from 25 to 7 and removed every follow-up and every late non-UI question, and a question gate on top (C) cut them to 4; but no request reached a ready Intent in any arm: with the questions gone, the shaper spends its three submissions on missing package files and other gate refusals and ends with a draft marked not ready. On five small held-out requests nothing changed: no arm asked a question, and readiness stayed at 10 of 10 for the release and for B and 9 of 10 for C on the ChatGPT lineup; the Claude-lineup held-out cells could not run because the subscription seat was over its weekly pace. Verdict: no. The registered rule needed 6 of 8 ready Intents and zero questions; B gives 0 of 8 and still asks 7. The question loop is real and the decide-instead-of-ask rule shortens it, but on these large requests it was hiding that the shapes do not reach readiness at all; the Claude lineup's own records on the same requests also end mostly in not-ready Intents, without the question loop.

Question

Unattended Builds answer every shaping question with its recommended option, at most three times. On a ChatGPT lineup (shaping with gpt-6.1-sol at low effort), eight ordinary feature requests ended without an Intent: each answer round brought a new question, three times over. Why do answered questions not converge, and does the rule under test ("shaping asks only UI/UX; a question the shaper can recommend an answer to is decided by the shaper and recorded as an assumption") end the loop and produce Intents, without harming ordinary small requests on either lineup?

Arms

  • A (release): Kogen release v0.1.0-dev.62015185, kogen-shaping `7bcfc8d`. Three of the eight requests first failed on the previous release (kogen-shaping `0484edf`); the loop is the same on both.
  • B (prompt only): A with the shaper prompt's question rule replaced (texts verbatim in data/arm-texts.json; no Go change, prompt goldens regenerated): a choice the shaper could offer with one option recommended is decided and recorded (assumption: ...); everything that is not UI/UX is decided from the request, the repository and probes; only a UI/UX choice with no defensible default, or a sign-in only the user can perform, may be asked; answers are final, with no follow-up on the details of an answered choice; a resource the shaping cannot reach is not asked for again after an answer: the Intent uses a stand-in, recorded as an assumption.
  • C (prompt and gate, exploratory): B plus a deterministic check in the question validator: a product question that marks a recommended option is refused as decided ("take that option, record it as an assumption, and continue to the Intent"); a product question with no recommended option may now be asked (today it is refused); sign-in handoffs unchanged. C changes behaviour that more than a dozen existing kogen-shaping question tests assert, so it is reported as a measurement only.

Method

  1. Model-free diagnosis of the eight failed Builds from their own logs (shaping output, answer log, events where kept): per session the questions asked, options, recommended option, the question check's classes, and how each session ended; every question classed by its text (UI/UX; non-UX the shaper should have decided; handoff or environment; follow-up of an earlier answer).
  2. Model-free replay of the 32 recorded sessions' questions through a mirror of the release's question validator and through B's rule (data/replay.json).
  3. Package checks. kogen-shaping's own setup and go test ./... on the pin and on B (Linux): the pin and B give the same result.
  4. Cells. Shape only (kogen intent shape) in a fresh copy of each repository at the requests' original base, 1800 s per cell over all its sessions.
    • The eight requests, one rep per arm, on the ChatGPT lineup exactly as the failed Builds (gpt-6.1-sol, low effort), with the unattended answerer the Builds use (all: recommended, at most three rounds).
    • Held-out: stage 3's five small requests (Python, Go and Node synthetic repositories), two reps, on both lineups (Claude Code with claude-sonnet-5-5 at high effort; gpt-6.1-sol at low effort), with no answerer: a cell that asks any question ends there and counts as not ready.
    • Side probe (descriptive): A and B on three of the eight requests with shaping effort high.
  5. Registered rule (Studio record first, before any cell): B passes if it reaches an Intent on at least 6 of the 8 requests; every shape ends within the round cap with no non-UX question after round 2; mean questions per Intent is 0 on the eight requests and on the held-out set (A is the control); at least 9 of 10 held-out cells ready on each lineup and not fewer than A; no new refusal class (in particular no refusal of an (assumption: ...) line); suite equal to the pin.

Results

Diagnosis of the eight failed Builds (model-free; their own logs; data/replay.json). Each Build ran four shaping sessions (the first and three answered rounds). Every one of the 32 sessions asked exactly one question, except two that spent all three submissions on questions the validator refused; no session asked two, and no question repeated an answered one. Of the 30 questions that reached the answerer, 20 were UI/UX placement, form or navigation choices, 6 were not UI/UX (record fields, a size-limit rule, a cut-preview ordering, an inline-versus-stored result, a launcher architecture, how a lesson is marked), and 4 were the same sign-in handoff asked four times. 15 of the 32 were follow-ups that name the option chosen in the previous answer ("In the chosen ... form", "On the new ... page you selected"). Every asked question carried exactly one recommended option, because the release requires it twice: the shaper prompt asks for "one marked recommended" (`shape/prompts/shaper.md:21`), and the question validator refuses any other form (`shape/shape.go:2182`). The replay through that validator: 30 of 30 valid, so all 30 were decidable under the rule (26 as an assumption, 4 handoffs as a stand-in). Kogen itself has no answer-round cap; the unattended Build answers at most three rounds and then stops for want of an Intent. Two release behaviours keep the loop going: a session that ends without a valid submission leaves the previous round's questions file in place (it is removed only when a shape succeeds, `shape/shape.go:799`), so the answerer answers an already answered round again; and for a sign-in the recommended answer ("I have completed the sign-in") is a claim the next probe disproves, so the same handoff returns.

The eight requests, ChatGPT lineup, unattended answerer (data/cells.json; tables printed by report.py):

set lineup arm cells ready questions asked mean per cell cells asking follow-ups of answers non-UX after round 2 refusals by gate
rows chatgpt A 8 0 25 3.12 8 11 4 question_check 44, validation 4
rows chatgpt B 8 0 7 0.88 5 0 0 decision_check 9, question_check 8, validation 7, intent 7, step_ownership 4, contract_review 2, literal_check 2, decision_evidence 1
rows chatgpt C 8 0 4 0.50 4 0 0 validation 7, question_check 7, intent 5, decision_check 2
  • A reproduces the failure on 8 of 8: no Intent anywhere; 25 questions (18 UI/UX, 3 non-UX, 4 handoff), 11 follow-ups of answers, 4 non-UX questions after round 2; where no question got past the model question check, the shaper kept submitting technical or product-policy questions until the session's three submissions were spent.
  • B cuts the questions to 7 (3 UI/UX, 4 non-UX; 0 follow-ups; 0 non-UX after round 2) and C to 4 (3 UI/UX questions without a recommended option, which stopped the unattended answerer as designed, and 1 sign-in handoff). Neither reaches a ready Intent on any row. With the questions gone the shaper still spends a submission on a technical question the check refuses, then runs out of its three submissions on package gates: missing acceptance.feature or step files, the literal check, a blocked report with no protected path, decision evidence it never read, and decision lines the request does not state. B writes a draft Intent marked not ready on 7 of 8 rows (A: 0), so in an unattended Build B moves these rows from "no Intent" at shaping to a not-ready Intent at approval, the state the Claude lineup mostly reaches on the same requests in the Builds' own records (where it asks at most one question in four sessions).
  • No refusal in any arm is of a line carrying (assumption: ...): the recorded assumptions are not refused by the decision check, the evidence gate or the Intent lint.
  • A second release defect bears on "answered questions converge": the decision check sees only the request and intent.md, never the answers (`shape/decision.go:18`), and exempts only lines marked as assumption or evidence (`:259-270`); a Decisions line tagged (source: answer N), the form the Intent template asks for, is refused as unsupported (3 such lines in these cells).
  • Effort high (side probe, A and B on three rows): with shaping effort high instead of low, A still asked 2-4 questions per request (9 in 3 cells) and reached no Intent (one cell at the 30-minute deadline); B asked none and reached no ready Intent either (two cells at the deadline, one not-ready draft). Low effort is not what keeps these requests from an Intent.

Held-out small requests, no answerer (a question would end the cell):

set lineup arm cells ready questions asked mean per cell cells asking follow-ups of answers non-UX after round 2 refusals by gate
held-out chatgpt A 10 10 0 0.00 0 0 0 decision_check 8, readiness 1
held-out chatgpt B 10 10 0 0.00 0 0 0 decision_check 4, readiness 2
held-out chatgpt C 10 9 0 0.00 0 0 0 decision_check 6, step_selfcheck 1

The Claude-lineup held-out cells (Claude Code with claude-sonnet-5-5, high effort; A and B, two reps) did not run: the one subscription seat this stage may use was over its weekly pace line for the whole window (0.91 of its week against a pace line of 0.84 at the last check), the second seat was past its 0.50 weekly stop, and no other Claude account was allowed. Stage 4's stored Claude cells on the same five requests (its candidate on an earlier release, 10 of 10 ready) are not this comparison.

Package checks. kogen-shaping with its own setup and go test ./... on Linux: the pin and B give the same result (every package passes except one test that depends on the host's temporary directory, failing on both); B changes no Go file, so every gate and stage 0's negative cases are unchanged by construction. C changes the question validator and would need more than a dozen existing question tests updated.

Verdict

No (registered rule, candidate B). Failing checks: rule 1, ready Intents on the eight requests: 0 of 8 (needed 6); rule 3, questions per Intent: B asks 7 on the eight requests (A 25), not 0, and 4 of the 7 are not UI/UX by their text. Holding: rule 2 (no B question after round 2 is non-UX; B asks no follow-up of an answer), rule 5 (no refusal of an assumption line; B's refusals are of gates the release also shows, though B reaches more of them because it gets past the questions; three blocked reports without a protected path may be the shaper's way out when it may not ask), rule 6 (suite equal to the pin, no Go change). Rule 4 (held-out): holds on the ChatGPT lineup (B 10 of 10 ready, A 10 of 10, no question in either); unresolved on the Claude lineup (cells not run, seat capacity). C (exploratory) does not change the picture: 0 of 8, 4 questions, and its gate changes tested behaviour. What the data supports instead: (1) the release's question contract makes every question decidable (a recommended option is mandatory) while the shaper prompt asks it to ask; (2) two release behaviours keep answer loops going (a stale questions file after a failed session; a sign-in handoff whose recommended answer is a claim); (3) the decision check refuses answer-sourced decision lines because it never sees the answers; (4) on these requests the ChatGPT shaper fails to produce a valid package within three submissions even when it asks nothing, at low and at high effort. No delivery REQUEST is filed from this stage.

Rerun

  1. Install release v0.1.0-dev.62015185 (arm A). For B or C, build Kogen `6201518` in cli/ with go mod edit -replace=github.com/KogenAI/kogen-shaping=<a copy of kogen-shaping 7bcfc8d with the arm's text from data/arm-texts.json> and go build -trimpath -buildvcs=false (Go 1.27.2), regenerate the prompt goldens with go test ./shape/ -run 'TestPromptsAreByteEqualToTheElixirFunctions|TestShaperPolicyGolden|TestShapeForceWireCompatibility' -update, and swap the release's bin/kogen.
  2. Held-out: make the three small repositories as in stage 3 (lab-shaping-small-requests-20261107, Rerun step 1) with .kogen/models.yaml shaping either executor: claude-code, model: claude-sonnet-5-5, effort: high or provider: chatgpt, model: gpt-6.1-sol, effort: low; run kogen intent shape <slug> <request file> once per request, arm, lineup and rep; a printed question ends the cell.
  3. Feature requests: use any feature request of your own on a web product repository whose .kogen/models.yaml shapes with provider: chatgpt, model: gpt-6.1-sol, effort: low; after a session that ends with questions, run printf 'all: recommended\n' | kogen intent shape <slug> - at most three times; read .kogen/local/shaping/<slug>/events.ndjson (question_check, submission_refused) and whether .kogen/intents/<slug>/intent.md exists. The eight requests of this finding are private; their per-cell counts are in data/cells.json.
  4. python3 -I report.py . prints every table above from data/cells.json and data/replay.json.