Published result
Preventing transient bytecode in shaping checks
A plain-language summary comes first, followed by the full text of the result's reproduction pack.
Result summary and pack text · reproduce/lab-shaping-bytecode-screen-20261107/README.md
- Finding id:
lab-shaping-bytecode-screen-20261107 - Verdict: No
- Finished: 7 November 2026
- Comparison: Kogen shaping component
- Reproduction pack: reproduce/lab-shaping-bytecode-screen-20261107/ (data, analysis script and rerun steps)
- Category page: categories/harness-levers.md
In plain terms
A small paired screen rose from six to seven ready Intents, but a regression failed the keep rule.
Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.
Full text of the reproduction pack
The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).
A prompt instruction asked generated acceptance steps to avoid transient Python bytecode writes in the checked project. The stored release produced 6 ready Intents on ten small requests; the candidate produced 7. Two requests changed from not ready to ready and one changed from ready to not ready. Neither arm asked a question. The regression fails the registered no-regression condition, and the net gain is within the previously observed paired flip range. The candidate was dropped. This is a screen, not a full-set result or a delivery qualification.
The readiness mutation gate remained unchanged. The instruction fired in one ready candidate: its generated step invoked Python with bytecode suppression. That request was ready in the release too. The regression ended at the decision check, and the gains involved other gates. These observations are not proof of causality. Intermediate refusals could be repaired; ready means terminal readiness without questions or terminal refusal.
Each request ran once per arm on the same model and low effort, without an answerer; the release cells were stored results. Times include failed attempts and are in the data. The shared credit balance did not move, but exact attributable spend is unknown. No held-out or complex tasks, hidden checks, or builds were used in this screen, so landing and hidden-check rates are not measured. A single repetition cannot separate these changes from ordinary variation.
The release uses Kogen source and shaping source. The candidate added only the initial instruction described above; source trees themselves are not included.
Rerun
Run python3 report.py to reproduce the paired counts from the included data. To repeat the experiment, use the small-request setup described in the small request finding, use the pinned release as control, add the bytecode instruction described above to the initial shaper prompt for the candidate, and shape the same ten climb requests once per arm with the same model, low effort and no answerer. Record terminal readiness, questions, gate refusals, generated step behavior and wall time; repeat pairs before claiming causality.