Published result
Setup evidence in contract review remains unproven
A plain-language summary comes first, followed by the full text of the result's reproduction pack.
Result summary and pack text · reproduce/shape-protected-setup-evidence-screen/README.md
- Finding id:
shape-protected-setup-evidence-screen - Verdict: Unresolved
- Finished: 8 November 2026
- Comparison: Kogen shaping component
- Reproduction pack: reproduce/shape-protected-setup-evidence-screen/ (data, analysis script and rerun steps)
- Category page: categories/harness-levers.md
In plain terms
A scratch evidence change matched the unpatched shaper on the only independently confirmed setup conflict and on ten readiness controls. The small retained sample does not establish an improvement.
Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.
Full text of the reproduction pack
The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).
The screen did not establish an improvement. The only independently verified setup contradiction in the recovered sample was refused by both the unpatched shaper and the candidate. Both cited the configured setup conflict; neither cell logged contract review before that refusal, so this case does not demonstrate that the added contract-review evidence fired. The ten control requests reached ready in both arms (10/10 versus 10/10), within the specified ±1-per-10 noise band.
The blocked Build cohort comprised 25 distinct slugs and 29 blocked dispositions. Latest builder reports by slug were 12 acceptance-fixture, four fresh-copy history, three dependency, three configured setup/check, two nondeterminism and one toolchain. These are reports, not 25 verified contradictions. Seventeen slugs later landed; two examined revisions changed their Intents and fixtures. Seven retained Intent versions from five slugs were available to measure. Only one of those versions has an independently confirmed setup contradiction. This is too small and selective to estimate a catch rate over 25 slugs, or to show the absence of a benefit in other cases.
The paired replay ran once per arm for each version and each control: 34 Sol-low cells, on the same Linux host per input. Six retained pairs terminated in both arms; a seventh baseline hit the existing 1,200-second limit. Candidate-only contract-review traces show the patched stage was reached in some cells, but the logged traces do not expose the inserted evidence sections, so their contents cannot be independently confirmed from those logs. The focused model-free regression exercises evidence inclusion and rejection of a scripted confirmed finding. It does not establish live detection.
Of the six terminal retained pairs, two were ready in both arms, two were non-ready in both, one was refused in both for the independently confirmed setup conflict, and one changed from baseline non-ready to candidate ready. The latter baseline failed the stage-format gate; it is not attributed to the modified contract-review stage. One of the non-ready pairs stopped at different gates: a generated acceptance check versus a reported missing Git object; the original probe was not retained, so the latter remains a model report. The baseline timeout is not a catch or an approval. None of these differences supports the candidate mechanism as a climb.
This was a shape-only screen: no Build, task landing, or hidden grading was run. Ready is not hidden-grader pass. The metrics file records hidden checks as 0/0 and landings as zero performed, not as measured failure rates; its valid-Intent counts are the lower bound represented by ready cells. The logged driver also misidentified a host path as a refusal on successful cells; this report normalizes successful cells to no refusal using their successful exit and ready receipt. Both arms used the same safety-adjusted packaging fixture, whose deletion commands were replaced by checks for fresh output paths; the setup still copied the relevant file. Shaping a saved Intent as a new request may revise it, so an outcome is not a direct approval decision on an unchanged original.
The baseline was Kogen 41d70052 with shaper f04edce. The candidate added configured setup, check and acceptance commands, protected-path names, and literally referenced tracked command files to contract-review evidence. The scratch diff SHA-256 was cafcae2e831455c1bd783f77fa5f2503e78c9fca1fcb782b08d0f01a616d2227. The patch does not follow indirect shell references. No product code was changed.
Disposition: drop this scratch candidate for delivery on this evidence. The original operational concern remains unresolved. The investigation is recorded in Studio document shape-protected-setup-cohort-regression-20261108.
Rerun
Run python3 report.py in this directory to reproduce the per-cell readiness and exit table and the control totals from results.json. Saved private Intent text is deliberately excluded; the public data supports reanalysis of the recorded results, not an independent replay of those private inputs. Per-arm wall totals include failed and timed-out cells and are not a speed comparison. Credit usage per cell was unavailable; known token counts and unknown-call counts are recorded in the data.