Published result
Static checks on shaped files stop 1 of 22 defects the builder refuses
A plain-language summary comes first, followed by the full text of the result's reproduction pack.
Result summary and pack text · reproduce/lab-c2-static-checks-20261105/README.md
- Finding id:
lab-c2-static-checks-20261105 - Verdict: No
- Finished: 5 November 2026
- Comparison: Kogen shaping component
- Reproduction pack: reproduce/lab-c2-static-checks-20261105/ (data, analysis script and rerun steps)
- Category page: categories/harness-levers.md
In plain terms
The repository's own format, vet and syntax checks, run on shaper-written acceptance files before approve, caught 1 of 22 builder-blocked defects (a Go driver that was not gofmt-formatted). With the 6 readiness refusals that already happen, 7 of 28 failures are stopped. The rest are semantic: contradictory steps, wrong expected values, fixtures that do not match the code, timing, string escaping. No false refusals in 30 landed Intents, and 1 in 12 in-scope Go packages. Shaped bytes survived for only 5 of the 28 failures, the rest were scored from logs and probes. The 60% bar is not met, so the shape-repair comparison is needed.
Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.
Full text of the reproduction pack
The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).
Question: when the shaper writes acceptance files with a defect the builder must then refuse (builder-blocked), would running the repository's own static checks on the shaped files at readiness, before approve, have caught it? Step 1 of a two-step Lab; model-free, no seat tokens. Studio Lab record lab-c2-static-checks-20261105 is the lever record lab-c2-static-checks-step1-20261105; assumption kogen/build-failures-c2-shaped-harness-defects-20261105.
Result: no, not enough. The checks caught 1 of 22 builder-blocked defects (4.5%), the Go driver that was not gofmt-clean. Counting the 6 report_invalid rows (5 Intents) that the readiness acceptance run already refused, 7 of 28 targeted failures (25%) are stopped before approve, but only the one is due to the new checks. The 60% bar is not met, so the Lab's step 2 (a shape-repair round fed the builder's report, against a fresh rebuild) is needed: about 24 Builds and 8 seat-hours, not run here. False refusals were not the problem: 0 of a 30-Intent random sample, 0 of 327 landed Intents on gofmt and node --check, and 1 of 12 in-scope Go packages under the configured go vet ./... (a driver that calls an API the Intent adds).
Method
- Arm (a): applied to the shaped files laid over the base, before approve: (1) the repo's
formatcheck,test -z "$(gofmt -l .)"(gofmt is per file, so replayed asgofmt -lon each shaped.gofile); (2) the repo'svetcheckgo vet ./...(replayed asgo list ./...to see whether the shaped Go package is in scope, thengo vetof that package, on the base tree with the shaped files laid over); (3)node --checkon shaped.jsand.cjs. - Fixtures: the 22 builder-blocked and 6 report_invalid rows of the 72-hour build-failure classification (
data/c2-table.json), and 333 Intents landed since 2 Nov, of which 30 were drawn at random (seed 20261105,data/green-sample30.json). Landed Intents are listed with their repository and landing commit indata/green-landed-intents.json; the shaped files are the files each landing commit added under the Intent folder and the steps folder. Source is public: github.com/KogenAI/<repo>/commit/<sha>. - Data limit: the shaped bytes of most failures no longer exist (a retry re-shapes, trees are reused, older run folders were removed). Shaped files were recovered for 5 of the 28 targeted failures (3 builder-blocked, 2 report_invalid), and for 4 of those 5 they are a later shape of the same Intent. The other 23 are scored from the logged defect text, backed by defect-class probes: one minimal file per logged defect, run through the same checks (
data/class-probes.json). - Tools: gofmt, go vet and go 1.27.2, node 24.21.0 from the build host's installs; git 2.54.0 (Apple Git). Go checks ran under the host's shared test lock after delivery work had drained, cheap checks earlier at low priority.
- What is already in flight: the request
k-shape-artifact-static-checkscovers arm items 1 and 2 (configured format, lint and vet checks on shaped paths in the readiness copy, a named refusal, recheck, unrelated-failure attribution). It does not runnode --check, which flagged nothing here.
Results
| Set | Checked | Flagged |
|---|---|---|
| builder-blocked defects (22) | arm (a) | 1 (gofmt-unclean Go driver) |
| recovered failing trees (5 of C2 plus the C10 tree, 12 JS files) | node --check | 0 |
| 327 landed Intents with checkable files (451 files) | gofmt, node --check | 0 |
| 30-Intent random sample (28 with checkable files) | gofmt, node --check, vet | 0 false refusals |
| 33 landed Intents that shaped Go files | go vet in ./... scope: 12 packages |
11 pass, 1 false refusal; 21 packages are outside ./... (testdata folders, ignore build tag, no module at the repo root) |
same 33, naive go vet of every shaped Go package |
not the configured check | 9 type errors against code not yet written, 11 not compilable alone |
Defect-class probes: a gofmt-unclean Go driver is flagged. Not flagged by gofmt, node --check or go vet: $$ through JS replace, double-escaped regex dots, JSON field case, fixture total 140 against 120, a trap and sleep timing loop, byte equality over elapsed time, a Go fixture calling t.Parallel twice.
Reading
Everything except formatting is semantic: contradictory steps, wrong expected values, fixtures that do not match the code under test, timing, string escaping. Static checks are cheap and should stay (they stop the one class they can), but they will not remove the cluster. Caution for any implementation: a shaped Go driver that type-checks only against code the Intent adds fails go vet on the base; that must be reported as pre-implementation, not as a shaped-file defect.
Rerun
Everything above is recomputed offline from the published data, with no model call:
python3 -I report.py data
It prints the catch rate over the 22 builder-blocked defects, the cheap-check and vet counts, the false refusals in the sample and in the in-scope Go packages, and the probe results. The raw checks were produced by running gofmt, node --check and go vet on the files named in data/green-landed-intents.json at their landing commits' parents; those commits are public.
Limits
Five real trees of 28; the rest is classification from logs plus probes. One host, one toolchain version. The report_invalid rows are counted as already caught because the readiness acceptance run is what refused them.