Published result

Unconditional kogen check at landing fails its rule: it refuses good commits and reproduces 1 of 8 breakers

A plain-language summary comes first, followed by the full text of the result's reproduction pack.

Result summary and pack text · reproduce/lab-prelanding-kogen-check-20261105/README.md

In plain terms

Replaying 48 component landings (8 breakers, 40 known-good) with kogen's go build, vet and test run on the landing-time kogen main after pinning the landed commit: the registered rule fails (12 of 31 good commits refused, limit 2). Strict recall looks high (7 of 8) but only 1 breaker is refused for the reported reason; the rest are refused by a hard-coded-pin test fixed in kogen on 4 Nov or by loop-module skew, and some are not reproduced at all. Good commits are also refused for coupling between components and for kogen tests that kogen edited with the move. About 4.8 minutes added per landing. Not built as specified; the tuple variant refuses good commits too, because kogen's tests lag legitimate component moves.

Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.

Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

Finding lab-prelanding-kogen-check-20261105. A model-free Lab (no model call, no seat, 5 Nov 2026) on one question: would a kogen consumer check that is never skipped (no "no exported interface", no fast-forward skip, no bare-loop skip) have stopped the commits that broke kogen's tests, without refusing commits that were fine? The Studio Lab record is lab-prelanding-kogen-check-20261105 (registered 5 Nov 09:49Z before the first run, the result is in a later revision).

Arms, on the same 48 component landings of 3 and 4 Nov:

  • A: the landing gate as it decided, read from the land logs (no re-run). All 48 landed GREEN. Kogen's own tests ran at landing for 1 of the 8 breakers (shaping 1e795ed, go test only); the other 7 were skipped as "no exported interface" (2), "fast-forward" (2) or "still requires the bare kogen-loop" (3).
  • B: the unconditional check. A scratch copy of kogen main as it was at the landing time; the landed commit pinned the way the adoption does it (go get and go mod tidy; for kogen-loop sync-kogen-loop.sh and the module rename); then gofmt -l, go build, go vet and go test ./... in cli/, including consumes/*, TestLoopPin and TestAssemblyUsesThePinnedComponents. A failure refuses the landing only if the same test passes on that kogen main without the pin (the base). A commit that cannot be pinned or built counts as red.

Sample (data/sample.tsv): the 8 breakers named in the 5 Nov adoption review (build 8852812 and 10f190b, coordinator 91d0a32 and bc5b726, shaping f04edce and 1e795ed, setup aeeb5be, review f4b7f54) and 40 known-good landings drawn with random.Random(20261105).sample from the 84 green landings whose first release is 41d70052 or d98cfe01. The good commits run in draw order, so the 31 that ran are themselves a random sample of the 40.

Registered pass rule: B refuses at least 6 of the 8 breakers and at most 2 of the 40 good commits.

Result

Reading Breakers refused Good refused Rule
A, the gate as it decided 0 of 8 0 of 40 fails (no recall)
B strict (as registered) 7 of 8 12 of 31 run FAIL (strict, as registered: 12 false refusals of 31 good run > 2)
B adjusted (post hoc: the one hard-coded-pin test is not counted) 4 of 8 8 of 31 FAIL (adjusted: hard-coded-pin test ignored: 8 false refusals of 31 good run > 2)
B behaviour only (post hoc: that test and loop-module skew are not counted) 1 of 8 8 of 31 FAIL (behaviour only: hard-coded-pin test and bare-loop skew ignored: 8 false refusals of 31 good run > 2)

The registered rule fails, and it was decided at the 11th good commit (a third false refusal; limit 2). The remaining good commits only refine the other two readings. Median cost: 4.8 minutes added to a landing (pin, gofmt, build, vet and test on one 2-vCPU host with a warm Go build cache; 4.8 with the wait for the shared Go lock, which was 0 in practice).

What the numbers are made of

  • False refusals, strict (12 of 31). By cause: 5 x API coupling: an unmoved component fails to build/vet against the pinned one; 4 x obsolete test: hard-coded component versions in assembly_test.go; 1 x kogen test change needed: TestApproveDeadlineHungCLI and TestApproveDeadlineSplitCLI fail again when re-run alone on the pinned tree and pass on the unpinned base; kogen changed cli/approve_deadline_test.go in 96b7438 (3 Nov 19:54Z), just before release 41d70052, so this landing needed a kogen-side change the adoption carried; 1 x same two timing-deadline tests as n=23 fail on the pin and pass on the base (not re-run); kogen changed cli/approve_deadline_test.go in 96b7438 before release 41d70052; 1 x kogen test fails on the pin only (TestResearchRemovalDefaults); kogen changed cli/research_removal_test.go in 96b7438 before release 41d70052, consistent with a kogen-side change the adoption carried (not verified test by test). The hard-coded-pin refusals come from TestAssemblyUsesThePinnedComponents, which in kogen until commit 896ae0c ("Make adoption tests follow component pins", 4 Nov 03:52Z) compared the packaged executable against a hard-coded map of component versions, so any pin move failed it. It fails on the pin and passes on the base at the kogen mains cf58cd64 and 5cf70c29 (3 Nov 20:22Z to 4 Nov 03:52Z) and passes at the earlier mains 9bf6073f and a1a04454. Refusals by kogen main: {"cf58cd64": 3, "5cf70c29": 1, "9bf6073f": 2, "a1a04454": 6}. Kogen's test follows the pins since 896ae0c.
  • False refusals that remain when that test is ignored (8 of 31). n=23 verification@57542e9 (kogen 9bf6073f): kogen test change needed: TestApproveDeadlineHungCLI and TestApproveDeadlineSplitCLI fail again when re-run alone on the pinned tree and pass on the unpinned base; kogen changed cli/approve_deadline_test.go in 96b7438 (3 Nov 19:54Z), just before release 41d70052, so this landing needed a kogen-side change the adoption carried; n=24 executor@70ceb5a (kogen a1a04454): API coupling: an unmoved component fails to build/vet against the pinned one; n=26 executor@64ffa58 (kogen a1a04454): API coupling: an unmoved component fails to build/vet against the pinned one; n=28 executor@923b1ad (kogen a1a04454): API coupling: an unmoved component fails to build/vet against the pinned one; n=31 executor@a4da7f8 (kogen a1a04454): API coupling: an unmoved component fails to build/vet against the pinned one; n=33 verification@292a2d1 (kogen 9bf6073f): same two timing-deadline tests as n=23 fail on the pin and pass on the base (not re-run); kogen changed cli/approve_deadline_test.go in 96b7438 before release 41d70052; n=34 executor@cd5e978 (kogen a1a04454): API coupling: an unmoved component fails to build/vet against the pinned one; n=36 loop@0c49f87 (kogen a1a04454): kogen test fails on the pin only (TestResearchRemovalDefaults); kogen changed cli/research_removal_test.go in 96b7438 before release 41d70052, consistent with a kogen-side change the adoption carried (not verified test by test) Two kinds: coupling (the pinned commit needs the other components to move with it, which the adoption does and a single pin does not) and kogen tests that had to change with the move (kogen edited them in 96b7438 before release 41d70052).
  • Strict recall (7 of 8) is mostly not recall. Refused for the reason the review reported: build@8852812. Refused only by the hard-coded-pin test: build@10f190b, coordinator@91d0a32, shaping@1e795ed. Refused because kogen main still vendors the bare kogen-loop while the pinned component requires github.com/KogenAI/kogen-loop, so the unmoved components no longer build (loop-module skew, the review's P cluster): setup@aeeb5be, review@f4b7f54, coordinator@bc5b726. Not refused: shaping@f04edce.
  • The breakers that were not reproduced. Two broke only kogen's acceptance suite (build 10f190b, shaping 1e795ed), which a go-level check does not run (the review: go level 6 of 9, acceptance needed for the rest). Shaping f04edce and coordinator 91d0a32 broke kogen's tests only in an adoption window that moved a tuple of components at once; pinned alone onto the landing-time kogen main they leave build, vet and test as they were. The three post-rename breakers cannot be pinned alone onto a pre-rename kogen main at all, and a loop-aligned single pin does not build either (the other components still import the bare path).
  • Ignored on every reading as pre-existing: TestZshDynamicWiring (the host has no zsh) and, on kogen mains up to a1a04454, TestShapeBudgetRoleContract fail on the unpinned base too.

Diagnostic: pin the tuple the next adoption would pin (post hoc)

The review's windows moved all components together. On the rows whose era allows it, the same steps with the tuple "landed commit plus every other component's main head at that time" (loop-free tuple where the heads mix old and new loop paths) give: 5 breakers run, 5 refused (strict), 2 for a kogen test or vet failure (coordinator 91d0a32: TestResearchRemovalDefaults; shaping f04edce: kogen's test code no longer compiles against the executor head, undefined: agent.Reviewer), 2 unbuildable through loop-module skew. It also refuses 6 of the 6 good commits tried, all for kogen tests that kogen itself edited before the release (TestApproveDeadline*, TestResearchRemovalDefaults, a vet error in a kogen test file; kogen 96b7438 and 896ae0c). So the tuple variant does not separate breakers from good commits either: at the landing-time kogen main, kogen's tests lag every tuple move, and the real adoption carried the kogen-side edit in the same pin. A tuple failure is also attributed to the tuple, not to the commit that triggered the check.

Verdict

No. The unconditional single-commit check, run as registered, fails the pass rule on false refusals, and it also fails on recall under both post hoc readings (4 and 1 of 8). Of the 8 breakers it reproduces 1 for the reported reason (build 8852812, consumes/kogen-build TestRecover*). No DELIVERY.md is written. What the data supports instead: (1) the cost is small (about 4.8 minutes); (2) a landing-time check cannot tell "a component broke kogen" from "kogen's tests lag a legitimate component move": in this window the adoption itself carried kogen-side test edits (96b7438, 896ae0c), coupled components need to move together (an executor change breaks the unmoved build module), and the tuple variant refuses the good commits as well, so any such gate needs the kogen-side change to travel with the pin or kogen's tests to be pin-agnostic; (3) it needs a kogen main whose tests follow the pins (true since 896ae0c) and which has finished the loop-module move; (4) acceptance-only breaks need the acceptance suite.

Limits

  • Arm B pins one commit onto kogen main as of the landing time and stops at kogen's go level; the landing times come from the land logs' done stamp and the last first-parent kogen commit at or before it.
  • 31 of the 40 good commits ran (random prefix); the adjusted and behaviour readings are estimates from those. The strict verdict does not depend on the rest.
  • The host is a 2-vCPU Linux box without zsh; one test fails on every base for that reason. Timings come from a host shared with other Labs and with delivery; load was recorded per run in the Lab log, not controlled.
  • The tuple diagnostic is post hoc, covers 11 rows, includes unqualified model-facing heads, and was not registered.
  • 4 of the 8 breakers' original failures were read from the adoption step outputs, not re-observed.

Rerun

Everything below runs offline from the published data files, with no model call and no kogen checkout:

python3 -I report.py data

It recounts the table above from data/summary.json (one row per sample commit with strict and adjusted verdicts, cause class, new and old failing tests, wall seconds) and data/sample.tsv (arm A's decision), and checks every count against replay.json. data/runs.json holds the raw B runs and the base facts (failing test ids per run, which tests passed on the unpinned base). To redo the measurement itself, scripts/prelanding-check.sh <component> <sha> <kogen-base-sha> <outdir> is the check (steps copied from delivery's adopt-pins-lib.sh and the adoption's go step); scripts/build_sample.py rebuilds the sample from the delivery mirrors and land logs, scripts/orch2.sh is the driver, and data/tuples/ holds the tuple files, and data/rerun-n23.txt the re-run of the two timing tests of row 23 on its pinned tree. The component and kogen sources are public: https://github.com/KogenAI/kogen and https://github.com/KogenAI/kogen-build (and the other kogen-* repositories) at the commits in data/sample.tsv.