Source summary
Findings
Published results, each with its own page and reproduction pack, followed by the fourteen family sections from FINDINGS.md.
Source-reported figures stay labelledFindings are narrative summaries. A number in this source text is not a verified outcome unless a linked claim-ledger row identifies it and carries its own evidence status.
Published results
Every published result has its own page, grouped by comparison, newest first. Verdicts: Yes means the result supports the change or claim tested; No means it does not; Unresolved means the evidence is not strong enough either way; Superseded means a later test replaced it.
kogen-loop vs Codex
- A shell that hands control back during long waits: no clear effect on hard tasks · Unresolved · 31 October 2026
- Adding an 'apply_patch' editing tool to kogen-loop did not reduce its output · No · 31 October 2026
- Asking the model for terser text does not change kogen-loop's cost · No · 30 October 2026
- kogen-loop's connection to the model streams as fast as Codex's · No · 30 October 2026
- kogen-loop costs less than Codex at high effort, on 14 Go tasks · Yes · 30 October 2026
- Report on whether kogen-loop is cheaper, as reliable and faster than Codex: goal not met · Unresolved · 30 October 2026
- Did kogen-loop's connection slow down on 30 October? Earlier data, now superseded · Superseded · 30 October 2026
- kogen-loop runs fewer shell commands than Codex in old runs · No · 30 October 2026
- Sending reasoning back to the model: promising cost, but unresolved on 8 tasks · Unresolved · 30 October 2026
- kogen-loop used about twice Codex's reasoning, for the same total cost · Unresolved · 30 October 2026
- On the subscription route kogen-loop used about 40% of Codex's tokens · Yes · 30 October 2026
- kogen-loop vs Codex on 8 Go tasks: cheaper at medium effort, no clear gap at high · Yes · 30 October 2026
- Short test of two kogen-loop 'several steps per request' tools (batch and exec) · Unresolved · 30 October 2026
- A 'batch' tool for kogen-loop packs more work per request but does not cut cost · No · 30 October 2026
Kogen vs the same agent used alone, on hard tasks
- Kogen with the shaping fix against Claude Code alone on five Rails tasks (smoke, round 2) · Unresolved · 7 November 2026
- Correction: three of the nine chosen hard tasks were chosen because of a grader defect · Yes · 1 November 2026
- Correction: how the grader affected the five hard-task results · Yes · 31 October 2026
- A newer Kogen driving Codex vs Codex alone: tied on the tasks that could be graded · Unresolved · 30 October 2026
- Kogen driving Codex vs Codex alone, 6 hard tasks, one MacBook: tied on success · Unresolved · 30 October 2026
- Kogen driving Codex vs Codex alone, 9 hard tasks, one laptop: tied on success · Unresolved · 30 October 2026
- Choosing hard tasks for Kogen against Codex alone · Unresolved · 30 October 2026
- What Kogen's independent review step did in hard-task runs · Unresolved · 30 October 2026
- Kogen driving Claude Code vs Claude Code alone, 9 hard tasks, one laptop · Unresolved · 30 October 2026
- Kogen driving Claude Code vs Claude Code alone, 6 hard tasks, one MacBook, second block · Unresolved · 30 October 2026
- Kogen driving Claude Code vs Claude Code alone, 6 hard tasks, one MacBook · Unresolved · 30 October 2026
kogen-loop speed, patches and connection routes
- kogen-loop is faster and lighter than Claude Code on the same API key · Yes · 5 November 2026
- What keeps the prompt cache warm (unconfirmed probe) · Unresolved · 4 November 2026
- Message Batches review of landed commits · Unresolved · 4 November 2026
- A change to Kogen's loop that labels tool errors sends the model exactly the same bytes · No · 3 November 2026
- Handing small repositories to kogen-loop up front still removes its look-around requests on three servers, with input flat on the Mac · Yes · 2 November 2026
- kogen-loop handing small repositories to the model up front removes its look-around requests · Yes · 2 November 2026
- kogen-loop's latest version passes 13 of 14 tasks; its new cost is shell polling, not failures · Unresolved · 1 November 2026
- kogen-loop's polling and refused-call fixes work: half the polling requests, no refused calls, no task broken · Yes · 1 November 2026
- kogen-loop waiting up to 45 seconds for a shell command removes result-fetching requests on hard tasks · Yes · 1 November 2026
- Starting tool calls while the model is still streaming does not make kogen-loop faster · No · 31 October 2026
- One extra instruction sentence made kogen-loop about 17% faster at the same pass rate · Yes · 31 October 2026
- What kogen-loop does when the model stream or its shell misbehaves · Unresolved · 31 October 2026
- The finish-right-away sentence caused no early stops on two hard review tasks, but only one could show it · Unresolved · 31 October 2026
- No faster route to the model was reachable with kogen-loop's own login · No · 30 October 2026
- Letting kogen-loop's file-writing tool carry a follow-up check: not used by the model · No · 30 October 2026
- Three speed patches for kogen-loop: none faster or cheaper beyond noise · No · 30 October 2026
Test setup, rules and sandboxing
- Approval delays lack phase attribution · Unresolved · 8 November 2026
- Alarm relays to main: one line per event removes every repeat · Yes · 5 November 2026
- Rule: loop-vs-Codex time and cost are measured only on the two Linux servers · Yes · 30 October 2026
- Test harness change: install dependencies once, retry timed-out approvals · Yes · 30 October 2026
- Why a Kogen approval timed out: the shared 120-second deadline under heavy load · Yes · 30 October 2026
- Four macOS sandbox tools measured against Kogen's old containment needs (historical) · Yes · 15 October 2026
Kogen shaping and delivery-build speed
- Six of nine Kogen test suites exceed their time budgets on a quiet host · Unresolved · 7 November 2026
- On macOS, Go tests started detached are scanned by XProtect; attached to ssh they are not · Yes · 6 November 2026
- The tests request's time budgets cannot yet be priced from recorded logs · Unresolved · 5 November 2026
- Unconditional kogen check at landing fails its rule: it refuses good commits and reproduces 1 of 8 breakers · No · 5 November 2026
- Reading a Build's final outcome fixes delivery's misreads · Yes · 5 November 2026
- Batching Kogen's approval step-binding check cuts approval time by over 90% · Yes · 5 November 2026
- Retained verification diagnostics and repair locations · Unresolved · 3 November 2026
- Verification timeout diagnostics remain unconfirmed in task runs · Unresolved · 3 November 2026
- Kogen's landed leftover-process sweep still misses a server started by a launcher that closes file descriptors · No · 3 November 2026
- Verification caller cancellation follow-up · Unresolved · 3 November 2026
- A rule asking Kogen's shaper for a protected property check saved a valid intent in 1 of 10 pairs against 4 for the adopted build, so we dropped it · No · 3 November 2026
- Shaping repair completed paired screen · Unresolved · 3 November 2026
- Shaping repair paired screen · Unresolved · 3 November 2026
- Letting Kogen's shaper decide non-product questions itself looks better on paper, but the change did not cause the gains · Unresolved · 3 November 2026
- With a stronger model, Kogen's data-stores tool and the adopted build both wrote a valid intent in all 13 pairs, and neither passed a hidden suite · Unresolved · 3 November 2026
- Kogen's re-landed data-stores tool wrote fewer valid intents than the adopted build, by the narrowest margin our rule counts · No · 3 November 2026
- Telling Kogen's shaper to decide reversible product questions itself did not get one stuck ticket to an Intent · No · 3 November 2026
- Moving Kogen's project setup step into its own component changed none of its code or prompt text · No · 3 November 2026
- Loop output line boundaries need a triggered confirmation · Unresolved · 3 November 2026
- Three-task Loop evidence wave · Unresolved · 3 November 2026
- Executor policy move cannot compile in the adopted assembly · Unresolved · 3 November 2026
- Coordinator validation feedback screen · Unresolved · 3 November 2026
- A readiness regression claim remains tentative after targeted repeats · Unresolved · 3 November 2026
- A bounded follow-up leaves the component update unqualified · No · 3 November 2026
- A coherent component update remains unqualified · Unresolved · 3 November 2026
- A coherent component update regresses task readiness · No · 3 November 2026
- A change to Kogen's review step that scopes its no-defect check made no difference in 2041 retained reviews · No · 3 November 2026
- Build changes remain unverified after an incomplete screen · Unresolved · 3 November 2026
- A bug in our Agents-on-Rails task converter would have hidden a correct solution's hidden checks on one new task; no published score is affected · No · 3 November 2026
- The adopted Kogen release saved a valid intent in 4 of 12 control runs and passed every hidden check in none · Unresolved · 3 November 2026
- Showing Kogen's builder a small repository's files in its first request cuts its reading steps to zero and build requests by 17%, but passed no more hidden tests (57 of 66 against 58) · No · 2 November 2026
- Kogen's shaper on Claude Opus 5.5 through Claude Code could not be judged: the test stopped after 47 pairs ran with both arms, too few to tell · Unresolved · 2 November 2026
- Kogen's data-stores change was refused at shaping again on a third machine: 0 of 8 valid intents pooled, against 8 of 8 for the build before it · Unresolved · 2 November 2026
- Kogen's data-stores change was refused at shaping in 4 of 4 account-cleanup runs, so its coverage is unmeasured · Unresolved · 2 November 2026
- Kogen split into component modules behaved like the build before it in three matched runs · Unresolved · 2 November 2026
- Letting Kogen's loop wait 45 seconds for shell commands passes the same hidden tests (55 of 78 each) and saves no requests on these tasks · No · 2 November 2026
- Frozen Kogen passed every hidden check in 0 of 18 runs on six Rails tasks, and shaped a valid intent in 6 · Unresolved · 2 November 2026
- Four more runs per task on one machine: current Kogen shaped a valid intent in 3 of 14 counted runs and passed every hidden check in none · Unresolved · 2 November 2026
- Current Kogen passed every hidden check in 1 of 18 runs on six Rails tasks, and shaped a valid intent in 8 · Unresolved · 2 November 2026
- A second block of Kogen on Claude Opus 5.5 passed every hidden check in 4 of 12 runs on six Rails tasks, and 11 of 12 wrote a valid intent · Unresolved · 2 November 2026
- With mise's trust setting applied, Kogen on Claude Opus 5.5 passed every hidden check in 1 of 6 runs on a third server, and 3 runs got no valid intent · Unresolved · 2 November 2026
- With mise's trust setting applied, Kogen on Claude Opus 5.5 passed every hidden check in 2 of 12 runs on six Rails tasks, and 7 runs were still refused at shaping · Unresolved · 2 November 2026
- Kogen on Claude Opus 5.5, third server: 3 of 6 runs landed a build and 1 passed every hidden check; pooled 2 of 18 · Unresolved · 2 November 2026
- Kogen on Claude Opus 5.5 passed every hidden check in 1 of 12 runs on six Rails tasks, and every valid intent it shaped landed · Unresolved · 2 November 2026
- A test-authoring paragraph in Kogen's builder prompt did not shorten builds on four Go tasks · No · 1 November 2026
- Kogen's new shaping example check refuses good specifications and loses 7 hidden passes in 65 shapes · No · 1 November 2026
- Turning on the request audit in Kogen shaping lost badly to unpatched Kogen over two Go repetitions and stays dropped · No · 1 November 2026
- The stop-when-checks-pass sentence in Kogen's builder prompt lost to unpatched Kogen over two Go repetitions · No · 1 November 2026
- The patched builder's losses were caused by defects written before it started, not by finishing early · Unresolved · 1 November 2026
- A coverage ledger in Kogen shaping did not earn its keep over two Go repetitions and is dropped · No · 1 November 2026
- A coverage-ledger patch for Kogen showed no gain on six hard Rails tasks, where no run passed · No · 1 November 2026
- An unstated-requirement-pass patch for Kogen showed no gain on six hard Rails tasks, where no run passed · No · 1 November 2026
- Updating Kogen's built-in loop to tonight's fixes removes refused shell calls but shows no gain in hidden passes (32 vs 31 of 52) · Unresolved · 1 November 2026
- High reasoning effort for Kogen's shaper passes no more hidden tests than medium (57 of 78 each) · No · 1 November 2026
- Kogen's current main passes more hidden tests than the build before its last nine commits (31 vs 23 of 39), mostly because the example check was removed · Yes · 1 November 2026
- Kogen's cold prompt-cache requests are conversation starts, and putting the testing rules first does not warm them · No · 1 November 2026
- Kogen's newer build matched the previous build on 13 Go tasks over three repetitions · Unresolved · 1 November 2026
- Kogen's newer build with the delivered fixes did no worse than the previous build on 13 Go tasks · Unresolved · 1 November 2026
- Which of six ways to reuse or skip checks in Kogen builds could save time · Unresolved · 31 October 2026
- Three shaping changes screened: the second-host half was invalid, the valid half shows no gain · Unresolved · 31 October 2026
- A clearer refusal message for shaping changed nothing: both versions accepted 25 of 26 · No · 31 October 2026
- Allowing only one resubmission broke Kogen's shaping step; allowing five matched the default · No · 31 October 2026
- A cheaper model at medium effort could not get Kogen's shaping step accepted (7 of 78) · No · 31 October 2026
- Switching on Kogen's request audit lost four net tasks on 13 Go tasks, so the patch was dropped · No · 31 October 2026
- A stop-when-checks-pass sentence in Kogen's builder prompt held the hidden result and cut effort on the first 13 Go tasks · Yes · 31 October 2026
- A coverage ledger in Kogen shaping did not show a gain on the first 13 Go tasks · Unresolved · 31 October 2026
- Adding review probes to Kogen lost two net tasks on 13 Go tasks, so the patch was dropped · No · 31 October 2026
- An unstated-requirement pass in Kogen shaping did not show a gain on the first 13 Go tasks · Unresolved · 31 October 2026
- An unstated-requirement pass in Kogen shaping lost to unpatched Kogen over two Go repetitions and is dropped · No · 31 October 2026
- Where the time goes in Kogen builds and shaping sessions · Unresolved · 31 October 2026
- A prototype that skips an already-passed starting check: safe, but small headroom · Unresolved · 31 October 2026
Kogen shaping component
- Setup evidence in contract review remains unproven · Unresolved · 8 November 2026
- Historical and current shaping releases · Unresolved · 8 November 2026
- Screen of shaping evidence and probe handling · No · 8 November 2026
- Evidence-path guidance before and after rejection · No · 8 November 2026
- Decision check and shaping readiness · Unresolved · 8 November 2026
- Combined evidence and transient-output guidance · No · 8 November 2026
- Guidance for carrying shaping decisions · No · 8 November 2026
- Variation in shaping readiness · Unresolved · 8 November 2026
- Shaping guidance for assumptions and evidence · No · 8 November 2026
- Existing-behavior evidence in shaping · No · 7 November 2026
- Initial evidence-path instruction in shaping · No · 7 November 2026
- Evidence-path instructions in shaping · No · 7 November 2026
- Deciding instead of asking shortens shaping's question loop but does not produce Intents · No · 7 November 2026
- Shaping readiness on a fixed request set · Unresolved · 7 November 2026
- Preventing transient bytecode in shaping checks · No · 7 November 2026
- Variation in shaping readiness · Unresolved · 7 November 2026
- Naming the decision line and controls.json gets Rails shapes through shaping · Yes · 7 November 2026
- Naming the gate forms in the shaper prompt gets small requests through shaping; the Rails tasks stop at later gates · No · 6 November 2026
- The review-size bound counts whole files the review never carries, and the one change that passes it removes the gate · No · 6 November 2026
- The decision-record gate costs one shaping attempt, but other gates end these shapes before readiness · No · 6 November 2026
- The second shaper text changes leave all six cells at the 30-minute deadline, with the first draft using most of it · Unresolved · 6 November 2026
- The shaper prompt change gets one of five refused tasks to readiness; the rest stop at prose statements and later gates · No · 6 November 2026
- Request-only shaping refusals are mostly prompt-versus-gate contradictions, and gate-side changes alone turn 2 of 5 · No · 6 November 2026
- Handoff questions stopped shaping before the question check, not after · No · 5 November 2026
- A model-free scope inventory names every owner-keyed store the hidden checks touch; the counted shaper cells were not run · Unresolved · 5 November 2026
- Static checks on shaped files stop 1 of 22 defects the builder refuses · No · 5 November 2026
- Shaping recovery on Claude: one rescue, otherwise a tie (tentative) · Unresolved · 5 November 2026
- Shaping recovery on Claude: second smoke run, unresolved · Unresolved · 5 November 2026
- Bounded shaping recovery confirmation · Unresolved · 4 November 2026
- Repair after shaping refusals remains tentative · Unresolved · 4 November 2026
Kogen build component
- A behaviour projection leaves every recorded false red of the budget-off test outside it · Yes · 5 November 2026
- Slow checks re-run on an unchanged tree: little to save, and already shipped · No · 5 November 2026
- Changed-first checks in build rounds: the targeted test run is not cheap enough · No · 5 November 2026
- An allow-list for the budget-off consumer test: not measured · Unresolved · 5 November 2026
- Build component check on Claude stays unresolved · Unresolved · 5 November 2026
- Extracting one release file cuts the biggest MacBook test's temp files by 46% · Yes · 5 November 2026
- Sharing a source copy does not cut MacBook temp-file load · No · 5 November 2026
- Verify timeouts and land-red under load stay unresolved · Unresolved · 5 November 2026
- Shared Go lock for small suites stays unresolved · Unresolved · 5 November 2026
- Build component head with a fixed Intent · Unresolved · 4 November 2026
- Build main head on Claude: no clear difference (tentative) · Unresolved · 4 November 2026
- Build head paired screen remains tentative · Unresolved · 4 November 2026
Kogen review component
- Why two review verdicts were refused · Unresolved · 8 November 2026
- A refused review submission should not make its requirement texts binding · Yes · 6 November 2026
- No review arm of the wide-change Lab passes Stage 0; today's review already cuts 9 parts to 3 · No · 5 November 2026
- Review component check on Claude stays unresolved · Unresolved · 5 November 2026
- Review component head with a fixed Intent · Unresolved · 4 November 2026
- Review head regression not confirmed · No · 4 November 2026
- Review component on Claude: no difference attributable (tentative) · Unresolved · 4 November 2026
- Saved-candidate reviews after four changes · Unresolved · 3 November 2026
Kogen verification component
- A shaper's probe and the readiness run used different environments · Yes · 7 November 2026
- Archiving per-Intent step files leaves the spec trace unchanged, but many of them already name tests that no longer exist · No · 5 November 2026
- The replay's Quint batch is fast, but its generator rejects 7 of 17 valid acceptance runs · No · 5 November 2026
- On Linux the wall-time deadline test does not fail; its pre-setup work is 4x shorter than on the Mac · Yes · 5 November 2026
- The wall-time deadline test starts its clock too early; the loaded run did not happen · Yes · 5 November 2026
- Isolated, parallel-safe generated acceptance: no detector with a payoff passes · No · 5 November 2026
- A probe copy with git history lets git-reading observers pass · Yes · 5 November 2026
- Reaping while the check runs ends the Linux helper's hangs, with no orphans · Yes · 5 November 2026
- Linux check helper leaves zombies; reaping while the check runs fixes it in a first run · Unresolved · 5 November 2026
Kogen Context component
- A new-nonce instruction detected 8 of 8 forced compactions in a small Claude Code test · Yes · 5 November 2026
- Context packet shows no time-to-patch gain in a screen · Unresolved · 5 November 2026
- Packet-only localization on 11 more cases · No · 5 November 2026
- Detecting compaction without a model: a transcript watcher always works, token drop only on long sessions · Yes · 5 November 2026
- A brief marker did not detect compaction; the hook did · No · 5 November 2026
- Hook-free compaction detection: markers fail, a self-check works, the hook needs the launcher · Unresolved · 5 November 2026
- Packet-only localization over the API · Unresolved · 4 November 2026
- Context packet did not speed up four tasks · Unresolved · 4 November 2026
Source summary by family
F01 · Shape, Intent and planOpen source summary and related recordsF02 · Isolation, landing and interfaceOpen source summary and related recordsF03 · Harness and loopOpen source summary and related recordsF04 · Models, efforts and routingOpen source summary and related recordsF05 · Context, Jev and cacheOpen source summary and related recordsF06 · Checks, review and recoveryOpen source summary and related recordsF07 · Tasks, grading and methodOpen source summary and related recordsF08 · Language and executable specOpen source summary and related recordsF09 · Campfire and web stacksOpen source summary and related recordsF10 · Agent-facing outputOpen source summary and related recordsF11 · Claimed cross-cutting resultsOpen source summary and related recordsF12 · Explicit lever and recipe armsOpen source summary and related recordsF13 · Late-September product mechanismsOpen source summary and related recordsF14 · Context-store and task policyOpen source summary and related records