Published result
A coherent component update regresses task readiness
A plain-language summary comes first, followed by the full text of the result's reproduction pack.
Result summary and pack text · reproduce/kogen-coherent-main-a1a0445/README.md
- Finding id:
kogen-coherent-main-a1a0445 - Verdict: No
- Finished: 3 November 2026
- Comparison: Kogen shaping and delivery-build speed
- Reproduction pack: reproduce/kogen-coherent-main-a1a0445/ (data, analysis script and rerun steps)
- Category page: categories/harness-levers.md
In plain terms
Candidate-only Terra medium screen of the current coherent component mains: 2/9 valid Intents, 2/9 landed, 0/9 hidden all-passes. Registered reading: WORSE. Task-level harms and both timeout sensitivities remain visible. No individual component effect is claimed.
Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.
Full text of the reproduction pack
The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).
The treatment is Kogen `a1a04454a891a8a8b823d5cdba5c77ab0301cc28` assembled with every component revision linked in source-commits.json. The tuple includes the shaping commit containing shape-repair-before-terminal-submit. No source patch or Kogen re-pin was made.
Nine full-pipeline tasks were assigned, six targets and three held-outs. The targets are paired individually against stored outcomes; held-outs have no matching stored cells and are descriptive. The three shape-only repetitions use the same registered export input and evaluator. Candidate roles use gpt-6.1-terra medium via native OpenAI. Intent validity, landing, hidden pass, review and acceptance separation are separate outcomes; acceptance separation was not independently replayed.
The registered rule applies per-task harm guards and the mean threshold, and shows primary readings with and without stored approval timeouts. The result is WORSE. The sample is small and historical, with host/toolchain variation; it supports no significance or general capability claim.
Rerun
Patch: none; the treatment is the commit tuple linked in source-commits.json. Run python3 report.py to replay the aggregate result from data.json. To repeat the experiment, use the registered candidate-only lanes protocol and qualified task kits with those linked commits, gpt-6.1-terra medium, and the registered rule. source-commits.json links every public source revision. A fresh measurement uses the existing qualified task kits, hidden evaluators and candidate-only lanes workflow with the registered assignment, model, effort and rule. Task hashes and raw cell receipts stay in private execution records; this pack contains no tasks, grader data, source copies, patches, hosts, accounts, paths or cell IDs.
Accounting
Cell wall includes failed attempts. Cached input is part of input; reasoning is part of output. Dollar cost is unpriced, not zero. Orchestration usage is unmetered. Historical missing hidden grades remain explicit in each stored denominator.