# Kogen Bench · bounded agent guide Data revision: 607dfaf490e2e2bb13f78470ee8962028e2ed179 Source: https://github.com/KogenAI/kogen-bench/tree/607dfaf490e2e2bb13f78470ee8962028e2ed179 Numeric values copied from source narrative remain source-reported. Claim values are transcribed from the claim ledger with their original status and evidence label. No results are recomputed or pooled here. ## Start here - [Findings](https://bench.kogen.dev/findings/) - [Hypotheses](https://bench.kogen.dev/hypotheses/) - [Experiments](https://bench.kogen.dev/rounds/) - [Comparisons](https://bench.kogen.dev/comparisons/) - [Evidence map](https://bench.kogen.dev/evidence/) - [Publication validation](https://bench.kogen.dev/publication-validation/) - [Methods](https://bench.kogen.dev/methods/) - [Reproduce](https://bench.kogen.dev/reproduce/) - [Verify published numbers](https://bench.kogen.dev/verify/) - [Rerun and regrade tasks](https://bench.kogen.dev/rerun/) - [Credits](https://bench.kogen.dev/credits/) - [Data](https://bench.kogen.dev/data/) ## Publication evidence status - B1 (resolved): All 5,020 captured graded IDs now match unique rows in the refreshed public official grade export by exact cell ID. The 128 post-cut rows were appended from the authoritative ledger; 25 invalid control_apply rows are controls, not model failures. - B2 (resolved): The locally sanitized v1.2 snapshot preserves cited content from original revision 1118f7fc0dbb042af8c8de2ffd1b85768cf9e2a0 and uses public-release sanitization revision 30d9b25e4eefa7117121de44298fed504b429f4e. Its published-byte SHA-256 manifest verifies, and all 134 pinned EVIDENCE-MAP fragments resolve locally. The decision ledger is absent, so comprehensive decision-to-clause coverage remains limited. - The refreshed public official grade export exact-joins all 5,020 of 5,020 captured graded IDs. Its 128 post-cut additions comprise 84 model PASS, 19 model FAIL, and 25 invalid control_apply rows (result=invalid/outcome=fail); controls are not model failures. Publication gate states are listed above; B2 locally verifies the pinned v1.2 specification fragments cited by EVIDENCE-MAP.md. The absent decision ledger means comprehensive decision-to-clause coverage remains limited. - [GRADE-JOIN.md](https://github.com/KogenAI/kogen-bench/blob/607dfaf490e2e2bb13f78470ee8962028e2ed179/results/GRADE-JOIN.md) documents the refreshed exact public matches and post-cut row classification. - [Full validation report](https://bench.kogen.dev/publication-validation/) ## Findings ### F01 — Shape, Intent and plan Source summary: https://bench.kogen.dev/findings/f01/ ### F02 — Isolation, landing and interface Source summary: https://bench.kogen.dev/findings/f02/ ### F03 — Harness and loop Source summary: https://bench.kogen.dev/findings/f03/ ### F04 — Models, efforts and routing Source summary: https://bench.kogen.dev/findings/f04/ ### F05 — Context, Jev and cache Source summary: https://bench.kogen.dev/findings/f05/ ### F06 — Checks, review and recovery Source summary: https://bench.kogen.dev/findings/f06/ ### F07 — Tasks, grading and method Source summary: https://bench.kogen.dev/findings/f07/ ### F08 — Language and executable spec Source summary: https://bench.kogen.dev/findings/f08/ ### F09 — Campfire and web stacks Source summary: https://bench.kogen.dev/findings/f09/ ### F10 — Agent-facing output Source summary: https://bench.kogen.dev/findings/f10/ ### F11 — Claimed cross-cutting results Source summary: https://bench.kogen.dev/findings/f11/ ### F12 — Explicit lever and recipe arms Source summary: https://bench.kogen.dev/findings/f12/ ### F13 — Late-September product mechanisms Source summary: https://bench.kogen.dev/findings/f13/ ### F14 — Context-store and task policy Source summary: https://bench.kogen.dev/findings/f14/ ## Hypothesis register - H01 (F01; UNRESOLVED; available same-model summaries are not a joined public comparison.): On matched tasks and model settings, the full Kogen Shape/Build/Review route has a higher pass rate than direct Codex. — https://bench.kogen.dev/hypotheses/h01/ - H02 (F01; UNRESOLVED; no public causal estimate isolates shaping.): Shaping improves final task completion enough to offset its added cost and time. — https://bench.kogen.dev/hypotheses/h02/ - H03 (F01; NOT ESTABLISHED; no validated guarantee cohort is available.): For a defined population of approved Intents that pass specified probes, Build completion can approach the declared reliability target. — https://bench.kogen.dev/hypotheses/h03/ - H04 (F01; UNRESOLVED; no public duration-by-human-input comparison.): A bounded Shaping time and human-input budget can minimize total cost while preserving completion and resolving material unknowns. — https://bench.kogen.dev/hypotheses/h04/ - H05 (F01; UNRESOLVED; no isolated public contrast.): Running probes that settle stated assumptions before Build reduces failed or stopped Builds against a matched no-probe condition. — https://bench.kogen.dev/hypotheses/h05/ - H06 (F01; UNRESOLVED; no public granularity comparison.): A task-granularity rule for Intents improves completion per unit of integration and coordination cost over alternative sizes. — https://bench.kogen.dev/hypotheses/h06/ - H07 (F01; UNRESOLVED; related records are exploratory or have unreconciled outcome joins.): More specific Intent detail raises builder success, with a larger effect for weaker builders than for stronger builders. — https://bench.kogen.dev/hypotheses/h07/ - H08 (F01; UNRESOLVED; no joined public route comparison.): A compact context pack and a strong-model plan allow a cheap builder to outperform its matched direct baseline without exceeding a direct strong-model cost target. — https://bench.kogen.dev/hypotheses/h08/ - H09 (F01; NOT ESTABLISHED; no complete public comparison identified.): One strong-model plan can guide multiple Luna-low builders and a low-cost integrator with equal or higher completion at lower total cost than a matched alternative. — https://bench.kogen.dev/hypotheses/h09/ - H10 (F01; NOT ESTABLISHED; adjacent escalation records do not test this advisor contrast.): Giving a weak builder access to a stronger advisor after a defined failure state improves completion per total cost over retrying or restarting with a strong model. — https://bench.kogen.dev/hypotheses/h10/ - H11 (F01; UNRESOLVED; cascade records do not yield a reconciled public verdict.): Escalation triggered by observed failure matches or exceeds text-based pre-routing on completion while reducing total cost. — https://bench.kogen.dev/hypotheses/h11/ - H12 (F01; NOT ESTABLISHED; no prospective repair-benefit estimate.): Checks, review, and actionable retries increase final completion over a reject-only gate under matched tasks and attempt budgets. — https://bench.kogen.dev/hypotheses/h12/ - H13 (F01; NOT ESTABLISHED; no matched public comparison identified.): Two complementary shapers resolve failures missed by either single shaper at an acceptable added cost. — https://bench.kogen.dev/hypotheses/h13/ - H14 (F01; UNRESOLVED; available R74 record is WITHDRAWN.): Sol planning with Luna-max building improves the success-cost frontier over an all-Luna route on matched tasks. — https://bench.kogen.dev/hypotheses/h14/ - H15 (F01; NOT ESTABLISHED; no controlled outcome comparison.): More complete Shaping plus executable specifications improves success or cost over a matched direct Codex or Grok Build. — https://bench.kogen.dev/hypotheses/h15/ - H16 (F02; NOT ESTABLISHED; no matched design comparison.): Under matched concurrent project workloads, worktree, copy-on-write, or another isolation design reduces cross-project interference and failure without unacceptable setup or Build-time cost. — https://bench.kogen.dev/hypotheses/h16/ - H17 (F02; NOT ESTABLISHED; no comparative reliability or cost study.): Rebase-based landing produces fewer unwanted base-history changes or recovery errors than automatic merge while remaining within a declared time and conflict budget. — https://bench.kogen.dev/hypotheses/h17/ - H18 (F02; NOT ESTABLISHED; no measured usability or runtime comparison.): A headless command-driven Shaping interface improves external-driver completion, latency, or error recovery over a Kogen TUI under the same tasks. — https://bench.kogen.dev/hypotheses/h18/ - H19 (F03; UNRESOLVED; harness-pick records do not reconstruct the scored cohort.): On matched ChatGPT models, task IDs, prompts, tools, and effort, Pi and Codex differ in success, time, cost, or cache use by a reproducible amount. — https://bench.kogen.dev/hypotheses/h19/ - H20 (F03; NOT ESTABLISHED; no complete comparable matrix.): Under matched prompts and tool access, at least one of Pi, OpenCode, Grok Build, Claude Code, or Codex has a better predeclared success-cost-time profile than the alternatives. — https://bench.kogen.dev/hypotheses/h20/ - H21 (F03; UNRESOLVED; associated receipts are interim or confounded.): A tuned custom Kogen harness matches or exceeds the selected external harness on the same task/model population and cost rule. — https://bench.kogen.dev/hypotheses/h21/ - H22 (F03; NOT ESTABLISHED; no joined end-to-end comparison.): End-to-end Kogen improves task completion enough to offset any slower loop overhead relative to a standalone harness. — https://bench.kogen.dev/hypotheses/h22/ - H23 (F03; NOT ESTABLISHED; Rust khr comparison is not established in the public receipt.): Elixir kh and Rust khr differ measurably in startup, tool-call, file-write, integration, or total Build time under equivalent work. — https://bench.kogen.dev/hypotheses/h23/ - H24 (F03; NOT ESTABLISHED; no isolated test identified.): Removing TUI code and using a command-only loop reduces runtime or latency while preserving task success. — https://bench.kogen.dev/hypotheses/h24/ - H25 (F03; NOT ESTABLISHED; no public one-factor prompt result.): A lean coding-specific system prompt lowers tokens or improves cache use without reducing task success. — https://bench.kogen.dev/hypotheses/h25/ - H26 (F03; NOT ESTABLISHED; no paired public comparison.): Replacing Claude Code’s prompt and constraining its tools changes its task outcomes in a way that separates harness effects from prompt/tool effects. — https://bench.kogen.dev/hypotheses/h26/ - H27 (F03; NOT ESTABLISHED; no completed matched comparison.): A Claude Code -p wrapper preserves the relevant subscription route while improving Kogen control without unacceptable quality or latency loss. — https://bench.kogen.dev/hypotheses/h27/ - H28 (F03; UNRESOLVED; available candidate records do not establish a scored cohort.): Claude Code’s task quality offsets its Kogen loop maintenance cost compared with a native loop. — https://bench.kogen.dev/hypotheses/h28/ - H29 (F03; UNRESOLVED; exploratory source summaries are not joined to exact public outcomes.): Parallel subagents in Shape, Develop, or Review improve completion or wall time enough to offset their added calls and tokens versus one editing agent. — https://bench.kogen.dev/hypotheses/h29/ - H30 (F03; NOT ESTABLISHED; no public non-inferiority comparison.): Omitting helper agents does not materially reduce task completion and lowers cost or complexity under a declared margin. — https://bench.kogen.dev/hypotheses/h30/ - H31 (F03; NOT ESTABLISHED; no isolated affordance ablation.): Mid-turn steering, external tool access, and recovery affordances explain a measurable portion of Pi/Codex quality differences. — https://bench.kogen.dev/hypotheses/h31/ - H32 (F03; UNRESOLVED; the public kh-climb receipt is partial and lacks exact cell joins.): A mechanical verification gate after the final edit reduces unverified completion without unacceptable cost or false blocking. — https://bench.kogen.dev/hypotheses/h32/ - H33 (F03; UNRESOLVED; exploratory surface records are partial.): Loop verbosity, output format, or prompt changes account for a measurable part of harness outcome differences. — https://bench.kogen.dev/hypotheses/h33/ - H34 (F03; UNRESOLVED; historical pooled comparison is not fully reconciled.): Shell-only editing tools outperform apply-patch-style tools on a matched task set under a predeclared completion and cost rule. — https://bench.kogen.dev/hypotheses/h34/ - H35 (F03; UNRESOLVED; public rows do not establish the source-reported comparison.): A strong plan plus shell builder matches direct Sol while reducing total cost on matched tasks. — https://bench.kogen.dev/hypotheses/h35/ - H36 (F03; NOT ESTABLISHED; no clean loop-isolation comparison.): Kogen’s own loop differs from Codex embedded as its builder after the builder model, tasks, and stage policy are held constant. — https://bench.kogen.dev/hypotheses/h36/ - H37 (F03; NOT ESTABLISHED; the formal Grok frontier has no run result.): A native Grok provider in Kogen gives a lower-cost or higher-success route than the matched alternative, including Grok Build as a separate comparator. — https://bench.kogen.dev/hypotheses/h37/ - H38 (F03; NOT ESTABLISHED; no comparative subscription matrix.): An alternative provider or subscription has a better cost-success profile than current subsidized options under matched tasks and limits. — https://bench.kogen.dev/hypotheses/h38/ - H39 (F04; UNRESOLVED; no complete role-by-role public comparison.): A role-specific model/effort allocation improves the declared completion-cost-time frontier over a fixed allocation on matched tasks. — https://bench.kogen.dev/hypotheses/h39/ - H40 (F04; NOT ESTABLISHED; model preference alone is not a measured comparison.): The newer Luna model has a better cost-success profile than the earlier Luna baseline on the same tasks and harness. — https://bench.kogen.dev/hypotheses/h40/ - H41 (F04; UNRESOLVED; no full difficulty-stratified effort curve.): Increasing Luna effort improves completion more on harder tasks and yields a measurable cost tradeoff. — https://bench.kogen.dev/hypotheses/h41/ - H42 (F04; UNRESOLVED; associated screens are exploratory.): The compared Sol, Luna, and Astra settings have distinguishable success, time, or cost profiles when builder role and tasks are held fixed. — https://bench.kogen.dev/hypotheses/h42/ - H43 (F04; NOT ESTABLISHED; no isolated role comparison.): The Sol low-effort setting outperforms Luna max in shaping, review, or synthesis roles enough to justify added cost. — https://bench.kogen.dev/hypotheses/h43/ - H44 (F04; NOT ESTABLISHED; no exact public comparison.): Luna low plus strong-model assistance exceeds matched direct Luna-low and direct-Sol options on a declared cost/time-success frontier. — https://bench.kogen.dev/hypotheses/h44/ - H45 (F04; UNRESOLVED; the public R66 record does not preserve the historical high-effort anchor cohort.): Sol medium has a higher pass rate than Codex on a predeclared hard-task population. — https://bench.kogen.dev/hypotheses/h45/ - H46 (F04; NOT ESTABLISHED; a small screen is insufficient for a policy.): Astra is not cost-effective for routine tasks but improves outcomes enough on a capped high-value task set to justify a targeted route. — https://bench.kogen.dev/hypotheses/h46/ - H47 (F04; UNRESOLVED; cascade summaries are not reconciled.): Escalating after an observed failure beats fixed direct models or text-based routing under equal task and cost rules. — https://bench.kogen.dev/hypotheses/h47/ - H48 (F04; NOT ESTABLISHED; offline routing agreement is not live Build success.): Routing from task text has worse accuracy or cost than a matched policy using observed outcomes or context. — https://bench.kogen.dev/hypotheses/h48/ - H49 (F04; NOT ESTABLISHED; operating preference is not a measured optimum.): Luna max for most implementation/operator work with Sol reserved for planning/rules is on the declared cost-success frontier. — https://bench.kogen.dev/hypotheses/h49/ - H50 (F04; NOT ESTABLISHED; available panels are not one matched frontier test.): A combined plan-level, role, reviewer, and contract route beats direct Sol on a preregistered completion-cost frontier. — https://bench.kogen.dev/hypotheses/h50/ - H51 (F05; UNRESOLVED; historical cache summaries use incompatible definitions and epochs.): Cache reuse, lean prompts, or context management reduces Kogen loop tokens or cost more than other tested runtime changes while preserving task success. — https://bench.kogen.dev/hypotheses/h51/ - H52 (F05; NOT ESTABLISHED; the public smoke did not meet the target.): A stable cache key/session affinity and stable prompt prefix raise the measured live hit rate to the declared gate on repeated requests. — https://bench.kogen.dev/hypotheses/h52/ - H53 (F05; NOT ESTABLISHED; no public compatible accounting table.): Cached, uncached, and input-token definitions can be reconciled across Kogen, Codex, and language arms to support a fair cost comparison. — https://bench.kogen.dev/hypotheses/h53/ - H54 (F05; UNRESOLVED; available trace records report no actual compactions.): A Jev classifier selects useful transcript lines for compaction with lower latency and no material loss of task-relevant context. — https://bench.kogen.dev/hypotheses/h54/ - H55 (F05; UNRESOLVED; source screens are not joined to public Build outcomes.): Deterministic retrieval plus Jev and Luna readers produces more useful source-grounded packets for a stronger worker at lower total cost than the baseline. — https://bench.kogen.dev/hypotheses/h55/ - H56 (F05; NOT ESTABLISHED; no public paired rescue test.): A bounded packet of code relationships rescues failures beyond what file retrieval alone achieves on matched cases. — https://bench.kogen.dev/hypotheses/h56/ - H57 (F05; NOT ESTABLISHED; offline diagnostics do not measure Build efficacy.): Prepared task-specific context improves Jev routing/classification accuracy over a bare-context baseline and improves downstream Build decisions. — https://bench.kogen.dev/hypotheses/h57/ - H58 (F05; UNRESOLVED; role inventory is incomplete and no general role winner is measured.): At least one Jev role among audit, routing, classification, review, compaction, or context preparation improves a predeclared outcome at acceptable cost. — https://bench.kogen.dev/hypotheses/h58/ - H59 (F05; UNRESOLVED; the public R54 receipt is interim and does not establish token savings.): Passing a planner’s context to the builder reduces builder tokens without reducing matched-task success. — https://bench.kogen.dev/hypotheses/h59/ - H60 (F05; NOT ESTABLISHED; Tidewave data is a partial diagnostic, not a live comparative result.): Runtime code inspection yields more useful task evidence than static retrieval on a fixed checkpoint and improves task outcomes. — https://bench.kogen.dev/hypotheses/h60/ - H61 (F05; NOT ESTABLISHED; no reconciled public control table.): Language comparisons hold cache accounting, prompt, and human steering constant well enough to attribute outcome differences to language. — https://bench.kogen.dev/hypotheses/h61/ - H62 (F06; NOT ESTABLISHED. No public matched before/after result measures both failure detection and work or time.): A selective, faster check set can preserve detection of real failures while reducing check time or cost. — https://bench.kogen.dev/hypotheses/h62/ - H63 (F06; NOT ESTABLISHED. No public controlled catalog or concurrency comparison is identified.): A catalog of safe parallel tests can reduce verification wall time without changing results. — https://bench.kogen.dev/hypotheses/h63/ - H64 (F06; NOT ESTABLISHED. No matched mechanics-test versus full-check result is published.): Targeted tests of Build mechanics can preserve defect detection at lower cost than repeated full shaping checks. — https://bench.kogen.dev/hypotheses/h64/ - H65 (F06; NOT ESTABLISHED. Public records include blocked and smoke-only workflows, but no isolated live recovery comparison.): Clear handoff, process custody, failure classes, and visible stage status can prevent repeated expensive Build attempts. — https://bench.kogen.dev/hypotheses/h65/ - H66 (F06; NOT ESTABLISHED. No check-ablation comparison is joined to official outcomes.): Strong deterministic checks can enable a low-effort model to complete tasks that weaker checks miss. — https://bench.kogen.dev/hypotheses/h66/ - H67 (F06; INTERIM, OFFLINE ONLY. Reviewer diagnostics are incomplete and are not a prospective live veto test.): A scoped reviewer veto improves acceptance quality enough to justify its review and repair cost. — https://bench.kogen.dev/hypotheses/h67/ - H68 (F06; INTERIM, OFFLINE ONLY. The public summaries do not include the patch-level decision ledger.): Executable review catches small-diff false accepts that other review methods miss. — https://bench.kogen.dev/hypotheses/h68/ - H69 (F06; INTERIM, OFFLINE ONLY. The published audit did not run a repair or Build rerun.): Contract audits detect missing requested behavior and can safely gate completion or trigger repair. — https://bench.kogen.dev/hypotheses/h69/ - H70 (F06; NOT ESTABLISHED. Offline audits identify test-quality concerns, but do not compare those check designs prospectively.): Explicit behavior checks outperform source-text or file-existence checks without losing relevant coverage. — https://bench.kogen.dev/hypotheses/h70/ - H71 (F06; NOT ESTABLISHED. The study census lists check research, but no public default-gate portfolio comparison is available.): The proposed static and domain-specific checks are accurate and fast enough for the default gate. — https://bench.kogen.dev/hypotheses/h71/ - H72 (F06; NOT ESTABLISHED. No matched repair-loop outcome is identified in the public records.): A post-change fix loop can repair a static-check failure rather than reject functionally passing work. — https://bench.kogen.dev/hypotheses/h72/ - H73 (F06; PARTIAL, OFFLINE ONLY. Saved-output and candidate audits are counterfactual evidence, not a prospective grading-policy test.): Verification should score observable behavior even when an implementation reports its own checks as successful. — https://bench.kogen.dev/hypotheses/h73/ - H74 (F06; NOT ESTABLISHED. No cross-project transfer or maintenance comparison is published.): Reusable duplicate, error, and type checks provide value beyond the Kogen project. — https://bench.kogen.dev/hypotheses/h74/ - H75 (F07; INTERIM. Several desktop Mac hill-climb iterations are reported, but cell-level outcomes and the final iteration are not fully reconciled in the public export.): A cheap starting task and one-lever-at-a-time climb identifies useful task and recipe changes without mistaking saturation for general success. — https://bench.kogen.dev/hypotheses/h75/ - H76 (F07; NOT ESTABLISHED. No matched comparison for this starting-model choice is linked to a public claim.): Luna or Grok is the better low-cost, fast starting builder for the intended task set. — https://bench.kogen.dev/hypotheses/h76/ - H77 (F07; NOT ESTABLISHED. Cross-stack task and implementation records exist, but no matched task-set study isolates stack or recency effects.): A benchmark limited to Rails tasks can misstate performance on Elixir/Phoenix or Rust work. — https://bench.kogen.dev/hypotheses/h77/ - H78 (F07; NOT ESTABLISHED. Public task provenance is useful for reproduction, but no task-quality comparison tests the proposition.): Public real-codebase tasks pinned to source revisions are more reproducible than hidden replay tasks. — https://bench.kogen.dev/hypotheses/h78/ - H79 (F07; NOT ESTABLISHED. No controlled database-task concurrency comparison is published.): SQLite tasks allow more reliable parallel execution than PostgreSQL-heavy tasks while a smaller PostgreSQL set preserves realism. — https://bench.kogen.dev/hypotheses/h79/ - H80 (F07; PARTIAL METHOD EVIDENCE. Public records preserve invalid grades and control lanes, but do not quantify the operating characteristics of this review policy.): Independent review and wrong-solution and valid-alternative controls expose benchmark or grader errors before costly runs. — https://bench.kogen.dev/hypotheses/h80/ - H81 (F07; METHOD REQUIREMENT, NOT A PERFORMANCE RESULT. Public records do not establish a controlled host-effect estimate.): Wall-time comparisons require equivalent stable hosts and controlled load rather than a single remote development link. — https://bench.kogen.dev/hypotheses/h81/ - H82 (F07; NOT ESTABLISHED. No completed public balance audit validates this pairing rule.): Pairing equivalent workers is valid when host, load, concurrency, and provider epoch are recorded and balanced. — https://bench.kogen.dev/hypotheses/h82/ - H83 (F07; NOT ESTABLISHED. No public operating-characteristic or cost comparison validates the savings and error rate.): Preliminary grading and reuse of frozen baseline results can reduce new cells without changing a decision. — https://bench.kogen.dev/hypotheses/h83/ - H84 (F07; NOT ESTABLISHED. The public snapshot does not isolate provider epoch from host, load, or revision.): A provider-load epoch can explain cross-day timing changes even when the harness is unchanged. — https://bench.kogen.dev/hypotheses/h84/ - H85 (F07; NOT ESTABLISHED. No public simulation or prospective validation measures decision error and savings together.): Sequential allocation, saturation pairs, single-repetition screens, and futility stops reduce study cost while preserving decisions. — https://bench.kogen.dev/hypotheses/h85/ - H86 (F07; PARTIAL REPORTING EVIDENCE. Public pages and ledgers supply some of these fields, but unresolved joins and missing protocol pins remain.): Results with pinned versions, pass rate, wall time, cost, invalid cases, and full setup are reproducible and interpretable. — https://bench.kogen.dev/hypotheses/h86/ - H87 (F07; PARTIAL PROCESS EVIDENCE. Public pages distinguish preregistered, descriptive, withdrawn, and incomplete cohorts; a complete registration-history audit is not available here.): Design and decision rules recorded before execution, with exploratory results labelled, reduce retrospective reinterpretation. — https://bench.kogen.dev/hypotheses/h87/ - H88 (F07; NOT ESTABLISHED AS A COMPLETE POLICY. Several unresolved ITT dispositions remain in the public exports.): Intention-to-treat scoring counts internal stops as failures and excludes only demonstrated environment or adapter faults. — https://bench.kogen.dev/hypotheses/h88/ - H89 (F07; PARTIAL REPORTING EVIDENCE. The public run ledger exposes missing fields, so completeness is not established.): Recording task, arm, revision, host, load, timeout, attempts, setup, and token categories is sufficient for later reproduction. — https://bench.kogen.dev/hypotheses/h89/ - H90 (F07; NOT ESTABLISHED. The public frontier lane has no released outcome cohort.): Kogen can solve strategically chosen tasks that direct Codex does not solve. — https://bench.kogen.dev/hypotheses/h90/ - H91 (F08; DESCRIPTIVE ONLY. Public cohorts are small and incomplete, and some original tasks have implementation confounds.): A matched comparison can identify the best implementation language for the Kogen agent loop across task success, reliability, cost, and implementation time. — https://bench.kogen.dev/hypotheses/h91/ - H92 (F08; NOT ESTABLISHED. The public comparison does not hold all of those factors constant.): Rust has a better runtime and concurrency tradeoff than Elixir OTP when logic, prompts, and repair opportunities are equivalent. — https://bench.kogen.dev/hypotheses/h92/ - H93 (F08; NOT ESTABLISHED. No before/after architecture comparison is identified.): Separating Mix applications into an umbrella makes tests and the workflow faster. — https://bench.kogen.dev/hypotheses/h93/ - H94 (F08; PARITY METHOD, NOT EFFICACY. Conformance can test parity but does not show improved task completion.): Language-agnostic executable acceptance specifications preserve the same semantics across implementations. — https://bench.kogen.dev/hypotheses/h94/ - H95 (F08; NOT ESTABLISHED. No controlled live comparison is identified.): Quint specifications written during planning improve Kogen or direct-model outcomes. — https://bench.kogen.dev/hypotheses/h95/ - H96 (F08; NOT ESTABLISHED. No public translation-audit outcome is linked.): Reviewing the translation from natural-language requirements into Quint catches omitted behavior. — https://bench.kogen.dev/hypotheses/h96/ - H97 (F08; METHOD REQUIREMENT. Existing language cohorts do not establish full parity.): A fair language comparison holds tasks, prompts, model, effort, steering, grader, and repair loop constant. — https://bench.kogen.dev/hypotheses/h97/ - H98 (F08; NOT ESTABLISHED. No public inclusion-rule test or Gleam cohort is available.): Gleam should remain eligible until a defined matched-comparison rule supports excluding it. — https://bench.kogen.dev/hypotheses/h98/ - H99 (F08; PARTIAL MEASUREMENT. Public timing and resource files exist, but their measurements are descriptive and do not by themselves rank full implementations.): Compile time, run wall time, token use, and task grade should all inform the language choice. — https://bench.kogen.dev/hypotheses/h99/ - H100 (F09; NOT ESTABLISHED. Both public Campfire lanes were withdrawn before formal cells.): A parity-checked Elixir/Phoenix implementation can approach or beat the selected Rust implementation on a matched Campfire feature. — https://bench.kogen.dev/hypotheses/h100/ - H101 (F09; NOT ESTABLISHED. No qualifying matched feature cohort ran.): A small representative feature can be compared fairly across stacks using SQLite, matched optimization, and disabled caching. — https://bench.kogen.dev/hypotheses/h101/ - H102 (F09; NOT ESTABLISHED. No public matched adapter or implementation comparison is available.): A stronger Exqlite or Ecto implementation changes the relative runtime result. — https://bench.kogen.dev/hypotheses/h102/ - H103 (F09; NOT ESTABLISHED. No public correctness-control results are available for the Campfire comparison.): Throughput measurements reject implementations that drop messages or otherwise fail correctness. — https://bench.kogen.dev/hypotheses/h103/ - H104 (F09; NOT ESTABLISHED. No completed public review of the feature-selection rule is linked.): The chosen feature is representative, reviewable, and suitable for a public web-stack benchmark. — https://bench.kogen.dev/hypotheses/h104/ - H105 (F09; NOT ESTABLISHED. The public Elixir-port round does not establish a matched cross-stack agent comparison.): Rails agent tasks can be ported to Elixir or Rust so agent performance can be compared on equivalent tasks. — https://bench.kogen.dev/hypotheses/h105/ - H106 (F10; NOT ESTABLISHED. No complete, matched parse-and-repair comparison is published.): A compact structured format can reduce tokens while remaining easier for agents to parse than JSON, YAML, or TOML and more useful than plain text. — https://bench.kogen.dev/hypotheses/h106/ - H107 (F10; NOT ESTABLISHED. No public round-trip and live task comparison is linked.): Translating tool output into a compact format, with a fallback when needed, improves agent comprehension. — https://bench.kogen.dev/hypotheses/h107/ - H108 (F10; NOT ESTABLISHED. No controlled trace-format repair outcome is available.): Compact stack traces and error reports help an agent find and fix the correct code. — https://bench.kogen.dev/hypotheses/h108/ - H109 (F10; NOT ESTABLISHED. No matched logging-format study is published.): Structured logs improve diagnosis without removing useful detail. — https://bench.kogen.dev/hypotheses/h109/ - H110 (F10; INTERIM. A partial output-feedback cohort exists, but does not establish a format preference or completion benefit.): Agent-facing output rules should include only formats and checks with measured improvements. — https://bench.kogen.dev/hypotheses/h110/ - H111 (F11; UNRESOLVED; historical summary is source-reported and not reproducible from public data.): Staged and direct model comparison. On a shared task set, a staged Kogen route may help a weaker builder solve failures while direct Sol high may have a higher pass rate, lower cost per pass, or longer wall time. Falsify with exact task-matched outcomes, cost, and wall receipts for every arm. — https://bench.kogen.dev/hypotheses/h111/ - H112 (F11; UNRESOLVED; historical cost frontier is source-reported and not reproducible from public data.): Route cost and quality frontier. Across a shared task set, direct Sol, route-v3, and direct Luna max may occupy different pass-rate and cost-per-pass points. Falsify with one reconciled cohort, compatible cost accounting, and task-level outcomes. — https://bench.kogen.dev/hypotheses/h112/ - H113 (F11; INTERIM; R72 is descriptive and does not establish superiority.): R72 same-model comparison. On R72 P1, the Kogen Luna-max ladder may outperform direct Codex Luna max. Falsify with a complete cell join and a prespecified paired analysis. — https://bench.kogen.dev/hypotheses/h113/ - H114 (F11; CONFOUNDED / DESCRIPTIVE; language cohorts remain separate.): R70 language choice. The R70 implementation-stack comparison and selected RvE sample may identify a better language for Kogen. Evaluate with a parity-controlled comparison whose task, prompt, harness, and repair loop are matched; a selected sample alone cannot determine a broad language winner. — https://bench.kogen.dev/hypotheses/h114/ - H115 (F11; INTERIM; historical cache summaries are unreconciled; T98 is telemetry only.): Cache-hit explanation. A Kogen cache-hit rate below Codex's may explain much of the token-spend difference. Falsify with request-level records from matched epochs and compatible cache/token definitions, then relate cache changes to cost. — https://bench.kogen.dev/hypotheses/h115/ - H116 (F11; NOT-RUN; the T100 lane has no result.): Cache-change effect. A cache change may increase cache use. Falsify with a defined metric and baseline, matched request traffic, and before/after records. — https://bench.kogen.dev/hypotheses/h116/ - H117 (F12; Not established as a distinct run.): Planner context pack. A compact context pack may produce a better or cheaper plan than a file-list input. Compare the two inputs with all other planner and builder settings fixed. — https://bench.kogen.dev/hypotheses/h117/ - H118 (F12; Not established as a distinct run.): Plan guard. A plan sanity/constraint gate may prevent plans that would otherwise cause Build failures. Compare downstream outcomes and gate cost with and without the guard. — https://bench.kogen.dev/hypotheses/h118/ - H119 (F12; Not established as a distinct run.): Plan examples. Relevant examples may improve transfer from plan to implementation. Hold the plan and builder settings fixed while changing example exposure. — https://bench.kogen.dev/hypotheses/h119/ - H120 (F12; Not established as a distinct run.): Plan reproduction evidence. Evidence that reproduces a failure may improve the plan's diagnosis and implementation. Compare with an otherwise identical plan without that evidence. — https://bench.kogen.dev/hypotheses/h120/ - H121 (F12; Not established as a distinct run.): Plan interfaces. Explicit cross-component interfaces may reduce integration failures. Compare matched tasks with interfaces present or absent. — https://bench.kogen.dev/hypotheses/h121/ - H122 (F12; Not established as a distinct run.): Planner turns. Varying the number of planner turns may change completion enough to justify its time and token cost. Compare fixed turn budgets. — https://bench.kogen.dev/hypotheses/h122/ - H123 (F12; Not established as a distinct run.): Plan debate. Competing plans may improve plan choice and downstream task success. Compare debate against a single-plan control. — https://bench.kogen.dev/hypotheses/h123/ - H124 (F12; A planning-arm cohort appears in R51; its summary is withheld pending cell-level reconciliation.): Plan specificity. Criteria, approach, and step-level plans may affect builders differently. Compare each level within builder and task strata. — https://bench.kogen.dev/hypotheses/h124/ - H125 (F12; Not established as a distinct run.): Review scope. A narrower review surface may reduce cost while preserving useful findings. Compare findings, false blocks, missed defects, and review cost. — https://bench.kogen.dev/hypotheses/h125/ - H126 (F12; No live, prospective executable-review benefit is established.): Executable review. Running review checks may detect defects that text-only review misses. Measure true and false review findings and prospective repair outcomes. — https://bench.kogen.dev/hypotheses/h126/ - H127 (F12; No prospective Build benefit is established.): Contract gate. Checking requirements before acceptance or escalation may catch incomplete work. Compare detection and false-block rates, then measure Build outcomes after any repair. — https://bench.kogen.dev/hypotheses/h127/ - H128 (F12; No deployable best-of-N benefit is established.): Best-of-N. Generating and selecting among candidates may increase successful completion enough to justify the additional attempts. Compare selected outcomes with the same direct baseline and report pass@N separately. — https://bench.kogen.dev/hypotheses/h128/ - H129 (F12; Not established as a distinct run.): Hybrid pipeline. Combining staged modes may improve the pass/cost frontier over its constituent recipes. Compare the complete bundle and every constituent arm. — https://bench.kogen.dev/hypotheses/h129/ - H130 (F12; No split-specific outcome comparison is established.): Split work. Dividing implementation among workers may increase successful throughput. Compare a fixed split plan with a solo arm, including coordination cost and wall time. — https://bench.kogen.dev/hypotheses/h130/ - H131 (F12; The historical route-v3 summary is source-reported and not reproducible from public data.): Routing policies. Dynamic, forced, or versioned routing may assign useful models more effectively than fixed routes. Compare on the same tasks and model/effort choices. — https://bench.kogen.dev/hypotheses/h131/ - H132 (F12; No validated cascade policy ranking is established.): Route cascade. Triggering escalation only after failure may preserve quality at lower cost than always escalating. Compare policy-level success and cost against fixed direct models. — https://bench.kogen.dev/hypotheses/h132/ - H133 (F12; No final gate ranking is established.): Cascade gate. Contract, review, or combined gates may differ in their ability to trigger useful escalation. Compare gate sensitivity, false triggers, success, and cost. — https://bench.kogen.dev/hypotheses/h133/ - H134 (F12; R54 reports a handoff vs plain-pack comparison; its round status remains INTERIM.): Context handoff. Passing prepared context to later stages may reduce repeated work or cost without reducing completion. Compare a handoff arm with a matched no-handoff arm. — https://bench.kogen.dev/hypotheses/h134/ - H135 (F12; Multiple labels occur across rounds, but the public recipe-to-cell index is incomplete.): Recipe attribution. Staged, direct, direct-shell, direct-escalate, plan-shell, and escalate-shell recipes may differ in outcome and cost. Each needs an explicit arm manifest and direct within-cohort comparison. — https://bench.kogen.dev/hypotheses/h135/ - H136 (F12; Several exploratory ladder rounds ran; no general best ladder is established.): Ladder composition. Luna-only, Sol-effort, and diverse ladders may differ in success and cost. Compare each ladder under a fixed task set and trigger rule. — https://bench.kogen.dev/hypotheses/h136/ - H137 (F12; No controlled version comparison is established.): Harness-version effect. Changes in Codex CLI or Claude Code versions may change earlier harness comparisons. Compare versions on identical tasks, requests, and host conditions. — https://bench.kogen.dev/hypotheses/h137/ - H138 (F12; No complete current-version matched set is established.): Matched model/effort baselines. Kogen configurations with Luna max or Sol low/medium/high builders, with and without Sol planning, may differ from direct baselines. Match the effective model and effort at each tested role. — https://bench.kogen.dev/hypotheses/h138/ - H139 (F12; No exact raw-vs-provided arm join is established.): Raw versus shaped Intent. Supplying a raw external-project Intent instead of a provided shaped Intent may change Kogen's apparent advantage. Compare both inputs against the same task and model baselines. — https://bench.kogen.dev/hypotheses/h139/ - H140 (F13; NO EMPIRICAL EVIDENCE; no controlled comparison is identified.): Cross-harness route. Claude as the primary builder with Codex adversarial review may outperform an agent-only route on completion quality or cost. Compare the full route with a matched single-harness control. — https://bench.kogen.dev/hypotheses/h140/ - H141 (F13; NO EMPIRICAL EVIDENCE; no controlled comparison is identified.): Jev audit during Shaping. A controller-guaranteed Jev audit during Shaping may catch feasibility or context errors before approval. Compare audit findings, false alarms, and downstream Build outcomes with the audit disabled. — https://bench.kogen.dev/hypotheses/h141/ - H142 (F13; NO EMPIRICAL EVIDENCE; no controlled comparison is identified.): Vertical integration. Building and integrating the same requested feature may reduce wasted or brittle partial implementations compared with plumbing work that is not tied to an end-to-end Intent. Compare completion and rework on matched tasks. — https://bench.kogen.dev/hypotheses/h142/ - H143 (F13; NO EMPIRICAL EVIDENCE; no controlled comparison is identified.): Stashed work reuse. Reusing valid existing work may reduce Build time without increasing correctness failures. Compare against a clean-start control with the same acceptance suite. — https://bench.kogen.dev/hypotheses/h143/ - H144 (F13; NO EMPIRICAL EVIDENCE; no controlled comparison is identified.): Shaping preflight. A bounded, low-cost preflight audit may verify material risks before longer Shaping work. Compare audit cost, detected risks, false blocks, and Build outcomes against a matched no-preflight control. — https://bench.kogen.dev/hypotheses/h144/ - H145 (F13; INTERIM; no joined reliability or cost comparison is available.): Build reliability. Reliability changes may prevent repeated costly cycles and classify environment failures separately from semantic failures. Compare retries, successful completion, and total cost under injected and naturally occurring failure classes. — https://bench.kogen.dev/hypotheses/h145/ - H146 (F13; Source-reported topology diagnostic only; the proposed second-host workflow is untested.): Parallel Builds on a second Mac. A second Mac may raise throughput while preserving Build ownership and verification. Compare serial and parallel operation on the same eligible workload, including coordination, queue, and grading time. — https://bench.kogen.dev/hypotheses/h146/ - H147 (F13; NO EMPIRICAL EVIDENCE; no alert/control comparison is identified.): Build status and alerts. Showing current stage/check state and offering a completion alert may reduce unattended waiting and missed failures. Measure detection time, missed events, and operator wait; this is not a model pass-rate hypothesis. — https://bench.kogen.dev/hypotheses/h147/ - H148 (F14; OFFLINE, SOURCE-REPORTED; retrieval diagnostics do not measure Build success.): New context data structure. A new structure for code and domain knowledge may provide more bounded, explainable context than a conventional graph, retrieval-augmented generation, or vector store. Compare retrieval and downstream task outcomes using the same queries and packet budget. — https://bench.kogen.dev/hypotheses/h148/ - H149 (F14; OFFLINE, SOURCE-REPORTED; freshness diagnostics do not measure Build success.): Change-centered persistence. Persisting changes, queries, or transformations as primary objects may handle code and requirement evolution better than storing atomic facts and edges. Compare update effort, stale-answer rate, and retrieval quality after controlled edits. — https://bench.kogen.dev/hypotheses/h149/ - H150 (F14; OFFLINE, SOURCE-REPORTED; no matched live Build result is available.): Bounded uncertain context. A context store may produce bounded task packets while exposing uncertainty, stale observations, and exceptions after edits. Compare packet size, calibration, and task outcomes under controlled repository changes. — https://bench.kogen.dev/hypotheses/h150/ - H151 (F14; INTERIM; the frontier result is incomplete and is not the proposed Luna-medium screen.): Targeted Luna-medium screen. Luna medium on simpler tasks where Codex failed may reveal a lower-cost harness improvement sooner than difficult Sol tasks. Compare on frozen prior-failure tasks, with a same-task direct baseline and complete cost/effort receipts. — https://bench.kogen.dev/hypotheses/h151/ - H152 (F14; NO EMPIRICAL EVIDENCE; no completed public fairness audit is identified.): TrackLine task fairness. TrackLine must be repaired and audited as a fair task before its results inform harness decisions. Establish task correctness, valid alternatives, reproducible setup, and grader sensitivity before scored use. — https://bench.kogen.dev/hypotheses/h152/ - H153 (F14; INTERIM; mixed-version inventory only, with no causal result.): Harness and prompt mining. Mining harness implementations and prompts may reveal mechanisms behind performance differences. Convert candidate mechanisms into a complete feature map and test them with isolated ablations rather than treating the inventory itself as causal evidence. — https://bench.kogen.dev/hypotheses/h153/ ## Round register - cache-key-api-v4 (VALID; family not linked): H-API: STATUS VALID; primary finding NOT SUPPORTED — https://bench.kogen.dev/rounds/cache-key-api-v4/ - cache-key-backend-v4 (VALID; family not linked): H-BACKEND: STATUS VALID; primary finding NOT SUPPORTED — https://bench.kogen.dev/rounds/cache-key-backend-v4/ - cache-replay-mechanism (DESCRIPTIVE; family not linked): Controlled prompt-cache replay — https://bench.kogen.dev/rounds/cache-replay-mechanism/ - campfire-lane1 (INCOMPLETE; family not linked): Campfire runtime comparison, lane 1 — https://bench.kogen.dev/rounds/campfire-lane1/ - campfire-lane2 (INCOMPLETE; family not linked): Campfire runtime comparison, lane 2 — https://bench.kogen.dev/rounds/campfire-lane2/ - candidate-counterfactual (DESCRIPTIVE; family not linked): Saved-candidate counterfactual audit — https://bench.kogen.dev/rounds/candidate-counterfactual/ - claim-b-rerun-1 (INVALID; F03): Claim B worker rerun — https://bench.kogen.dev/rounds/claim-b-rerun-1/ - claim-b-screen-1 (INVALID; F03): Claim B transport screen — https://bench.kogen.dev/rounds/claim-b-screen-1/ - confirm-combined-1 (INCOMPLETE; F03): Combined kh confirmation — https://bench.kogen.dev/rounds/confirm-combined-1/ - core4-pilots (DESCRIPTIVE; family not linked): Core4 pilots — https://bench.kogen.dev/rounds/core4-pilots/ - core4b-ksub-max (DESCRIPTIVE; family not linked): Core4b and Ksub Luna max — https://bench.kogen.dev/rounds/core4b-ksub-max/ - ctx-store (DESCRIPTIVE; F14): context store — https://bench.kogen.dev/rounds/ctx-store/ - e06-context (PILOT; F05): e06 context — https://bench.kogen.dev/rounds/e06-context/ - e09-tidewave (PILOT; F05): e09 locators and Tidewave — https://bench.kogen.dev/rounds/e09-tidewave/ - frontier (INCOMPLETE; F07, F12, F14): R74 Kogen frontier lane — https://bench.kogen.dev/rounds/frontier/ - fv-pilot-2026-10-10 (DESCRIPTIVE; family not linked): Lean versus Quint verification pilot — https://bench.kogen.dev/rounds/fv-pilot-2026-10-10/ - gleam (DESCRIPTIVE; family not linked): Gleam language round — https://bench.kogen.dev/rounds/gleam/ - grok-frontier (INCOMPLETE; family not linked): Grok frontier — https://bench.kogen.dev/rounds/grok-frontier/ - hard1-calibrate-1 (INVALID; F03): Hard-task harness calibration — https://bench.kogen.dev/rounds/hard1-calibrate-1/ - harness-compare-1 (DESCRIPTIVE; F03): Four-harness comparison — https://bench.kogen.dev/rounds/harness-compare-1/ - harness-pick-2026-09-29 (DESCRIPTIVE; F03): September harness pick — https://bench.kogen.dev/rounds/harness-pick-2026-09-29/ - hc-1 (DESCRIPTIVE; F07, F12): Studio hill climb iteration 1 — https://bench.kogen.dev/rounds/hc-1/ - hc-2 (DESCRIPTIVE; F07, F12): Studio hill climb iteration 2 — https://bench.kogen.dev/rounds/hc-2/ - hc-3 (DESCRIPTIVE; F07, F12): Studio hill climb iteration 3 — https://bench.kogen.dev/rounds/hc-3/ - hc-4 (DESCRIPTIVE; F07, F12): Studio hill climb iteration 4 — https://bench.kogen.dev/rounds/hc-4/ - hc01-small32 (INCOMPLETE; family not linked): HC01 small32 — https://bench.kogen.dev/rounds/hc01-small32/ - intent-coverage-audit (DESCRIPTIVE; family not linked): Intent coverage audit — https://bench.kogen.dev/rounds/intent-coverage-audit/ - jev-route-ctx (DESCRIPTIVE; family not linked): Jev route with prepared context — https://bench.kogen.dev/rounds/jev-route-ctx/ - jev-triage-ctx (INCOMPLETE; family not linked): Jev failure triage with prepared context — https://bench.kogen.dev/rounds/jev-triage-ctx/ - kh-climb (DESCRIPTIVE; F03): kh climb — https://bench.kogen.dev/rounds/kh-climb/ - kh-compare-2026-09-29 (INVALID; F03): September kh comparison — https://bench.kogen.dev/rounds/kh-compare-2026-09-29/ - kogen-rs-ladder-luna-r74x (DESCRIPTIVE; family not linked): Kogen-RS R74 sandbox comparison — https://bench.kogen.dev/rounds/kogen-rs-ladder-luna-r74x/ - l0-reconcile (VALID; F11, F12): L0 historical claim reconciliation — https://bench.kogen.dev/rounds/l0-reconcile/ - l1-task-grader-trust (VALID; family not linked): L1 — Task and grader trust — https://bench.kogen.dev/rounds/l1-task-grader-trust/ - l2-plan-specificity (VALID; family not linked): L2 plan specificity × builder strength — https://bench.kogen.dev/rounds/l2-plan-specificity/ - l3-repair (DESCRIPTIVE; F01): L3 actionable self-verification repair — https://bench.kogen.dev/rounds/l3-repair/ - l3b-repair-vs-continue (VALID; family not linked): L3b repair vs continue — https://bench.kogen.dev/rounds/l3b-repair-vs-continue/ - l4-context-packet (DESCRIPTIVE; family not linked): L4 deterministic context packet — https://bench.kogen.dev/rounds/l4-context-packet/ - l4b-confirm (VALID; family not linked): L4b independent context-packet replication — https://bench.kogen.dev/rounds/l4b-confirm/ - l5-sol-comparators (DESCRIPTIVE; family not linked): L5 — direct Sol-medium comparators — https://bench.kogen.dev/rounds/l5-sol-comparators/ - l6-auditor-replay (VALID; F06): L6: scoped-auditor replay — https://bench.kogen.dev/rounds/l6-auditor-replay/ - lang-sol-replication (DESCRIPTIVE; F08, F11): Sol-medium language replication — https://bench.kogen.dev/rounds/lang-sol-replication/ - llm-latency-long (DESCRIPTIVE; family not linked): LLM latency probe: long reasoning prompts — https://bench.kogen.dev/rounds/llm-latency-long/ - llm-latency-short (DESCRIPTIVE; family not linked): LLM latency probe: short prompts — https://bench.kogen.dev/rounds/llm-latency-short/ - luna-medium (DESCRIPTIVE; family not linked): Luna medium on core4 — https://bench.kogen.dev/rounds/luna-medium/ - night-2026-10-01 (DESCRIPTIVE; F03, F07): benchmark night 2026-10-01 — https://bench.kogen.dev/rounds/night-2026-10-01/ - offline-aa-bo3 (DESCRIPTIVE; family not linked): Offline A/A and best-of-three analysis — https://bench.kogen.dev/rounds/offline-aa-bo3/ - offline-contract-audit (DESCRIPTIVE; family not linked): Offline contract audit — https://bench.kogen.dev/rounds/offline-contract-audit/ - offline-review (DESCRIPTIVE; family not linked): Historical offline review audit — https://bench.kogen.dev/rounds/offline-review/ - offline-review3 (INCOMPLETE; family not linked): Frozen offline reviewer comparison — https://bench.kogen.dev/rounds/offline-review3/ - overstrict-replay (DESCRIPTIVE; family not linked): Over-strict test replay — https://bench.kogen.dev/rounds/overstrict-replay/ - pilot73 (DESCRIPTIVE; family not linked): pilot73 — recovered shaper source round — https://bench.kogen.dev/rounds/pilot73/ - r1 (INVALID; family not linked): r1 — https://bench.kogen.dev/rounds/r1/ - r10 (DESCRIPTIVE; family not linked): r10 — https://bench.kogen.dev/rounds/r10/ - r11 (DESCRIPTIVE; family not linked): r11 — https://bench.kogen.dev/rounds/r11/ - r12 (INVALID; family not linked): r12 — https://bench.kogen.dev/rounds/r12/ - r13 (DESCRIPTIVE; family not linked): r13 — https://bench.kogen.dev/rounds/r13/ - r14 (DESCRIPTIVE; family not linked): r14 — https://bench.kogen.dev/rounds/r14/ - r15 (DESCRIPTIVE; family not linked): r15 — https://bench.kogen.dev/rounds/r15/ - r16 (DESCRIPTIVE; family not linked): r16 — https://bench.kogen.dev/rounds/r16/ - r17 (DESCRIPTIVE; family not linked): r17 — https://bench.kogen.dev/rounds/r17/ - r18 (DESCRIPTIVE; family not linked): r18 — https://bench.kogen.dev/rounds/r18/ - r19 (DESCRIPTIVE; family not linked): r19 — https://bench.kogen.dev/rounds/r19/ - r2 (DESCRIPTIVE; family not linked): r2 — https://bench.kogen.dev/rounds/r2/ - r20 (DESCRIPTIVE; family not linked): r20 — https://bench.kogen.dev/rounds/r20/ - r21 (DESCRIPTIVE; family not linked): r21 — https://bench.kogen.dev/rounds/r21/ - r22 (DESCRIPTIVE; family not linked): r22 — https://bench.kogen.dev/rounds/r22/ - r23 (DESCRIPTIVE; family not linked): r23 — https://bench.kogen.dev/rounds/r23/ - r24 (DESCRIPTIVE; family not linked): r24 — https://bench.kogen.dev/rounds/r24/ - r25 (DESCRIPTIVE; family not linked): r25 — https://bench.kogen.dev/rounds/r25/ - r26 (DESCRIPTIVE; family not linked): r26 — https://bench.kogen.dev/rounds/r26/ - r27 (DESCRIPTIVE; family not linked): r27 — https://bench.kogen.dev/rounds/r27/ - r28 (INVALID; family not linked): r28 — https://bench.kogen.dev/rounds/r28/ - r29 (DESCRIPTIVE; family not linked): r29 — https://bench.kogen.dev/rounds/r29/ - r3 (INVALID; family not linked): r3 — https://bench.kogen.dev/rounds/r3/ - r30 (DESCRIPTIVE; family not linked): r30 — https://bench.kogen.dev/rounds/r30/ - r31 (INVALID; family not linked): r31 — https://bench.kogen.dev/rounds/r31/ - r32 (DESCRIPTIVE; family not linked): r32 — https://bench.kogen.dev/rounds/r32/ - r32a (INCOMPLETE; family not linked): r32a — https://bench.kogen.dev/rounds/r32a/ - r32b (DESCRIPTIVE; family not linked): r32b — https://bench.kogen.dev/rounds/r32b/ - r33 (DESCRIPTIVE; family not linked): r33 — https://bench.kogen.dev/rounds/r33/ - r34 (DESCRIPTIVE; family not linked): r34 — https://bench.kogen.dev/rounds/r34/ - r34b (DESCRIPTIVE; family not linked): r34b — https://bench.kogen.dev/rounds/r34b/ - r35 (DESCRIPTIVE; family not linked): r35 — https://bench.kogen.dev/rounds/r35/ - r36 (INVALID; family not linked): r36 — https://bench.kogen.dev/rounds/r36/ - r37 (INVALID; family not linked): r37 — https://bench.kogen.dev/rounds/r37/ - r37c (DESCRIPTIVE; family not linked): r37c — https://bench.kogen.dev/rounds/r37c/ - r38 (DESCRIPTIVE; family not linked): r38 — https://bench.kogen.dev/rounds/r38/ - r39 (INVALID; family not linked): r39 — https://bench.kogen.dev/rounds/r39/ - r39b (DESCRIPTIVE; family not linked): r39b — https://bench.kogen.dev/rounds/r39b/ - r4 (DESCRIPTIVE; family not linked): r4 — https://bench.kogen.dev/rounds/r4/ - r40 (DESCRIPTIVE; family not linked): r40 — https://bench.kogen.dev/rounds/r40/ - r41 (INVALID; family not linked): r41 — https://bench.kogen.dev/rounds/r41/ - r42 (INVALID; family not linked): r42 — https://bench.kogen.dev/rounds/r42/ - r43 (DESCRIPTIVE; family not linked): r43 — https://bench.kogen.dev/rounds/r43/ - r44 (DESCRIPTIVE; family not linked): r44 — https://bench.kogen.dev/rounds/r44/ - r45 (INVALID; family not linked): r45 — https://bench.kogen.dev/rounds/r45/ - r46 (DESCRIPTIVE; family not linked): r46 — https://bench.kogen.dev/rounds/r46/ - r47 (DESCRIPTIVE; family not linked): r47 — https://bench.kogen.dev/rounds/r47/ - r48 (DESCRIPTIVE; family not linked): r48 — https://bench.kogen.dev/rounds/r48/ - r48b (DESCRIPTIVE; family not linked): r48b — https://bench.kogen.dev/rounds/r48b/ - r49 (INVALID; family not linked): r49 — https://bench.kogen.dev/rounds/r49/ - r49b (DESCRIPTIVE; family not linked): r49b — https://bench.kogen.dev/rounds/r49b/ - r5 (DESCRIPTIVE; family not linked): r5 — https://bench.kogen.dev/rounds/r5/ - r50 (INCOMPLETE; family not linked): r50 — https://bench.kogen.dev/rounds/r50/ - r51 (INCOMPLETE; family not linked): r51 — https://bench.kogen.dev/rounds/r51/ - r52 (PILOT; family not linked): r52 — https://bench.kogen.dev/rounds/r52/ - r53 (INVALID; family not linked): r53 — https://bench.kogen.dev/rounds/r53/ - r53b (INCOMPLETE; family not linked): r53b — https://bench.kogen.dev/rounds/r53b/ - r54 (INCOMPLETE; family not linked): r54 — https://bench.kogen.dev/rounds/r54/ - r55 (INCOMPLETE; family not linked): r55 — https://bench.kogen.dev/rounds/r55/ - r56 (INVALID; family not linked): r56 — https://bench.kogen.dev/rounds/r56/ - r56b (INCOMPLETE; family not linked): r56b — https://bench.kogen.dev/rounds/r56b/ - r56c (INCOMPLETE; family not linked): r56c — https://bench.kogen.dev/rounds/r56c/ - r56d (INCOMPLETE; family not linked): r56d — https://bench.kogen.dev/rounds/r56d/ - r56p2 (INCOMPLETE; family not linked): r56p2 — https://bench.kogen.dev/rounds/r56p2/ - r57 (INVALID; family not linked): r57 — https://bench.kogen.dev/rounds/r57/ - r57-studio (PILOT; F07): R57 Rails Studio smoke lane — https://bench.kogen.dev/rounds/r57-studio/ - r57-t90-studio (PILOT; F07): R57 T90 Studio retry lane — https://bench.kogen.dev/rounds/r57-t90-studio/ - r57b (INCOMPLETE; family not linked): r57b — https://bench.kogen.dev/rounds/r57b/ - r57c (INCOMPLETE; family not linked): r57c — https://bench.kogen.dev/rounds/r57c/ - r57d (INCOMPLETE; family not linked): r57d — https://bench.kogen.dev/rounds/r57d/ - r57e (INCOMPLETE; family not linked): r57e — https://bench.kogen.dev/rounds/r57e/ - r58 (PILOT; family not linked): r58 — https://bench.kogen.dev/rounds/r58/ - r58b (INVALID; family not linked): r58b — https://bench.kogen.dev/rounds/r58b/ - r58x-studio (PILOT; F07, F12): R58x Studio Elixir ladder lane — https://bench.kogen.dev/rounds/r58x-studio/ - r59 (INCOMPLETE; family not linked): r59 — https://bench.kogen.dev/rounds/r59/ - r6 (DESCRIPTIVE; family not linked): r6 — https://bench.kogen.dev/rounds/r6/ - r60 (INVALID; family not linked): r60 — https://bench.kogen.dev/rounds/r60/ - r60b (INVALID; family not linked): r60b — https://bench.kogen.dev/rounds/r60b/ - r61 (INVALID; family not linked): r61 — https://bench.kogen.dev/rounds/r61/ - r62 (INVALID; family not linked): r62 — https://bench.kogen.dev/rounds/r62/ - r62b (INVALID; family not linked): r62b — https://bench.kogen.dev/rounds/r62b/ - r63 (INCOMPLETE; family not linked): r63 — https://bench.kogen.dev/rounds/r63/ - r63b (INCOMPLETE; family not linked): r63b — https://bench.kogen.dev/rounds/r63b/ - r64 (DESCRIPTIVE; family not linked): r64 — https://bench.kogen.dev/rounds/r64/ - r64b (INCOMPLETE; family not linked): r64b — https://bench.kogen.dev/rounds/r64b/ - r64c (DESCRIPTIVE; family not linked): r64c — https://bench.kogen.dev/rounds/r64c/ - r64d (DESCRIPTIVE; family not linked): r64d — https://bench.kogen.dev/rounds/r64d/ - r64e (INCOMPLETE; family not linked): r64e — https://bench.kogen.dev/rounds/r64e/ - r65 (INVALID; family not linked): r65 — https://bench.kogen.dev/rounds/r65/ - r65b (INVALID; family not linked): r65b — https://bench.kogen.dev/rounds/r65b/ - r66 (DESCRIPTIVE; family not linked): r66 — https://bench.kogen.dev/rounds/r66/ - r67 (INVALID; family not linked): r67 — https://bench.kogen.dev/rounds/r67/ - r67b (INVALID; family not linked): r67b — https://bench.kogen.dev/rounds/r67b/ - r68 (INCOMPLETE; family not linked): r68 — https://bench.kogen.dev/rounds/r68/ - r68b (DESCRIPTIVE; family not linked): r68b — https://bench.kogen.dev/rounds/r68b/ - r69 (INVALID; family not linked): r69 — https://bench.kogen.dev/rounds/r69/ - r7 (INVALID; family not linked): r7 — https://bench.kogen.dev/rounds/r7/ - r70 (INCOMPLETE; family not linked): r70 — https://bench.kogen.dev/rounds/r70/ - r70-compile (DESCRIPTIVE; family not linked): r70 compilation timing — https://bench.kogen.dev/rounds/r70-compile/ - r70-rve (INVALID; family not linked): Round 70 original RvE cohort — https://bench.kogen.dev/rounds/r70-rve/ - r70-rve-ext (VALID; family not linked): r70-RvE language extension — https://bench.kogen.dev/rounds/r70-rve-ext/ - r70-rve-rerun (DESCRIPTIVE; family not linked): Round 70 fixed-skeleton rerun — https://bench.kogen.dev/rounds/r70-rve-rerun/ - r70-rve-task8v2 (INCOMPLETE; F11): Round 70 task 8 v2 smoke lane — https://bench.kogen.dev/rounds/r70-rve-task8v2/ - r70-rve2 (INVALID; family not linked): r70-rve2: Rust and Elixir round — https://bench.kogen.dev/rounds/r70-rve2/ - r70-rve3 (INVALID; family not linked): r70-rve3: Rust and Elixir round — https://bench.kogen.dev/rounds/r70-rve3/ - r70-spot1 (DESCRIPTIVE; family not linked): r70-spot1: Four-language round — https://bench.kogen.dev/rounds/r70-spot1/ - r70-task8 (PILOT; family not linked): Round 70 task 8 extension — https://bench.kogen.dev/rounds/r70-task8/ - r71 (INCOMPLETE; family not linked): r71 — https://bench.kogen.dev/rounds/r71/ - r72 (INCOMPLETE; F11, F12): R72 same-model ladder comparison — https://bench.kogen.dev/rounds/r72/ - r73 (INCOMPLETE; family not linked): r73 proper shaping — https://bench.kogen.dev/rounds/r73/ - r74 (INCOMPLETE; family not linked): r74 Kogen builder configurations versus Codex — https://bench.kogen.dev/rounds/r74/ - r8 (DESCRIPTIVE; family not linked): r8 — https://bench.kogen.dev/rounds/r8/ - r9 (DESCRIPTIVE; family not linked): r9 — https://bench.kogen.dev/rounds/r9/ - race1 (DESCRIPTIVE; family not linked): Race 1: timed conformance race — https://bench.kogen.dev/rounds/race1/ - race2 (DESCRIPTIVE; family not linked): Race 2: timed conformance race — https://bench.kogen.dev/rounds/race2/ - race3-sprint (DESCRIPTIVE; family not linked): Race 3 sprint — descriptive; experimental benchmark spec, not Kogen's released core — https://bench.kogen.dev/rounds/race3-sprint/ - race4 (DESCRIPTIVE; family not linked): Race 4 — descriptive; experimental benchmark spec (suite-v3), not Kogen's released core — https://bench.kogen.dev/rounds/race4/ - race5 (DESCRIPTIVE; family not linked): Race 5 — language comparison (descriptive) — https://bench.kogen.dev/rounds/race5/ - race6 (DESCRIPTIVE; family not linked): Race 6 — spec-format pilot (descriptive) — https://bench.kogen.dev/rounds/race6/ - l1-model-effort (DESCRIPTIVE; family not linked): l1-model-effort — https://bench.kogen.dev/rounds/l1-model-effort/ - rails-catalogue (DESCRIPTIVE; family not linked): rails-catalogue — https://bench.kogen.dev/rounds/rails-catalogue/ - rtg1 (DESCRIPTIVE; family not linked): RTG1 core4 replications — https://bench.kogen.dev/rounds/rtg1/ - rz1 (VALID; family not linked): rz1: Zig vs Rust on Round-70 Kogen-subsystem tasks — https://bench.kogen.dev/rounds/rz1/ - shape1-wave1 (DESCRIPTIVE; family not linked): Shape1 first wave — https://bench.kogen.dev/rounds/shape1-wave1/ - sol-medium (DESCRIPTIVE; family not linked): Sol medium — https://bench.kogen.dev/rounds/sol-medium/ - stack-oneshot-conformance (DESCRIPTIVE; family not linked): One-shot stack conformance — https://bench.kogen.dev/rounds/stack-oneshot-conformance/ - studio-kogen (DESCRIPTIVE; F06, F07): Studio Kogen smoke lane — https://bench.kogen.dev/rounds/studio-kogen/ - studio-kogen-80a4 (DESCRIPTIVE; F06, F07): Studio Kogen 80a4 smoke lane — https://bench.kogen.dev/rounds/studio-kogen-80a4/ - studio-shaper-pilot (PILOT; F06, F07): Studio shaper pilot — https://bench.kogen.dev/rounds/studio-shaper-pilot/ - t100-cache-lane (INCOMPLETE; family not linked): T100 cache lane — https://bench.kogen.dev/rounds/t100-cache-lane/ - t98-cache-smoke (PILOT; family not linked): T98 prompt-cache smoke — https://bench.kogen.dev/rounds/t98-cache-smoke/ - unmapped (INVALID; family not linked): Unmapped public results — https://bench.kogen.dev/rounds/unmapped/ - x-compaction (DESCRIPTIVE; F05): x compaction — https://bench.kogen.dev/rounds/x-compaction/ - x-context (DESCRIPTIVE; F05): x context — https://bench.kogen.dev/rounds/x-context/ - x-continuation (INCOMPLETE; F03, F06): x continuation — https://bench.kogen.dev/rounds/x-continuation/ - x-ladder (DESCRIPTIVE; F04): x ladder — https://bench.kogen.dev/rounds/x-ladder/ - x-minikogen (DESCRIPTIVE; F03): x minikogen — https://bench.kogen.dev/rounds/x-minikogen/ - x-mining (DESCRIPTIVE; F07): x mining — https://bench.kogen.dev/rounds/x-mining/ - x-models (DESCRIPTIVE; F04): x models — https://bench.kogen.dev/rounds/x-models/ - x-parallel (DESCRIPTIVE; F03): x parallel — https://bench.kogen.dev/rounds/x-parallel/ - x-race (DESCRIPTIVE; F03): x race — https://bench.kogen.dev/rounds/x-race/ - x-recovery (DESCRIPTIVE; F06): x recovery — https://bench.kogen.dev/rounds/x-recovery/ - x-review (DESCRIPTIVE; F06): x review — https://bench.kogen.dev/rounds/x-review/ - x-roles (DESCRIPTIVE; F03, F04): x roles — https://bench.kogen.dev/rounds/x-roles/ - x-surface (INCOMPLETE; F03, F10): x surface — https://bench.kogen.dev/rounds/x-surface/ - arch-elixir-vs-go (DESCRIPTIVE; family not linked): ARCH-20261011 — Elixir core vs Go core — https://bench.kogen.dev/rounds/arch-elixir-vs-go/ - lab01-analysis-tooling-screen (DESCRIPTIVE; family not linked): LAB-01: analysis-tooling screen — https://bench.kogen.dev/rounds/lab01-analysis-tooling-screen/ - containment-small-eu (DESCRIPTIVE; family not linked): Small containment comparison — https://bench.kogen.dev/rounds/containment-small-eu/ - containment-small-eu-v2 (DESCRIPTIVE; family not linked): Small containment comparison: three configured setups — https://bench.kogen.dev/rounds/containment-small-eu-v2/ - containment-small-eu-helper (DESCRIPTIVE; family not linked): Small containment comparison: Kogen's helper — https://bench.kogen.dev/rounds/containment-small-eu-helper/ - review-convergence-1 (DESCRIPTIVE; family not linked): Code review that never ends: does the reviewer model decide? — https://bench.kogen.dev/rounds/review-convergence-1/ - screen-kogen-1-stage-d (DESCRIPTIVE; family not linked): Kogen mechanisms, one at a time: tasks too easy to show gains — https://bench.kogen.dev/rounds/screen-kogen-1-stage-d/ - screen-kogen-2 (DESCRIPTIVE; family not linked): Kogen mechanisms, one at a time, on four tasks the unchanged setup did not all solve — https://bench.kogen.dev/rounds/screen-kogen-2/ - screen-kogen-1-stage-1 (INVALID; family not linked): Kogen mechanism screen, first baseline runs: invalid — https://bench.kogen.dev/rounds/screen-kogen-1-stage-1/ - screen-kogen-2-first-run (INVALID; family not linked): Kogen mechanisms on four tasks: invalid first run — https://bench.kogen.dev/rounds/screen-kogen-2-first-run/ - baseline-variance-1 (DESCRIPTIVE; family not linked): Repeating the unchanged Kogen configuration: the screen's gains were probably chance — https://bench.kogen.dev/rounds/baseline-variance-1/ - review-timing-1 (DESCRIPTIVE; family not linked): Reviewing only once: cheaper, and wrong code gets through — https://bench.kogen.dev/rounds/review-timing-1/ - shaping-model-1 (DESCRIPTIVE; family not linked): Writing the specification inside the attempt: a stronger model did not help — https://bench.kogen.dev/rounds/shaping-model-1/ - builder-effort-1 (DESCRIPTIVE; family not linked): More builder effort: fewer repairs and less input in these attempts — https://bench.kogen.dev/rounds/builder-effort-1/ - confirm-builder-effort-1 (DESCRIPTIVE; family not linked): Builder effort confirmed on fresh tasks: same correctness, about a quarter less input — https://bench.kogen.dev/rounds/confirm-builder-effort-1/ - repair-stop-1 (DESCRIPTIVE; family not linked): Repair limit: the six-repair stop did not hide successes on the Elixir task — https://bench.kogen.dev/rounds/repair-stop-1/ - review-bound-1 (DESCRIPTIVE; family not linked): Review round limits: stopping review after one or two rounds would have let wrong code through — https://bench.kogen.dev/rounds/review-bound-1/ - review-bound-2 (DESCRIPTIVE; family not linked): Review round limits, extended: a one-round limit would have let wrong code through in a quarter of attempts — https://bench.kogen.dev/rounds/review-bound-2/ - retention-1 (DESCRIPTIVE; family not linked): Builder context trimming: dropping old tool output saved almost no input — https://bench.kogen.dev/rounds/retention-1/ - review-independence-1 (DESCRIPTIVE; family not linked): Review independence: a reviewer shown the builder's conversation approved later and used more input — https://bench.kogen.dev/rounds/review-independence-1/ - repair-cost-1 (DESCRIPTIVE; family not linked): Repair cost: a stronger builder, or the reviewer's model as fixer, needed fewer repairs than the low-effort builder — https://bench.kogen.dev/rounds/repair-cost-1/ - planner-direct-1 (DESCRIPTIVE; family not linked): Dropping the planner: planner off matched planner on at 87% of the input, on tasks this builder solves every time (one attempt per task) — https://bench.kogen.dev/rounds/planner-direct-1/ - claude-roles-1 (DESCRIPTIVE; family not linked): Claude roles: a Sonnet reviewer approved wrong code twice; Haiku and all-Sonnet settings were correct on all seven tasks (one attempt each) — https://bench.kogen.dev/rounds/claude-roles-1/ - fixer-model-1 (DESCRIPTIVE; family not linked): Fixer model: in one attempt per task, Sol medium kept the Sol high fixer's correctness at 70% of the input on the four tasks the baseline finished, and a Luna medium fixer was weaker; the run stopped at its input budget (19 of 24 attempts scored) — https://bench.kogen.dev/rounds/fixer-model-1/ - planner-direct-2 (DESCRIPTIVE; family not linked): Dropping the planner on harder tasks: both settings were correct on all four; planner off used 90% of the input, nearly all of the difference from one task (one attempt per task) — https://bench.kogen.dev/rounds/planner-direct-2/ - claude-builder-2 (DESCRIPTIVE; family not linked): Claude builders on harder tasks: Sonnet 5.5 reached the repair cap on three of four tasks and Haiku 5.5 on one of three scored, and the GPT-6.1 Sol builder on one of four (one attempt per task) — https://bench.kogen.dev/rounds/claude-builder-2/ - builder-subagents-1 (DESCRIPTIVE; family not linked): Builder helpers: the offered helper tool was never called, so no helper ran (one attempt per task and setting) — https://bench.kogen.dev/rounds/builder-subagents-1/ - review-fix-1 (DESCRIPTIVE; family not linked): Review and fix, new flow only: all 9 reviewed attempts ended correct and the 3 failures stopped before review; the earlier loop was not run (one attempt per task and setting) — https://bench.kogen.dev/rounds/review-fix-1/ - shape-default-1 (DESCRIPTIVE; family not linked): Generated shaping with the default Sol high shaper, on a Lab build whose shaping is not Kogen-equivalent: 3 of 4 correct against 4 of 4 with a supplied specification (one attempt per task and setting) — https://bench.kogen.dev/rounds/shape-default-1/ - blocker-stop-1 (DESCRIPTIVE; family not linked): Builder blocker stop against a six-repair cap: inconclusive, because all 12 scored attempts were correct, the blocker tool was never called and no stop was reached (one attempt per task and setting) — https://bench.kogen.dev/rounds/blocker-stop-1/ - claude-review-fix-1 (DESCRIPTIVE; family not linked): Review and fix, exploratory (two repetitions per task, small counts): in two prose-specification runs the reviewers fixed 4 of 8 wrong builds (Sonnet 3 of 5, Sol high 1 of 1, Haiku 0 of 2); with supplied scenarios no wrong code reached review — https://bench.kogen.dev/rounds/claude-review-fix-1/ - builder-subagents-2 (DESCRIPTIVE; family not linked): Builder helpers, second run: with a hint the builder called the helper 0 times, with a required call it called it once per attempt and the helper did little (one attempt per task and setting) — https://bench.kogen.dev/rounds/builder-subagents-2/ - repeat-screen-1 (DESCRIPTIVE; family not linked): Repeat screen on Kogen's defaults, two repetitions per task: all 19 scored attempts were correct, a Sol medium reviewer used less input and the planner did not help (19 of 24 attempts scored) — https://bench.kogen.dev/rounds/repeat-screen-1/ - review-fix-2 (DESCRIPTIVE; family not linked): Review and fix against the earlier flow: with a weak builder the new flow ended with approved correct code in 4 of 4 attempts and the earlier flow in 0 of 4; with Kogen's builder both did 4 of 4 (one attempt per task and setting) — https://bench.kogen.dev/rounds/review-fix-2/ - shaping-2 (DESCRIPTIVE; family not linked): Shaper helpers and generated specifications — https://bench.kogen.dev/rounds/shaping-2/ - prose-vs-gherkin-1 (DESCRIPTIVE; family not linked): Prose versus Gherkin: Gherkin 10/10, prose 8/10; two wrong prose submissions approved unchanged — https://bench.kogen.dev/rounds/prose-vs-gherkin-1/ - mechanism-split-1 (DESCRIPTIVE; family not linked): Which flow mechanism helps a weak builder? Follow-up correct submissions were 6/7 versus 1/8 — https://bench.kogen.dev/rounds/mechanism-split-1/ - review-named-defect-1 (DESCRIPTIVE; family not linked): Named-defect review did not stop polish edits in either run — https://bench.kogen.dev/rounds/review-named-defect-1/ - builder-low-1 (DESCRIPTIVE; family not linked): Builder effort at low vs medium: correctness held, total input fell 4% — https://bench.kogen.dev/rounds/builder-low-1/ - reviewer-effort-weak-1 (DESCRIPTIVE; family not linked): Lower reviewer effort with a weak builder: rescues held in this screen — https://bench.kogen.dev/rounds/reviewer-effort-weak-1/ - cheap-profile-1 (DESCRIPTIVE; family not linked): A lower-cost profile kept correctness across three development runs — https://bench.kogen.dev/rounds/cheap-profile-1/ - confirm-cheap-profile-1 (DESCRIPTIVE; family not linked): Cheap profile confirmation on fresh tasks: UNRESOLVED — https://bench.kogen.dev/rounds/confirm-cheap-profile-1/ - reviewer-polish-1 (DESCRIPTIVE; family not linked): Reviewer polish screen: medium effort did not polish more — https://bench.kogen.dev/rounds/reviewer-polish-1/ - confirm-reviewer-medium-1 (DESCRIPTIVE; family not linked): Medium reviewer confirmation on fresh tasks: UNRESOLVED — https://bench.kogen.dev/rounds/confirm-reviewer-medium-1/ - confirm-reviewer-medium-2 (DESCRIPTIVE; family not linked): Second medium reviewer confirmation: UNRESOLVED at about 84.4% of high input — https://bench.kogen.dev/rounds/confirm-reviewer-medium-2/ - review-off-hard-1 (DESCRIPTIVE; family not linked): Does the default builder need review on harder tasks? — https://bench.kogen.dev/rounds/review-off-hard-1/ - intent-short-1 (DESCRIPTIVE; family not linked): Short intent with complete acceptance scenarios — https://bench.kogen.dev/rounds/intent-short-1/ - confirm-intent-short-1 (DESCRIPTIVE; family not linked): Short intent on a fresh specification — https://bench.kogen.dev/rounds/confirm-intent-short-1/ - review-off-hard-1-kogen (DESCRIPTIVE; family not linked): Does real Kogen reproduce the review-off screen? — https://bench.kogen.dev/rounds/review-off-hard-1-kogen/ - chatgpt-session-cache-1 (DESCRIPTIVE; family not linked): Does a stable session id fix prompt caching on a ChatGPT subscription? — https://bench.kogen.dev/rounds/chatgpt-session-cache-1/ - review-value-2 (DESCRIPTIVE; family not linked): Does Kogen's review step earn its cost? Seeded defects that acceptance misses — https://bench.kogen.dev/rounds/review-value-2/ - review-value-2-clean (DESCRIPTIVE; family not linked): Review on correct code: clean controls — https://bench.kogen.dev/rounds/review-value-2-clean/ - review-value-2-natural-off (DESCRIPTIVE; family not linked): Natural builds with review off — https://bench.kogen.dev/rounds/review-value-2-natural-off/ - review-value-2-natural-reviewed (DESCRIPTIVE; family not linked): Natural builds with review on: three review settings — https://bench.kogen.dev/rounds/review-value-2-natural-reviewed/ - review-value-2-openai (DESCRIPTIVE; family not linked): Seeded defects again, with an OpenAI reviewer — https://bench.kogen.dev/rounds/review-value-2-openai/ - long-tasks-reset-fix-1-before (DESCRIPTIVE; family not linked): Long tasks on Kogen, before the reset fix — https://bench.kogen.dev/rounds/long-tasks-reset-fix-1-before/ - long-tasks-reset-fix-1-after (DESCRIPTIVE; family not linked): Long tasks on Kogen, after the reset fix — https://bench.kogen.dev/rounds/long-tasks-reset-fix-1-after/ - long-tasks-reset-fix-1-after-studio (DESCRIPTIVE; family not linked): Long tasks on Kogen after the reset fix: two more repetitions on a second machine — https://bench.kogen.dev/rounds/long-tasks-reset-fix-1-after-studio/ - reviewer-models-seeded-1 (DESCRIPTIVE; family not linked): Faster review models on seeded defects: Terra and Opus against Sol high — https://bench.kogen.dev/rounds/reviewer-models-seeded-1/ - reviewer-models-clean-1 (DESCRIPTIVE; family not linked): Faster review models on correct code: do they block or break it? (partial) — https://bench.kogen.dev/rounds/reviewer-models-clean-1/ - shaper-models-1 (DESCRIPTIVE; family not linked): Which model should shape the requirements? Five shapers on 16 tasks — https://bench.kogen.dev/rounds/shaper-models-1/ - terra-builder-1 (DESCRIPTIVE; family not linked): OpenAI GPT-6.1 Terra as the builder: medium and max effort against Sol medium — https://bench.kogen.dev/rounds/terra-builder-1/ - terra-builder-effort-1 (DESCRIPTIVE; family not linked): OpenAI GPT-6.1 Terra as the builder: low, medium and high effort against Sol medium — https://bench.kogen.dev/rounds/terra-builder-effort-1/ - loop-vs-executor-1 (DESCRIPTIVE; family not linked): Same builder model: Kogen's own build loop against the Claude Code executor (interim) — https://bench.kogen.dev/rounds/loop-vs-executor-1/ - executor-lineup-dry-1 (DESCRIPTIVE; family not linked): The full Claude Code lineup runs end to end (dry cells) — https://bench.kogen.dev/rounds/executor-lineup-dry-1/ - sonnet-long-builder-1 (DESCRIPTIVE; family not linked): Claude Sonnet 5.5 as Kogen's builder on long tasks, with Sonnet or Opus review — https://bench.kogen.dev/rounds/sonnet-long-builder-1/ - terra-vs-sonnet-builder-1 (DESCRIPTIVE; family not linked): Terra high vs Sonnet medium as the builder (interim) — https://bench.kogen.dev/rounds/terra-vs-sonnet-builder-1/ - shape-flawed-requests-1 (DESCRIPTIVE; family not linked): Catching flawed requests in Shape: baseline, levers and other models — https://bench.kogen.dev/rounds/shape-flawed-requests-1/ - claude-code-fix-rechecks-1 (DESCRIPTIVE; family not linked): Re-checks of three Kogen fixes on the Claude Code lineup — https://bench.kogen.dev/rounds/claude-code-fix-rechecks-1/ - terra-reviewer-seeded-clean-1 (DESCRIPTIVE; family not linked): Terra as Kogen's reviewer: seeded faults and clean controls — https://bench.kogen.dev/rounds/terra-reviewer-seeded-clean-1/ ## Claim ledger index - CL-WA-harness-pick-2026-09-29-public-capture-identity (INTERIM; reproducible from public data for candidate capture counts; source-reported, not reproducible from public data for the historical design variants; candidate_delivery_capture): https://bench.kogen.dev/claims/cl-wa-harness-pick-2026-09-29-public-capture-identity/ - CL-WA-kh-compare-2026-09-29-public-capture-identity (INTERIM; reproducible from public data for candidate capture counts; source-reported, not reproducible from public data for the historical design variants; candidate_delivery_capture): https://bench.kogen.dev/claims/cl-wa-kh-compare-2026-09-29-public-capture-identity/ - CL-WA-claim-b-screen-1-public-capture-identity (CONFOUNDED; reproducible from public data for candidate capture counts; source-reported, not reproducible from public data for the historical design variants; candidate_delivery_capture): https://bench.kogen.dev/claims/cl-wa-claim-b-screen-1-public-capture-identity/ - CL-WA-claim-b-rerun-1-public-capture-identity (CONFOUNDED; reproducible from public data for candidate capture counts; source-reported, not reproducible from public data for the historical design variants; candidate_delivery_capture): https://bench.kogen.dev/claims/cl-wa-claim-b-rerun-1-public-capture-identity/ - CL-WA-confirm-combined-1-public-capture-identity (INTERIM; reproducible from public data for candidate capture counts; source-reported, not reproducible from public data for the historical design variants; candidate_delivery_capture): https://bench.kogen.dev/claims/cl-wa-confirm-combined-1-public-capture-identity/ - CL-WA-harness-compare-1-public-capture-identity (INTERIM; reproducible from public data for candidate capture counts; source-reported, not reproducible from public data for the historical design variants; candidate_delivery_capture): https://bench.kogen.dev/claims/cl-wa-harness-compare-1-public-capture-identity/ - CL-WA-hard1-calibrate-1-public-capture-identity (CONFOUNDED; reproducible from public data for candidate capture counts; source-reported, not reproducible from public data for the historical design variants; candidate_delivery_capture): https://bench.kogen.dev/claims/cl-wa-hard1-calibrate-1-public-capture-identity/ - CL-WA-kh-climb-reported-kh-climb-baseline (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-kh-climb-reported-kh-climb-baseline/ - CL-WA-kh-climb-reported-kh-climb-control (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-kh-climb-reported-kh-climb-control/ - CL-WA-kh-climb-reported-kh-climb-v1 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-kh-climb-reported-kh-climb-v1/ - CL-WA-kh-climb-reported-kh-climb-v2 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-kh-climb-reported-kh-climb-v2/ - CL-WA-kh-climb-reported-kh-climb-v3 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-kh-climb-reported-kh-climb-v3/ - CL-WA-kh-climb-reported-kh-climb-v4 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-kh-climb-reported-kh-climb-v4/ - CL-WA-kh-climb-reported-kh-climb-v5 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-kh-climb-reported-kh-climb-v5/ - CL-WA-x-ladder-reported-x-ladder-model-screen (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-ladder-reported-x-ladder-model-screen/ - CL-WA-x-ladder-reported-x-ladder-matrix (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-ladder-reported-x-ladder-matrix/ - CL-WA-x-ladder-reported-x-ladder-time-first (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-ladder-reported-x-ladder-time-first/ - CL-WA-x-ladder-reported-x-ladder-patient (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-ladder-reported-x-ladder-patient/ - CL-WA-x-ladder-reported-x-ladder-checkpoints (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-ladder-reported-x-ladder-checkpoints/ - CL-WA-x-models-reported-x-models-overall (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-models-reported-x-models-overall/ - CL-WA-x-models-reported-x-models-primary-screen (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-models-reported-x-models-primary-screen/ - CL-WA-x-models-reported-x-models-adjacent-panel (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-models-reported-x-models-adjacent-panel/ - CL-WA-x-models-reported-x-models-astra (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-models-reported-x-models-astra/ - CL-WA-x-roles-reported-x-roles-worker-pilot (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-roles-reported-x-roles-worker-pilot/ - CL-WA-x-roles-reported-x-roles-implementation (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-roles-reported-x-roles-implementation/ - CL-WA-x-roles-reported-x-roles-reviewer (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-roles-reported-x-roles-reviewer/ - CL-WA-x-roles-reported-x-roles-decision (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-roles-reported-x-roles-decision/ - CL-WA-x-review-reported-x-review-checklist (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-review-reported-x-review-checklist/ - CL-WA-x-review-reported-x-review-repair (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-review-reported-x-review-repair/ - CL-WA-x-review-reported-x-review-semantic-repair (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-review-reported-x-review-semantic-repair/ - CL-WA-x-race-reported-x-race (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-race-reported-x-race/ - CL-WA-x-race-reported-x-race-selector-check (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-race-reported-x-race-selector-check/ - CL-WA-x-race-reported-x-race-decision (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-race-reported-x-race-decision/ - CL-WA-x-parallel-reported-x-parallel-v03 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-parallel-reported-x-parallel-v03/ - CL-WA-x-parallel-reported-x-parallel-v03-cost (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-parallel-reported-x-parallel-v03-cost/ - CL-WA-x-parallel-reported-x-parallel-token-summary (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-parallel-reported-x-parallel-token-summary/ - CL-WA-x-minikogen-reported-x-minikogen-mechanism-pairs (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-minikogen-reported-x-minikogen-mechanism-pairs/ - CL-WA-x-minikogen-reported-x-minikogen-test-first-removal (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-minikogen-reported-x-minikogen-test-first-removal/ - CL-WA-x-minikogen-reported-x-minikogen-frozen-decision (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-minikogen-reported-x-minikogen-frozen-decision/ - CL-WA-x-surface-reported-x-surface-feedback-format (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-surface-reported-x-surface-feedback-format/ - CL-WA-x-surface-reported-x-surface-fixture-repair (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-surface-reported-x-surface-fixture-repair/ - CL-WA-x-surface-reported-x-surface-remaining-rep (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-surface-reported-x-surface-remaining-rep/ - CL-WA-x-context-reported-x-context (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-context-reported-x-context/ - CL-WA-x-context-reported-x-context-admission-guard (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-context-reported-x-context-admission-guard/ - CL-WA-x-compaction-reported-x-compaction-traces (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-compaction-reported-x-compaction-traces/ - CL-WA-x-compaction-reported-x-compaction-peak-input (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-compaction-reported-x-compaction-peak-input/ - CL-WA-x-compaction-reported-x-compaction-elision-review (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-compaction-reported-x-compaction-elision-review/ - CL-WA-x-compaction-reported-e06-context-pilot-link (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-compaction-reported-e06-context-pilot-link/ - CL-WA-x-continuation-reported-x-continuation-live (NOT-RUN; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-continuation-reported-x-continuation-live/ - CL-WA-x-continuation-reported-x-continuation-scripted-provider (NOT-RUN; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-continuation-reported-x-continuation-scripted-provider/ - CL-WA-x-recovery-reported-x-recovery (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-recovery-reported-x-recovery/ - CL-WA-x-mining-reported-x-mining-observation-inventory (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-mining-reported-x-mining-observation-inventory/ - CL-WA-x-mining-reported-x-mining-usage-audit (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-mining-reported-x-mining-usage-audit/ - CL-WA-x-mining-reported-x-mining-x09 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-x-mining-reported-x-mining-x09/ - CL-WA-e06-context-reported-e06-context-pilot (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-e06-context-reported-e06-context-pilot/ - CL-WA-ctx-store-reported-ctx-store-ranked-paths (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-ctx-store-reported-ctx-store-ranked-paths/ - CL-WA-ctx-store-reported-ctx-store-recall (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-ctx-store-reported-ctx-store-recall/ - CL-WA-ctx-store-reported-ctx-store-freshness (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-ctx-store-reported-ctx-store-freshness/ - CL-WA-e09-tidewave-reported-e09-locator-gates (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-e09-tidewave-reported-e09-locator-gates/ - CL-WA-e09-tidewave-reported-e09-tidewave-runtime (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-e09-tidewave-reported-e09-tidewave-runtime/ - CL-WA-night-2026-10-01-reported-night-overall (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-night-2026-10-01-reported-night-overall/ - CL-WA-night-2026-10-01-reported-night-p1 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-night-2026-10-01-reported-night-p1/ - CL-WA-night-2026-10-01-reported-night-p2 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-night-2026-10-01-reported-night-p2/ - CL-WA-night-2026-10-01-reported-night-p3a (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-night-2026-10-01-reported-night-p3a/ - CL-WA-night-2026-10-01-reported-night-p3c (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-night-2026-10-01-reported-night-p3c/ - CL-WA-night-2026-10-01-reported-night-p5 (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-night-2026-10-01-reported-night-p5/ - CL-WA-night-2026-10-01-reported-night-parallel-read (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-night-2026-10-01-reported-night-parallel-read/ - CL-WA-night-2026-10-01-reported-night-pi-mining (INTERIM; source-reported, not reproducible from public data; documentary_experiment_record): https://bench.kogen.dev/claims/cl-wa-night-2026-10-01-reported-night-pi-mining/ - CL-WA-hc-1-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; hill_climb_lifecycle_and_outcome): https://bench.kogen.dev/claims/cl-wa-hc-1-lifecycle-and-outcome/ - CL-WA-hc-2-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; hill_climb_lifecycle_and_outcome): https://bench.kogen.dev/claims/cl-wa-hc-2-lifecycle-and-outcome/ - CL-WA-hc-3-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; hill_climb_lifecycle_and_outcome): https://bench.kogen.dev/claims/cl-wa-hc-3-lifecycle-and-outcome/ - CL-WA-hc-4-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; hill_climb_lifecycle_and_outcome): https://bench.kogen.dev/claims/cl-wa-hc-4-lifecycle-and-outcome/ - CL-WA-r72-lifecycle-and-outcome (INTERIM; Reproducible from public L0 reconciliation CSVs for R72 P1 (64/84 vs 60/84) and partial P3 (12/13 vs 10/11); not independently joined to the canonical official outcome export (0 matching scored rows). Initial design and complete lifecycle remain source-reported, not reproducible from public data.; early_stop_round_denominators): https://bench.kogen.dev/claims/cl-wa-r72-lifecycle-and-outcome/ - CL-WA-frontier-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; frontier_lane_lifecycle): https://bench.kogen.dev/claims/cl-wa-frontier-lifecycle-and-outcome/ - CL-WA-studio-kogen-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; smoke_only_lifecycle): https://bench.kogen.dev/claims/cl-wa-studio-kogen-lifecycle-and-outcome/ - CL-WA-studio-kogen-80a4-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; smoke_only_lifecycle): https://bench.kogen.dev/claims/cl-wa-studio-kogen-80a4-lifecycle-and-outcome/ - CL-WA-studio-shaper-pilot-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; smoke_only_lifecycle): https://bench.kogen.dev/claims/cl-wa-studio-shaper-pilot-lifecycle-and-outcome/ - CL-WA-r57-studio-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; setup_invalid_pilot): https://bench.kogen.dev/claims/cl-wa-r57-studio-lifecycle-and-outcome/ - CL-WA-r57-t90-studio-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; smoke_only_lifecycle): https://bench.kogen.dev/claims/cl-wa-r57-t90-studio-lifecycle-and-outcome/ - CL-WA-r58x-studio-lifecycle-and-outcome (INTERIM; source-reported, not reproducible from public data; partial_round_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r58x-studio-lifecycle-and-outcome/ - CL-WA-r70-rve-r70-original-rve-elixir-full-pass (CONFOUNDED; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-r70-original-rve-elixir-full-pass/ - CL-WA-r70-rve-r70-original-rve-go-full-pass (CONFOUNDED; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-r70-original-rve-go-full-pass/ - CL-WA-r70-rve-r70-original-rve-rust-full-pass (CONFOUNDED; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-r70-original-rve-rust-full-pass/ - CL-WA-r70-rve-r70-original-rve-ts-bun-full-pass (CONFOUNDED; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-r70-original-rve-ts-bun-full-pass/ - CL-WA-r70-rve-rerun-r70-rve-rerun-elixir-full-pass (CONFOUNDED; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-rerun-r70-rve-rerun-elixir-full-pass/ - CL-WA-r70-rve-rerun-r70-rve-rerun-go-full-pass (CONFOUNDED; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-rerun-r70-rve-rerun-go-full-pass/ - CL-WA-r70-rve-rerun-r70-rve-rerun-rust-full-pass (CONFOUNDED; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-rerun-r70-rve-rerun-rust-full-pass/ - CL-WA-r70-rve-rerun-r70-rve-rerun-ts-bun-full-pass (CONFOUNDED; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-rerun-r70-rve-rerun-ts-bun-full-pass/ - CL-WA-r70-rve-ext-r70-rve-ext-elixir-full-pass (VALID; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-ext-r70-rve-ext-elixir-full-pass/ - CL-WA-r70-rve-ext-r70-rve-ext-go-full-pass (VALID; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-ext-r70-rve-ext-go-full-pass/ - CL-WA-r70-rve-ext-r70-rve-ext-rust-full-pass (VALID; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-ext-r70-rve-ext-rust-full-pass/ - CL-WA-r70-rve-ext-r70-rve-ext-ts-bun-full-pass (VALID; reproducible from public data; official_full_pass): https://bench.kogen.dev/claims/cl-wa-r70-rve-ext-r70-rve-ext-ts-bun-full-pass/ - CL-WA-r70-rve-ungraded-delivery-itt-gap (CONFOUNDED; reproducible from public data for 40 scored cells; source-reported, not reproducible from public data for the five extra deliveries; unmatched_ungraded_delivery_count): https://bench.kogen.dev/claims/cl-wa-r70-rve-ungraded-delivery-itt-gap/ - CL-WA-r70-task8v1-smoke-codex-gpt-6-luna-max-default-r70-8-go-r90 (DESCRIPTIVE; reproducible from public data; smoke_test_fraction): https://bench.kogen.dev/claims/cl-wa-r70-task8v1-smoke-codex-gpt-6-luna-max-default-r70-8-go-r90/ - CL-WA-r70-task8v1-smoke-codex-gpt-6-luna-max-default-r70-8-rust-r90 (DESCRIPTIVE; reproducible from public data; smoke_test_fraction): https://bench.kogen.dev/claims/cl-wa-r70-task8v1-smoke-codex-gpt-6-luna-max-default-r70-8-rust-r90/ - CL-WA-r70-task8v2-smoke-codex-gpt-6-luna-max-default-r70-8-go-v2-r1 (DESCRIPTIVE; reproducible from public data; smoke_test_fraction): https://bench.kogen.dev/claims/cl-wa-r70-task8v2-smoke-codex-gpt-6-luna-max-default-r70-8-go-v2-r1/ - CL-WA-r70-task8v2-smoke-codex-gpt-6-luna-max-default-r70-8-rust-v2-r1 (DESCRIPTIVE; reproducible from public data; smoke_test_fraction): https://bench.kogen.dev/claims/cl-wa-r70-task8v2-smoke-codex-gpt-6-luna-max-default-r70-8-rust-v2-r1/ - CL-WA-campfire-lane1-formal-cell-lifecycle (WITHDRAWN; source-reported, not reproducible from public data; withdrawn_formal_lifecycle): https://bench.kogen.dev/claims/cl-wa-campfire-lane1-formal-cell-lifecycle/ - CL-WA-campfire-lane2-formal-cell-lifecycle (WITHDRAWN; source-reported, not reproducible from public data; withdrawn_formal_lifecycle): https://bench.kogen.dev/claims/cl-wa-campfire-lane2-formal-cell-lifecycle/ - CL-WA-llm-latency-short-request-counts-and-endpoints (DESCRIPTIVE; reproducible from public raw logs; client_visible_latency): https://bench.kogen.dev/claims/cl-wa-llm-latency-short-request-counts-and-endpoints/ - CL-WA-llm-latency-long-request-counts-and-endpoints (DESCRIPTIVE; reproducible from public raw logs; client_visible_latency): https://bench.kogen.dev/claims/cl-wa-llm-latency-long-request-counts-and-endpoints/ - CL-WA-intent-coverage-audit-reported-observation (INTERIM; source-reported, not reproducible from public data; offline_intent_coverage_audit): https://bench.kogen.dev/claims/cl-wa-intent-coverage-audit-reported-observation/ - CL-WA-overstrict-replay-reported-observation (INTERIM; source-reported, not reproducible from public data; offline_overstrict_test_replay): https://bench.kogen.dev/claims/cl-wa-overstrict-replay-reported-observation/ - CL-WA-candidate-counterfactual-reported-observation (INTERIM; source-reported, not reproducible from public data; saved_candidate_counterfactual): https://bench.kogen.dev/claims/cl-wa-candidate-counterfactual-reported-observation/ - CL-WA-offline-review-reported-observation (INTERIM; source-reported, not reproducible from public data; historical_offline_review_proxy): https://bench.kogen.dev/claims/cl-wa-offline-review-reported-observation/ - CL-WA-offline-review3-reported-observation (INTERIM; source-reported, not reproducible from public data; frozen_offline_reviewer_comparison): https://bench.kogen.dev/claims/cl-wa-offline-review3-reported-observation/ - CL-WA-offline-contract-audit-reported-observation (INTERIM; source-reported, not reproducible from public data; offline_contract_audit): https://bench.kogen.dev/claims/cl-wa-offline-contract-audit-reported-observation/ - CL-WA-offline-aa-bo3-reported-observation (INTERIM; source-reported, not reproducible from public data; offline_false_promotion_and_selector_replay): https://bench.kogen.dev/claims/cl-wa-offline-aa-bo3-reported-observation/ - CL-WA-jev-route-ctx-reported-observation (INTERIM; source-reported, not reproducible from public data; offline_route_agreement): https://bench.kogen.dev/claims/cl-wa-jev-route-ctx-reported-observation/ - CL-WA-jev-triage-ctx-reported-observation (INTERIM; source-reported, not reproducible from public data; offline_failure_triage): https://bench.kogen.dev/claims/cl-wa-jev-triage-ctx-reported-observation/ - CL-WA-t98-cache-smoke-reported-observation (INTERIM; source-reported, not reproducible from public data; cache_telemetry_smoke): https://bench.kogen.dev/claims/cl-wa-t98-cache-smoke-reported-observation/ - CL-WA-t100-cache-lane-reported-observation (NOT-RUN; source-reported, not reproducible from public data; cache_lane_status): https://bench.kogen.dev/claims/cl-wa-t100-cache-lane-reported-observation/ - CL-WA-r53-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r53-public-export-coverage/ - CL-WA-r53b-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r53b-public-export-coverage/ - CL-WA-r56-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r56-public-export-coverage/ - CL-WA-r56b-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r56b-public-export-coverage/ - CL-WA-r56c-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r56c-public-export-coverage/ - CL-WA-r56d-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r56d-public-export-coverage/ - CL-WA-r56p2-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r56p2-public-export-coverage/ - CL-WA-r57-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r57-public-export-coverage/ - CL-WA-r57b-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r57b-public-export-coverage/ - CL-WA-r57c-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r57c-public-export-coverage/ - CL-WA-r57d-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r57d-public-export-coverage/ - CL-WA-r57e-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r57e-public-export-coverage/ - CL-WA-r64-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r64-public-export-coverage/ - CL-WA-r64b-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r64b-public-export-coverage/ - CL-WA-r64c-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r64c-public-export-coverage/ - CL-WA-r64d-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r64d-public-export-coverage/ - CL-WA-r64e-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r64e-public-export-coverage/ - CL-WA-r69-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r69-public-export-coverage/ - CL-WA-r70-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r70-public-export-coverage/ - CL-WA-r71-public-export-coverage (INTERIM; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r71-public-export-coverage/ - CL-WA-r74-public-export-coverage (WITHDRAWN; reproducible from public data for public export counts; source-reported, not reproducible from public data for historical/planned denominator variants; separate_public_export_row_counts): https://bench.kogen.dev/claims/cl-wa-r74-public-export-coverage/ - CL-WA-rails-catalogue-plan-shell-public-export-vs-reported (INTERIM; reproducible from public data for export counts/cost arithmetic; source-reported, not reproducible from public data for historical denominators and cost claims; retrospective_rails_export_pass_rate_and_denominator): https://bench.kogen.dev/claims/cl-wa-rails-catalogue-plan-shell-public-export-vs-reported/ - CL-WA-rails-catalogue-sol-high-public-export-vs-reported (INTERIM; reproducible from public data for export counts/cost arithmetic; source-reported, not reproducible from public data for historical denominators and cost claims; retrospective_rails_export_pass_rate_and_denominator): https://bench.kogen.dev/claims/cl-wa-rails-catalogue-sol-high-public-export-vs-reported/ - CL-WA-rails-catalogue-luna-max-public-export-vs-reported (INTERIM; reproducible from public data for export counts/cost arithmetic; source-reported, not reproducible from public data for historical denominators and cost claims; retrospective_rails_export_pass_rate_and_denominator): https://bench.kogen.dev/claims/cl-wa-rails-catalogue-luna-max-public-export-vs-reported/ - CL-WA-r53-source-reported-denominator-variants (INTERIM; source-reported, not reproducible from public data; historical_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r53-source-reported-denominator-variants/ - CL-WA-r56-source-reported-denominator-variants (INTERIM; source-reported, not reproducible from public data; historical_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r56-source-reported-denominator-variants/ - CL-WA-r57-source-reported-denominator-variants (INTERIM; Reproducible from the public L0 reconciliation CSV and public outcome export for the recovered 311-cell pooled population and published interim analysis. One planned Studio shell-only slot has no recovered cell ID or outcome, so the full planned population remains scoped and incomplete. INTERIM. The complete planned 312-cell population remains source-reported, not reproducible from public data.; historical_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r57-source-reported-denominator-variants/ - CL-WA-r64-source-reported-denominator-variants (INTERIM; source-reported, not reproducible from public data; historical_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r64-source-reported-denominator-variants/ - CL-WA-r69-source-reported-denominator-variants (INTERIM; Reproducible from the public L0 reconciliation CSV for the recovered 400-row assignment, 396 included ITT rows, arm counts, and intervals. The reconstruction has 327 internal official grades, of which 150 now match the refreshed public official outcome export; the official join remains partial. INTERIM. The full historical ITT partition remains source-reported, not reproducible from public data.; historical_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r69-source-reported-denominator-variants/ - CL-WA-r70-source-reported-denominator-variants (INTERIM; source-reported, not reproducible from public data; historical_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r70-source-reported-denominator-variants/ - CL-WA-r71-source-reported-denominator-variants (INTERIM; source-reported, not reproducible from public data; historical_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r71-source-reported-denominator-variants/ - CL-WA-r74-source-reported-denominator-variants (WITHDRAWN; Reproducible from public L0 reconciliation CSVs for the recovered 167 internal scored rows, 9 smoke attempts, and coverage counts. These rows are not independently joined to the canonical official outcome export; 231 retained slots lack recovered cell IDs and terminal statuses. WITHDRAWN. 231 retained slots remain source-reported, not reproducible from public data.; historical_denominator_conflict): https://bench.kogen.dev/claims/cl-wa-r74-source-reported-denominator-variants/ - CL-H111-historical-staged-and-direct-comparison (INTERIM; source-reported, not reproducible from public data; historical_staged_and_direct_comparison_claim): https://bench.kogen.dev/claims/cl-h111-historical-staged-and-direct-comparison/ - CL-H112-historical-route-cost-frontier (INTERIM; source-reported, not reproducible from public data; historical_route_cost_frontier_claim): https://bench.kogen.dev/claims/cl-h112-historical-route-cost-frontier/ - CL-H115-historical-cache-hit-comparison (INTERIM; source-reported, not reproducible from public data; historical_cache_hit_comparison_claim): https://bench.kogen.dev/claims/cl-h115-historical-cache-hit-comparison/ - CL-H116-historical-cache-change (INTERIM; source-reported, not reproducible from public data; historical_cache_change_claim): https://bench.kogen.dev/claims/cl-h116-historical-cache-change/ - CL-L3-REPAIR-RESCUE-RATE (VALID; Reproducible from six published aggregate repair rows; individual test-set preservation is unverifiable.; repair_rescue_count): https://bench.kogen.dev/claims/cl-l3-repair-rescue-rate/ - CL-L6-AUDITOR-ADVISORY-REPLAY (VALID; Reproducible from the published offline case and call ledgers; not a live Build-gate estimate.; offline_auditor_target_check): https://bench.kogen.dev/claims/cl-l6-auditor-advisory-replay/ - CL-SOL-LANG-REPLICATION-POSTHOC-EQUAL-TASK (VALID; Reproducible from the published round-local CSV; equal-task formula is post-hoc because the original combined formula was not recorded.; posthoc_equal_task_pass_rate): https://bench.kogen.dev/claims/cl-sol-lang-replication-posthoc-equal-task/ ## Evidence map - 1.1 Process model: https://bench.kogen.dev/evidence/11-process-model/ - 2.1 Intent .kogen/intents//intent.md: https://bench.kogen.dev/evidence/21-intent-kogenintentsintentmd/ - 2.4 Acceptance tests and stack adapters: https://bench.kogen.dev/evidence/24-acceptance-tests-and-stack-adapters/ - 2.5 Approval, protection and refs: https://bench.kogen.dev/evidence/25-approval-protection-and-refs/ - 3.0 Rules: https://bench.kogen.dev/evidence/30-rules/ - 3.1 The recipe: ladder: https://bench.kogen.dev/evidence/31-the-recipe-ladder/ - 3.7 Verification and base-relative checks: https://bench.kogen.dev/evidence/37-verification-and-base-relative-checks/ - 3.8 Verdict, audit, selection: https://bench.kogen.dev/evidence/38-verdict-audit-selection/ - 3.9 Commit and landing: https://bench.kogen.dev/evidence/39-commit-and-landing/ - 4.1 Provider port and test seams: https://bench.kogen.dev/evidence/41-provider-port-and-test-seams/ - 4.9 Provider requests and prompt caching: https://bench.kogen.dev/evidence/49-provider-requests-and-prompt-caching/ - 4.10 Grok provider: https://bench.kogen.dev/evidence/410-grok-provider/ - 5.1 Process custody (success-relevant: deadlines, hangs, ARG_MAX): https://bench.kogen.dev/evidence/51-process-custody-success-relevant-deadlines-hangs-arg_max/ - 6.1 Out of core v1: https://bench.kogen.dev/evidence/61-out-of-core-v1/ - C.1 Definition: https://bench.kogen.dev/evidence/c1-definition/ - Sources and precedence: https://bench.kogen.dev/evidence/sources-and-precedence/ ## Data and methods - [METHOD.md](https://github.com/KogenAI/kogen-bench/blob/607dfaf490e2e2bb13f78470ee8962028e2ed179/METHOD.md) - [STANDARD.md](https://github.com/KogenAI/kogen-bench/blob/607dfaf490e2e2bb13f78470ee8962028e2ed179/STANDARD.md) - [reproduce/README.md](https://github.com/KogenAI/kogen-bench/blob/607dfaf490e2e2bb13f78470ee8962028e2ed179/reproduce/README.md) - [Downloads](https://bench.kogen.dev/data/) - [Machine index](https://bench.kogen.dev/index.json) - [Snapshot hashes](https://bench.kogen.dev/snapshot.json)