F12

F12 — Explicit lever and recipe arms

The authored family record is reproduced below. Structured links connect its hypotheses, rounds, and claim-ledger rows.

Source-reported narrativeNumeric statements in this family document remain source-reported unless an exact claim-ledger row is linked and its status allows that interpretation.

Family source document · hypotheses/f12-lever-recipes.md

F12 — Explicit lever and recipe arms

Status: INTERIM. A flag or recipe name is not evidence that a distinct arm ran. Historical screens often bundle changes, use different task/host cohorts, or lack a cell-to-arm join. The register below records what would falsify each proposition and whether a separate, attributable comparison is established in the public record.

Hypotheses

  • H117 — Planner context pack. A compact context pack may produce a better or cheaper plan than a file-list input. Compare the two inputs with all other planner and builder settings fixed.
  • H118 — Plan guard. A plan sanity/constraint gate may prevent plans that would otherwise cause Build failures. Compare downstream outcomes and gate cost with and without the guard.
  • H119 — Plan examples. Relevant examples may improve transfer from plan to implementation. Hold the plan and builder settings fixed while changing example exposure.
  • H120 — Plan reproduction evidence. Evidence that reproduces a failure may improve the plan's diagnosis and implementation. Compare with an otherwise identical plan without that evidence.
  • H121 — Plan interfaces. Explicit cross-component interfaces may reduce integration failures. Compare matched tasks with interfaces present or absent.
  • H122 — Planner turns. Varying the number of planner turns may change completion enough to justify its time and token cost. Compare fixed turn budgets.
  • H123 — Plan debate. Competing plans may improve plan choice and downstream task success. Compare debate against a single-plan control.
  • H124 — Plan specificity. Criteria, approach, and step-level plans may affect builders differently. Compare each level within builder and task strata.
  • H125 — Review scope. A narrower review surface may reduce cost while preserving useful findings. Compare findings, false blocks, missed defects, and review cost.
  • H126 — Executable review. Running review checks may detect defects that text-only review misses. Measure true and false review findings and prospective repair outcomes.
  • H127 — Contract gate. Checking requirements before acceptance or escalation may catch incomplete work. Compare detection and false-block rates, then measure Build outcomes after any repair.
  • H128 — Best-of-N. Generating and selecting among candidates may increase successful completion enough to justify the additional attempts. Compare selected outcomes with the same direct baseline and report pass@N separately.
  • H129 — Hybrid pipeline. Combining staged modes may improve the pass/cost frontier over its constituent recipes. Compare the complete bundle and every constituent arm.
  • H130 — Split work. Dividing implementation among workers may increase successful throughput. Compare a fixed split plan with a solo arm, including coordination cost and wall time.
  • H131 — Routing policies. Dynamic, forced, or versioned routing may assign useful models more effectively than fixed routes. Compare on the same tasks and model/effort choices.
  • H132 — Route cascade. Triggering escalation only after failure may preserve quality at lower cost than always escalating. Compare policy-level success and cost against fixed direct models.
  • H133 — Cascade gate. Contract, review, or combined gates may differ in their ability to trigger useful escalation. Compare gate sensitivity, false triggers, success, and cost.
  • H134 — Context handoff. Passing prepared context to later stages may reduce repeated work or cost without reducing completion. Compare a handoff arm with a matched no-handoff arm.
  • H135 — Recipe attribution. Staged, direct, direct-shell, direct-escalate, plan-shell, and escalate-shell recipes may differ in outcome and cost. Each needs an explicit arm manifest and direct within-cohort comparison.
  • H136 — Ladder composition. Luna-only, Sol-effort, and diverse ladders may differ in success and cost. Compare each ladder under a fixed task set and trigger rule.
  • H137 — Harness-version effect. Changes in Codex CLI or Claude Code versions may change earlier harness comparisons. Compare versions on identical tasks, requests, and host conditions.
  • H138 — Matched model/effort baselines. Kogen configurations with Luna max or Sol low/medium/high builders, with and without Sol planning, may differ from direct baselines. Match the effective model and effort at each tested role.
  • H139 — Raw versus shaped Intent. Supplying a raw external-project Intent instead of a provided shaped Intent may change Kogen's apparent advantage. Compare both inputs against the same task and model baselines.

Lever register

“Not established as a distinct run” means no exact public arm manifest and cell join establish that the named contrast ran independently. It does not assert that the flag was never used. Adjacent bundles and offline studies are listed as context, not as proof of the isolated lever.

H Candidate flag or recipe Status in the public evidence Adjacent records and remaining contrast
H117 Compact pack vs ls-files Not established as a distinct run. R56 and x-context are bundled or incompletely mapped; the public record does not identify a clean pack-only contrast. See R56, x-context.
H118 --plan-guard Not established as a distinct run. No exact guard-only arm and downstream Build join identified.
H119 --plan-examples Not established as a distinct run. No exact examples-only arm identified.
H120 --plan-repro Not established as a distinct run. No exact reproduction-evidence-only arm identified.
H121 --plan-interfaces Not established as a distinct run. No exact interface-only arm identified.
H122 --plan-agentic N Not established as a distinct run. No isolated planner-turn dose comparison identified.
H123 --plan-debate Not established as a distinct run. Debate appears in broader exploratory reports, but no public, reconciled debate-only cohort is identified.
H124 --plan-level criteria/approach/steps A planning-arm cohort appears in R51; its summary is withheld pending cell-level reconciliation. R51 records planning labels crossed with Luna low, Luna max, and Sol low on eight listed tasks. Its public page does not support a single reconciled comparison. See R51, measurement contract, and the public cell and delivery exports.
H125 --review-scope Not established as a distinct run. No matched narrow-vs-full review surface with reconciled cost and defect outcomes identified.
H126 --review-exec No live, prospective executable-review benefit is established. Offline review studies are diagnostic and not a live Build repair comparison. See offline review, offline review 3.
H127 --contract No prospective Build benefit is established. Contract audits and historical gates do not isolate a repair effect. See offline contract audit, R49b.
H128 --best-of N No deployable best-of-N benefit is established. Offline selection analysis is not a live selection policy comparison; see offline A/A and best-of-three.
H129 --hybrid Not established as a distinct run. No complete manifest maps a hybrid label to a matched constituent set and outcome join.
H130 --split N No split-specific outcome comparison is established. The separate x-parallel pilot compared solo with a planned DAG; it does not establish an effect for this flag or for a general split policy. See x-parallel.
H131 --route, --route-v3, --route-forced The historical route-v3 summary is source-reported and not reproducible from public data. Keep the H112 route summary separate from offline route-prediction accuracy; see `CL-H112-historical-route-cost-frontier` (INTERIM) and Jev route context.
H132 --route-cascade, --route-cascade-contract No validated cascade policy ranking is established. R49 and R49b have different capture/grade coverage and no recovered common decision rule. See R49, R49b.
H133 --model-cascade --mc-gate contract/review/both No final gate ranking is established. R49b contains contract/review arms, but their cross-cohort outcomes do not isolate gate sensitivity or policy-level efficacy. See R49b.
H134 --ctx-handoff R54 reports a handoff vs plain-pack comparison; its round status remains INTERIM. The public page does not establish a rereading or cost reduction. Keep the comparison attached to R54; do not infer a benefit. See R54.
H135 staged, direct, direct-shell, direct-escalate, plan-shell, escalate-shell Multiple labels occur across rounds, but the public recipe-to-cell index is incomplete. Keep R53, R56, R57, R64, R65b, and other cohorts separate. R53/R56/R57/R64 historical summaries remain partly unreconciled; see their round pages and claim-ledger status rows.
H136 ladder, ladder-luna, ladder-sol-low/medium/high, ladder-diverse Several exploratory ladder rounds ran; no general best ladder is established. The hill-climb iterations, R72, and R74 have separate lifecycle records; R72 is descriptive, the final hill-climb outcome is unresolved, and R74 is WITHDRAWN. See hc-1, hc-2, hc-3, hc-4, R72, R74.
H137 Codex CLI / Claude Code version comparisons No controlled version comparison is established. Versions occur in separate dated cohorts; version, task, host, and setup changes are not separated in a public version A/B. See harness-pick, R70 RvE.
H138 Matched Codex and Kogen model/effort baselines No complete current-version matched set is established. R72 is descriptive on a superseded revision; R74 is withdrawn; frontier is incomplete. Do not pool them. See R72, R74, frontier.
H139 INTENT_SOURCE=raw vs provided shaped Intent No exact raw-vs-provided arm join is established. R69 includes Intent arms but does not establish this specific contrast in the public export. See R69 and its source-denominator row in the claim ledger.

Experiment and cohort register

This table keeps runs separate from the hypotheses and from the named flags. Counts shown in round pages are not silently promoted to claim-ledger outcomes. Unknown fields are shown explicitly; the linked reproduction sections contain the most complete available run manifests.

experiment_id Arm; task set; model/effort Revision; host/epoch planned / started / finished / graded / ITT n Outcome and status Correction or supersession Claim ID and public receipt
R51 planning labels (H124) None/approach/criteria/steps crossed with Luna low, Luna max, Sol low; the task list is on the round page. Per-cell Kogen/harness revisions are not joined to an interpretable comparison; environment details are incomplete. Public deliveries and official grades are recorded in the snapshot; the planned/start/finish/ITT partition is not recovered. Public verdict withheld; INTERIM. Earlier plan-level headlines were removed pending the cell-level crosswalk. No R51 claim ID is assigned in the claim ledger; R51, cells, run records.
R54 context handoff (H134) pack-plan-handoff vs pack-plan; task list is on the round page; per-cell model/effort not recovered. Harness/Kogen pin not recovered; US and EU public host labels appear in run records, but pair balance/epoch is not established. Planned n not recorded; public capture/grade counts and the exact ITT partition are not reconciled. Outcome remains INTERIM; no demonstrated pass or cost benefit. Do not infer reuse from token or outcome summaries; see the round's limitations. No R54 claim ID is assigned in the claim ledger; R54; run records.
R56 stage/context ablation (H117, H118, H125–H127, H129) Full pipeline and removed-stage arms; exact manifests and per-cell task/model mapping not reconciled in the public page. Source commits, host/epoch pairing, and effective effort not established for a single contrast. Public capture and official export counts differ; historical analysis denominator is separately source-reported. Historical stage deltas are source-reported, not reproducible from public data; one-factor effects were underpowered and did not meet the stated retention rule. Keep the R56 full pipeline, ablations, and later R56 variants separate. `CL-WA-r56-public-export-coverage` (INTERIM); `CL-WA-r56-source-reported-denominator-variants` (INTERIM); R56.
R57 shell recipe comparison (H135) Shell-only vs default Kogen tool surface; historical pooled task/arm mapping remains incomplete. No per-cell task/host balance is established for the historical pooled summary. L0 reproduces a scoped 311-row recovered population with matching public outcomes; one planned desktop Mac shell-only slot remains without a recovered cell ID or outcome. Shell-only 114/155 and default tools 111/156 plus the published interim bound reproduce from L0 CSVs. INTERIM; scoped recovered population; no general shell superiority claim. Do not merge R57 with R57b–R57e or desktop Mac pilots without a registered rule. `CL-WA-r57-public-export-coverage` (INTERIM); `CL-WA-r57-source-reported-denominator-variants` (INTERIM); R57; L0 reconciliation.
R49/R49b cascade and gate captures (H132, H133) Model cascade labels; R49 task list and R49b task list are distinct; exact model/effective-effort mapping is incomplete. Harness/revision and host differences are not reconciled into one comparison. R49 public page lists captured deliveries; R49b has a separate capture and analysis. Overall lifecycle partitions are unresolved. No common policy verdict. R49b's contract/review outcomes do not identify which gate is better. INTERIM. Preserve each round and its task/arm cohort separately. R49; R49b; R49b measurement sources.
hc-1 (H136) Ladder recipes on a development task set; exact task/model/effort details are in the round receipt and not a benchmark panel. desktop Mac; exact host epoch and complete per-cell pin join are not present in the public export. 40 grades are source-reported; ITT and task-level cell join are incomplete. Outcome figures are source-reported; INTERIM / RECONCILIATION_QUEUE. Keep separate from hc-2 to hc-4 and R72. `CL-WA-hc-1-lifecycle-and-outcome`; hc-1.
hc-2 (H136) Ladder, diverse ladder, and floor control; v2 decision tasks plus separate v1 guards. desktop Mac; Kogen revision recorded in the round receipt; full cell join remains incomplete. 90 grades reported including scored cells and guards; ITT is not a promotion denominator. Exploratory screen; ladder retained within this dev sequence, diverse and floor not promoted. INTERIM. Guards are not pooled with scored v2 cells. `CL-WA-hc-2-lifecycle-and-outcome`; hc-2.
hc-3 (H136) Ladder vs generated edge-test ladder, with separate guards; v2 development set. desktop Mac; per-cell source joins remain incomplete. 60 grades reported across scored and guard cells; exact ITT partition unresolved. Exploratory screen; edge-test variant not promoted. INTERIM. Keep guards and scored cells distinct; do not extrapolate to held-out tasks. `CL-WA-hc-3-lifecycle-and-outcome`; hc-3.
hc-4 (H136, H138) Ladder with coupled planner/shaper effort variants; v2 development and guard tasks. desktop Mac; exact final per-cell manifest and full revision join unavailable. A trimmed design expected 43 retained cells, including diagnostics; closed scored/graded n is not verified. INTERIM; outcome unresolved. Do not use an early checkpoint as final closure. `CL-WA-hc-4-lifecycle-and-outcome`; hc-4.
R72/R74 ladder and model-effort comparisons (H136, H138) Distinct R72 P1/P2/P3 and R74 arms; see R72 and R74 for the recorded model, task, revision, and reproduction limits. R72 uses a superseded Kogen revision; R74 is withdrawn and has no public tagged cells. L0 reproduces R72 P1 (64/84 vs 60/84) and partial P3 (12/13 vs 10/11); these CSV rows do not match the canonical public outcome export. L0 reconstructs selected R74 counts, but 231 retained slots lack recovered IDs and terminal statuses. R72 remains INTERIM, descriptive only with no superiority claim; R74 remains WITHDRAWN and is not a full cohort reconstruction. Never aggregate across R72, R74, hill-climb iterations, or frontier. `CL-WA-r72-lifecycle-and-outcome` (INTERIM); `CL-WA-r74-source-reported-denominator-variants` (WITHDRAWN); R72; R74; L0 reconciliation.
R64 plan-shell and direct baselines (H135, H138) Plan-shell vs Codex direct arms; a 17-task historical Rails summary is not reproduced by public capture. Kogen is not used by the Codex baseline; harness source commit is not recorded; task/host capture is incomplete. Public R64 capture and official outcomes are smaller than the historical 85-cell summary. Historical values are source-reported, not reproducible from public data; no plan-shell superiority claim. Preserve R64, R64b–R64e as distinct cohorts. `CL-WA-r64-public-export-coverage` (INTERIM); `CL-WA-r64-source-reported-denominator-variants` (INTERIM); R64.

Design implications

  • Require a frozen recipe manifest with each lever's exact flag, model role, prompt/config hash, task IDs, effective effort, revision, and stop behavior.
  • Use a one-factor contrast when estimating a lever; register bundle and interaction tests separately.
  • Give claim-bearing model, routing, and speed comparisons matched tasks, effective settings, hosts, and epochs, with exact cell IDs and complete ITT disposition.
  • Keep developer-set tuning rounds separate from held-out evidence and report selection history.

Explicit non-implications

  • A named command-line flag does not prove that the flag was separately tested or caused an outcome.
  • A bundle that works on one selected task does not establish a transferable recipe.
  • An offline reviewer or selector diagnostic does not show that a live Build improves.
  • A screen that does not show a lift does not prove a model ceiling or equivalence.
  • R72, R74, and hill-climb iteration results are not interchangeable ladder observations.

Smallest open tests

  • Freeze a public recipe manifest for each candidate lever and map it to immutable cell IDs before grading.
  • Run one registered one-factor comparison per arm on the same task set, with direct and bundle controls, matched effective model/effort, and full request, cost, wall, and ITT records.
  • For routing and cascade gates, compare policies on a held-out cohort and include threshold calibration, false-trigger rates, and all helper costs.
  • For shell or split claims, measure correctness, test coverage, requests, and wall on matched tasks; do not substitute task throughput for completion quality.