F04

F04 — Models, efforts and routing

The authored family record is reproduced below. Structured links connect its hypotheses, rounds, and claim-ledger rows.

Source-reported narrativeNumeric statements in this family document remain source-reported unless an exact claim-ledger row is linked and its status allows that interpretation.

Family source document · hypotheses/f04-models-routing.md

F04 — Models, efforts and routing

Coverage: H39–H50. Model and effort effects are task-, role-, and version-specific. Reported small screens remain separate from official joined outcomes. Wave B claim IDs link to `results/claim-ledger.jsonl`; a source-reported claim is not reproducible simply because it has a stable claim ID.

Hypotheses

ID Falsifiable proposition Current state
H39 A role-specific model/effort allocation improves the declared completion-cost-time frontier over a fixed allocation on matched tasks. UNRESOLVED; no complete role-by-role public comparison.
H40 The newer Luna model has a better cost-success profile than the earlier Luna baseline on the same tasks and harness. NOT ESTABLISHED; model preference alone is not a measured comparison.
H41 Increasing Luna effort improves completion more on harder tasks and yields a measurable cost tradeoff. UNRESOLVED; no full difficulty-stratified effort curve.
H42 The compared Sol, Luna, and Astra settings have distinguishable success, time, or cost profiles when builder role and tasks are held fixed. UNRESOLVED; associated screens are exploratory.
H43 The Sol low-effort setting outperforms Luna max in shaping, review, or synthesis roles enough to justify added cost. NOT ESTABLISHED; no isolated role comparison.
H44 Luna low plus strong-model assistance exceeds matched direct Luna-low and direct-Sol options on a declared cost/time-success frontier. NOT ESTABLISHED; no exact public comparison.
H45 Sol medium has a higher pass rate than Codex on a predeclared hard-task population. UNRESOLVED; the public R66 record does not preserve the historical high-effort anchor cohort.
H46 Astra is not cost-effective for routine tasks but improves outcomes enough on a capped high-value task set to justify a targeted route. NOT ESTABLISHED; a small screen is insufficient for a policy.
H47 Escalating after an observed failure beats fixed direct models or text-based routing under equal task and cost rules. UNRESOLVED; cascade summaries are not reconciled.
H48 Routing from task text has worse accuracy or cost than a matched policy using observed outcomes or context. NOT ESTABLISHED; offline routing agreement is not live Build success.
H49 Luna max for most implementation/operator work with Sol reserved for planning/rules is on the declared cost-success frontier. NOT ESTABLISHED; operating preference is not a measured optimum.
H50 A combined plan-level, role, reviewer, and contract route beats direct Sol on a preregistered completion-cost frontier. NOT ESTABLISHED; available panels are not one matched frontier test.

Experiment register

experiment_id Arm / task set Model / effort Revision; host / epoch Planned / started / finished / graded / ITT n Outcome, status, correction or supersession Claim ID Public receipt
x-models Primary and adjacent model screens, plus separate Astra panel; task sets are not interchangeable. Luna/Sol/Astra combinations are in the receipt; role and effort vary. Exact cell pins and task-level host mapping are unavailable. Wave B records a documentary inventory, not exact joined cells. INTERIM; no model or effort default follows. Primary screen is labelled inconclusive in the receipt. CL-WA-x-models-reported-x-models-overall; CL-WA-x-models-reported-x-models-primary-screen; CL-WA-x-models-reported-x-models-adjacent-panel; CL-WA-x-models-reported-x-models-astra (all INTERIM). round, reported data, claim ledger
x-ladder Model/effort screens, ladder profiles, time-first and patient variants; separate scans remain distinct. Luna and Sol settings described in the receipt. Exact cell revisions and host/epoch map unavailable. Exact joined population not recovered. INTERIM; small source-reported screens and checkpoint inventory do not identify a best effort or ladder. CL-WA-x-ladder-reported-x-ladder-model-screen; CL-WA-x-ladder-reported-x-ladder-matrix; CL-WA-x-ladder-reported-x-ladder-time-first; CL-WA-x-ladder-reported-x-ladder-patient; CL-WA-x-ladder-reported-x-ladder-checkpoints (all INTERIM). round, reported data, claim ledger
x-roles Worker, implementation, reviewer, and decision screens; keep each role-specific population separate. Luna/Sol combinations; review effort and builder effort are not fully aligned. Exact pins and host/epoch joins unavailable. No exact public task/cell join. INTERIM; no model-role allocation was promoted. CL-WA-x-roles-reported-x-roles-worker-pilot; CL-WA-x-roles-reported-x-roles-implementation; CL-WA-x-roles-reported-x-roles-reviewer; CL-WA-x-roles-reported-x-roles-decision (all INTERIM). round, reported data, claim ledger
r25–r30 Early route/model combinations; each task and arm table remains on its separate round page. See each receipt; the family is not one matched model/effort panel. Different rounds and revisions; do not pool epochs. See separate receipts; no common lifecycle denominator. Exploratory/historical records; no overall role or model winner is supported. No H39–H50-linked Wave B claim ID. r25, r26, r27, r28, r29, r30
r48b–r52 Cascade, model, and effort variants; r49/r49b are separate and retain their own arm labels. Model/effort settings and clamp notes are per-round; see receipts. Per-round pins and hosts differ. Round-specific counts are not a single pooled panel. INTERIM; source-reported comparisons do not establish a cost frontier or a cascade gate. No H39–H50-linked Wave B claim ID. r48b, r49, r49b, r50, r51, r52
r59 Luna high versus medium and a historical max anchor; separate cohorts. Luna high/medium; historical max is not a matched contemporaneous arm. See receipt; anchor epoch differs. Receipt-specific population; no common matched lifecycle denominator. Descriptive only; no full effort curve or deployment threshold. No H39–H50-linked Wave B claim ID. r59
r64 / r64b Plan-shell and direct anchors on the Rails panel. Kogen requested max is clamped to xhigh; direct Sol setting is listed in the receipt. Receipts describe the source and host panels; model-effort mismatch remains relevant. Public captures do not reconstruct the source-reported ITT population. INTERIM; historical comparison and cost/pass are source-reported, not reproducible from public data; do not claim equivalence to direct Sol. No H39–H50-linked Wave B claim ID. r64, r64b
r65b Plan-shell/escalate-shell versus direct Codex anchors on the listed held-out set. Kogen and direct model settings are on the receipt; retain direct anchors separately. See receipt; host and stop exclusions are part of its population. Scored-cell rows exist, but round-specific reconciliations and exact hypothesis mapping remain separate from this page. Descriptive receipt; no all-task frontier or general model-routing conclusion. No H39–H50-linked Wave B claim ID. r65b
r66 Sol medium, historical Sol high, and Luna max; historical high anchor is not identified by the public filter. Sol medium / Sol high / Luna max; see round corrections. Per-cell settings and revision differences are described in the receipt. Current medium cells are available descriptively; historical anchor cohort is not publicly identified. INTERIM; do not infer a hard-task Sol-medium win from the prior headline. No H39–H50-linked Wave B claim ID. r66
r68b Plan-shell kh and direct-model anchors on a held-out panel. Exact effective effort and clamp notes are on the receipt. Separate host/version context; do not merge with r64/r65b. Receipt-specific; no shared lifecycle population. CONFOUNDED in the family map; descriptive counts do not settle H44/H50. No H39–H50-linked Wave B claim ID. r68b
r72 Same-model ladder comparison; P1/P2/P3 remain separate. P1 Luna max, P3 direct Sol high per the source-reported page. Superseded Kogen pin; task IDs and comparator harness are incomplete. Exact ITT partition not recovered. INTERIM and descriptive; no current-version ladder advantage. CL-WA-r72-lifecycle-and-outcome (INTERIM; Wave B maps it to H113/H136/H138). r72, claim ledger
r74 Planned ladder/configuration comparison against direct Codex; no public per-cell arms. Model-role design is recorded; no per-cell public model rows. Planned Kogen pin only; delivered-cell and harness pins are unreconciled. Planned cohort is documented; the public export has no tagged outcomes. WITHDRAWN; no comparative conclusion. CL-WA-r74-public-export-coverage; CL-WA-r74-source-reported-denominator-variants (WITHDRAWN). r74, claim ledger
jev-route-ctx Offline context-prepared routing diagnostic; not a live Build arm. Offline route models are described in the receipt. Evaluation version and analysis records are not included in the public tree. Route decisions are the analysis unit; no live Build cells. INTERIM; route agreement is not accuracy or Build success. CL-WA-jev-route-ctx-reported-observation (INTERIM; no H IDs assigned in Wave B). round, claim ledger
jev-triage-ctx Offline failure-locus labeling diagnostic; planned full pass did not run. Model/effort is in the receipt; no live builder arm. Source records are absent from the public page. Offline sample only; no live Build cells. INTERIM; progression gate failed, so this is not a task-routing or efficacy result. CL-WA-jev-triage-ctx-reported-observation (INTERIM; no H IDs assigned in Wave B). round, claim ledger

Design implications

Compare model and effort at a fixed role, task set, harness, prompt, attempt budget, and host/epoch. Record requested and effective effort separately; an effort clamp changes the treatment. Include cost and wall time as separate outcomes and preserve stopped or invalid assignments in the declared analysis population.

What this evidence does not show

The exploratory x-models and x-ladder summaries do not establish a default model, effort curve, Astra policy, or model cascade. R59’s historical anchor, R64’s effort clamp, R66’s unidentified historical control, R68b’s confounds, and withdrawn R74 cannot be combined into one frontier. Routing agreement alone does not establish better Build outcomes.

Smallest open tests

Run a preregistered task-stratified matrix with Luna and Sol effort levels plus a capped Astra arm, recording effective settings, usage, grades, time, and cost on exact IDs. Separately compare one fixed direct route, text-based routing, and failure-triggered escalation under the same per-task budget and a held-out confirmation set.