F07
F07 — Tasks, grading, and method
The authored family record is reproduced below. Structured links connect its hypotheses, rounds, and claim-ledger rows.
Family source document · hypotheses/f07-tasks-method.md
F07 Tasks, grading, and method
This page records evidence for H75–H90. These propositions concern task selection, grading, study design, and reproducibility. A methodological recommendation is not itself a measured Build effect.
Hypotheses and current status
| ID | Falsifiable proposition | Current evidence status |
|---|---|---|
| H75 | A cheap starting task and one-lever-at-a-time climb identifies useful task and recipe changes without mistaking saturation for general success. | INTERIM. Several desktop Mac hill-climb iterations are reported, but cell-level outcomes and the final iteration are not fully reconciled in the public export. |
| H76 | Luna or Grok is the better low-cost, fast starting builder for the intended task set. | NOT ESTABLISHED. No matched comparison for this starting-model choice is linked to a public claim. |
| H77 | A benchmark limited to Rails tasks can misstate performance on Elixir/Phoenix or Rust work. | NOT ESTABLISHED. Cross-stack task and implementation records exist, but no matched task-set study isolates stack or recency effects. |
| H78 | Public real-codebase tasks pinned to source revisions are more reproducible than hidden replay tasks. | NOT ESTABLISHED. Public task provenance is useful for reproduction, but no task-quality comparison tests the proposition. |
| H79 | SQLite tasks allow more reliable parallel execution than PostgreSQL-heavy tasks while a smaller PostgreSQL set preserves realism. | NOT ESTABLISHED. No controlled database-task concurrency comparison is published. |
| H80 | Independent review and wrong-solution and valid-alternative controls expose benchmark or grader errors before costly runs. | PARTIAL METHOD EVIDENCE. Public records preserve invalid grades and control lanes, but do not quantify the operating characteristics of this review policy. |
| H81 | Wall-time comparisons require equivalent stable hosts and controlled load rather than a single remote development link. | METHOD REQUIREMENT, NOT A PERFORMANCE RESULT. Public records do not establish a controlled host-effect estimate. |
| H82 | Pairing equivalent workers is valid when host, load, concurrency, and provider epoch are recorded and balanced. | NOT ESTABLISHED. No completed public balance audit validates this pairing rule. |
| H83 | Preliminary grading and reuse of frozen baseline results can reduce new cells without changing a decision. | NOT ESTABLISHED. No public operating-characteristic or cost comparison validates the savings and error rate. |
| H84 | A provider-load epoch can explain cross-day timing changes even when the harness is unchanged. | NOT ESTABLISHED. The public snapshot does not isolate provider epoch from host, load, or revision. |
| H85 | Sequential allocation, saturation pairs, single-repetition screens, and futility stops reduce study cost while preserving decisions. | NOT ESTABLISHED. No public simulation or prospective validation measures decision error and savings together. |
| H86 | Results with pinned versions, pass rate, wall time, cost, invalid cases, and full setup are reproducible and interpretable. | PARTIAL REPORTING EVIDENCE. Public pages and ledgers supply some of these fields, but unresolved joins and missing protocol pins remain. |
| H87 | Design and decision rules recorded before execution, with exploratory results labelled, reduce retrospective reinterpretation. | PARTIAL PROCESS EVIDENCE. Public pages distinguish preregistered, descriptive, withdrawn, and incomplete cohorts; a complete registration-history audit is not available here. |
| H88 | Intention-to-treat scoring counts internal stops as failures and excludes only demonstrated environment or adapter faults. | NOT ESTABLISHED AS A COMPLETE POLICY. Several unresolved ITT dispositions remain in the public exports. |
| H89 | Recording task, arm, revision, host, load, timeout, attempts, setup, and token categories is sufficient for later reproduction. | PARTIAL REPORTING EVIDENCE. The public run ledger exposes missing fields, so completeness is not established. |
| H90 | Kogen can solve strategically chosen tasks that direct Codex does not solve. | NOT ESTABLISHED. The public frontier lane has no released outcome cohort. |
Run and claim record
Live counts are reported as planned / started / finished / graded / ITT. Source-reported summary values are not reproduced from exact public cell IDs unless the linked ledger states otherwise. Exact task, model, revision, host, and epoch details are included only where the cited round page makes them recoverable.
| experiment_id | arm and task set | model / effort | Kogen and harness revision | host / epoch | planned / started / finished / graded / ITT | outcome | status and correction | claim ID | public receipt and raw records |
|---|---|---|---|---|---|---|---|---|---|
| hc-1 | Kogen ladder and shorter ladder; task set is in the round page | Sol high shaping; Luna max builder versus Sol medium builder | Kogen 1d567772446a66ea75fd04ae0e9c514b835f2d20; separate comparator harness revision not recovered | desktop Mac; epoch not recorded | 40 / 40 / 40 / 40 / not reconciled | Ladder 19/20; shorter ladder 17/20. | INTERIM; source-reported, not reproducible from public data. The shorter ladder was discarded in the source decision. | CL-WA-hc-1-lifecycle-and-outcome | round page, measurement record, claim ledger |
| hc-2 | Kogen ladder, diverse ladder, floor arm, and v1 guards; task set is in the round page | Exact per-cell model and effort not recoverable from the public summary | Kogen c056dccb9ee7534490102c8ae316122983b6f17b; separate comparator harness revision not recovered | desktop Mac; epoch not recorded | 90 / 90 / 90 / 90 / not reconciled | V2 ladder 16/20, diverse 13/20, floor 13/20; v1 guard ladder 9/10, diverse 10/10, floor 7/10. | INTERIM; source-reported, not reproducible from public data. Guard cells are not promoted as decision outcomes. | CL-WA-hc-2-lifecycle-and-outcome | round page, measurement record, claim ledger |
| hc-3 | Plain ladder and generated edge-test ladder, with guards; task set is in the round page | Exact per-cell model and effort not recoverable from the public summary | Kogen d8f29faab1a28a3b52c2cc70ec308cbd9f360450; separate comparator harness revision not recovered | desktop Mac; epoch not recorded | 60 / 60 / 60 / 60 / not reconciled | V2 ladder 16/20, edge-test arm 13/20; generated 197 tests, kept 50, dropped 147; four repairs triggered and none selected. | INTERIM; source-reported, not reproducible from public data. Edge-test arm was discarded in the source decision. | CL-WA-hc-3-lifecycle-and-outcome | round page, measurement record, claim ledger |
| hc-4 | Effort variants for shaping and planning on the v2 discriminator and guard set | Luna max builder; exact planner and shaping requests not fully verified | Kogen cde7a380455e9793cb1fc8315791bc9beffb863b; separate comparator harness revision not recovered | desktop Mac; epoch not recorded | 43 expected retained; 9 started diagnostics; exact finished / graded / ITT unresolved | No closed scored result can be reproduced from the public snapshot. | INTERIM; six cells were stopped; final scored denominator and outcomes remain unreconciled. | CL-WA-hc-4-lifecycle-and-outcome | round page, measurement record, claim ledger |
| frontier | Kogen frontier prepared from prior direct-builder failures; active-storage closure unverified | Kogen builder model and effort not established for a closed cohort | Kogen cde7a380455e9793cb1fc8315791bc9beffb863b; separate harness revision not recovered | Worker host and epoch not reconciled | 70 prepared / 0 released; no scored ITT population | No released frontier outcome. | INTERIM lifecycle record; do not interpret preparation as performance. | CL-WA-frontier-lifecycle-and-outcome | round page, measurement record, claim ledger |
| night-2026-10-01 overall | Multiple lanes, including distinct P1, P2, P3, and P5 subsets; exact task joins are incomplete | Exact per-cell model and effort not recoverable for all subsets | Exact per-arm Kogen and harness revisions not recovered | Four hosts are reported; provider epoch and balanced host map are unavailable | Planned and ITT denominators not recovered; invalid-grade disposition remains incomplete | The source record reports invalid-grade and invalid-task strata; lane results are kept separate in the claim ledger. | INTERIM; source-reported, not reproducible from public data. | CL-WA-night-2026-10-01-reported-night-overall | round page, reported data, claim ledger |
| night-2026-10-01 P1 | P1 lane; exact task set and planned denominator not recovered | Model and effort unavailable | Kogen and harness revisions unavailable | Host and epoch map unavailable | Planned / started / finished / graded / ITT cells not recovered | No outcome summary was recovered. | INTERIM; source-reported, not reproducible from public data. | CL-WA-night-2026-10-01-reported-night-p1 | round page, reported data, claim ledger |
| night-2026-10-01 P2 | P2 lane; exact task IDs not available | Exact model and effort not recovered | Exact Kogen and harness revisions not recovered | Host and epoch map unavailable | Admitted task count is source-reported; planned / started / finished / graded / ITT cells not joined | Valid-only summary exists without exact public cells; not pooled with other lanes. | INTERIM; source-reported, not reproducible from public data. | CL-WA-night-2026-10-01-reported-night-p2 | round page, reported data, claim ledger |
| night-2026-10-01 P3a and P3c | Separate exploratory lanes; exact task mapping unavailable | Exact model and effort not recovered | Exact Kogen and harness revisions not recovered | Host and epoch map unavailable | Planned / started / finished / graded / ITT cells not joined | Distinct summaries are preserved separately; do not pool the lanes. | INTERIM; source-reported, not reproducible from public data. | CL-WA-night-2026-10-01-reported-night-p3a; CL-WA-night-2026-10-01-reported-night-p3c | round page, reported data, claim ledger |
| night-2026-10-01 P5 and parallel-read | Selected failures and a separate parallel-read subset; exact task and arm maps unavailable | Exact model and effort not recovered | Exact Kogen and harness revisions not recovered | Host and epoch map unavailable | Planned / started / finished / graded / ITT cells not joined | Selected-subset and request-count summaries are not comparable with the other night lanes. | INTERIM; source-reported, not reproducible from public data. | CL-WA-night-2026-10-01-reported-night-p5; CL-WA-night-2026-10-01-reported-night-parallel-read | round page, reported data, claim ledger |
| night-2026-10-01 Pi mining | Mining installation, not a benchmark arm | No benchmark model execution | No scored Kogen or harness revision | Not applicable | Not run as a benchmark arm | No model comparison. | NOT-RUN; ledger status INTERIM; source-reported, not reproducible from public data. | CL-WA-night-2026-10-01-reported-night-pi-mining | round page, reported data, claim ledger |
| x-mining observation and usage inventories | Mixed-version historical records; not a matched arm comparison | Historical model and effort fields are incomplete | Mixed revisions; exact cohort is not reconstructed | Host and provider epoch incomplete | Counts and ITT mapping are not a single cohort | The reports identify incomplete outcome and usage metadata; no result here is promoted to a pass rate. | INTERIM; source-reported, not reproducible from public data. | CL-WA-x-mining-reported-x-mining-observation-inventory; CL-WA-x-mining-reported-x-mining-usage-audit | round page, reported data, claim ledger |
| x-mining-x09 | Separate diagnostic screen; exact task IDs unavailable | Model and effort unavailable | Exact Kogen and harness revisions unavailable | Host and epoch unavailable | Completed diagnostic count is source-reported; scored Build and ITT counts are unavailable | The source summary labels this screen inconclusive. | INTERIM; source-reported, not reproducible from public data. | CL-WA-x-mining-reported-x-mining-x09 | round page, reported data, claim ledger |
| studio-kogen and studio-kogen-80a4 | Preparation and retry smokes; no bulk scored cells | Model and effort are not fully recoverable | Exact commit unavailable for studio-kogen; the 80a4 lane label is not a verified Kogen pin; harness revisions unavailable | desktop Mac; epoch not recorded | Smoke-only lifecycle; no bulk scored population | Smoke receipts do not establish recovery or benchmark performance. | INTERIM; source-reported, not reproducible from public data. | CL-WA-studio-kogen-lifecycle-and-outcome; CL-WA-studio-kogen-80a4-lifecycle-and-outcome | studio-kogen, studio-kogen-80a4, claim ledger |
| studio-shaper-pilot | Shaper preparation smoke; no scored cohort was released | Exact model and effort not fully recoverable | Exact Kogen and harness revisions unavailable | desktop Mac; epoch not recorded | Smoke-only lifecycle; no scored release | The smoke result is not a Build outcome. | INTERIM; source-reported, not reproducible from public data. | CL-WA-studio-shaper-pilot-lifecycle-and-outcome | round page, claim ledger |
| r57-studio | Catalogue preparation; no scored release | Exact model and effort not fully recoverable | Kogen 6747368333e55a694338d32312eb1f49ec2b4813; harness revision unavailable | desktop Mac; epoch not recorded | Smoke/setup record only; no scored release | Setup limitations prevent an outcome estimate. | INTERIM; source-reported, not reproducible from public data. | CL-WA-r57-studio-lifecycle-and-outcome | round page, claim ledger |
| r57-t90-studio | Catalogue preparation; no scored release | Exact model and effort not fully recoverable | Kogen 8c705841d66023dcc7025dba9a24208afd794a02; harness revision unavailable | desktop Mac; epoch not recorded | Smoke/setup record only; no scored release | Setup limitations prevent an outcome estimate. | INTERIM; source-reported, not reproducible from public data. | CL-WA-r57-t90-studio-lifecycle-and-outcome | round page, claim ledger |
| r58x-studio | Cross-stack desktop Mac pilot; exact task/arm-level final outcomes unreconciled | Exact per-cell model and effort not reconciled | Kogen d8f29faab1a28a3b52c2cc70ec308cbd9f360450; separate harness revision not recovered | desktop Mac; epoch not recorded | Later checkpoint has 36 finished scored rows, 18 per arm; ITT reconciliation unresolved | Earlier checkpoint had no released cells; later outcomes remain unreconciled. | INTERIM; preserve the checkpoints separately. | CL-WA-r58x-studio-lifecycle-and-outcome | round page, measurement record, claim ledger |
Design implications
Task difficulty, stack, host, provider epoch, and implementation revision can each change what a comparison means. Current records justify keeping task sets, guard cells, smokes, stopped slots, official grades, and counterfactual outcomes separate. They do not validate a single adaptive allocation policy or a broad Kogen advantage claim.
What this evidence does not show
The hill-climb summaries do not establish that one lever caused its observed outcome or that its apparent benefit transfers beyond the selected tasks. A cross-stack cohort does not establish task-set representativeness. Prepared or stopped cells are not scored results. Reporting completeness is not evidence that the recorded comparison was balanced.
Smallest open tests
Publish exact, immutable task/arm/cell joins and reconcile invalid cases before estimating ITT. For H75 and H83–H85, preregister an allocation rule and evaluate both decision quality and resource use against a fixed-allocation baseline. For H77–H79, balance source project, stack, database, task difficulty, and model protocol. For H80–H89, audit controls, registration order, host and provider epochs, revision pins, and stop dispositions against the raw records.
Evidence links
See the round register, result-ledger scope and known gaps, and the research census. Reproduction and exact revision records are maintained at round level; their missing fields are retained as gaps here.