F07

F07 — Tasks, grading, and method

The authored family record is reproduced below. Structured links connect its hypotheses, rounds, and claim-ledger rows.

Source-reported narrativeNumeric statements in this family document remain source-reported unless an exact claim-ledger row is linked and its status allows that interpretation.

Family source document · hypotheses/f07-tasks-method.md

F07 Tasks, grading, and method

This page records evidence for H75–H90. These propositions concern task selection, grading, study design, and reproducibility. A methodological recommendation is not itself a measured Build effect.

Hypotheses and current status

ID Falsifiable proposition Current evidence status
H75 A cheap starting task and one-lever-at-a-time climb identifies useful task and recipe changes without mistaking saturation for general success. INTERIM. Several desktop Mac hill-climb iterations are reported, but cell-level outcomes and the final iteration are not fully reconciled in the public export.
H76 Luna or Grok is the better low-cost, fast starting builder for the intended task set. NOT ESTABLISHED. No matched comparison for this starting-model choice is linked to a public claim.
H77 A benchmark limited to Rails tasks can misstate performance on Elixir/Phoenix or Rust work. NOT ESTABLISHED. Cross-stack task and implementation records exist, but no matched task-set study isolates stack or recency effects.
H78 Public real-codebase tasks pinned to source revisions are more reproducible than hidden replay tasks. NOT ESTABLISHED. Public task provenance is useful for reproduction, but no task-quality comparison tests the proposition.
H79 SQLite tasks allow more reliable parallel execution than PostgreSQL-heavy tasks while a smaller PostgreSQL set preserves realism. NOT ESTABLISHED. No controlled database-task concurrency comparison is published.
H80 Independent review and wrong-solution and valid-alternative controls expose benchmark or grader errors before costly runs. PARTIAL METHOD EVIDENCE. Public records preserve invalid grades and control lanes, but do not quantify the operating characteristics of this review policy.
H81 Wall-time comparisons require equivalent stable hosts and controlled load rather than a single remote development link. METHOD REQUIREMENT, NOT A PERFORMANCE RESULT. Public records do not establish a controlled host-effect estimate.
H82 Pairing equivalent workers is valid when host, load, concurrency, and provider epoch are recorded and balanced. NOT ESTABLISHED. No completed public balance audit validates this pairing rule.
H83 Preliminary grading and reuse of frozen baseline results can reduce new cells without changing a decision. NOT ESTABLISHED. No public operating-characteristic or cost comparison validates the savings and error rate.
H84 A provider-load epoch can explain cross-day timing changes even when the harness is unchanged. NOT ESTABLISHED. The public snapshot does not isolate provider epoch from host, load, or revision.
H85 Sequential allocation, saturation pairs, single-repetition screens, and futility stops reduce study cost while preserving decisions. NOT ESTABLISHED. No public simulation or prospective validation measures decision error and savings together.
H86 Results with pinned versions, pass rate, wall time, cost, invalid cases, and full setup are reproducible and interpretable. PARTIAL REPORTING EVIDENCE. Public pages and ledgers supply some of these fields, but unresolved joins and missing protocol pins remain.
H87 Design and decision rules recorded before execution, with exploratory results labelled, reduce retrospective reinterpretation. PARTIAL PROCESS EVIDENCE. Public pages distinguish preregistered, descriptive, withdrawn, and incomplete cohorts; a complete registration-history audit is not available here.
H88 Intention-to-treat scoring counts internal stops as failures and excludes only demonstrated environment or adapter faults. NOT ESTABLISHED AS A COMPLETE POLICY. Several unresolved ITT dispositions remain in the public exports.
H89 Recording task, arm, revision, host, load, timeout, attempts, setup, and token categories is sufficient for later reproduction. PARTIAL REPORTING EVIDENCE. The public run ledger exposes missing fields, so completeness is not established.
H90 Kogen can solve strategically chosen tasks that direct Codex does not solve. NOT ESTABLISHED. The public frontier lane has no released outcome cohort.

Run and claim record

Live counts are reported as planned / started / finished / graded / ITT. Source-reported summary values are not reproduced from exact public cell IDs unless the linked ledger states otherwise. Exact task, model, revision, host, and epoch details are included only where the cited round page makes them recoverable.

experiment_id arm and task set model / effort Kogen and harness revision host / epoch planned / started / finished / graded / ITT outcome status and correction claim ID public receipt and raw records
hc-1 Kogen ladder and shorter ladder; task set is in the round page Sol high shaping; Luna max builder versus Sol medium builder Kogen 1d567772446a66ea75fd04ae0e9c514b835f2d20; separate comparator harness revision not recovered desktop Mac; epoch not recorded 40 / 40 / 40 / 40 / not reconciled Ladder 19/20; shorter ladder 17/20. INTERIM; source-reported, not reproducible from public data. The shorter ladder was discarded in the source decision. CL-WA-hc-1-lifecycle-and-outcome round page, measurement record, claim ledger
hc-2 Kogen ladder, diverse ladder, floor arm, and v1 guards; task set is in the round page Exact per-cell model and effort not recoverable from the public summary Kogen c056dccb9ee7534490102c8ae316122983b6f17b; separate comparator harness revision not recovered desktop Mac; epoch not recorded 90 / 90 / 90 / 90 / not reconciled V2 ladder 16/20, diverse 13/20, floor 13/20; v1 guard ladder 9/10, diverse 10/10, floor 7/10. INTERIM; source-reported, not reproducible from public data. Guard cells are not promoted as decision outcomes. CL-WA-hc-2-lifecycle-and-outcome round page, measurement record, claim ledger
hc-3 Plain ladder and generated edge-test ladder, with guards; task set is in the round page Exact per-cell model and effort not recoverable from the public summary Kogen d8f29faab1a28a3b52c2cc70ec308cbd9f360450; separate comparator harness revision not recovered desktop Mac; epoch not recorded 60 / 60 / 60 / 60 / not reconciled V2 ladder 16/20, edge-test arm 13/20; generated 197 tests, kept 50, dropped 147; four repairs triggered and none selected. INTERIM; source-reported, not reproducible from public data. Edge-test arm was discarded in the source decision. CL-WA-hc-3-lifecycle-and-outcome round page, measurement record, claim ledger
hc-4 Effort variants for shaping and planning on the v2 discriminator and guard set Luna max builder; exact planner and shaping requests not fully verified Kogen cde7a380455e9793cb1fc8315791bc9beffb863b; separate comparator harness revision not recovered desktop Mac; epoch not recorded 43 expected retained; 9 started diagnostics; exact finished / graded / ITT unresolved No closed scored result can be reproduced from the public snapshot. INTERIM; six cells were stopped; final scored denominator and outcomes remain unreconciled. CL-WA-hc-4-lifecycle-and-outcome round page, measurement record, claim ledger
frontier Kogen frontier prepared from prior direct-builder failures; active-storage closure unverified Kogen builder model and effort not established for a closed cohort Kogen cde7a380455e9793cb1fc8315791bc9beffb863b; separate harness revision not recovered Worker host and epoch not reconciled 70 prepared / 0 released; no scored ITT population No released frontier outcome. INTERIM lifecycle record; do not interpret preparation as performance. CL-WA-frontier-lifecycle-and-outcome round page, measurement record, claim ledger
night-2026-10-01 overall Multiple lanes, including distinct P1, P2, P3, and P5 subsets; exact task joins are incomplete Exact per-cell model and effort not recoverable for all subsets Exact per-arm Kogen and harness revisions not recovered Four hosts are reported; provider epoch and balanced host map are unavailable Planned and ITT denominators not recovered; invalid-grade disposition remains incomplete The source record reports invalid-grade and invalid-task strata; lane results are kept separate in the claim ledger. INTERIM; source-reported, not reproducible from public data. CL-WA-night-2026-10-01-reported-night-overall round page, reported data, claim ledger
night-2026-10-01 P1 P1 lane; exact task set and planned denominator not recovered Model and effort unavailable Kogen and harness revisions unavailable Host and epoch map unavailable Planned / started / finished / graded / ITT cells not recovered No outcome summary was recovered. INTERIM; source-reported, not reproducible from public data. CL-WA-night-2026-10-01-reported-night-p1 round page, reported data, claim ledger
night-2026-10-01 P2 P2 lane; exact task IDs not available Exact model and effort not recovered Exact Kogen and harness revisions not recovered Host and epoch map unavailable Admitted task count is source-reported; planned / started / finished / graded / ITT cells not joined Valid-only summary exists without exact public cells; not pooled with other lanes. INTERIM; source-reported, not reproducible from public data. CL-WA-night-2026-10-01-reported-night-p2 round page, reported data, claim ledger
night-2026-10-01 P3a and P3c Separate exploratory lanes; exact task mapping unavailable Exact model and effort not recovered Exact Kogen and harness revisions not recovered Host and epoch map unavailable Planned / started / finished / graded / ITT cells not joined Distinct summaries are preserved separately; do not pool the lanes. INTERIM; source-reported, not reproducible from public data. CL-WA-night-2026-10-01-reported-night-p3a; CL-WA-night-2026-10-01-reported-night-p3c round page, reported data, claim ledger
night-2026-10-01 P5 and parallel-read Selected failures and a separate parallel-read subset; exact task and arm maps unavailable Exact model and effort not recovered Exact Kogen and harness revisions not recovered Host and epoch map unavailable Planned / started / finished / graded / ITT cells not joined Selected-subset and request-count summaries are not comparable with the other night lanes. INTERIM; source-reported, not reproducible from public data. CL-WA-night-2026-10-01-reported-night-p5; CL-WA-night-2026-10-01-reported-night-parallel-read round page, reported data, claim ledger
night-2026-10-01 Pi mining Mining installation, not a benchmark arm No benchmark model execution No scored Kogen or harness revision Not applicable Not run as a benchmark arm No model comparison. NOT-RUN; ledger status INTERIM; source-reported, not reproducible from public data. CL-WA-night-2026-10-01-reported-night-pi-mining round page, reported data, claim ledger
x-mining observation and usage inventories Mixed-version historical records; not a matched arm comparison Historical model and effort fields are incomplete Mixed revisions; exact cohort is not reconstructed Host and provider epoch incomplete Counts and ITT mapping are not a single cohort The reports identify incomplete outcome and usage metadata; no result here is promoted to a pass rate. INTERIM; source-reported, not reproducible from public data. CL-WA-x-mining-reported-x-mining-observation-inventory; CL-WA-x-mining-reported-x-mining-usage-audit round page, reported data, claim ledger
x-mining-x09 Separate diagnostic screen; exact task IDs unavailable Model and effort unavailable Exact Kogen and harness revisions unavailable Host and epoch unavailable Completed diagnostic count is source-reported; scored Build and ITT counts are unavailable The source summary labels this screen inconclusive. INTERIM; source-reported, not reproducible from public data. CL-WA-x-mining-reported-x-mining-x09 round page, reported data, claim ledger
studio-kogen and studio-kogen-80a4 Preparation and retry smokes; no bulk scored cells Model and effort are not fully recoverable Exact commit unavailable for studio-kogen; the 80a4 lane label is not a verified Kogen pin; harness revisions unavailable desktop Mac; epoch not recorded Smoke-only lifecycle; no bulk scored population Smoke receipts do not establish recovery or benchmark performance. INTERIM; source-reported, not reproducible from public data. CL-WA-studio-kogen-lifecycle-and-outcome; CL-WA-studio-kogen-80a4-lifecycle-and-outcome studio-kogen, studio-kogen-80a4, claim ledger
studio-shaper-pilot Shaper preparation smoke; no scored cohort was released Exact model and effort not fully recoverable Exact Kogen and harness revisions unavailable desktop Mac; epoch not recorded Smoke-only lifecycle; no scored release The smoke result is not a Build outcome. INTERIM; source-reported, not reproducible from public data. CL-WA-studio-shaper-pilot-lifecycle-and-outcome round page, claim ledger
r57-studio Catalogue preparation; no scored release Exact model and effort not fully recoverable Kogen 6747368333e55a694338d32312eb1f49ec2b4813; harness revision unavailable desktop Mac; epoch not recorded Smoke/setup record only; no scored release Setup limitations prevent an outcome estimate. INTERIM; source-reported, not reproducible from public data. CL-WA-r57-studio-lifecycle-and-outcome round page, claim ledger
r57-t90-studio Catalogue preparation; no scored release Exact model and effort not fully recoverable Kogen 8c705841d66023dcc7025dba9a24208afd794a02; harness revision unavailable desktop Mac; epoch not recorded Smoke/setup record only; no scored release Setup limitations prevent an outcome estimate. INTERIM; source-reported, not reproducible from public data. CL-WA-r57-t90-studio-lifecycle-and-outcome round page, claim ledger
r58x-studio Cross-stack desktop Mac pilot; exact task/arm-level final outcomes unreconciled Exact per-cell model and effort not reconciled Kogen d8f29faab1a28a3b52c2cc70ec308cbd9f360450; separate harness revision not recovered desktop Mac; epoch not recorded Later checkpoint has 36 finished scored rows, 18 per arm; ITT reconciliation unresolved Earlier checkpoint had no released cells; later outcomes remain unreconciled. INTERIM; preserve the checkpoints separately. CL-WA-r58x-studio-lifecycle-and-outcome round page, measurement record, claim ledger

Design implications

Task difficulty, stack, host, provider epoch, and implementation revision can each change what a comparison means. Current records justify keeping task sets, guard cells, smokes, stopped slots, official grades, and counterfactual outcomes separate. They do not validate a single adaptive allocation policy or a broad Kogen advantage claim.

What this evidence does not show

The hill-climb summaries do not establish that one lever caused its observed outcome or that its apparent benefit transfers beyond the selected tasks. A cross-stack cohort does not establish task-set representativeness. Prepared or stopped cells are not scored results. Reporting completeness is not evidence that the recorded comparison was balanced.

Smallest open tests

Publish exact, immutable task/arm/cell joins and reconcile invalid cases before estimating ITT. For H75 and H83–H85, preregister an allocation rule and evaluate both decision quality and resource use against a fixed-allocation baseline. For H77–H79, balance source project, stack, database, task difficulty, and model protocol. For H80–H89, audit controls, registration order, host and provider epochs, revision pins, and stop dispositions against the raw records.

Evidence links

See the round register, result-ledger scope and known gaps, and the research census. Reproduction and exact revision records are maintained at round level; their missing fields are retained as gaps here.