F03

F03 — Harness and loop

The authored family record is reproduced below. Structured links connect its hypotheses, rounds, and claim-ledger rows.

Source-reported narrativeNumeric statements in this family document remain source-reported unless an exact claim-ledger row is linked and its status allows that interpretation.

Family source document · hypotheses/f03-harness-loop.md

F03 — Harness and loop

Coverage: H19–H38. The propositions below define observable comparisons; an associated receipt is not itself evidence of a winner. Wave B linked claim IDs are preserved below. Public delivery-capture counts are not graded task outcomes. Source summaries without an exact public join remain labelled source-reported, not reproducible from public data in the claim ledger.

Hypotheses

ID Falsifiable proposition Current state
H19 On matched ChatGPT models, task IDs, prompts, tools, and effort, Pi and Codex differ in success, time, cost, or cache use by a reproducible amount. UNRESOLVED; harness-pick records do not reconstruct the scored cohort.
H20 Under matched prompts and tool access, at least one of Pi, OpenCode, Grok Build, Claude Code, or Codex has a better predeclared success-cost-time profile than the alternatives. NOT ESTABLISHED; no complete comparable matrix.
H21 A tuned custom Kogen harness matches or exceeds the selected external harness on the same task/model population and cost rule. UNRESOLVED; associated receipts are interim or confounded.
H22 End-to-end Kogen improves task completion enough to offset any slower loop overhead relative to a standalone harness. NOT ESTABLISHED; no joined end-to-end comparison.
H23 Elixir kh and Rust khr differ measurably in startup, tool-call, file-write, integration, or total Build time under equivalent work. NOT ESTABLISHED; Rust khr comparison is not established in the public receipt.
H24 Removing TUI code and using a command-only loop reduces runtime or latency while preserving task success. NOT ESTABLISHED; no isolated test identified.
H25 A lean coding-specific system prompt lowers tokens or improves cache use without reducing task success. NOT ESTABLISHED; no public one-factor prompt result.
H26 Replacing Claude Code’s prompt and constraining its tools changes its task outcomes in a way that separates harness effects from prompt/tool effects. NOT ESTABLISHED; no paired public comparison.
H27 A Claude Code -p wrapper preserves the relevant subscription route while improving Kogen control without unacceptable quality or latency loss. NOT ESTABLISHED; no completed matched comparison.
H28 Claude Code’s task quality offsets its Kogen loop maintenance cost compared with a native loop. UNRESOLVED; available candidate records do not establish a scored cohort.
H29 Parallel subagents in Shape, Develop, or Review improve completion or wall time enough to offset their added calls and tokens versus one editing agent. UNRESOLVED; exploratory source summaries are not joined to exact public outcomes.
H30 Omitting helper agents does not materially reduce task completion and lowers cost or complexity under a declared margin. NOT ESTABLISHED; no public non-inferiority comparison.
H31 Mid-turn steering, external tool access, and recovery affordances explain a measurable portion of Pi/Codex quality differences. NOT ESTABLISHED; no isolated affordance ablation.
H32 A mechanical verification gate after the final edit reduces unverified completion without unacceptable cost or false blocking. UNRESOLVED; the public kh-climb receipt is partial and lacks exact cell joins.
H33 Loop verbosity, output format, or prompt changes account for a measurable part of harness outcome differences. UNRESOLVED; exploratory surface records are partial.
H34 Shell-only editing tools outperform apply-patch-style tools on a matched task set under a predeclared completion and cost rule. UNRESOLVED; historical pooled comparison is not fully reconciled.
H35 A strong plan plus shell builder matches direct Sol while reducing total cost on matched tasks. UNRESOLVED; public rows do not establish the source-reported comparison.
H36 Kogen’s own loop differs from Codex embedded as its builder after the builder model, tasks, and stage policy are held constant. NOT ESTABLISHED; no clean loop-isolation comparison.
H37 A native Grok provider in Kogen gives a lower-cost or higher-success route than the matched alternative, including Grok Build as a separate comparator. NOT ESTABLISHED; the formal Grok frontier has no run result.
H38 An alternative provider or subscription has a better cost-success profile than current subsidized options under matched tasks and limits. NOT ESTABLISHED; no comparative subscription matrix.

Experiment register

experiment_id Arm / task set Model / effort Revision; host / epoch Planned / started / finished / graded / ITT n Outcome, status, correction or supersession Claim ID Public receipt
harness-pick-2026-09-29 Public manifest has task IDs and capture labels, but not the scored comparison arms. Not joined to the capture. Harness/Kogen pins and epoch are not recovered in the public capture. Public manifest records 32 rows, including 25 exact delivery IDs; official grade joins = 0. These are capture counts, not scored n. INTERIM; early comparison is exploratory and the corrected scored cohort is unavailable. CL-WA-harness-pick-2026-09-29-public-capture-identity (INTERIM). round, manifest, claim ledger
kh-compare-2026-09-29 Public capture spans task/venue records that do not match the described small desktop Mac comparison; arms are absent. Not joined to the capture. Harness/Kogen pins and epoch are not recovered for the described comparison. Public manifest records 33 rows, including 32 exact delivery IDs; official grade joins = 0. These are capture counts, not scored n. INTERIM; Sonnet path-check correction is not mapped to grades; Rust khr is not established as run. CL-WA-kh-compare-2026-09-29-public-capture-identity (INTERIM). round, manifest, claim ledger
claim-b-screen-1 Transport-confounded harness comparison; exact arms/tasks are not in the public capture. Not recovered. Not recovered. No exact public delivery or grade crosswalk; historical design count is source-reported only. CONFOUNDED; transport interruption was classified as model failure in the source summary. No efficacy claim. CL-WA-claim-b-screen-1-public-capture-identity (CONFOUNDED). round, claim ledger
claim-b-rerun-1 Worker rerun with setup amendment; the public capture does not expose the exact arm/task map. Not recovered. Host-as-repetition limits interpretation; full execution pin is not recovered. No exact public delivery or grade crosswalk. CONFOUNDED; source-reported arms and setup failures are not an exact public comparison. CL-WA-claim-b-rerun-1-public-capture-identity (CONFOUNDED). round, claim ledger
confirm-combined-1 Internal kh combination/one-factor variants; manifest does not identify arms. Not recovered. Not recovered. Public manifest records 16 exact delivery IDs and no official grade joins; this does not reproduce the larger source comparison. INTERIM; the registered decision did not select the combined variant; no general effect is established. CL-WA-confirm-combined-1-public-capture-identity (INTERIM). round, manifest, claim ledger
harness-compare-1 Four-harness comparison; exact public arm/task identities are absent. Not recovered. Not recovered. No exact public delivery or official-grade crosswalk. INTERIM; surrounding task defects were identified, but the affected comparison cells are unknown. CL-WA-harness-compare-1-public-capture-identity (INTERIM). round, claim ledger
hard1-calibrate-1 Hard-task harness calibration; exact cell-to-task correction map is absent. Not recovered. Not recovered. No exact public delivery or grade crosswalk. CONFOUNDED; floor/ceiling tasks and false blockers prevent a general harness interpretation. CL-WA-hard1-calibrate-1-public-capture-identity (CONFOUNDED). round, claim ledger
kh-climb B0, unchanged H0, and V1–V5 harness changes; the receipt separates each attempt. Luna configurations are in the receipt; effective max may be clamped. Exact cell revisions and host/epoch mapping are unavailable. Documentary arm counts vary; V1 has a denominator conflict and V4/V5 are partial. No exact task/cell join. INTERIM; no change was promoted from these source-reported screens. Do not pool baseline, partial variants, or unrun V3. CL-WA-kh-climb-reported-kh-climb-baseline; CL-WA-kh-climb-reported-kh-climb-control; CL-WA-kh-climb-reported-kh-climb-v1; CL-WA-kh-climb-reported-kh-climb-v2; CL-WA-kh-climb-reported-kh-climb-v3; CL-WA-kh-climb-reported-kh-climb-v4; CL-WA-kh-climb-reported-kh-climb-v5 (all INTERIM). round, reported data, claim ledger
x-roles Worker, implementation, reviewer, and policy screens; do not combine these panels. Luna/Sol arms are in the reported-data file. Exact cell revisions and venue epoch are not joined. Public cell/task join unavailable. INTERIM; small source-reported panels, no role policy selected. CL-WA-x-roles-reported-x-roles-worker-pilot; CL-WA-x-roles-reported-x-roles-implementation; CL-WA-x-roles-reported-x-roles-reviewer; CL-WA-x-roles-reported-x-roles-decision (all INTERIM). round, reported data, claim ledger
x-race Race/selection and selector checks; task-by-arm raw rows are absent. See reported-data file; not joined to public cells. Not recovered. The ledger reports a source cohort but no exact public outcome join. INTERIM; selected-candidate checks do not establish a general multi-agent gain. CL-WA-x-race-reported-x-race; CL-WA-x-race-reported-x-race-selector-check; CL-WA-x-race-reported-x-race-decision (INTERIM). round, reported data, claim ledger
x-parallel Solo versus planned DAG and separate cost/token summaries. Not fully recovered. Not recovered. Source-reported summaries have no exact public task/cell join. INTERIM; small exploratory panel, not a verified throughput or quality gain. CL-WA-x-parallel-reported-x-parallel-v03; CL-WA-x-parallel-reported-x-parallel-v03-cost; CL-WA-x-parallel-reported-x-parallel-token-summary (INTERIM). round, reported data, claim ledger
x-minikogen Small mechanism pairs and a separate test-first removal screen. Not fully recovered. Not recovered. Arm map and task-level rows are incomplete. INTERIM; source-reported small screens do not establish a steering or verification-gate effect. CL-WA-x-minikogen-reported-x-minikogen-mechanism-pairs; CL-WA-x-minikogen-reported-x-minikogen-test-first-removal; CL-WA-x-minikogen-reported-x-minikogen-frozen-decision (INTERIM). round, reported data, claim ledger
x-surface Feedback-format cohort and separate fixture/replication records. Models are listed in the receipt; effort alignment is incomplete. Not recovered. Partial cohort only; remaining replication was not run at the cited cutoff. INTERIM; no prompt, output, or steering effect is established. CL-WA-x-surface-reported-x-surface-feedback-format; CL-WA-x-surface-reported-x-surface-fixture-repair; CL-WA-x-surface-reported-x-surface-remaining-rep (INTERIM). round, reported data, claim ledger
x-continuation Live continuation bridge versus scripted provider mechanics. Live model lane did not execute. Not recovered. Live lane has no real-model attempt; offline mechanical checks are a separate population. NOT-RUN for model-backed efficacy; mechanical checks do not establish H31. CL-WA-x-continuation-reported-x-continuation-live; CL-WA-x-continuation-reported-x-continuation-scripted-provider (both NOT-RUN). round, reported data, claim ledger
night-2026-10-01 P1–P5 exploratory lanes; lane/task mapping is not complete. Multiple harness/model arms; see receipt. Four public host identifiers; epoch and balanced assignment are not established. Approximate source inventory only; invalid grades and lane populations are kept separate in the round receipt. INTERIM; no validated improvement. Pi installation/mining is not a benchmark arm. CL-WA-night-2026-10-01-reported-night-overall; CL-WA-night-2026-10-01-reported-night-p1; CL-WA-night-2026-10-01-reported-night-p2; CL-WA-night-2026-10-01-reported-night-p3a; CL-WA-night-2026-10-01-reported-night-p3c; CL-WA-night-2026-10-01-reported-night-p5; CL-WA-night-2026-10-01-reported-night-parallel-read; CL-WA-night-2026-10-01-reported-night-pi-mining (all INTERIM). round, reported data, claim ledger
r27–r30 Early Kogen route and harness arms; task/arm specifics remain on separate receipts. See each receipt; do not infer matched effort across rounds. Different rounds and snapshots; no pooled epoch. See individual receipts; no common lifecycle denominator. Exploratory/interim pages; no single route winner is supported by this family register. No H19–H38-linked Wave B claim ID. r27, r28, r29, r30
r57 family Shell-only and default-tool arms; the family page lists pooled and component rounds separately. See component receipts; model/effort and effective settings must be matched before comparison. Receipts retain host and snapshot distinctions. Public export and source summary have a missing closure cell; do not merge the component rounds. INTERIM; the public decision page withholds a fully reproducible non-inferiority claim. CL-WA-r57-public-export-coverage; CL-WA-r57-source-reported-denominator-variants (INTERIM). r57, decision record, claim ledger
r64 / r64b / r65b / r68b Plan-shell and related route arms; each round has its own tasks, anchors, and correction history. Requested/effective model effort differs in at least one Kogen arm; see each receipt. Do not pool hosts, task panels, or harness versions. See each receipt; no shared denominator. Descriptive and interim round records do not establish that plan-shell matches direct Sol at lower cost. No H19–H38-linked Wave B claim ID. r64, r64b, r65b, r68b
grok-frontier Planned Grok frontier; no formal task arm completed. Grok Build and Kogen-native provider are distinct roles. Pin/host details are on the page. No formal or scored cells. NOT-RUN; no Grok efficacy conclusion. No H19–H38-linked Wave B claim ID. Grok frontier

Design implications

Keep harness choice, provider, prompt, tool surface, model effort, and whole-route policy as separate factors. For task outcomes, compare exact cell IDs and official grades; for runtime, report host, load, epoch, and paired balance. Public delivery manifests establish only that a candidate record was captured, not that the task passed.

What this evidence does not show

These records do not establish a general Pi, Codex, kh, khr, OpenCode, Claude Code, shell-only, parallel-agent, or Grok winner. A source-reported pilot, a candidate-delivery count, an offline/scripted check, and a completed graded Build are distinct evidence types. The listed interim and confounded rows cannot support a measured superiority claim.

Smallest open tests

Run a preregistered matched-task harness panel with exact model/effort, prompts, tools, retries, host/epoch, official grades, usage, and all attempted-cell dispositions. Then use one-factor arms for the verification gate, tool surface, prompt, steering, and helper topology, with a transfer task set held out from tuning.