F03
F03 — Harness and loop
The authored family record is reproduced below. Structured links connect its hypotheses, rounds, and claim-ledger rows.
Family source document · hypotheses/f03-harness-loop.md
F03 — Harness and loop
Coverage: H19–H38. The propositions below define observable comparisons; an associated receipt is not itself evidence of a winner. Wave B linked claim IDs are preserved below. Public delivery-capture counts are not graded task outcomes. Source summaries without an exact public join remain labelled source-reported, not reproducible from public data in the claim ledger.
Hypotheses
| ID | Falsifiable proposition | Current state |
|---|---|---|
| H19 | On matched ChatGPT models, task IDs, prompts, tools, and effort, Pi and Codex differ in success, time, cost, or cache use by a reproducible amount. | UNRESOLVED; harness-pick records do not reconstruct the scored cohort. |
| H20 | Under matched prompts and tool access, at least one of Pi, OpenCode, Grok Build, Claude Code, or Codex has a better predeclared success-cost-time profile than the alternatives. | NOT ESTABLISHED; no complete comparable matrix. |
| H21 | A tuned custom Kogen harness matches or exceeds the selected external harness on the same task/model population and cost rule. | UNRESOLVED; associated receipts are interim or confounded. |
| H22 | End-to-end Kogen improves task completion enough to offset any slower loop overhead relative to a standalone harness. | NOT ESTABLISHED; no joined end-to-end comparison. |
| H23 | Elixir kh and Rust khr differ measurably in startup, tool-call, file-write, integration, or total Build time under equivalent work. | NOT ESTABLISHED; Rust khr comparison is not established in the public receipt. |
| H24 | Removing TUI code and using a command-only loop reduces runtime or latency while preserving task success. | NOT ESTABLISHED; no isolated test identified. |
| H25 | A lean coding-specific system prompt lowers tokens or improves cache use without reducing task success. | NOT ESTABLISHED; no public one-factor prompt result. |
| H26 | Replacing Claude Code’s prompt and constraining its tools changes its task outcomes in a way that separates harness effects from prompt/tool effects. | NOT ESTABLISHED; no paired public comparison. |
| H27 | A Claude Code -p wrapper preserves the relevant subscription route while improving Kogen control without unacceptable quality or latency loss. |
NOT ESTABLISHED; no completed matched comparison. |
| H28 | Claude Code’s task quality offsets its Kogen loop maintenance cost compared with a native loop. | UNRESOLVED; available candidate records do not establish a scored cohort. |
| H29 | Parallel subagents in Shape, Develop, or Review improve completion or wall time enough to offset their added calls and tokens versus one editing agent. | UNRESOLVED; exploratory source summaries are not joined to exact public outcomes. |
| H30 | Omitting helper agents does not materially reduce task completion and lowers cost or complexity under a declared margin. | NOT ESTABLISHED; no public non-inferiority comparison. |
| H31 | Mid-turn steering, external tool access, and recovery affordances explain a measurable portion of Pi/Codex quality differences. | NOT ESTABLISHED; no isolated affordance ablation. |
| H32 | A mechanical verification gate after the final edit reduces unverified completion without unacceptable cost or false blocking. | UNRESOLVED; the public kh-climb receipt is partial and lacks exact cell joins. |
| H33 | Loop verbosity, output format, or prompt changes account for a measurable part of harness outcome differences. | UNRESOLVED; exploratory surface records are partial. |
| H34 | Shell-only editing tools outperform apply-patch-style tools on a matched task set under a predeclared completion and cost rule. | UNRESOLVED; historical pooled comparison is not fully reconciled. |
| H35 | A strong plan plus shell builder matches direct Sol while reducing total cost on matched tasks. | UNRESOLVED; public rows do not establish the source-reported comparison. |
| H36 | Kogen’s own loop differs from Codex embedded as its builder after the builder model, tasks, and stage policy are held constant. | NOT ESTABLISHED; no clean loop-isolation comparison. |
| H37 | A native Grok provider in Kogen gives a lower-cost or higher-success route than the matched alternative, including Grok Build as a separate comparator. | NOT ESTABLISHED; the formal Grok frontier has no run result. |
| H38 | An alternative provider or subscription has a better cost-success profile than current subsidized options under matched tasks and limits. | NOT ESTABLISHED; no comparative subscription matrix. |
Experiment register
| experiment_id | Arm / task set | Model / effort | Revision; host / epoch | Planned / started / finished / graded / ITT n | Outcome, status, correction or supersession | Claim ID | Public receipt |
|---|---|---|---|---|---|---|---|
harness-pick-2026-09-29 |
Public manifest has task IDs and capture labels, but not the scored comparison arms. | Not joined to the capture. | Harness/Kogen pins and epoch are not recovered in the public capture. | Public manifest records 32 rows, including 25 exact delivery IDs; official grade joins = 0. These are capture counts, not scored n. | INTERIM; early comparison is exploratory and the corrected scored cohort is unavailable. | CL-WA-harness-pick-2026-09-29-public-capture-identity (INTERIM). |
round, manifest, claim ledger |
kh-compare-2026-09-29 |
Public capture spans task/venue records that do not match the described small desktop Mac comparison; arms are absent. | Not joined to the capture. | Harness/Kogen pins and epoch are not recovered for the described comparison. | Public manifest records 33 rows, including 32 exact delivery IDs; official grade joins = 0. These are capture counts, not scored n. | INTERIM; Sonnet path-check correction is not mapped to grades; Rust khr is not established as run. | CL-WA-kh-compare-2026-09-29-public-capture-identity (INTERIM). |
round, manifest, claim ledger |
claim-b-screen-1 |
Transport-confounded harness comparison; exact arms/tasks are not in the public capture. | Not recovered. | Not recovered. | No exact public delivery or grade crosswalk; historical design count is source-reported only. | CONFOUNDED; transport interruption was classified as model failure in the source summary. No efficacy claim. | CL-WA-claim-b-screen-1-public-capture-identity (CONFOUNDED). |
round, claim ledger |
claim-b-rerun-1 |
Worker rerun with setup amendment; the public capture does not expose the exact arm/task map. | Not recovered. | Host-as-repetition limits interpretation; full execution pin is not recovered. | No exact public delivery or grade crosswalk. | CONFOUNDED; source-reported arms and setup failures are not an exact public comparison. | CL-WA-claim-b-rerun-1-public-capture-identity (CONFOUNDED). |
round, claim ledger |
confirm-combined-1 |
Internal kh combination/one-factor variants; manifest does not identify arms. | Not recovered. | Not recovered. | Public manifest records 16 exact delivery IDs and no official grade joins; this does not reproduce the larger source comparison. | INTERIM; the registered decision did not select the combined variant; no general effect is established. | CL-WA-confirm-combined-1-public-capture-identity (INTERIM). |
round, manifest, claim ledger |
harness-compare-1 |
Four-harness comparison; exact public arm/task identities are absent. | Not recovered. | Not recovered. | No exact public delivery or official-grade crosswalk. | INTERIM; surrounding task defects were identified, but the affected comparison cells are unknown. | CL-WA-harness-compare-1-public-capture-identity (INTERIM). |
round, claim ledger |
hard1-calibrate-1 |
Hard-task harness calibration; exact cell-to-task correction map is absent. | Not recovered. | Not recovered. | No exact public delivery or grade crosswalk. | CONFOUNDED; floor/ceiling tasks and false blockers prevent a general harness interpretation. | CL-WA-hard1-calibrate-1-public-capture-identity (CONFOUNDED). |
round, claim ledger |
kh-climb |
B0, unchanged H0, and V1–V5 harness changes; the receipt separates each attempt. | Luna configurations are in the receipt; effective max may be clamped. | Exact cell revisions and host/epoch mapping are unavailable. | Documentary arm counts vary; V1 has a denominator conflict and V4/V5 are partial. No exact task/cell join. | INTERIM; no change was promoted from these source-reported screens. Do not pool baseline, partial variants, or unrun V3. | CL-WA-kh-climb-reported-kh-climb-baseline; CL-WA-kh-climb-reported-kh-climb-control; CL-WA-kh-climb-reported-kh-climb-v1; CL-WA-kh-climb-reported-kh-climb-v2; CL-WA-kh-climb-reported-kh-climb-v3; CL-WA-kh-climb-reported-kh-climb-v4; CL-WA-kh-climb-reported-kh-climb-v5 (all INTERIM). |
round, reported data, claim ledger |
x-roles |
Worker, implementation, reviewer, and policy screens; do not combine these panels. | Luna/Sol arms are in the reported-data file. | Exact cell revisions and venue epoch are not joined. | Public cell/task join unavailable. | INTERIM; small source-reported panels, no role policy selected. | CL-WA-x-roles-reported-x-roles-worker-pilot; CL-WA-x-roles-reported-x-roles-implementation; CL-WA-x-roles-reported-x-roles-reviewer; CL-WA-x-roles-reported-x-roles-decision (all INTERIM). |
round, reported data, claim ledger |
x-race |
Race/selection and selector checks; task-by-arm raw rows are absent. | See reported-data file; not joined to public cells. | Not recovered. | The ledger reports a source cohort but no exact public outcome join. | INTERIM; selected-candidate checks do not establish a general multi-agent gain. | CL-WA-x-race-reported-x-race; CL-WA-x-race-reported-x-race-selector-check; CL-WA-x-race-reported-x-race-decision (INTERIM). |
round, reported data, claim ledger |
x-parallel |
Solo versus planned DAG and separate cost/token summaries. | Not fully recovered. | Not recovered. | Source-reported summaries have no exact public task/cell join. | INTERIM; small exploratory panel, not a verified throughput or quality gain. | CL-WA-x-parallel-reported-x-parallel-v03; CL-WA-x-parallel-reported-x-parallel-v03-cost; CL-WA-x-parallel-reported-x-parallel-token-summary (INTERIM). |
round, reported data, claim ledger |
x-minikogen |
Small mechanism pairs and a separate test-first removal screen. | Not fully recovered. | Not recovered. | Arm map and task-level rows are incomplete. | INTERIM; source-reported small screens do not establish a steering or verification-gate effect. | CL-WA-x-minikogen-reported-x-minikogen-mechanism-pairs; CL-WA-x-minikogen-reported-x-minikogen-test-first-removal; CL-WA-x-minikogen-reported-x-minikogen-frozen-decision (INTERIM). |
round, reported data, claim ledger |
x-surface |
Feedback-format cohort and separate fixture/replication records. | Models are listed in the receipt; effort alignment is incomplete. | Not recovered. | Partial cohort only; remaining replication was not run at the cited cutoff. | INTERIM; no prompt, output, or steering effect is established. | CL-WA-x-surface-reported-x-surface-feedback-format; CL-WA-x-surface-reported-x-surface-fixture-repair; CL-WA-x-surface-reported-x-surface-remaining-rep (INTERIM). |
round, reported data, claim ledger |
x-continuation |
Live continuation bridge versus scripted provider mechanics. | Live model lane did not execute. | Not recovered. | Live lane has no real-model attempt; offline mechanical checks are a separate population. | NOT-RUN for model-backed efficacy; mechanical checks do not establish H31. | CL-WA-x-continuation-reported-x-continuation-live; CL-WA-x-continuation-reported-x-continuation-scripted-provider (both NOT-RUN). |
round, reported data, claim ledger |
night-2026-10-01 |
P1–P5 exploratory lanes; lane/task mapping is not complete. | Multiple harness/model arms; see receipt. | Four public host identifiers; epoch and balanced assignment are not established. | Approximate source inventory only; invalid grades and lane populations are kept separate in the round receipt. | INTERIM; no validated improvement. Pi installation/mining is not a benchmark arm. | CL-WA-night-2026-10-01-reported-night-overall; CL-WA-night-2026-10-01-reported-night-p1; CL-WA-night-2026-10-01-reported-night-p2; CL-WA-night-2026-10-01-reported-night-p3a; CL-WA-night-2026-10-01-reported-night-p3c; CL-WA-night-2026-10-01-reported-night-p5; CL-WA-night-2026-10-01-reported-night-parallel-read; CL-WA-night-2026-10-01-reported-night-pi-mining (all INTERIM). |
round, reported data, claim ledger |
r27–r30 |
Early Kogen route and harness arms; task/arm specifics remain on separate receipts. | See each receipt; do not infer matched effort across rounds. | Different rounds and snapshots; no pooled epoch. | See individual receipts; no common lifecycle denominator. | Exploratory/interim pages; no single route winner is supported by this family register. | No H19–H38-linked Wave B claim ID. | r27, r28, r29, r30 |
r57 family |
Shell-only and default-tool arms; the family page lists pooled and component rounds separately. | See component receipts; model/effort and effective settings must be matched before comparison. | Receipts retain host and snapshot distinctions. | Public export and source summary have a missing closure cell; do not merge the component rounds. | INTERIM; the public decision page withholds a fully reproducible non-inferiority claim. | CL-WA-r57-public-export-coverage; CL-WA-r57-source-reported-denominator-variants (INTERIM). |
r57, decision record, claim ledger |
r64 / r64b / r65b / r68b |
Plan-shell and related route arms; each round has its own tasks, anchors, and correction history. | Requested/effective model effort differs in at least one Kogen arm; see each receipt. | Do not pool hosts, task panels, or harness versions. | See each receipt; no shared denominator. | Descriptive and interim round records do not establish that plan-shell matches direct Sol at lower cost. | No H19–H38-linked Wave B claim ID. | r64, r64b, r65b, r68b |
grok-frontier |
Planned Grok frontier; no formal task arm completed. | Grok Build and Kogen-native provider are distinct roles. | Pin/host details are on the page. | No formal or scored cells. | NOT-RUN; no Grok efficacy conclusion. | No H19–H38-linked Wave B claim ID. | Grok frontier |
Design implications
Keep harness choice, provider, prompt, tool surface, model effort, and whole-route policy as separate factors. For task outcomes, compare exact cell IDs and official grades; for runtime, report host, load, epoch, and paired balance. Public delivery manifests establish only that a candidate record was captured, not that the task passed.
What this evidence does not show
These records do not establish a general Pi, Codex, kh, khr, OpenCode, Claude Code, shell-only, parallel-agent, or Grok winner. A source-reported pilot, a candidate-delivery count, an offline/scripted check, and a completed graded Build are distinct evidence types. The listed interim and confounded rows cannot support a measured superiority claim.
Smallest open tests
Run a preregistered matched-task harness panel with exact model/effort, prompts, tools, retries, host/epoch, official grades, usage, and all attempted-cell dispositions. Then use one-factor arms for the verification gate, tool surface, prompt, steering, and helper topology, with a transfer task set held out from tuning.