Round fv-pilot-2026-10-10

Lean versus Quint verification pilot

Status · DESCRIPTIVEReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/fv-pilot-2026-10-10/README.md

Round date: 2026-10-10 Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Recomputation status: FULLY RECOMPUTABLE Why not VALID: CEILING, SMALL_SAMPLE, DECLARED_AMENDMENTS, INCOMPLETE_CAPTURE — Both arms hit the success ceiling; this small pilot has declared execution amendments and incomplete Standard capture. It establishes no winner.

STATUS: DESCRIPTIVE

An agent repaired four small Elixir applications with formal feedback from Lean or Quint. Both arms repaired 12 of 12 valid trials, so the protocol's main measure does not separate them. Effort differences are secondary observations.

Lean is a proof assistant that checks formal definitions and proofs. Quint is a specification language with model checking for state transitions. Both received representations regenerated from the same application source, fixed contracts, starter tests and equivalent source-location feedback.

An arm is one comparison group: Lean or Quint. Elixir is the application programming language. A fixture is a small test application. The bridge translates application code into formal representations; its frontend reads the code and its emitters produce Lean or Quint definitions. An oracle is an independent reference checker of expected behavior. The harness runs and measures trials; admission means it passed readiness checks. A smoke test is a short operational check. A cycle regenerates the formal representation, checks its laws and runs starter tests. Tokens are units of model input and output; capped tokens count uncached input plus output, while raw tokens include cached input.

How and where it ran

All 24 valid trials ran on one Hetzner Linux host (2 vCPU) in Europe on 10 October 2026, using Codex CLI 0.161.0 and requested gpt-6.1-sol with high effort. The same account and admitted harness were used throughout. Fresh sessions used equal caps of 900 seconds, 60,000 uncached input plus output tokens and ten complete check cycles. Model requests can overshoot the token cap before their usage report arrives; raw totals remain available. The scheduled task/repeat order was fixed after randomization. At most two trials ran concurrently; the remaining trials ran sequentially. After initial admission, harness launch parameters were changed to specify the account configuration directory, expected account and host-specific readiness approval. Launch used Europe’s approval; overall approval remained false because the second host was pending. Recorded harness versions in the schedule were updated without changing trial order, and deployed files were reconciled before launch. The requested settings are pinned; provider-effective receipts are absent.

Four initial attempts failed before any command could execute and were repeated after infrastructure repair. Their rows and consumed budgets remain. Two power losses on the operator’s local workstation did not affect trials, which ran on the host. The results give exact UTC intervals, every deviation, per-task summaries and all individual valid trials. The amendments distinguish pre-trial choices from decisions during execution.

Trial accounting

There were 24 planned cells and 28 attempts: 24 valid completions and four infrastructure exclusions. Planned, started, finished, graded and counted populations are separate; attempt counts include the replacements.

Population Count
Planned cells 24
Started attempts 28
Finished attempts 28
Graded cells 24
Planned task/tool/repeat combinations counted in the analysis 24

Machine-readable counts and their definitions are in trial accounting.

Required reproduction metadata

  • Kogen commit: not applicable; these trials used direct Codex sessions.
  • Harness commit: not recorded as a Git commit; the source-reported tree fingerprint is e0454a97ba050c52cdc3f4567b6c7bd72f6b6aed plus Codex CLI 0.161.0.
  • Model and effort: requested gpt-6.1-sol, high; provider-effective receipts are absent.
  • Task IDs: F01, W01, W04, X01.
  • Raw records: sanitized lifecycle ledger and Standard records.
  • Historical command: runner --schedule <schedule> --cell <cell> --codex-home <account-home> --expect-account <account> --admission-receipt <receipt>; installation locations and account identity are withheld.

Public evidence and reproduction

Run python3 rounds/fv-pilot-2026-10-10/reproduce.py from the repository root to recompute RESULTS.md from the public ledger. This replays the analysis, not model calls or final grading. The private evaluator and its inputs are withheld. Public runtime locations were edited for publication; the admitted private hashes do not authenticate these edited copies. See the execution source notes for the required external interface. For a public feedback check on a fresh Linux host with the pinned tools, run python3 rounds/fv-pilot-2026-10-10/assemble.py F01 lean work-f01-lean, then enter work-f01-lean and run ./fv check. Select another task or arm as needed. This standalone check has no agent, trial meter or final evaluator; the defective starting app is expected to fail its formal laws. The full independent historical evaluation cannot be repeated from this bundle.

What's missing and why

  • circumstances.cap_end (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Boundary telemetry not retained (m08).
  • circumstances.cap_start (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Boundary telemetry not retained (m08).
  • circumstances.concurrent_cells_end (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Host-wide concurrent cell count was not captured (m66).
  • circumstances.concurrent_cells_start (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Host-wide concurrent cell count was not captured (m66).
  • circumstances.dispatcher_id (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Dispatcher receipt unavailable (m12).
  • circumstances.incidents (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell incident receipt was not captured (m78).
  • circumstances.load1_end (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Boundary telemetry not retained (m08).
  • circumstances.load1_start (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Boundary telemetry not retained (m08).
  • circumstances.load_samples (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No matching controller samples retained (m29).
  • circumstances.queue (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Queue receipt unavailable (m49).
  • cost.calculator_version (28 missing): Billed cost and pricing inputs were not retained; tokens are reported without a dollar estimate. Reason: Cost calculator version not recorded (m61).
  • cost.price_table_version (28 missing): Billed cost and pricing inputs were not retained; tokens are reported without a dollar estimate. Reason: Price table version not recorded (m89).
  • cost.usd (28 missing): Billed cost and pricing inputs were not retained; tokens are reported without a dollar estimate. Reason: Per-cell cost and pricing inputs are not present in the public round data (m70).
  • effort.effective (28 missing): The runner pinned the requested setting; a provider-effective receipt is absent from the published evidence. Reason: Per-cell effective-effort receipt not published (m73).
  • environment.account_class (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No owner-approved class-only account receipt is available for the cell (m100).
  • environment.cpu_model (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No per-cell CPU receipt (m34).
  • environment.kernel (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No per-cell kernel receipt (m36).
  • environment.network.allowlist_hosts (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell network allowlist not present in the public round data (m79).
  • environment.network.profile (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell network profile not present in the public round data (m80).
  • environment.ram_gib (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No per-cell RAM receipt (m35).
  • environment.toolchains.elixir (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • environment.toolchains.erlang (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • environment.toolchains.node (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • environment.toolchains.other_inventory (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • environment.toolchains.python (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • environment.toolchains.ruby (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • environment.toolchains.rust (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • grade.grader (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell grader software version is not in the public data (m77).
  • grade.timestamp (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell official repair-grade timestamp was not retained in the public record (m101).
  • host.cpu (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No per-cell CPU receipt (m34).
  • host.kernel (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No per-cell kernel receipt (m36).
  • host.ram_gib (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No per-cell RAM receipt (m35).
  • host.spec_ref (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: No measured host-spec reference (m30).
  • model.effective (28 missing): The runner pinned the requested setting; a provider-effective receipt is absent from the published evidence. Reason: Per-cell effective-model receipt not published (m74).
  • sandbox.egress_allow (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell egress allowlist is not present in the public round data (m75).
  • sandbox.egress_profile (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell egress profile is not present in the public round data (m76).
  • sandbox.profile (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell sandbox profile is not present in the public round data (m87).
  • sandbox.profile_sha256 (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell sandbox profile fingerprint is not present in the public round data (m86).
  • setup.deps_source (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell dependency source is not recorded (m71).
  • setup.sandbox_profile_sha256 (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell sandbox fingerprint is not present in the public round data (m84).
  • setup.task_base.hash (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Original base commit/tree hash absent; fresh_base_commit is not a base hash (m42).
  • task.base_revision.hash (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Original base commit/tree hash absent; fresh_base_commit is not a base hash (m42).
  • timestamps.attempts (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Attempt boundary receipts unavailable (m06).
  • timestamps.phases.develop.end_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.develop.start_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.gate.end_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.gate.start_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.grade.end_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.grade.start_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.plan.end_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.plan.start_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.review.end_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.review.start_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.setup.end_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.setup.start_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.shape.end_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.shape.start_utc (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timing.phases_s.develop (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase wall not emitted or not separable (m45).
  • timing.phases_s.gate (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase wall not emitted or not separable (m45).
  • timing.phases_s.grade (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase wall not emitted or not separable (m45).
  • timing.phases_s.plan (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase wall not emitted or not separable (m45).
  • timing.phases_s.review (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase wall not emitted or not separable (m45).
  • timing.phases_s.setup (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase wall not emitted or not separable (m45).
  • timing.phases_s.shape (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Phase wall not emitted or not separable (m45).
  • tokens.phases.develop.cached_input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.develop.input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.develop.output (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.develop.reasoning (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.gate.cached_input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.gate.input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.gate.output (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.gate.reasoning (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.grade.cached_input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.grade.input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.grade.output (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.grade.reasoning (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.plan.cached_input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.plan.input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.plan.output (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.plan.reasoning (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.review.cached_input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.review.input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.review.output (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.review.reasoning (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.setup.cached_input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.setup.input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.setup.output (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.setup.reasoning (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.shape.cached_input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.shape.input (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.shape.output (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tokens.phases.shape.reasoning (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-phase token counter not emitted (m44).
  • tools.grader (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Per-cell grader software version is not in the public data (m77).
  • tools.toolchains.elixir (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • tools.toolchains.erlang (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • tools.toolchains.node (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • tools.toolchains.other_inventory (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • tools.toolchains.python (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • tools.toolchains.ruby (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).
  • tools.toolchains.rust (28 missing): This field was not captured in the public trial evidence; no value is inferred. Reason: Toolchain version/inventory not recorded (m54).

Update 13 October 2026

Sinkhole for kogen.dev unverified for this run (the host's hosts file was not readable by the agent user); the agents' own transcripts, where preserved, show no access; transcripts for this round are no longer available.

Round results · rounds/fv-pilot-2026-10-10/RESULTS.md

Both arms reached 12 correct repairs out of 12 valid trials. This is a ceiling: the protocol's main measure does not separate them. The effort observations below are secondary and descriptive. There are only three repeats per task and arm, with 12 trials per arm across four small tasks. These observations support no broad winner claim.

Lean is a proof assistant that checks formal definitions and proofs. Quint is a specification language with model checking for state transitions. Both worked from the same Elixir applications, contracts and starter tests.

How and where it ran

All valid trials ran on one Hetzner Linux host (2 vCPU) in Europe, with Codex CLI 0.161.0, requested gpt-6.1-sol and high effort, using the same account throughout. Each trial used a fresh session. The fixed randomized schedule specified three repeats for each task and arm. Concurrency was at most two trials; remaining trials ran sequentially.

Valid trial UTC interval: 2026-10-10T13:59:16Z to 2026-10-10T14:57:28Z. Infrastructure-invalid UTC interval: 2026-10-10T13:39:55Z to 2026-10-10T13:47:13Z.

Equal caps were 900 seconds, 60,000 uncached input plus output tokens, and ten complete check cycles. A cycle regenerates the representation, checks the fixed laws and runs starter tests; failed checks consume cycles. Reported wall time covers the agent trial, not construction or later grading.

Per-task results

Each bracket gives the observed minimum–maximum; preceding values are medians. Diagnosis counts are match / partial / miss, using the fixed grader rule: both property and function phrases = match, one = partial, neither = miss. This is a deterministic text score, not a human assessment of explanation quality. Regression counts are reported final-grade failures in protected behavior and checks, not a claim about all possible behavior.

Task Arm Correct / 3 Diagnosis M/P/X Before patch Cycles median [range] (sum) Wall seconds median [range] Capped tokens median [range] Raw tokens median [range] Regressions
F01: Tenant access Lean 3/3 3/0/0 3/3 2 [2–2] (6) 78.9 [69.2–80.3] 21,971 [18,581–39,132] 220,499 [194,140–222,357] 0
F01: Tenant access Quint 3/3 3/0/0 3/3 4 [3–4] (11) 119.8 [103.0–135.6] 31,957 [26,272–32,882] 227,360 [208,853–319,346] 0
W01: Exact approval Lean 3/3 3/0/0 3/3 2 [2–5] (9) 88.6 [84.8–120.7] 29,040 [22,313–30,974] 194,558 [191,856–277,161] 0
W01: Exact approval Quint 3/3 2/1/0 3/3 3 [2–6] (11) 119.4 [100.6–178.8] 35,442 [18,694–50,078] 274,162 [229,126–479,646] 0
W04: Preservation Lean 3/3 3/0/0 3/3 4 [3–4] (11) 110.4 [97.3–126.9] 28,138 [25,030–39,743] 290,495 [259,910–327,914] 0
W04: Preservation Quint 3/3 3/0/0 3/3 4 [3–4] (11) 145.5 [114.6–156.6] 43,767 [37,401–52,675] 362,999 [320,323–450,201] 0
X01: Cache identity Lean 3/3 3/0/0 3/3 2 [2–2] (6) 90.1 [85.2–92.5] 26,860 [22,306–37,728] 245,100 [217,634–315,232] 0
X01: Cache identity Quint 3/3 3/0/0 3/3 2 [2–3] (7) 109.2 [108.1–147.2] 29,167 [21,499–29,528] 313,967 [201,723–315,480] 0

Arm summaries and secondary effort differences

Arm Correct / 12 Diagnosis M/P/X Before patch Cycles median [range] (sum) Wall seconds median [range] Capped tokens median [range] Raw tokens median [range] Regressions
Lean 12/12 12/0/0 12/12 2 [2–5] (32) 89.3 [69.2–126.9] 27,499 [18,581–39,743] 233,728.5 [191,856–327,914] 0
Quint 12/12 11/1/0 12/12 3 [2–6] (40) 119.6 [100.6–178.8] 32,419.5 [18,694–52,675] 314,723.5 [201,723–479,646] 0

Both arms had diagnosis before the first patch in 12/12 trials (100%). Lean had 12 matches; Quint had 11 matches and one partial (W01, repeat 1). The match-rate difference is 8.3 percentage points, one trial. No valid trial reached a cap or reported a regression.

Descriptive differences below are Quint minus Lean. Ratios are Quint / Lean; they compare arm medians, not causal treatment effects.

Secondary measure Median difference Median ratio
Check cycles +1.0 1.500×
Wall seconds +30.3 1.339×
Capped tokens +4,920.5 1.179×
Raw tokens +80,995.0 1.347×

Lean used 32 total cycles and Quint 40: eight more cycles for Quint (25%). Total agent wall time was 1,124.9 seconds for Lean and 1,538.3 seconds for Quint, a difference of 413.4 seconds (36.8%). Backend runtime is part of the observed effort. Task difficulty, agent choices, cache use, host sharing and the small sample limit interpretation. No significance test or general ranking is warranted.

Total cost per successful repair

There is no billed dollar record. Tokens measure usage, not money; cached input and output have different pricing. No dollar cost is estimated. For each arm, total cost per successful repair in tokens means the sum of all valid-trial usage divided by the number of correct repairs. Every valid trial succeeded here, so this is the arithmetic mean, not the median. Construction, qualification and smoke usage are outside this trial metric.

Task Arm Total capped tokens Total raw tokens Capped tokens / correct repair Raw tokens / correct repair
F01 Lean 79,684 636,996 26,561.3 212,332.0
F01 Quint 91,111 755,559 30,370.3 251,853.0
W01 Lean 82,327 663,575 27,442.3 221,191.7
W01 Quint 104,214 982,934 34,738.0 327,644.7
W04 Lean 92,911 878,319 30,970.3 292,773.0
W04 Quint 133,843 1,133,523 44,614.3 377,841.0
X01 Lean 86,894 777,966 28,964.7 259,322.0
X01 Quint 80,194 831,170 26,731.3 277,056.7
All tasks Lean 341,816 2,956,856 28,484.7 246,404.7
All tasks Quint 409,362 3,703,186 34,113.5 308,598.8

Capped tokens = input − cached input + output. Raw tokens = input + output. Reasoning is already included in output and is not added again. Cache-write input is retained when reported and is already part of input. The token cap is observed only when an atomic request usage report arrives; overshoot is retained rather than truncated. These trials had no overshoot.

Infrastructure failures consumed additional usage. Counting those attempts as operational overhead gives the following totals per successful repair; they remain excluded from the 12-trial success and effort distributions.

Arm Invalid attempts Additional capped / raw tokens Inclusive capped tokens / success Inclusive raw tokens / success
Lean 2 5,748 / 49,396 28,963.7 250,521.0
Quint 2 6,948 / 74,660 34,692.5 314,820.5

Deviations, amendments and infrastructure history

  1. Before scored trials, the token cap was interpreted as uncached input plus output, excluding cached input. Raw usage remains reported.
  2. Before scored trials, both packages removed precomputed checker output. Authored bridge replay was restricted to identical source locations only. Native backend diagnostics were retained. Checking was mediated by a broker, project setup was frozen, and API/completion checks were hardened.
  3. The operator waived a second model’s review of the fixes to save time. The supervisor inspected the changes instead, and the full qualification checks were repeated.
  4. The four excluded attempts were repeat 2 of tenant access and exact approval, each with Lean and Quint. The command executor was missing from the agent’s execution environment. Correction rows retain the original usage and mark those attempts as infrastructure failures; their artifacts remain archived privately. Replacement attempts used the same scheduled combinations. Valid completed trials were retained. Before restarting, an operational test had to demonstrate command execution, documentation access, a counted check and prevention of bypassing the counted checker. Caps and grading were unchanged. The operational test also exposed local socket restrictions and broker-control-file inspection; local file transport and cleanup corrected these before scored restart.
  5. During the run, the operator moved the second half of the planned two-host split to Europe. All 24 valid trials ran on one Hetzner Linux host in Europe, with two virtual CPUs and at most two simultaneous trials, instead of the planned 12 trials in Europe and 12 in the United States. The same harness, readiness approval, Codex binary and account were used for all 24. Schedule rows 13–24 moved to that host and its existing account configuration; the earlier allocation was retained. The United States host ran no trials. Task order, packages, model, effort, caps, checking, isolation and grading were unchanged.
  6. After initial admission, harness launch parameters were changed to specify the account configuration directory, expected account and host-specific readiness approval. Launch used Europe’s approval; overall approval remained false because the second host was pending. Recorded harness versions in the schedule were updated without changing trial order, and deployed files were reconciled before launch.
  7. Two power losses on the operator’s local workstation had no effect on trials because they ran on the European Linux host. Exact power-loss times were not retained here.
  8. Two interrupted qualification evaluation attempts were repeated sequentially with unchanged sources after an output-redirection fault. No completed application failure was rerun to obtain a passing answer.
  9. Standard capture is incomplete: no billed cost, phase-level usage, host boundary telemetry or provider-effective receipt is supplied. The package declares these gaps and remains DESCRIPTIVE.

The protocol and amendments are preserved as sanitized publication copies. The sanitized ledger retains all 59 lifecycle rows, including running entries, superseded infrastructure completions and four correction rows. Analysis takes each cell’s latest status, then excludes infra-invalid attempts; their usage is shown separately. Final grades and diagnosis scores are source-reported receipts. The private final evaluator is not published, so this bundle recomputes the analysis but cannot repeat final grading.

Individual valid trials

Task Arm Repeat Correct Diagnosis Before patch Cycles Wall s Capped tokens Raw tokens Regressions
F01 Lean 1 PASS match yes 2 78.9 18,581 222,357 0
F01 Lean 2 PASS match yes 2 69.2 39,132 194,140 0
F01 Lean 3 PASS match yes 2 80.3 21,971 220,499 0
F01 Quint 1 PASS match yes 3 103.0 26,272 227,360 0
F01 Quint 2 PASS match yes 4 135.6 32,882 319,346 0
F01 Quint 3 PASS match yes 4 119.8 31,957 208,853 0
W01 Lean 1 PASS match yes 5 120.7 22,313 277,161 0
W01 Lean 2 PASS match yes 2 84.8 29,040 191,856 0
W01 Lean 3 PASS match yes 2 88.6 30,974 194,558 0
W01 Quint 1 PASS partial yes 3 119.4 35,442 274,162 0
W01 Quint 2 PASS match yes 6 178.8 50,078 479,646 0
W01 Quint 3 PASS match yes 2 100.6 18,694 229,126 0
W04 Lean 1 PASS match yes 4 126.9 39,743 290,495 0
W04 Lean 2 PASS match yes 4 110.4 25,030 259,910 0
W04 Lean 3 PASS match yes 3 97.3 28,138 327,914 0
W04 Quint 1 PASS match yes 4 156.6 43,767 362,999 0
W04 Quint 2 PASS match yes 3 114.6 52,675 320,323 0
W04 Quint 3 PASS match yes 4 145.5 37,401 450,201 0
X01 Lean 1 PASS match yes 2 92.5 22,306 217,634 0
X01 Lean 2 PASS match yes 2 85.2 37,728 315,232 0
X01 Lean 3 PASS match yes 2 90.1 26,860 245,100 0
X01 Quint 1 PASS match yes 2 109.2 29,167 313,967 0
X01 Quint 2 PASS match yes 3 147.2 29,528 315,480 0
X01 Quint 3 PASS match yes 2 108.1 21,499 201,723 0

Next steps

Both tools achieved 12/12 correct repairs. This small pilot establishes no winner or tool-selection decision; effort differences are descriptive. A separately registered follow-up could use harder tasks or remove source-location feedback, with equal inputs and caps and complete execution and billed-cost records.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
rounds/fv-pilot-2026-10-10/reproduce.py
python3 rounds/fv-pilot-2026-10-10/reproduce.py
Inputs
Full execution replay
Unavailable
Scope
Recomputes per-task and arm results from the sanitized lifecycle ledger; model calls and independent final evaluation are not replayable.

View the complete per-round reproduction inventory · Repository reproduction guide