F06

F06 — Checks, review, and recovery

The authored family record is reproduced below. Structured links connect its hypotheses, rounds, and claim-ledger rows.

Source-reported narrativeNumeric statements in this family document remain source-reported unless an exact claim-ledger row is linked and its status allows that interpretation.

Family source document · hypotheses/f06-checks-review-recovery.md

F06 Checks, review, and recovery

This page records evidence for H62–H74. The propositions below are hypotheses, not established findings. Published measurements remain attached to their individual runs and claim IDs. Where a B1 claim has no hypothesis mapping, it is cited as contextual evidence without changing that ledger mapping.

Hypotheses and current status

ID Falsifiable proposition Current evidence status
H62 A selective, faster check set can preserve detection of real failures while reducing check time or cost. NOT ESTABLISHED. No public matched before/after result measures both failure detection and work or time.
H63 A catalog of safe parallel tests can reduce verification wall time without changing results. NOT ESTABLISHED. No public controlled catalog or concurrency comparison is identified.
H64 Targeted tests of Build mechanics can preserve defect detection at lower cost than repeated full shaping checks. NOT ESTABLISHED. No matched mechanics-test versus full-check result is published.
H65 Clear handoff, process custody, failure classes, and visible stage status can prevent repeated expensive Build attempts. NOT ESTABLISHED. Public records include blocked and smoke-only workflows, but no isolated live recovery comparison.
H66 Strong deterministic checks can enable a low-effort model to complete tasks that weaker checks miss. NOT ESTABLISHED. No check-ablation comparison is joined to official outcomes.
H67 A scoped reviewer veto improves acceptance quality enough to justify its review and repair cost. INTERIM, OFFLINE ONLY. L6 misses both targets in the selected sample and is not a prospective live veto test.
H68 Executable review catches small-diff false accepts that other review methods miss. INTERIM, OFFLINE ONLY. The public summaries do not include the patch-level decision ledger.
H69 Contract audits detect missing requested behavior and can safely gate completion or trigger repair. INTERIM, OFFLINE ONLY. The published audit did not run a repair or Build rerun.
H70 Explicit behavior checks outperform source-text or file-existence checks without losing relevant coverage. NOT ESTABLISHED. Offline audits identify test-quality concerns, but do not compare those check designs prospectively.
H71 The proposed static and domain-specific checks are accurate and fast enough for the default gate. NOT ESTABLISHED. The study census lists check research, but no public default-gate portfolio comparison is available.
H72 A post-change fix loop can repair a static-check failure rather than reject functionally passing work. NOT ESTABLISHED. No matched repair-loop outcome is identified in the public records.
H73 Verification should score observable behavior even when an implementation reports its own checks as successful. PARTIAL, OFFLINE ONLY. Saved-output and candidate audits are counterfactual evidence, not a prospective grading-policy test.
H74 Reusable duplicate, error, and type checks provide value beyond the Kogen project. NOT ESTABLISHED. No cross-project transfer or maintenance comparison is published.

Run and claim record

Live counts refers to planned, started, finished, graded, and ITT Build cells. Offline rows have no live Build population. Where a value is reported but its underlying cases are absent, the linked claim remains INTERIM and is labeled source-reported, not reproducible from public data.

experiment_id arm and task set model / effort Kogen and harness revision host / epoch planned / started / finished / graded / ITT outcome status and correction claim ID public receipt and raw records
x-review-checklist Checklist and risk-directed review; exact diagnostic cases unavailable Exact model and effort unavailable Offline review; no live Kogen or harness revision Not applicable Offline analysis; live counts not applicable Each method was reported at 0/14 false accepts, 0/10 false blocks, and 14/14 supported-defect recall; case-level join unavailable. INTERIM; source-reported, not reproducible from public data. CL-WA-x-review-reported-x-review-checklist round page, reported data, claim ledger
x-review-repair Partial review-and-repair workflow; exact tasks unavailable Exact model and effort unavailable No live Kogen revision recovered Not applicable Offline workflow; live counts not applicable 1/4 sessions reportedly emitted the required final result; 0/8 fresh confirmation calls occurred. INTERIM; partial repair workflow, not task completion efficacy; source-reported, not reproducible from public data. CL-WA-x-review-reported-x-review-repair round page, reported data, claim ledger
x-review-semantic-repair Separate semantic-repair screen; exact task and arm mapping unavailable Sol high is reported; exact model build unavailable No live Kogen revision recovered Not recovered Small source-reported screen; live counts not applicable Complete deliveries reportedly failed the confirmation gate at 1/2 versus 0/2; do not combine with the diagnostic case set. INTERIM; source-reported, not reproducible from public data. CL-WA-x-review-reported-x-review-semantic-repair round page, reported data, claim ledger
offline-review3 Tool-less versus executable final-patch review; frozen patch IDs unavailable Sol high is reported for both arms; exact model build unavailable Offline review; no live Kogen revision Not applicable Offline analysis of a planned patch set; live counts not applicable Reported paired first runs: 63/80. Tool-less: precision 19/23, recall 19/32, false vetoes 4/31. Executable: precision 21/29, recall 21/32, false vetoes 8/31. INTERIM; budget-stopped partial study; source-reported, not reproducible from public data. Earlier interim summary superseded. CL-WA-offline-review3-reported-observation round page, claim ledger
offline-contract-audit Sol medium and Luna max audits of known false accepts and passing controls; exact case IDs unavailable Sol medium and Luna max; model builds unavailable Offline audit; no live Kogen revision Not applicable Offline analysis; live counts not applicable Exact defect found in 1/5 versus 0/5 cases; passing controls flagged in 10/10 versus 6/10. No repair or Build rerun followed. INTERIM; source-reported, not reproducible from public data. CL-WA-offline-contract-audit-reported-observation round page, claim ledger
intent-coverage-audit Saved failure outputs and controls from R58, R61, R62, and R62b; no new model arm No new model execution Offline audit; original implementation revisions unavailable Not applicable Offline analysis; live counts not applicable The report attributes 8/24 failed-test occurrences to dropped or weakened requirements, 15/24 to requirements retained but still failing, and 1/24 to stricter-than-request behavior. INTERIM; descriptive saved-output audit, not a shaping effect. Source-reported, not reproducible from public data. CL-WA-intent-coverage-audit-reported-observation round page, claim ledger
overstrict-replay Recovered ExUnit tests replayed against saved implementations from R61–R71 No model execution Offline replay; original source revisions unavailable Not applicable Offline test replay; live counts not applicable Reported 168/1,160 replayed tests as over-strict; the provided-R71 subset was 9/23. INTERIM; source-reported, not reproducible from public data. The replay does not show that a particular Build stop was caused by an over-strict test. CL-WA-overstrict-replay-reported-observation round page, claim ledger
candidate-counterfactual Saved candidate diffs from R67b and R71; not official final outcomes No new model execution Offline counterfactual audit; original cell revisions not supplied Not applicable Offline candidate grades; not a live Build cohort Seven saved candidates were graded and all seven passed; the audit classified six cells as over-blocked and found zero genuine misses among those six. INTERIM; counterfactual only, not an official pass or ITT result; source-reported, not reproducible from public data. CL-WA-candidate-counterfactual-reported-observation round page, claim ledger
l6-auditor-replay Request-scoped auditor on 40 selected r70 patches; two auditor models Luna max and Sol low Offline request runner; fingerprints and source-case IDs are in the round page 80 requests; 79 valid verdicts; one timeout; live grade and ITT counts are zero Neither auditor meets both predeclared precision and false-demotion targets; advisory only for this sample. VALID offline replay; not a live gate qualification. CL-L6-AUDITOR-ADVISORY-REPLAY (STATUS_ONLY) L6 auditor replay, claim ledger
x-continuation-live Live continuation path; task IDs unavailable No real-model attempt reported Exact Kogen and harness revisions unavailable Not recovered Blocked before model execution; no scored live population No live continuation result. NOT-RUN; source-reported, not reproducible from public data. CL-WA-x-continuation-reported-x-continuation-live round page, reported data, claim ledger
x-continuation-scripted-provider Scripted-provider mechanical path; exact task IDs unavailable Scripted provider; no live model Exact Kogen and harness revisions unavailable Not recovered Offline mechanical test; live counts not applicable A source summary reports 3/6 checks before and 6/6 after a mechanical change; this is not a task outcome. NOT-RUN for live efficacy; source-reported, not reproducible from public data. CL-WA-x-continuation-reported-x-continuation-scripted-provider round page, reported data, claim ledger
x-recovery Recovery lifecycle; exact task set unavailable Model and effort unavailable Exact Kogen and harness revisions unavailable Not recovered Planned, started, finished, graded, and ITT counts unavailable Ledger identified; per-arm extraction remains required. INTERIM; no outcome estimate. CL-WA-x-recovery-reported-x-recovery round page, reported data, claim ledger

Design implications

The public evidence supports treating test quality and review false blocks as risks to measure. It does not qualify a default review veto, a fixed list of fast checks, automatic test demotion, or an automatic contract-audit gate. Keep offline diagnostics, saved-candidate counterfactuals, and live Build outcomes separate.

What this evidence does not show

It does not establish that a reviewer improves live acceptance, that additional checks preserve defect detection while reducing time, or that a recovery loop increases completion. Offline counts are not live Build rates. A candidate that passes a saved grading counterfactual is not an official round pass.

Smallest open tests

Freeze a blinded set of true defects, valid alternatives, and passing controls; compare each proposed check or reviewer with a matched baseline; record exact case IDs, false accepts, false blocks, time, cost, and fresh confirmation outcomes. For H65 and H72, run a prospective recovery test that records process custody and stage transitions and counts Kogen-internal stops under the declared ITT rule.

Evidence links

Related round histories: R37, R48b, R56, R67b. The research census records related check and review studies as index metadata only; no per-study synthesis is used here.