Published result

Saved-candidate reviews after four changes

A plain-language summary comes first, followed by the full text of the result's reproduction pack.

Result summary and pack text · reproduce/review-climb-saved-candidates-20261103/README.md

In plain terms

A completed 36-review screen detects 4 versus 3 held-out target mechanisms. Repair verification and the clean-control result remain unresolved, so no improvement is qualified.

Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.

Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

Verdict: unresolved. All 36 prescribed reviews completed: nine correlated saved cases, two repetitions per arm. HELD-OUT target mechanisms detected were A 4/14 and B 3/14. This screen does not establish that B catches more defects without extra false blocks.

DEV was scored and frozen before the first HELD-OUT call. Every assigned case remains in the tables. Repetitions are not additional independent tasks, and this is a component-phase screen with no significance or end-to-end rescue claim.

What the counts mean

A detection requires concrete evidence for the predeclared mechanism, including a corrective source edit that identifies it. Reported findings and attempted repairs are separate columns. Repairs remain unverified: the shipped standalone command refuses external verification after edits. A submitted approval followed by that refusal is not a final approval of the original defective tree. No hidden grader was run on the edited trees.

The case labelled clean has a real fixture failure on the execution environment and an additional demonstrated unlabelled defect. Its observed blocks are retained, but a false-block rate on a genuinely clean tree is not established. Some scoring-key expectations are narrower than the explicit prose contract; those misses remain in the primary totals and are diagnosed separately. The initial adapter evidence declares a green pass with no executed checks; it is not a measured qualification receipt.

Source and runtime

A is kogen-review 889ed92. B is kogen-review 3e1f93f, whose review implementation is unchanged from f01c92e. B includes the four changes under examination. Both arms use the same dependency pins and observation adapter, gpt-6.1-terra medium through ChatGPT, and the same toolchain and Git. Binary and adapter hashes are in data/versions.json. This comparison does not isolate the four changes individually.

Lever observations

All 36 configured and effective model/effort/backend observations match, and the four initial trees match within every case. B submitted nonempty requirement ledgers in 18/18 runs. Maximum logical requests were A 9, B 12; forced submission at or after request 13 fired 0 times. Recorded repair rounds reached 0; the adapter refusal prevents this screen from exercising the nonconverging-repair bound. These observations do not isolate individual item effects.

Cost and wall time

All review walls include failed attempts and the tools invoked within a review. Delivery waits and preparation are outside per-review wall measurements. The known route subtotal, including one apparatus-invalid setup attempt, is $8.671432 at the guard proxy rate. This is not an actual subscription bill or a published Terra price. Helper/orchestration model cost is unavailable, so a complete whole-workflow monetary total cannot be claimed. The extra attempt's requests, wall and guarded cost are retained in data/workflow-extras.json.

The last counted result file was written at 2026-11-03T20:52:37.964115404Z. Scoring completed at 2026-11-03T20:53:11.459590+00:00; feedback followed the last climb item by 3.970 hours.

Rerun

Recompute every published per-case and per-arm table with Python's standard library:

python3 report.py data/scores.tsv

The command checks all 36 assigned arm/repetition rows before reporting. data/scores.tsv uses anonymous case labels; it contains no candidate source, evaluator label text or private-reference patches. The public commits above allow inspection of the component changes.

Reproduction scope is the published analysis. Exact live replay requires the evaluator-custodied candidates, scoring key and identical observation adapter; they are not redistributed here. The original command was the standalone kogen-review entry point with a candidate base, approved Intent slug, fresh root, model, provider and effort supplied explicitly.

Disposition

Drop this exact screen as qualification evidence. No automatic extension, component revert or adoption hold is asserted. The comparison does not qualify an improvement claim. A separate mechanism-only follow-up addresses evidence citations that fail to establish a promised postcondition. Normal repository and release gates remain authoritative.

Paired descriptive tables

split case arm n detected targets missed targets requests wall seconds guarded cost USD
DEV D1 A 2 0 0 13 151.044 0.480727
DEV D1 B 2 0 0 7 96.365 0.434036
DEV D2 A 2 0 2 11 146.327 0.303443
DEV D2 B 2 0 2 10 148.430 0.473339
DEV D3 A 2 0 2 13 136.775 0.478570
DEV D3 B 2 0 2 9 157.356 0.468327
DEV D4 A 2 2 0 11 129.779 0.459388
DEV D4 B 2 1 1 11 160.024 0.483804
DEV D5 A 2 2 0 5 30.813 0.419450
DEV D5 B 2 2 0 10 86.445 0.518624
HELD-OUT H1 A 2 0 2 11 143.032 0.453691
HELD-OUT H1 B 2 0 2 8 123.415 0.450487
HELD-OUT H2 A 2 0 4 12 126.537 0.449937
HELD-OUT H2 B 2 0 4 10 129.204 0.449158
HELD-OUT H3 A 2 2 0 17 138.573 0.507009
HELD-OUT H3 B 2 1 1 17 207.820 0.556518
HELD-OUT H4 A 2 2 4 15 132.741 0.471552
HELD-OUT H4 B 2 2 4 20 289.447 0.595746
split arm n detected targets missed targets requests wall seconds guarded cost USD
DEV A 10 4 4 53 594.737 2.141578
DEV B 10 3 5 47 648.622 2.378130
HELD-OUT A 8 4 10 55 540.883 1.882189
HELD-OUT B 8 3 11 55 749.886 2.051909
ALL A 18 8 14 108 1135.620 4.023767
ALL B 18 6 16 102 1398.507 4.430039

All attempted repairs remain unverified. Guard estimates are not subscription bills.