> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [results/publication-validation.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/publication-validation.md)
- [results/publication-blockers.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/publication-blockers.json)
- [results/GRADE-JOIN.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/GRADE-JOIN.md)
- [EVIDENCE-MAP.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/EVIDENCE-MAP.md)

# Publication validation

Publication status is **ready**. B1 is resolved: All 5,020 captured graded IDs now match unique rows in the refreshed public official grade export by exact cell ID. The 128 post-cut rows were appended from the authoritative ledger; 25 invalid control_apply rows are controls, not model failures. B2 is resolved: The locally sanitized v1.2 snapshot preserves cited content from original revision 1118f7fc0dbb042af8c8de2ffd1b85768cf9e2a0 and uses public-release sanitization revision 30d9b25e4eefa7117121de44298fed504b429f4e. Its published-byte SHA-256 manifest verifies, and all 134 pinned EVIDENCE-MAP fragments resolve locally. The decision ledger is absent, so comprehensive decision-to-clause coverage remains limited. The full validation report and structured evidence-gate register are below.

# Publication validation

This report distinguishes repository consistency checks from the configured release gate. `validate_repo.py` checks repository indexes, records, cross-references, covered numbers, privacy rules, and authored relative links. `validate_release.py` validates the historical round-label inventory, applies strict Standard-record checks only to rounds dated 2026-10-09 or later, checks the registered publication gates, and then runs `validate_repo.py`. It does not run scrutiny, require independent-review receipts, or invoke official-grade round reproducers.

## Public grade-export scope

**B1 is resolved.** The refreshed public `grades.final.jsonl` export exact-joins **5,020/5,020** unique captured graded IDs using `source_record_id == public_cell_id == grades.final.jsonl cell_id`. The 2026-10-07 refresh appended the 128 authoritative official rows missing from the prior snapshot; the exact join now has zero capture-reported-only mappings. See [results/GRADE-JOIN.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/GRADE-JOIN.md) for the builder output and per-round counts.

Historical “official rows before” counts in the table below refer to the pre-refresh snapshot at benchmark commit `58c9ca5ee40fb958c61c61e381191a922886f88f`, committed `2026-10-07T16:18:30+03:00` (export SHA-256 `cbf85effb6334c650fa0204da6e2777ed8a5a699f14ab854529d998b5177703a`). The refreshed export SHA-256 is `4d639a3848b3661dcb5646630b6a4914e5053ff4b94ab029424d47d040e3c977`.

The 128 additions contain **84 model PASS, 19 model FAIL, and 25 invalid `control_apply` rows**. The controls carry `result=invalid` and `outcome=fail`; they are invalid controls, not model failures. The refreshed cells export preserves them as invalid and keeps their ITT treatment under the existing control policy.

The before/after tables classify rows by source `kind` and `result`. The previous `cells.*` export displayed the two pre-existing invalid controls in these target rounds as `outcome=fail`; the refreshed export now displays their official `result=invalid` classification. This classification is applied to all 199 `control_apply` rows in the refreshed input, including the 25 additions.

| Round | Official rows before (snapshot 58c9ca5, 2026-10-07T16:18:30+03:00) | Added | Official rows after | Added model PASS / FAIL | Added invalid controls |
|---|---:|---:|---:|---:|---:|
| r69 | 95 | 55 | 150 | 46 / 9 | 0 |
| r67b | 1 | 42 | 43 | 17 / 8 | 17 |
| r71 | 18 | 31 | 49 | 21 / 2 | 8 |
| **Total** | **4,892** | **128** | **5,020** | **84 / 19** | **25** |

The round verdicts remain INTERIM. r69's registered rule requires 400 official grades plus a gain of at least 15 percentage points and one-sided task-stratified p < 0.05; its refreshed public export verifies 150 of the 327 official grades in the recovered L0 cohort. r67b has no recovered public predeclared decision rule and its task/arm assignments remain unresolved. r71 has no located promotion threshold and only 40 model outcomes for 48 planned model slots; its nine invalid controls are not model failures. No decision or verdict changed.

The repository exact-join check verifies unique IDs on both sides and every capture ID against the refreshed export. The specification-fragment and round-inclusion checks are recorded below.

## Validation output

- `python3 reproduce/build_grade_join.py`: PASS — wrote 5,020 exact public official-export matches and 0 capture-reported-only mappings.
- The B1 exact-join predicate in `validate_repo.py`: PASS — no B1 join, count, report, or status errors were emitted.
- `python3 reproduce/validate_spec_snapshot.py`: PASS — 134 linked v1.2 fragments resolve; provenance, local navigation, and published-byte hashes verify.
- `python3 reproduce/test_validate_release.py`: PASS — six policy cases cover acceptance and rejection conditions for release-included-with-label.
- `python3 reproduce/validate_repo.py`: PASS — all five r67b task mappings reconcile to official grade rows and exact source-crosswalk identities; the pinned specification checks pass.
- `python3 reproduce/validate_release.py`: PASS — 243 round pages: 179 historical rounds are labelled, and 64 rounds dated from the strict cutoff of 2026-10-09 are checked under the new-round rule. None of those 64 has complete Standard capture; each carries a DESCRIPTIVE or INVALID label rather than VALID. The registered publication gates and repository checks pass. This result does not mean scrutiny or independent review receipts were checked.

## Specification and decision sources

The locally sanitized [specification snapshot](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/spec/README.md) preserves the v1.2 clause-reference content cited at original revision `1118f7fc0dbb042af8c8de2ffd1b85768cf9e2a0`; the public-release sanitization basis is revision `30d9b25e4eefa7117121de44298fed504b429f4e`. `spec/SOURCE.json` separates original-source hashes from hashes of the exact published bytes, and the SHA-256 manifest verifies the complete snapshot. All 134 linked v1.2 fragments in EVIDENCE-MAP.md are locally verified. **B2 is resolved for this cited-fragment subset.** The decision ledger is absent, so comprehensive decision-to-clause coverage remains limited.

## Scored-round record and release status


- **Named-defect review:** The [descriptive screen](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/review-named-defect-1/README.md) retains the first run's prose specification gap and reports a weak-builder follow-up with 16 of 16 attempts scored, seven cap stops, one rescue, and no wrong submissions.
- **Low-effort builder screen:** The [descriptive screen](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/builder-low-1/README.md) scored 8/8 in each setting with no repairs or wrong submissions. Builder input fell about 38% (651,878 to 407,155 tokens), while total input fell 4% (2.42M to 2.33M).
- **Prose versus Gherkin:** The [round page](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/prose-vs-gherkin-1/README.md) retains the correction that its original prose omitted a key-order requirement. In the weak-builder follow-up, both settings scored 4 of 4, wrong code reached review in 3 of 4 per setting, and the reviewer-fixer rescued six wrong trees.

The separate `validate_round.py --round ID --strict` command can check Standard-record structure and declared gaps/deviations for [language replication](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/lang-sol-replication/README.md), [L3 repair](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l3-repair/README.md), and [L3b repair vs continue](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l3b-repair-vs-continue/README.md). Their records declare 1,782, 390, and 193 missing Standard capture slots respectively, with four, four, and two deviations. These rounds predate the configured strict cutoff, so `validate_release.py` does not run those strict checks for them and does not invoke their official-grade reproducers. The historical round pages retain their own evidence limits and status labels; missing receipts have not been reconstructed.

## Archive claims and limits

- **Sol-medium language replication:** The raw combined totals and post-hoc equal-task rates are reproduced from the round CSV. The original formula and the four-pass scale are unrecorded, so the equal-task calculation is a sensitivity analysis, not an executable registered branch. See [round results](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/lang-sol-replication/RESULTS.md) and [claim ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/claim-ledger.jsonl).
- **L3 repair:** The aggregate CSV reproduces two rescues among six repairs; the three-rescue threshold was not met. Per-test pass-set preservation is unverifiable, and the sample does not establish causal repair benefit. See [round results](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l3-repair/RESULTS.md).
- **L3b repair vs continue:** The sanitized 24-cell table reproduces 3/8 REPAIR, 1/8 CONTINUE, and 2/8 RESTART rescues. The registered REPAIR-minus-CONTINUE threshold is met, and the private per-test no-regression result is published as a boolean. The result remains descriptive because of the disclosed DEV-1 overlap, hidden-suite exposure, and run-record gaps. See [round results](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/l3b-repair-vs-continue/RESULTS.md).
- **L6 auditor replay:** In the offline 40-patch replay, neither auditor met both predeclared precision and false-demotion targets. The 79 valid verdicts among 80 requests are auditor outputs, not official benchmark grades. The result supports advisory-only use for this sample, not a live Build gate.
- **R70 extension outcomes:** The public test-count ledger supports the pre-registered reps 31–32 full-pass counts for this extension: Rust 5/6, Elixir 4/6, Go 4/6, and TypeScript/Bun 6/6. Keep rep 33 separate as a post-hoc addition; the small, one-model cohorts do not support a stack ranking.
- **L1 grader controls:** Official controls found that all 14 wrong mutants failed and all 7 valid alternatives passed across the seven admitted variants. Two mutants failed at build and are weaker controls; each affected variant also had a behavior-level mutant failure. This supports only the listed task/grader control outcomes, not general grader fairness or Build success.
- **Latency probes:** The short and long probes support descriptive client-visible Codex CLI timing summaries from their public raw logs. Report 77 short-probe attempts and 76 uncensored completions, and 18 long-probe attempts and 17 uncensored completions. These do not establish provider transport latency, task success, production limits, or a model/effort ranking.
- **Task-8 smokes:** Four public task-8 smoke rows each record 1/25 hidden-test assertions and are explicitly unscored. They support smoke diagnostics only, with no task-8 efficacy or language-comparison claim.
- **L0 reconciliation:** `python3 reproduce/reconcile_l0.py` reproduces the published sanitized CSV analyses. It does not make each row an independently joined official outcome. R74 remains WITHDRAWN, and R72/R69/R57 remain bounded by their existing cohort status and join coverage.

Family summaries remain bounded. They do not establish a general comparative Build-success effect, and interim, confounded, withdrawn, not-run, or source-reported historical counts are not promoted into verified comparisons.

## Validation contract

- `python3 reproduce/validate_repo.py` checks indexes, records, cross-references, covered numbers, privacy constraints, and authored relative links. It does not independently rerun historical model calls or hidden grading.
- `python3 reproduce/validate_round.py --round ID --strict` checks one round's Standard record structure, exact gap/deviation declarations, and the strict eligibility predicate. It is a separate command; `validate_release.py` does not call it for historical rounds.
- `python3 reproduce/validate_release.py` checks that round pages and `rounds/STATUS.md` agree on historical labels, badges, reasons, dates, and recomputation states; applies strict `validate_round.py` checks only to rounds dated 2026-10-09 or later; verifies that the publication-blocker register is well-formed and resolved; and runs `validate_repo.py`. It does not run `scrutinize.py`, inspect or require independent-review receipts, or run official-grade round reproducers. On this snapshot it reports 179 historical rounds and 64 rounds under the strict rule, none with complete Standard capture and each labelled DESCRIPTIVE or INVALID, and `RELEASE GATE: PASS`.

- **Reviewer effort with a weak builder:** The [screen](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/reviewer-effort-weak-1/README.md) started 24 attempts and scored 23; Sol high rescued 11/11 wrong trees and Sol medium 6/6, with thinner rescue evidence for medium. Review stage time (Lab bundle) was 2,619 s for high and 1,173 s for medium (about 55% less); medium received fewer code snapshots failing hidden tests before review.
- **Lower-cost profile:** The [screen](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/cheap-profile-1/README.md) now covers three runs on eight development tasks; both profiles scored 12/12, total input fell 37%, and the sum of recorded stage times fell 32%. The third run added four tasks: 4/4 correct per profile, eight attempts total, with about 20.9% less input; wrong code did not reach review.
- **Cheap profile confirmation:** The [fresh-task confirmation](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/confirm-cheap-profile-1/README.md) scored 16 attempts across eight task–repetition pairs, each tested with both profiles; each profile was correct on all eight. The lower-cost profile used about 95.8% of default input against the fixed 75% ceiling; the decision is UNRESOLVED.
- **Reviewer polish:** The [development screen](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/reviewer-polish-1/README.md) scored 16/16 attempts, 8/8 per reviewer. Medium edited one already-correct tree against high's two, with about 6% less reviewer input and 35% less review stage time; no wrong code reached review.
- **First medium reviewer confirmation:** The [fresh-task confirmation](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/confirm-reviewer-medium-1/README.md) scored 16/16 attempts, 8/8 correct per reviewer. High rescued 4/4 wrong trees and medium 3/3; medium used 56% of high's input but received only three wrong trees against the fixed minimum of four. The decision is UNRESOLVED.
- **Second medium reviewer confirmation:** The [fresh-task confirmation](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/confirm-reviewer-medium-2/README.md) scored all 24 attempts; high was correct in 9/12 and medium in 11/12, with four builder stops before review. High rescued 4/4 wrong trees and medium 5/5, but medium used about 84.4% of high's input against the 70% limit. The decision is UNRESOLVED; there were no provider refusals.

- **Review value by builder strength:** The [pooled analysis](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/review-value-1.md) reports no genuinely wrong first-green trees among 169 included Sol-builder attempts; eight were submitted without review. The screen was NOT INFORMATIVE because 0 of 16 were wrong, below the four-build threshold; the weak-builder sample still records 31/31 rescues among 57 attempts reaching review. The 57 include one unscored provider-refusal attempt and one attempt stopped during review on already-correct code; seven attempts stopped before review and are excluded.
- **Hard-task review screen:** The [descriptive round](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/review-off-hard-1/README.md) scored 16 of 16 attempts; both settings were correct in 8 of 8 and no first-green build was wrong (0 of 16 against 4 required). Fixed decision: NOT INFORMATIVE; review-off used 1.01M input against 2.11M with review on; the screen does not establish whether review should be disabled.
- **Short intent development screen:** The [round](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/intent-short-1/README.md) scored 16 of 16 attempts; full and short were each correct in 8 of 8, with no wrong submissions, stops, provider refusals, or wrong first-green builds. Short used 19% less input; fixed decision SHORT HOLDS.
- **Short intent fresh-task confirmation:** The [round](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/confirm-intent-short-1/README.md) scored 12 of 12 attempts across six paired blocks; both arms were correct in 6 of 6, with no wrong submissions, stops, refusals, or wrong first-green builds. Short used 18% less input; the fixed rule returns CONFIRMED.


## Structured evidence-gate register

- B1 (resolved): All 5,020 captured graded IDs now match unique rows in the refreshed public official grade export by exact cell ID. The 128 post-cut rows were appended from the authoritative ledger; 25 invalid control_apply rows are controls, not model failures.
- B2 (resolved): The locally sanitized v1.2 snapshot preserves cited content from original revision 1118f7fc0dbb042af8c8de2ffd1b85768cf9e2a0 and uses public-release sanitization revision 30d9b25e4eefa7117121de44298fed504b429f4e. Its published-byte SHA-256 manifest verifies, and all 134 pinned EVIDENCE-MAP fragments resolve locally. The decision ledger is absent, so comprehensive decision-to-clause coverage remains limited.

See [GRADE-JOIN.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/GRADE-JOIN.md) and the [evidence map](/evidence/).
