Round offline-aa-bo3

Offline A/A and best-of-three analysis

Status · DESCRIPTIVEReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/offline-aa-bo3/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Why not VALID: The historical inventory retains DESCRIPTIVE because of POST_HOC_ANALYSIS, RAW_EVIDENCE_MISSING. See the round evidence and limitations below. Recomputation status: OBSERVED SOURCE ONLY

Recomputation limit: Published summary figures: planned/observed counts, outcome totals/rates, and numeric estimates as present in README.md and RESULTS.md. These figures cannot be recomputed from repository inputs because no public per-round analysis script with a complete, identified input bundle is published. See the round README, MEASURED.md, and MISSING.md for recorded scope and gaps; historical execution inputs/toolchains/grading are not bundled.

Status

DESCRIPTIVE

Why not VALID

  • POST_HOC_ANALYSIS: This is an offline selector replay and simulation, not a pre-registered live comparison.
  • RAW_EVIDENCE_MISSING: The underlying grade rows and selector inputs are absent.

Required reproduction metadata

  • Kogen commit: not recorded in the cited round source.
  • Harness commit: not recorded in the cited round source as a Git commit; any adapter or runtime fingerprints are not Git commits.

STATUS: INTERIM — Offline replay and simulation; descriptive only. Study figures in the result section are source-reported, not reproducible from public data. The accounting table classifies this as an offline analysis; its zero live Build counts are not measured outcomes. The source result file is not included in this public page.

Scope and reported result

The source separates a false-promotion simulation over 34 historical tasks from a best-of-three selector comparison on 17 eligible direct Luna-max tasks. For the simulation, reported directional false-promotion rates were 0.95% for a threshold of at least 4/5 versus at most 1/5, 2.04% for a gap of at least 3 passes, 7.79% for a gap of at least 2 passes, and 3.17% for 3/3 versus at most 1/3. The report estimates 85 repetitions per arm for a 20 percentage-point effect or 41 for a 30 percentage-point effect at its stated error and power settings; those settings are not included in the supplied summary.

In the separate selector comparison, reported equal-task pass@1 was 33.3%, deterministic selector pass@3 was 23.5%, and oracle pass@3 was 47.1%, with three times the dollars and summed wall time for the selector. Visible-test exit data were unknown for 75 of 84 eligible repetitions. This is evidence about that specific selector and snapshot, not all candidate ensembles.

Live Build accounting

planned_n started_n finished_n graded_n itt_n
0 0 0 0 0

This page covers offline replay and simulation, not a live Build cohort. The historical task and repetition counts above describe separate analysis units.

Reproduction record

Field Record
Harness / Kogen commit The analysis is an offline replay; no harness or Kogen execution is described. Original cell revisions are not listed in the supplied public input.
Model and effort No model call is described for the analysis. The underlying historical cohort's complete model/effort roster is not supplied.
Task IDs Exact task and cell IDs are not supplied; the source describes separate 34-task and 17-task cohorts.
Command The exact replay/simulation command is not recovered from the supplied public input.
Raw records The source inventory names results.json; underlying grade rows and the selector inputs are not included in the public data tree.

Until the task IDs, analysis settings and input grades are published, all figures remain source-reported, not reproducible from public data.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
unavailable
Script
No per-round analysis script is published.
Inputs
No identified public input bundle.
Full execution replay
Unavailable
Scope
No public per-round analysis script with a complete, identified input bundle is published. See the round README, MEASURED.md, and MISSING.md for recorded scope and gaps; historical execution inputs/toolchains/grading are not bundled.

View the complete per-round reproduction inventory · Repository reproduction guide