Round l4-context-packet

L4 deterministic context packet

Status · DESCRIPTIVEReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/l4-context-packet/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Why not VALID: The historical inventory retains DESCRIPTIVE because of CONDITIONS_MISMATCH, POST_HOC_AMENDMENT. See the round evidence and limitations below. Recomputation status: FULLY RECOMPUTABLE

Status

DESCRIPTIVE

Why not VALID

  • POST_HOC_AMENDMENT: Amendment 1 was written after 6 of 12 packet outcomes were known.
  • CONDITIONS_MISMATCH: The US controls were not interleaved with packet cells.
  • relabelled 2026-10-08 after scrutiny: Previously labelled VALID; the README documents a post-result amendment and non-interleaved controls.

STATUS: DESCRIPTIVE

Pre-registered: yes; original design frozen 7 October 2026 before the first scored packet cell. Amendment 1 was written after 6 of 12 packet outcomes were known; its disclosure is retained in the decision rule. Label: descriptive pilot; n=3 per arm per variant. The thresholds are coarse and support no general claim. Question: Do frozen public context packets increase official full-suite passes over same-seed contemporaneous no-packet controls on four L1-admitted variants? n: 24 official scored cells (12 packet, 12 control; 4 variants × 3 seeds per arm); 20 historical cells are context only. Headline (descriptive; not a validated registered finding): observed KEEP for packets in specification sections 3.2 and 4.9: packet-minus-control = +6 passes, 6 paired rescues, 0 losses, and no regression. The US controls were not interleaved. Configuration: direct Codex, gpt-6-luna at max, 3,600-second cap, zero retries; Elixir and Rust on EU, Go on US; official r70-macbook-window-v1 grades. Tokens are uncached input + cached input + output; total is their sum. Official runner wrapper: Codex 0.159.3, SHA-256 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300; this same fingerprint appears on all scored cells. Limit: outcome-selected tasks, n=3 per arm per variant, coarse thresholds, and mixed scheduling; the result supports no general claim.

Lifecycle count Count
Planned scored cells 24
Started scored cells 24
Finished scored cells 24
Officially graded scored cells 24
ITT denominator for scored cells 24

Sources: decision rule and Amendment 1, results, paired-cell data, historical baseline data, and reproducer.

Reproduce

Public base revisions

Kogen commit: source 470ba65c1d3c19191a2190a032cd9ebf24e71975 and packet 08a1ca88b8dfd67760586f433fc773a6b23842bb are exact linked examples; all per-task source and packet SHAs are listed above.

Harness commit: not recorded as a Git commit in the official aggregate grade rows. The lane dispatcher SHA-256 is f23824e32fb74f42e95e3a0c106df4070fac5425b9c8981dfbd293859acfdfee; the common wrapper fingerprint recorded on all scored rows is SHA-256 08019d8aab7c15481f6d14126645eebd7b2c2c98025f94118baeee2167e4a300.

Model and effort: direct Codex, gpt-6-luna at max; official grade route r70-macbook-window-v1; 3,600-second cap and zero retries. The lane handoff lists worker Codex CLI pin 0.160.0 and Python 3.14.7. Packet authoring used codex-cli 0.160.1 in an ephemeral public-only workspace. The reported runner wrapper version and fingerprint appear in the CSV-backed summary above.

Task IDs: packet tasks r70-2-elixir-l4pk, r70-2-go-l4pk, r70-4-elixir-fe2-l4pk, and r70-7-rust-l4pk; control tasks r70-2-elixir, r70-2-go, r70-4-elixir-fe2, and r70-7-rust. Exact cell IDs, reps, and seeds are in the paired-cell CSV.

Historical execution command: not recorded as a shell transcript. Reproduction command, run from the SOT repository root:

python3 rounds/l4-context-packet/reproduce/l4-context-packet.py

The script uses only the standard library and checks every reported outcome number against rounds/l4-context-packet/data/paired-cells.csv and rounds/l4-context-packet/data/historical-baselines.csv.

Raw records: these two CSV files contain the latest official aggregate row for each exact scored cell ID and the registered historical baseline cell IDs. The token total per cell is uncached input + cached input + output; arm totals sum all cell totals.

Recomputed from raw records

`recomputed.json`, mined cell records, and the token audit back observed official outcomes, per-arm pass counts/rates, manifest token counters, and wall medians. The source contains 6 rows (fail 1, pass 5); overall pass rate is 83.3% (5/6) among pass/fail grades. Per-arm detail and coverage counts are in the JSON.

No shared cell IDs are present in the committed public grade export; the source-only cohort is checked against the page's reported figures below. No shared public-export cell IDs are available for a paired outcome comparison. No complete published metadata/manifest token pairs are available for this round (0 missing manifest usage; 6 missing published metadata). Missing counters remain unknown, not zero. This round also has 6 raw-only or non-public rows; they are kept separate from the published-export denominator.

Round results · rounds/l4-context-packet/RESULTS.md

STATUS: DESCRIPTIVE (see the round README: post-result amendment, non-interleaved controls)

The primary comparison follows Amendment 1: packet and contemporaneous no-packet controls are paired by variant and seed. Each cell is the latest official aggregate grade row for its exact cell ID, from the MacBook grading route.

Paired outcomes

Source variant Seed Packet cell No-packet control Pair
r70-2-go 1 PASS 25/25 PASS 25/25 both pass
r70-2-go 2 PASS 25/25 FAIL 21/25 rescue
r70-2-go 3 PASS 25/25 PASS 25/25 both pass
r70-2-elixir 1 PASS 25/25 FAIL 21/25 rescue
r70-2-elixir 2 PASS 25/25 FAIL 21/25 rescue
r70-2-elixir 3 PASS 25/25 PASS 25/25 both pass
r70-4-elixir-fe2 1 FAIL 17/18 FAIL 17/18 both fail
r70-4-elixir-fe2 2 FAIL 17/18 FAIL 17/18 both fail
r70-4-elixir-fe2 3 FAIL 17/18 FAIL 17/18 both fail
r70-7-rust 1 PASS 24/24 FAIL 23/24 rescue
r70-7-rust 2 PASS 24/24 FAIL 23/24 rescue
r70-7-rust 3 PASS 24/24 FAIL 23/24 rescue

Pass counts, test counts, and tokens

A full pass is an official pass outcome. Mean tests_passed is the arithmetic mean across the three cells in that variant and arm. One cell's total tokens are uncached input + cached input + output; total is the sum. Tokens per pass is all tokens consumed by every cell in that arm divided by its full-pass count, so failed-cell tokens remain in the numerator.

Source variant Arm Successful cells Mean tests_passed Total tokens Tokens per successful cell
r70-2-go packet 3/3 25.00 1,922,619 640,873
r70-2-go control 2/3 23.67 1,320,610 660,305
r70-2-elixir packet 3/3 25.00 24,525,211 8,175,070.3
r70-2-elixir control 1/3 22.33 27,080,262 27,080,262
r70-4-elixir-fe2 packet 0/3 17.00 3,043,793 — (0 passes)
r70-4-elixir-fe2 control 0/3 17.00 3,256,526 — (0 passes)
r70-7-rust packet 3/3 24.00 1,053,801 351,267
r70-7-rust control 0/3 23.00 928,201 — (0 passes)
All variants packet 9/12 22.75 30,545,424 3,393,936
All variants control 3/12 21.50 32,585,599 10,861,866.3

Historical baselines (context only)

These are the registered historical Luna-max cells. Amendment 1 makes the contemporaneous paired controls the decision basis; the historical baseline is shown only as context.

Source variant Historical successes Eligible cells
r70-2-go 3/4 4
r70-2-elixir 2/4 4
r70-4-elixir-fe2 0/5 5
r70-7-rust 6/7 7

Decision and protocol notes

The net packet-minus-control change is +6 full passes. There are 6 paired rescues and 0 paired losses. Packet pass counts are no lower than controls on 4/4 variants. The regression rule did not fire; the worst mean tests_passed comparison is 17.00 packet versus 17.00 control on r70-4-elixir-fe2 (difference 0.00).

Verdict: KEEP packets in specification sections 3.2 and 4.9, as a descriptive pilot. The result is based on four outcome-selected variants and three cells per arm per variant; the rule thresholds are coarse and support no general claim.

The EU controls followed the seeded order recorded for the lane. One EU control was launched while a grade-window drain was in progress; this delayed the window only and affected no result. The three US controls ran after their packet cells had finished and were not interleaved.

A separate confirmation round, l4b-confirm, is running on three new variants; it is not included here.

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
rounds/l4-context-packet/reproduce/l4-context-packet.py
python3 rounds/l4-context-packet/reproduce/l4-context-packet.py
Inputs
Full execution replay
Unavailable
Scope
The public aggregate CSVs support analysis replay only; benchmark execution and official grading are not replayed.

View the complete per-round reproduction inventory · Repository reproduction guide