Published result
Context packet shows no time-to-patch gain in a screen
A plain-language summary comes first, followed by the full text of the result's reproduction pack.
Result summary and pack text · reproduce/kontext-matched-2-verified-patch-time-20261105/README.md
- Finding id:
kontext-matched-2-verified-patch-time-20261105 - Verdict: Unresolved
- Finished: 5 November 2026
- Comparison: Kogen Context component
- Reproduction pack: reproduce/kontext-matched-2-verified-patch-time-20261105/ (data, analysis script and rerun steps)
- Category page: categories/harness-levers.md
In plain terms
In 10 matched pairs on five real tasks (three in Zig, two in Rust), the agent produced a patch that passed the hidden check in every run with and without the Kogen Context packet. The median time ratio was 1.02 on Zig and 1.00 on Rust, so the claimed shortening did not appear. Most runs ended at the 300 second cap with the fix already made, so the times mostly measure the cap. This is a screen with one model, not a speed claim.
Verdict key: Yes, The result supports the change or claim that was tested. No, The result does not support it, so nothing changes. Unresolved, The evidence is not strong enough either way. Superseded, A later test replaced this result.
Full text of the reproduction pack
The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).
Verdict: UNRESOLVED. Under the screen rule fixed before the first cell, Kogen Context neither shortened nor lengthened the time to an independently verified patch. All 10 matched pairs produced a verified patch in both arms, and the stack medians of the per-case time ratio were 1.02 (Zig, Crane) and 1.00 (Rust), against the 0.85 the rule needs. The cells also cannot say much about speed: 13 of the 20 arm-runs ended at the 300 s attempt cap with the fix already on disk, because the agent was still running the repository's whole test suite. Their time is the cap plus grading, not the agent's speed.
Claim and rule
Claim under test: Kogen Context shortens the time from a real task to an independently verified patch, on Crane (Zig) and on one other stack (Rust).
- Arm A
direct: Claude Code 2.1.318, claude-sonnet-5-5, medium effort, the issue text and a bounded failing-test output, normal tools. - Arm B
assisted: arm A plus the deterministic packet fromkogen-context evidence ... --deadline 1s --bytes 6500 --ranges 6. Same model and effort in both arms. 300 s per attempt, one follow-up allowed. - Time to verified patch T = the harness row's
total_s(arm start to final grade: packet retrieval, every attempt, every grade) when the hidden check passes (outcomegreenorgreen_after_followup); otherwise T is censored at 1200 s. The hidden check is run by the harness, never by the agent. Timings include failed attempts and grading. - Per case: ratio = median T(B) / median T(A) over two repetitions (the second with
--rotate 1, which changes the arm order). - CLEAR LOSS: two or more cases where B is red in both repetitions and A green in both; or B verified patches at least 2 below A; or median ratio >= 1.15 with B verified <= A verified. PROMISING: B verified >= A verified, no such loss case, and on Crane and on Rust separately median ratio <= 0.85 with B verified >= A verified. Otherwise UNRESOLVED.
Result
| case | stack | verified A / B | median T A (s) | median T B (s) | ratio B/A | packet (bytes) |
|---|---|---|---|---|---|---|
| crane-xmlpe | Zig | 2 / 2 | 213.0 | 231.8 | 1.088 | 5,531 |
| crane-cssesc | Zig | 2 / 2 | 377.2 | 237.9 | 0.631 | 4,793 |
| crane-cssshape | Zig | 2 / 2 | 324.4 | 331.8 | 1.023 | 6,381 |
| bat2201 | Rust | 2 / 2 | 306.6 | 306.2 | 0.999 | 3,682 |
| bat2650 | Rust | 2 / 2 | 307.4 | 307.8 | 1.001 | 3,558 |
Stack medians of the per-case ratio: Zig 1.023 (verified 6 / 6), Rust 1.000 (verified 4 / 4); all cases 1.001. Matched pairs 10, dropped 0.
Per pair (T in seconds, A then B; "cap" = Claude hit the 300 s limit with no result event):
| pair | T A | T B | A | B |
|---|---|---|---|---|
| xmlpe r1 | 230.5 | 232.6 | finished | finished |
| xmlpe r2 | 195.5 | 231.0 | finished | finished |
| cssesc r1 | 385.8 | 106.3 | cap | finished at 30 s, no test run |
| cssesc r2 | 368.5 | 369.6 | cap | cap |
| cssshape r1 | 276.9 | 299.7 | finished | finished |
| cssshape r2 | 371.8 | 363.8 | cap | cap |
| bat2201 r1 | 307.6 | 305.8 | cap | cap |
| bat2201 r2 | 305.6 | 306.7 | cap | cap |
| bat2650 r1 | 308.4 | 308.5 | cap | cap |
| bat2650 r2 | 306.4 | 307.0 | cap | cap |
What the transcripts show (model-free):
- In every one of the 13 capped arm-runs the first edit to the fixed file was made within about 35 s, and the harness's hidden check passed on the final tree. The time then ran to the cap because the agent had started
cargo testorzig build test -Dspec=cssover the whole suite and that command was still running when the process was stopped. Time to verified patch therefore measures that choice plus the cap in 13 of 20 runs. - Both arms located and edited the reference file in every Crane run (and in the Rust runs on bat2650). On bat2201 the four runs per arm edited different files (A:
app.rsin r1 andclap_app.rsin r2; B:app.rsplusclap_app.rsin r1 andapp.rsin r2), the reference changesclap_app.rs, and every one of those fixes passed the hidden check. No pair shows the packet changing where the agent looked in a way that changed its time. - The only large gap is cssesc r1 (385.8 s vs 106.3 s): both arms edited the same file; B never started the long test run and stopped at 30 s, A did and hit the cap. That is variation in the agent's test habits with the same localization, so it is not a packet effect; it alone produces the 0.631 ratio.
- Uncapped pairs (xmlpe r1, r2; cssshape r1) differ by +1% to +18% (B slower) with the same edited file.
- Tokens (input + cache creation + cache read + output) exist only for the 7 arm-runs that wrote a result event: A 363.7k median on xmlpe, 594.4k on cssshape; B 343.7k and 502.6k; cssesc B r1 266.1k. They are descriptive only.
- Packets were valid and non-empty in all 10 assisted rows (bytes in the table).
Custody
Every unit ran Claude Code 2.1.318 with claude-sonnet-5-5 only (all 305 assistant messages and all 20 session inits report that model), the frozen binary kogen-context with SHA-256 59ee239c2abf96b1737cc809063136fd3334df368ff0637a8b1e6aac7b69f747, grader scripts/qualify-m4.py (SHA-256 373c98bb32619435fe481aa7ec9db4de1e8ca7ef4d784355432811b1cf72f066) and harness scripts/build_bench-confirmation.py (SHA-256 52d805f8800476e5a17621c4fb1c5c093e2a6d71847018d155a1c0ead93211b6). Zig 0.16.0 for the Crane cases. scripts/report_m4.py (SHA-256 3529e41a54f1bf99d7d3534873382a68d1d49fcf72810f89b719e0018cfa1028) produced data/report.json. The case builder used for the screen had SHA-256 270d27ad6343bd33f5454236d6c62ea0e74cf8c3c994ee886a3a5c6a91617197; scripts/mk-crane-case.sh differs from it only in taking its work directory from $WORK. The harness file's hash was checked on the run host after the screen; data/custody.csv lists the Claude, model, packet-tool and grader hashes recorded for each unit.
Excluded infrastructure runs: 7 earlier units were refused by the Claude client with a weekly usage limit in at least one arm; they are not results and are not in the data. No unit was ungraded and no unit needed an infrastructure rerun.
Cases (public revisions)
| case | repository | base | fix | hidden check |
|---|---|---|---|---|
| crane-xmlpe | github.com/zig-whatwg/crane | 70bb3b6487ce | 60e2f035f720 | tests/xml/xm1_entity_declaration_test.zig (5 tests) |
| crane-cssesc | github.com/zig-whatwg/crane | c6c132d18191 | 3b3dadf622f2 | tests/css/om1_name_eql_escapes_test.zig (8 tests) |
| crane-cssshape | github.com/zig-whatwg/crane | c6c1522b8fd1 | 5b0343a8df53 | tests/css/ci1_shape_scalar_extended_units_test.zig (1 test) |
| bat2201 | SWE-bench Multilingual sharkdp__bat-2201 (sharkdp/bat) |
instance base commit | instance patch | the instance's target test |
| bat2650 | SWE-bench Multilingual sharkdp__bat-2650 (sharkdp/bat) |
instance base commit | instance patch | the instance's target test |
The base of each Crane case is the parent of its fix commit. The hidden check is the fix's own new test file, run with zig build test -Dspec=<dir> -Dtest-file=<file> --summary all; it calls only functions already present on the base, so it checks behaviour, not guessed names. Selection rule: among fix commits since 2026-09-19, a fix whose tests are a separate new file calling only APIs present on the base, buildable without V8. briefs/ holds the symptom-only issue texts for the three Crane cases (written by us from the public commit and its tests after seeing the fix; they name no file, function or module). Bat cases use the SWE-bench issue text. Case kits and patches are not shipped.
The published scripts/qualify-m4.py and scripts/mk-crane-case.sh name the fix's source diff fix.patch; the run used the same code with that file named differently, which is the only change. The executed qualify-m4.py had SHA-256 373c98bb32619435fe481aa7ec9db4de1e8ca7ef4d784355432811b1cf72f06 (the custody hash above); the published copy has 4ca70efb6c2485bf02a254f4490afc4500c48fe2dca6b8c453c0d135aed5b6e5. The diff comes from the public Crane fix commit named in the table.
Rerun
- Tools: Zig 0.16.0 from ziglang.org; Claude Code 2.1.318 (
npm i -g @anthropic-ai/[email protected]) logged in, orANTHROPIC_API_KEYset in the environment; Rust stable with cargo; Python 3. The packet tool is the public source at github.com/KogenAI/kogen-context (it has no release tag; build theevidencesubcommand version and compareshasum -a 256of the binary with the hash above). - Crane cases:
git clone --bare https://github.com/zig-whatwg/crane $WORK/crane.git, then for each row of the table:WORK=$WORK bash scripts/mk-crane-case.sh <case> <fix sha> <test file>. This writes$WORK/kit/cases/<case>/{base,test.patch,fix.patch,origin}; add the brief frombriefs/<case>.txt. Bat cases: takesharkdp__bat-2201andsharkdp__bat-2650from the SWE-bench Multilingual dataset (base commit, test patch, instance patch). - Red/green controls:
python3 -I scripts/qualify-m4.py <case> base --out <dir>must be red and... ref --out <dir>green before any model run. - Screen, per case and repetition (rep 2 adds
--rotate 1):python3 -I scripts/build_bench-confirmation.py --cases <case> --context <kogen-context binary> --context-sha256 59ee239c2abf96b1737cc809063136fd3334df368ff0637a8b1e6aac7b69f747 --qualify scripts/qualify-m4.py --zig-dir <zig 0.16.0 dir> --summary-name result.json --diag-name diagnostic.txt --red <case>=<red receipt dir> --green <case>=<green receipt dir> --out <runs>/kxm4-screen-<case>-r<rep>(--preflightchecks packets without a model.) Use shared Zig caches for both arms and the grader so neither pays a cold build. - Report: copy each
summary.jsoninto one directory per unit namedkxm4-screen-<case>-r<rep>/and runpython3 -I scripts/report_m4.py <that directory> --out report.json. To check this pack without any model, rebuild the layout fromdata/units/<case>-r<rep>.json(they are the harness summaries with host paths removed) and run the same report; its verdict line isUNRESOLVED.
Files
data/units/*.json: harness rows per unit (host paths and account names removed).data/rows.csv: one line per arm-run.data/custody.csv: versions and hashes per unit.data/per_case.csv: the case table.data/report.json: the report.scripts/: our harness, grader, case builder and report script.briefs/: Crane issue texts.
Limits
Screen only: two repetitions on five cases, one model, one effort. No general speed claim. The 300 s cap and the agent's habit of running the whole test suite dominate most times, so a rerun with a different cap or a test-scoped instruction could give different times. Matched-1 (five cases, every row green in both arms) and the earlier pre-screen are separate and are not pooled here.