> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [reproduce/kogen-context-failure-packet-screen-20261104/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/kogen-context-failure-packet-screen-20261104/README.md)
- [categories/tools-jev-and-review.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/tools-jev-and-review.md)
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)

# Context packet did not speed up four tasks

- **Finding id:** `kogen-context-failure-packet-screen-20261104`
- **Verdict:** Unresolved
- **Finished:** 4 November 2026
- **Comparison:** [Kogen Context component](/comparisons/kogen-context/)
- **Reproduction pack:** [reproduce/kogen-context-failure-packet-screen-20261104/](https://github.com/KogenAI/kogen-bench/tree/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/kogen-context-failure-packet-screen-20261104) (data, analysis script and rerun steps)
- **Category page:** [categories/tools-jev-and-review.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/tools-jev-and-review.md)

## In plain terms

In a matched screen of four public coding tasks, two repetitions each, every cell passed in both arms. Agents given an automatic Kogen Context packet were not faster (median 46.0 s against 43.6 s) and used 23% more uncached input tokens. On both Rust tasks the packet missed the file the agent edited, and an earlier speedup on one of them did not replicate. Unresolved.

Verdict key: **Yes**, The result supports the change or claim that was tested. **No**, The result does not support it, so nothing changes. **Unresolved**, The evidence is not strong enough either way. **Superseded**, A later test replaced this result.

## Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

**Question.** When a coding agent starts from a failing check, does an automatic Kogen Context source packet help it finish the task correctly, sooner or with fewer tokens than the same agent without the packet?

**Result: unresolved.** Every cell passed its hidden check in both arms. On the four decision tasks the packet arm was not faster: median model time was 43.6 s without the packet and 46.0 s with it (+5.6%). It also used more uncached input tokens: 99,066 against 122,166 in total (+23.3%). On both Rust tasks the packet did not name the file the agent finally edited. The earlier single-host speedup on ripgrep 2208 did not replicate here: 39.3 s and 31.3 s without the packet, against 45.2 s and 46.8 s with it. The registered rule (PROMISING needs outcomes at least equal plus 15% faster or cheaper; CLEAR LOSS needs the packet arm red where the plain arm is green in both repetitions) gives UNRESOLVED. Nothing here supports a general speed, cost or localization claim.

## Setup
- Tasks: four public SWE-bench Multilingual cases (Rust: ripgrep 2208 and ripgrep 2574; Go: a gin header-binding feature; JavaScript: an axios timeout case), plus one calibration case that is reported but kept out of the decision. Each task's hidden check was confirmed red on the parent commit and green on the reference fix on the same machine before any cell.
- Arms, matched in everything else: the same model (claude-sonnet-5-5, medium effort), client version, issue text, bounded initial failure output (4,000 bytes), tool allow-list, time cap (300 s per attempt) and one bounded follow-up after a red check. The packet arm also received the packet from a frozen Kogen Context research build (failure-evidence mode, 6,500-byte and 6-range caps, 1 s deadline). Packets were 3.1 to 5.5 KB and took under 0.31 s.
- Size: 5 tasks x 2 arms x 2 repetitions = 20 cells on one machine, run one at a time. The arm that went first was swapped in repetition 2. Only claude-sonnet-5-5 requests were seen, and list-price cost was $2.23.

## Results (model seconds and uncached input tokens per cell)

| task | rep | no packet | packet | packet named the edited file |
| --- | --- | --- | --- | --- |
| ripgrep 2208 | 1 | pass, 39.3 s, 16,288 | pass, 45.2 s, 16,669 | no |
| ripgrep 2208 | 2 | pass, 31.3 s, 13,867 | pass, 46.8 s, 17,611 | no |
| ripgrep 2574 | 1 | pass, 53.8 s, 16,786 | pass, 40.2 s, 17,383 | no |
| ripgrep 2574 | 2 | pass, 47.8 s, 14,965 | pass, 39.9 s, 18,009 | no |
| gin header binding | 1 | pass, 30.7 s, 10,380 | pass, 29.5 s, 10,973 | 1 of 4 files |
| gin header binding | 2 | pass, 29.4 s, 8,930 | pass after one follow-up, 59.5 s, 22,818 | 1 of 5 files |
| axios timeout | 1 | pass, 156.1 s, 10,377 | pass, 146.1 s, 9,273 | yes |
| axios timeout | 2 | pass, 146.8 s, 7,473 | pass, 143.8 s, 9,430 | yes |
| calibration (not in decision) | 1 | pass, 25.8 s, 24,559 | pass, 28.6 s, 28,185 | yes |
| calibration (not in decision) | 2 | pass, 27.2 s, 25,335 | pass, 19.9 s, 26,529 | yes |

## Limits
- Small sample: 4 decision tasks, 2 repetitions each, one machine. Single-cell timing differences of this size flip between repetitions (ripgrep 2574 and gin both did).
- Every task passed in both arms, so this screen cannot show an outcome effect. The tasks were not hard enough to separate the arms.
- The packet came from a frozen research build, not the current product binary.

## Rerun
From the published data: `python3 report.py cells.json` prints the table, the medians, the token totals and the registered verdict. `cells.json` has one record per cell: outcome, attempts, model, grade and packet seconds, token counts, list cost, models seen, packet bytes and packet recall. A fresh run needs the four public SWE-bench Multilingual instances above at their parent commits, each instance's own test patch as the hidden check, and Claude Code running both arms with the settings in Setup.
