> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [reproduce/npm-preflight-cache/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/npm-preflight-cache/README.md)
- [categories/head-to-head.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/head-to-head.md)
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)

# Test harness change: install dependencies once, retry timed-out approvals

- **Finding id:** `npm-preflight-cache`
- **Verdict:** Yes
- **Finished:** 30 October 2026
- **Comparison:** [Test setup, rules and sandboxing](/comparisons/test-setup-and-sandboxing/)
- **Reproduction pack:** [reproduce/npm-preflight-cache/](https://github.com/KogenAI/kogen-bench/tree/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/npm-preflight-cache) (data, analysis script and rerun steps)
- **Category page:** [categories/head-to-head.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/head-to-head.md)

## In plain terms

A change to the test harness, not a model result; no model was called. It installs a task's locked package dependencies once and restores a private copy for each run, and it defers (instead of failing) a run whose approval step hit its 120-second deadline. It does not fix a stalled snapshot step: the patched harness still refuses at the deadline and only retries. The page lists the exact patches and a model-free proof.

Verdict key: **Yes**, The result supports the change or claim that was tested. **No**, The result does not support it, so nothing changes. **Unresolved**, The evidence is not strong enough either way. **Superseded**, A later test replaced this result.

## Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

A methods change to the run harness, not a model result. Two changes, deployed on both Mac runners on 30 October 2026:

1. **Stage locked npm dependencies once** (`patches/npm-stage-once.diff`). A task's `npm ci --ignore-scripts --no-audit --no-fund` setup step used to run again inside Kogen's 120-second approval deadline for every preflight and every run. A stalled install then used up the deadline and refused the whole unit. The harness now installs once per lockfile into an external cache (keyed on the lockfile and package inputs, the command, npm and node versions, OS and architecture), and each preflight/run restores a private copy. A missing cache or changed inputs fail closed; npm is never called inside the deadline. Task kits, task inputs and plan hashes are untouched.
2. **Defer timed-out approvals instead of failing the unit** (`patches/defer-approval-timeouts.diff`, applied on top of the first). A deadline that Kogen actually hit defers the whole task (all lanes and repetitions) behind the ordinary runs, retries it once through the normal host gate with the same 120-second deadline, and otherwise leaves its cells in the plan as an infrastructure gap (dispatch exit 5, incomplete report) and writes a request that the next unit drains. A real refusal is still fatal. Kogen, its approval and snapshot behaviour, and all deadlines are unchanged.

What the change does **not** fix: a slow snapshot step. With a stalled `git add -A -- .` the patched harness still refuses at the deadline (126.756 s); the harness only defers and retries it. See [`reproduce/kogen-approve-deadline-load/`](/findings/kogen-approve-deadline-load/) for the goose-labels-go measurements.

## Tools

No model is called. Real, pinned Kogen builds do the approvals (one pinned artifact per lane, hashes recorded before and after); the run executor in the deferral proof is the model-free `lanes/tests/fixtures/fake-lanes`. Task kits: `task-tags-go`, `bytefmt-go`, `goose-labels-go` (see [tasks.txt](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/npm-preflight-cache/tasks.txt); directories under `tasks/`). Host: two Macs (aliases mac-a and mac-b), Python 3.11+ and the npm and node versions recorded in each cache manifest.

## What is shipped

- `patches/npm-stage-once.diff`, `patches/defer-approval-timeouts.diff`: the `lanes/` changes (new `lanes/lanes/node_modules.py`, `lanes/bin/lanes-stage`, `lanes/lanes/approval.py`, `lanes/lanes/carry.py`, `lanes/bin/lanes-carry`, edits to `cell.py`, `preflight.py`, `dispatch.py`, `report.py`, and the new tests). Both apply in order to the `lanes/` tree as it stood before commit `cd24859c` (its parent `cd24859c^`, the published `7a0e22ad` tree); `git apply --check` passes there for both, in order (checked 31 October 2026). The current `lanes/` already contains their result and more (the `lanes/releases/` diff and later files), so they do not apply to it and need no applying. The host operations scripts that route new launches to the release (`run-queued`, `cc-prep`, `cg-prep`, the two day runners) are in [`lanes/ops/`](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/lanes/ops/), with the host paths read from `LANES_*` variables; `LANES_NPMFIX_HARNESS` points `cc-prep.sh` and `cg-prep.sh` at the patched harness.
- `faulttests/`: the four fault-test drivers as run (provenance copies with the paths turned into environment variables; they need the pinned Kogen artifacts and the private task kits and run directories, so they are not runnable standalone): `npm-stall-preflight.py` (npm stalls for 600 s, old and new harness), `git-add-stall-preflight.py` (only `git add -A -- .` stalls), `approval-deferral-fault.py` (setup stalls 125 s for one lane) and `completed-resume.py` (a completed task is not re-approved).
- `runs.csv`: one row per approval check: change, host alias, harness (patched or unpatched), fault, task, lane, stage, outcome, passed (= approved), seconds, deadline, npm calls observed, timed out. `tests.csv`: one row per test run. No logs, paths or hostnames.
- `analyze.py`, `expected-output.txt`.

## Rerun

Check out the tree the patches were made against, apply them and run the lane test suite offline (no model call):

```sh
git worktree add ../kogen-bench-pre cd24859c^ && cd ../kogen-bench-pre && git apply --check "$OLDPWD/reproduce/npm-preflight-cache/patches/npm-stage-once.diff" && git apply "$OLDPWD/reproduce/npm-preflight-cache/patches/npm-stage-once.diff" && git apply --check "$OLDPWD/reproduce/npm-preflight-cache/patches/defer-approval-timeouts.diff" && git apply "$OLDPWD/reproduce/npm-preflight-cache/patches/defer-approval-timeouts.diff" && lanes/test.sh
```

The `apply --check` steps and the two applications were run on 31 October 2026 and all succeed; `lanes/test.sh` was not run there. In the patched tree the new files `node_modules.py`, `carry.py`, the two `bin/` scripts and the new tests are byte-identical to the current `lanes/`; `cell.py`, `dispatch.py`, `preflight.py`, `report.py` and `approval.py` differ because later changes in `cd24859c` built on them.

The fault tests need a lanes run directory with a pinned Kogen artifact for the task. With `FAULT_ROOT`, `LANES_PRIVATE`, `NPM_STAGE_CACHE` and `OLD_LANES_RELEASE` set (see the drivers' headers), the npm stall test is

```sh
python3 reproduce/npm-preflight-cache/faulttests/npm-stall-preflight.py old && python3 reproduce/npm-preflight-cache/faulttests/npm-stall-preflight.py new
```

(`old` must refuse at the deadline with npm calls recorded; `new` must approve with zero npm calls), and the deferral test is `python3 reproduce/npm-preflight-cache/faulttests/approval-deferral-fault.py`, which asserts its own expectations and prints `FAULT PROOF PASSED`. Run these as the unprivileged benchmark user; use the default timeouts; each takes a few minutes. The unit tests in the original runs passed on both hosts (66 and 95 tests); they were **not** rerun against this repository's `lanes/` tree for this pack, only the patch application was checked.

## Reproduce the numbers

```sh
python3 -I reproduce/npm-preflight-cache/analyze.py | diff - reproduce/npm-preflight-cache/expected-output.txt
```

Verified 30 October 2026: this reproduces every published time, count and test total exactly.

## Expected results

Preflight with npm stalled for 600 s (deadline 120 s):

| Harness | Host | Task | Seconds | npm calls | Result |
|---|---|---|---:|---:|---|
| patched | mac-a | task-tags-go | 54.485 | 0 | approved |
| patched | mac-a | bytefmt-go | 14.609 | 0 | approved |
| patched | mac-b | task-tags-go | 36.191 | 0 | approved |
| patched | mac-b | bytefmt-go | 5.886 | 0 | approved |
| unpatched | mac-a | task-tags-go | 192.361 | 1 | refused at setup-1 (npm ci) |
| unpatched | mac-a | bytefmt-go | 120.431 | 2 | refused at setup-1 (npm ci) |

Total seconds include workspace preparation and cleanup; the approval deadline is still 120 s. Normal cached goose-labels-go preflight: 23.659 s, approved, 0 npm calls. Stalled `git add -A -- .`: refused after 126.756 s, `approve refused (timed out): no output`. The literal historical `status 143` text was not reproduced, because process-group cancellation suppressed it in that run.

Deferral proof (setup stalled 125 s for `task-tags-go`, lane a0, real Kogen approvals, model-free executor): bytefmt-go approved at 8.665 s and 7.775 s; task-tags-go a0 timed out at 132.848 s while its other lane was approved (53.135 s) so the whole task was deferred; after the two ordinary cells ran, a0 timed out again (128.133 s); on a later invocation both lanes approved (44.420 s, 50.770 s) and all four planned cells started with none re-run. First dispatch exit 5, later exit 0, 2 deadline failures, 0 provider calls, total 483.566 s. The published table lists 8 of the 10 approval rows; the other two are bytefmt-go re-checks at the start of the later invocation (4.829 s, 5.034 s; the final source skips approvals for completed tasks, proved separately by `completed-resume.py`: zero new approvals, zero new cells).

Tests: 66 passed on each host (87.923 s and 71.476 s) for the first change, 95 for the second (46.761 s; 41.751 s with one skip because `go` was not on the SSH `PATH`, passing at 0.525 s in the runner environment). Dependency parity: staged `node_modules` equal to a fresh install for task-tags-go, bytefmt-go and goose-labels-go on both hosts (2,510 entries each; file list, SHA-256, modes and symlink targets; lockfile hashes matched).

## Cost

No model calls, no API or subscription spend. The proofs used one machine for a few minutes each.

## Limits

Two hosts, two real task kits plus goose-labels-go, one fault at a time. The fault tests show the approval deadline is no longer consumed by npm and that a timed-out approval no longer loses a task; they do not show that approvals can never time out, and they do not locate the cause of the original goose-labels-go timeout.
