> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [reproduce/s3-audit-on-go-screen/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/s3-audit-on-go-screen/README.md)
- [categories/harness-levers.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/harness-levers.md)
- [site/content/finding-pages.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/finding-pages.json)

# Switching on Kogen's request audit lost four net tasks on 13 Go tasks, so the patch was dropped

- **Finding id:** `s3-audit-on-go-screen`
- **Verdict:** No
- **Finished:** 31 October 2026
- **Comparison:** [Kogen shaping and delivery-build speed](/comparisons/kogen-shaping-and-builds/)
- **Reproduction pack:** [reproduce/s3-audit-on-go-screen/](https://github.com/KogenAI/kogen-bench/tree/baf7c2f4307d3a989a8fb9363df8000fda3d9310/reproduce/s3-audit-on-go-screen) (data, analysis script and rerun steps)
- **Category page:** [categories/harness-levers.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/categories/harness-levers.md)

## In plain terms

A patch that runs Kogen's request audit before shaping was compared with unpatched Kogen, same commit and model (gpt-6.1-terra, medium effort), one trial per task. On 13 Go tasks the hidden checks passed on 11 tasks without the patch and 7 with it: no task gained and four were lost, against a keep rule of at least three net wins. Six of the 13 patched cells were refused at the shaping step, mostly because the shaped test lines were not in the required form.

Verdict key: **Yes**, The result supports the change or claim that was tested. **No**, The result does not support it, so nothing changes. **Unresolved**, The evidence is not strong enough either way. **Superseded**, A later test replaced this result.

## Full text of the reproduction pack

The text below is the pack's README as published in the repository (a system folder name may show without its leading slash, so that the site's privacy scan passes).

Switching on Kogen's request-audit step before shaping made things worse. **On 13 Go tasks unpatched Kogen passed the hidden checks on 11, the patched build on 7**: it gained no task and lost four, net -4, against a keep rule of at least +3. Verdict: **no** (patch dropped; Kogen stays as it is). Finished 31 October 2026.

## What was run

Same Kogen commit 3e759e88 (vendored Loop pin 8a2fcdae) in both arms, model gpt-6.1-terra at medium effort for shaping, building and review, request-only tasks (the request is the only input), one trial per arm per task, cells interleaved, one Linux laptop-class host.

| Arm | What differs | Patch sha256 |
| --- | --- | --- |
| b0 | nothing (empty patch) | none |
| s3 | the request audit runs before shaping (`patch/s3-audit-on.diff`) | 1dcd926000ff7aa055e58b7da8e166b83b1d274faca0df9bdee8422538d63f25 |

## Result

| Arm | Hidden pass | Wall s | Tokens | Requests |
| --- | --- | --- | --- | --- |
| b0 | 11/13 | 3158 | 2983814 | 197 |
| s3 | 7/13 | 2717 | 2336704 | 163 |

Wall, tokens and requests are summed per cell and **include cells that ended refused or build_blocked**. The patched arm is cheaper only because 6 of its 13 cells stopped at the shaping step (unpatched: 1). Pairs that differ: G01, G09, G10 and G12 pass unpatched and are refused with the patch. G08 is refused in both arms and G03 fails in both.

Why the patched arm refused (shaping stage, first-hand from the cell logs): three cells (G01, G10, G12) stopped because the shaper's `tests:` lines were not in the required `- <rule>: <test to add or update> (<layer>)` form; one (G09) because the request audit answer was not a valid payload ("a cell is malformed"); one (G03) because the Intent was not ready (a syntax error in the generated step file; the unpatched arm also failed that task); one (G08) because the shaper ended without calling a tool, as the unpatched arm did too.

## Limits

- One trial per arm per task, so a single task is within run-to-run variance; the verdict rests on the pre-set rule, which the result misses by seven.
- A repetition (same patch, same tasks) was queued after this one; if it finishes it will be published under its own id.
- Intent validity and acceptance separation are not scored here; the numbers are the hidden-check results only. The model has no listed price, so cost is token counts.
- Cold-cache laptop-class host, so wall times do not compare with other hosts.

## Reproduce the numbers

```sh
python3 -I reproduce/s3-audit-on-go-screen/report.py reproduce/s3-audit-on-go-screen/data s3-audit-on-go s3 | diff - reproduce/s3-audit-on-go-screen/expected-output.txt
```

`report.py` uses only the standard library. `data/s3-audit-on-go_cells.tsv` has one row per cell (G01 to G13 are the tasks in alphabetical order; names and hidden suites are not included).

## Rerun

The table reruns from `data/`. To run the cells again you need Kogen at 3e759e88 with `patch/s3-audit-on.diff` applied, the same 13 Go tasks and a ChatGPT login.
