> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/README.md)
- [site/content/research.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/site/content/research.md)
- [results/publication-blockers.json](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/publication-blockers.json)
- [results/publication-validation.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/publication-validation.md)
- [results/GRADE-JOIN.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/GRADE-JOIN.md)

# Kogen Bench

The research behind Kogen. What helps a coding agent finish software correctly, and at what cost? Explore experiments on models, plans, context, checks and recovery.

## What we are testing

Kogen is being built at [Optimum Tech](https://optimum.ba) to carry a shaped software feature through implementation, checks, review and rework. Bench records the experiments that inform that work.

- **Models and reasoning effort.** Which combinations finish the task correctly, how long they take, and what they cost.
- **Plans and context.** How much detail a builder needs, and whether a prepared packet of relevant project information helps it work.
- **Agent design.** The prompts, tools and execution loops around a model, including comparisons involving Kogen, Codex and Grok Build. Each record states what actually ran.
- **Checks and recovery.** How we grade working software, where review helps or gets in the way, and what to do when a build fails.
- **Languages and implementation choices.** Experiments across Rust, Go, TypeScript and Elixir, with task and environment differences kept visible.

## Hill-climbing, with a record of the steps

Hill-climbing means making a change, measuring what happens, and using that result to choose the next experiment. We have been testing plan detail, prepared context, builder strength and other choices in the software-building process.

A promising result is a reason to test again. Our [context-packet pilot](/rounds/l4-context-packet/) supported keeping prepared context on its selected tasks; the [independent replication](/rounds/l4b-confirm/) did not confirm an advantage on its new US tasks. Both belong in the record.

The archive also includes [failed-build recovery experiments](/rounds/l3b-repair-vs-continue/) and [cache-key checks](/rounds/cache-key-api-v4/). Further work will test promising changes on fresh tasks and improve the rerun kit. New results will be added after their evidence is reviewed; this is a research direction, not a release schedule.


## Start with a question

- [Read findings](/findings/)
- [Language comparison](/languages/)
- [Browse hypotheses](/hypotheses/)
- [Find an experiment](/rounds/)
- [Trace a specification clause to evidence](/evidence/)

## Read results with their status

The findings pages summarize the source record. A status such as INTERIM, CONFOUNDED, WITHDRAWN, or NOT-RUN limits what can be concluded. A missing value stays missing. Claim ledger rows retain their own evidence status and exact cohort; the site does not turn a source-reported number into a verified result by displaying it.

## For agents

- [Machine-readable index](/index.json)
- [Agent guide](/llms.txt)
- [Full bounded guide](/llms-full.txt)
- [Snapshot and source hashes](/snapshot.json)

## Publication status

Publication status is **ready**. B1 is resolved: All 5,020 captured graded IDs now match unique rows in the refreshed public official grade export by exact cell ID. The 128 post-cut rows were appended from the authoritative ledger; 25 invalid control_apply rows are controls, not model failures. B2 is resolved: The locally sanitized v1.2 snapshot preserves cited content from original revision 1118f7fc0dbb042af8c8de2ffd1b85768cf9e2a0 and uses public-release sanitization revision 30d9b25e4eefa7117121de44298fed504b429f4e. Its published-byte SHA-256 manifest verifies, and all 134 pinned EVIDENCE-MAP fragments resolve locally. The decision ledger is absent, so comprehensive decision-to-clause coverage remains limited. See [the grade-join scope](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/GRADE-JOIN.md) and [publication validation](/publication-validation/).
