The research behind Kogen
Kogen Bench
What helps a coding agent finish software correctly, and at what cost? Explore the experiments shaping Kogen, from model choices and plans to checks, recovery and hill-climbing.
What we are testing
Kogen is being built at Optimum Tech to carry a shaped software feature through implementation, checks, review and rework. Bench records the experiments that inform that work.
- Models and reasoning effort. Which combinations finish the task correctly, how long they take, and what they cost.
- Plans and context. How much detail a builder needs, and whether a prepared packet of relevant project information helps it work.
- Agent design. The prompts, tools and execution loops around a model, including comparisons involving Kogen, Codex and Grok Build. Each record states what actually ran.
- Checks and recovery. How we grade working software, where review helps or gets in the way, and what to do when a build fails.
- Languages and implementation choices. Experiments across Rust, Go, TypeScript and Elixir, with task and environment differences kept visible.
Hill-climbing, with a record of the steps
Hill-climbing means making a change, measuring what happens, and using that result to choose the next experiment. We have been testing plan detail, prepared context, builder strength and other choices in the software-building process.
A promising result is a reason to test again. Our context-packet pilot supported keeping prepared context on its selected tasks; the independent replication did not confirm an advantage on its new US tasks. Both belong in the record.
The archive also includes failed-build recovery experiments and cache-key checks. Further work will test promising changes on fresh tasks and improve the rerun kit. New results will be added after their evidence is reviewed; this is a research direction, not a release schedule.
- B1 · resolved · Official-export verification of captured grade IDs: All 5,020 captured graded IDs now match unique rows in the refreshed public official grade export by exact cell ID. The 128 post-cut rows were appended from the authoritative ledger; 25 invalid control_apply rows are controls, not model failures. Evidence source.
- B2 · resolved · Pinned Kogen specification snapshot and clause anchors: The locally sanitized v1.2 snapshot preserves cited content from original revision 1118f7fc0dbb042af8c8de2ffd1b85768cf9e2a0 and uses public-release sanitization revision 30d9b25e4eefa7117121de44298fed504b429f4e. Its published-byte SHA-256 manifest verifies, and all 134 pinned EVIDENCE-MAP fragments resolve locally. The decision ledger is absent, so comprehensive decision-to-clause coverage remains limited. Evidence source.
Choose a path
Start with the findings, then follow a question to its experiment and evidence.
For agents and researchers
Static, same-origin files. No account, server query, or JavaScript is required to read the record.
Compact guide and status rules/index.json
Hypotheses, rounds, claims, and their URLs/llms-full.txt
Bounded Markdown guide/snapshot.json
Source commit and file hashes
Source revision baf7c2f4307d3a989a8fb9363df8000fda3d9310