The research behind Kogen

Kogen Bench

What helps a coding agent finish software correctly, and at what cost? Explore the experiments shaping Kogen, from model choices and plans to checks, recovery and hill-climbing.

What we are testing

Kogen is being built at Optimum Tech to carry a shaped software feature through implementation, checks, review and rework. Bench records the experiments that inform that work.

  • Models and reasoning effort. Which combinations finish the task correctly, how long they take, and what they cost.
  • Plans and context. How much detail a builder needs, and whether a prepared packet of relevant project information helps it work.
  • Agent design. The prompts, tools and execution loops around a model, including comparisons involving Kogen, Codex and Grok Build. Each record states what actually ran.
  • Checks and recovery. How we grade working software, where review helps or gets in the way, and what to do when a build fails.
  • Languages and implementation choices. Experiments across Rust, Go, TypeScript and Elixir, with task and environment differences kept visible.

Hill-climbing, with a record of the steps

Hill-climbing means making a change, measuring what happens, and using that result to choose the next experiment. We have been testing plan detail, prepared context, builder strength and other choices in the software-building process.

A promising result is a reason to test again. Our context-packet pilot supported keeping prepared context on its selected tasks; the independent replication did not confirm an advantage on its new US tasks. Both belong in the record.

The archive also includes failed-build recovery experiments and cache-key checks. Further work will test promising changes on fresh tasks and improve the rerun kit. New results will be added after their evidence is reviewed; this is a research direction, not a release schedule.

How to read the recordEach experiment explains what ran, what happened and what the result can support. Small pilots, incomplete records and comparisons made under different conditions keep their limits visible.
Publication evidence status
  • B1 · resolved · Official-export verification of captured grade IDs: All 5,020 captured graded IDs now match unique rows in the refreshed public official grade export by exact cell ID. The 128 post-cut rows were appended from the authoritative ledger; 25 invalid control_apply rows are controls, not model failures. Evidence source.
  • B2 · resolved · Pinned Kogen specification snapshot and clause anchors: The locally sanitized v1.2 snapshot preserves cited content from original revision 1118f7fc0dbb042af8c8de2ffd1b85768cf9e2a0 and uses public-release sanitization revision 30d9b25e4eefa7117121de44298fed504b429f4e. Its published-byte SHA-256 manifest verifies, and all 134 pinned EVIDENCE-MAP fragments resolve locally. The decision ledger is absent, so comprehensive decision-to-clause coverage remains limited. Evidence source.
Read the publication validation report

Choose a path

Start with the findings, then follow a question to its experiment and evidence.

For agents and researchers

Static, same-origin files. No account, server query, or JavaScript is required to read the record.

Source revision baf7c2f4307d3a989a8fb9363df8000fda3d9310