> Data revision: baf7c2f4307d3a989a8fb9363df8000fda3d9310
> Source files:
- [hypotheses/f08-language-spec.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/hypotheses/f08-language-spec.md)
- [hypotheses/README.md](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/hypotheses/README.md)

# F08 — Language and executable specification

> Family source text is narrative from the repository. Any numeric value in that text remains source-reported unless an exact claim ledger entry supplies it.

# F08 Language and executable specification

This page records evidence for H91–H99. Language choice is a design decision informed by bounded observations; the available task cohorts do not establish a general language ranking.

## Hypotheses and current status

| ID | Falsifiable proposition | Current evidence status |
|---|---|---|
| H91 | A matched comparison can identify the best implementation language for the Kogen agent loop across task success, reliability, cost, and implementation time. | DESCRIPTIVE ONLY. Public cohorts are small and incomplete, and some original tasks have implementation confounds. |
| H92 | Rust has a better runtime and concurrency tradeoff than Elixir OTP when logic, prompts, and repair opportunities are equivalent. | NOT ESTABLISHED. The public comparison does not hold all of those factors constant. |
| H93 | Separating Mix applications into an umbrella makes tests and the workflow faster. | NOT ESTABLISHED. No before/after architecture comparison is identified. |
| H94 | Language-agnostic executable acceptance specifications preserve the same semantics across implementations. | PARITY METHOD, NOT EFFICACY. Conformance can test parity but does not show improved task completion. |
| H95 | Quint specifications written during planning improve Kogen or direct-model outcomes. | NOT ESTABLISHED. No controlled live comparison is identified. |
| H96 | Reviewing the translation from natural-language requirements into Quint catches omitted behavior. | NOT ESTABLISHED. No public translation-audit outcome is linked. |
| H97 | A fair language comparison holds tasks, prompts, model, effort, steering, grader, and repair loop constant. | METHOD REQUIREMENT. Existing language cohorts do not establish full parity. |
| H98 | Gleam should remain eligible until a defined matched-comparison rule supports excluding it. | NOT ESTABLISHED. No public inclusion-rule test or Gleam cohort is available. |
| H99 | Compile time, run wall time, token use, and task grade should all inform the language choice. | PARTIAL MEASUREMENT. Public timing and resource files exist, but their measurements are descriptive and do not by themselves rank full implementations. |

## Run and claim record

The direct language cohorts were built by Codex from task scaffolds. Kogen commit is therefore not applicable to those runs. The ledger currently files the official full-pass claims under H114 for reconciliation; those rows are cross-family evidence cited here without reassigning their hypothesis IDs.

| experiment_id | arm and task set | model / effort | Kogen and harness revision | host / epoch | planned / started / finished / graded / ITT | outcome | status and correction | claim ID | public receipt and raw records |
|---|---|---|---|---|---|---|---|---|---|
| r70-original-rve | Rust, Elixir, Go, TypeScript/Bun; tasks 1, 3, 4, and 6 | Direct Codex, gpt-6-luna, max | Kogen not applicable; exact harness Git commit not preserved | Rust and Elixir on kogen-bench-eu; Go and TypeScript/Bun on kogen-bench-us; provider epoch not balanced | 40 / 40 / 40 / 40; ITT unresolved because five additional deliveries lack joinable cell IDs | Rust 8/10, Elixir 2/10, Go 7/10, TypeScript/Bun 7/10. Claims: CL-WA-r70-rve-r70-original-rve-rust-full-pass CONFOUNDED; CL-WA-r70-rve-r70-original-rve-elixir-full-pass CONFOUNDED; CL-WA-r70-rve-r70-original-rve-go-full-pass CONFOUNDED; CL-WA-r70-rve-r70-original-rve-ts-bun-full-pass CONFOUNDED; CL-WA-r70-rve-ungraded-delivery-itt-gap CONFOUNDED. | CONFOUNDED; descriptive only. Task and front-end confounds remain, and original grades are not replaced by later cohorts. | B1 claim rows above | [original cohort page](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r70-rve/README.md), [test-count ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/test-counts.jsonl), [claim ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/claim-ledger.jsonl), [cost/time ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/cost-time.jsonl) |
| r70-rve-rerun | Fixed-skeleton sensitivity cohort; task 1 Elixir and task 4 on each of four stacks | Direct Codex, gpt-6-luna, max | Kogen not applicable; verified per-cell harness Git revision unavailable | EU and US hosts by stack; provider epoch not balanced | 15 / 15 / 15 / 15; rerun ITT linkage remains tied to its own cohort | Rust 3/3, Go 3/3, Elixir 3/6, TypeScript/Bun 2/3. Claims: CL-WA-r70-rve-rerun-r70-rve-rerun-rust-full-pass CONFOUNDED; CL-WA-r70-rve-rerun-r70-rve-rerun-elixir-full-pass CONFOUNDED; CL-WA-r70-rve-rerun-r70-rve-rerun-go-full-pass CONFOUNDED; CL-WA-r70-rve-rerun-r70-rve-rerun-ts-bun-full-pass CONFOUNDED. | CONFOUNDED and DESCRIPTIVE. Admission controls failed for task 4 on three stacks; do not pool with the original cohort as a causal result. | B1 claim rows above | [rerun page](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r70-rve-rerun/README.md), [test-count ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/test-counts.jsonl), [control ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/controls.jsonl), [claim ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/claim-ledger.jsonl) |
| r70-rve-ext | Rust, Elixir, Go, TypeScript/Bun; tasks 2, 5, and 7 | Direct Codex, gpt-6-luna, max | Kogen not applicable; verified per-cell harness Git revision unavailable | Rust and Elixir on kogen-bench-eu; Go and TypeScript/Bun on kogen-bench-us; wall comparisons are within-host | 24 preregistered cells for reps 31–32 plus five post-hoc rep-33 cells; official grades 29; ITT is cohort-specific | All extension grades: Rust 6/7, Elixir 5/8, Go 6/8, TypeScript/Bun 6/6. Claims: CL-WA-r70-rve-ext-r70-rve-ext-rust-full-pass VALID; CL-WA-r70-rve-ext-r70-rve-ext-elixir-full-pass VALID; CL-WA-r70-rve-ext-r70-rve-ext-go-full-pass VALID; CL-WA-r70-rve-ext-r70-rve-ext-ts-bun-full-pass VALID. | VALID ledger rows, DESCRIPTIVE cohort analysis. Small per-task samples and author-selected task templates do not support a broad ranking. | B1 claim rows above | [extension page](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r70-rve-ext/README.md), [test-count ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/test-counts.jsonl), [cost/time ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/cost-time.jsonl), [claim ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/claim-ledger.jsonl) |
| r70-task8v2 | Rust and Go smoke lane; exact task IDs are listed on the round page | Direct Codex with Luna max | Kogen not applicable; exact harness Git commit not recorded | Host and epoch are given in the round receipt | No scored Build cohort; smoke results remain outside scored ITT | The smoke gate failed for both arms, so no scored language comparison followed. | NOT-RUN for scored cells; DESCRIPTIVE smoke observations only. | No B1 claim ID assigned for this lane | [smoke lane](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r70-rve-task8v2/README.md), [test-count ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/test-counts.jsonl), [control ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/controls.jsonl) |
| r70-compile | Compile and execution timing across the listed tool/task combinations | No agent model execution | No Kogen Build revision; timing harness revision is documented on the round page | Single host; two timing windows | Timing rows are not live Build cells; Build planned / started / finished / graded / ITT counts are not applicable | Raw timing ledgers and method are public; no compiled timing headline is repeated on this page. | VALID round status; DESCRIPTIVE timing analysis. | No B1 claim ID assigned for this lane | [compile page](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r70-compile/README.md), [main timing ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r70-compile/raw/timings.jsonl), [supplemental timing ledger](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r70-compile/raw/timings-supplement.jsonl), [results](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/r70-compile/RESULTS.md) |

### Sol-medium language replication

The [language replication round](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/rounds/lang-sol-replication/README.md) reports 27 new cells and 31 reused cells on six task identities. Raw combined results are Rust 18/20, Go 18/20, and TS-Bun 15/18. A post-hoc unweighted mean of the six task pass fractions is Rust 90.3%, Go 88.9%, and TS-Bun 83.3% ([CL-SOL-LANG-REPLICATION-POSTHOC-EQUAL-TASK](https://github.com/KogenAI/kogen-bench/blob/baf7c2f4307d3a989a8fb9363df8000fda3d9310/results/claim-ledger.jsonl), **STATUS_ONLY**). The original equalization formula and scale for the four-pass threshold were not recorded; this sensitivity does not execute that branch. The claim is bounded to the selected Sol-medium replication cohort and does not establish a broad language ranking.

## Design implications

The public records support keeping the language cohorts separate and retaining a language choice as a bounded product decision. The original cohort is confounded; the fixed-skeleton rerun is a sensitivity check with failed admission controls; the extension is descriptive. Executable specs can test that implementations follow a defined contract, but parity is not evidence that one language or specification method improves agent outcomes.

## What this evidence does not show

These records do not show that Rust is broadly superior to Elixir, Go, TypeScript/Bun, or Gleam; that runtime or concurrency caused a pass-rate difference; or that Quint improves planning or task completion. The cost/time ledger is incomplete for several language cohorts, and compilation measurements are not a task-performance comparison.

## Smallest open tests

For language choice, freeze identical tasks, prompts, model and effort, tool access, retry policy, repair loop, grader, and resource accounting across stacks; include a predeclared rule for adding or excluding Gleam. For executable specifications, audit translation against blinded valid alternatives and omitted-behavior controls, then test whether the checks change live repair and completion outcomes.


## Related structured records

### Related hypotheses

- [H91](/hypotheses/h91/)
- [H92](/hypotheses/h92/)
- [H93](/hypotheses/h93/)
- [H94](/hypotheses/h94/)
- [H95](/hypotheses/h95/)
- [H96](/hypotheses/h96/)
- [H97](/hypotheses/h97/)
- [H98](/hypotheses/h98/)
- [H99](/hypotheses/h99/)

### Related experiments

- [lang-sol-replication](/rounds/lang-sol-replication/)

### Related claims

- [CL-SOL-LANG-REPLICATION-POSTHOC-EQUAL-TASK](/claims/cl-sol-lang-replication-posthoc-equal-task/)
