Round r70-compile

r70 compilation timing

Status · DESCRIPTIVEReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/r70-compile/README.md

Round date: HISTORICAL (before 2026-10-09) Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Why not VALID: The historical inventory retains DESCRIPTIVE because of DESCRIPTIVE_ONLY, EVIDENCE_SCOPE, NO_PREREG. See the round evidence and limitations below. Recomputation status: OBSERVED SOURCE ONLY

Recomputation limit: Published summary figures: planned/observed counts, outcome totals/rates, and numeric estimates as present in README.md and RESULTS.md. These figures cannot be recomputed from repository inputs because no public per-round analysis script with a complete, identified input bundle is published. See the round README, MEASURED.md, and MISSING.md for recorded scope and gaps; historical execution inputs/toolchains/grading are not bundled.

Status

DESCRIPTIVE

  • NO_PREREG — The README explicitly says no pre-registration.
  • DESCRIPTIVE_ONLY — The round measures compile timing and does not have a pre-registered inferential decision rule.
  • EVIDENCE_SCOPE — Retained timing summaries support description but do not establish the full VALID criteria.
  • relabelled 2026-10-08 after scrutiny

Required reproduction metadata

  • Kogen commit: not recorded in the cited round source.
  • Harness commit: not recorded in the cited round source as a Git commit; any adapter or runtime fingerprints are not Git commits.
  • Model and effort: a complete per-cell mapping is not recorded in the cited round source; values present in the source summary remain as stated above.
  • Task IDs: not recorded in the cited round source.
  • Historical execution command: not recorded in the cited round source.
  • Raw records: the cited sources do not provide a complete public round-specific request and grade bundle; public delivery and outcome exports are linked above where applicable.

Terminology: public round glossary.

STATUS: DESCRIPTIVE

Data completeness: N/A (0 deliveries; NO DELIVERED DATA). Measurement contract; declared gaps.

Pre-registered: no Label: DESCRIPTIVE Question: How long does each candidate language take to compile Kogen-like code? n: 300 timing rows; n=5 per tool/task combination and timed phase. Headline: The two timing windows each contain 150 rows and are marked valid in their raw summaries; the timing table is in RESULTS.md. Limit: One host, two timing windows, five repetitions per cell; descriptive timing only. Headline-number note: Counts are not re-derived here; the raw timing ledgers and summaries are linked below.

Sources: method; main raw timing ledger; main run summary; supplement timing ledger; supplement run summary.

Public delivery and outcome reconciliation

No standard scored benchmark cells are attributed to this round. The timing study contains 300 rows; see the main timing ledger, supplement timing ledger, and timing results for its n and measurements.

Round results · rounds/r70-compile/RESULTS.md

Measured 6 October 2026 on kogen-bench-eu in exclusive quiet windows: main run 11:35:43–11:43:51Z (8 min 8 s), supplement 11:46:03–11:47:58Z (1 min 55 s). Seconds, median [min–max], n=5 per cell. Method: METHOD.md. Raw rows: raw/.

Toolchains: rustc/cargo 1.97.1 (release profile), Go 1.27.1, TypeScript 7.0.2 (native compiler) + Bun 1.4.2, Elixir 1.20.2 on Erlang/OTP 29.

Host specification and load context:

Reported host snapshot: AMD EPYC-Milan, 2 vCPU, 7.6 GiB, kernel 6.8.0-142-generic. Median pre-cold 1-minute load was 2.8 on 2 vCPUs, mostly from preceding builds; stacks were interleaved per rep so the drift was shared.

Stack Task Clean build incl. deps Project-only rebuild (deps warm) One-file incremental
Rust (cargo build --release) 1 7.84 [7.74–12.40] 1.02 [1.02–1.02] 1.02 [1.02–1.02]
4 0.47 [0.42–0.47] 0.42 [0.41–0.42] 0.42 [0.42–0.42]
6 7.19 [7.14–11.56] 0.87 [0.87–0.87] 0.87 [0.87–1.33]
Go (go build) 1 8.50 [8.40–8.86] 0.11 [0.06–0.27] 0.27 [0.27–0.27]
4 6.75 [6.59–11.03] 0.06 [0.06–0.22] 0.22 [0.21–0.48]
6 7.96 [7.90–8.21] 0.11 [0.11–0.27] 0.27 [0.26–0.27]
Elixir (mix compile) 1 12.02 [11.91–17.39] 0.56 [0.51–0.72] 0.62 [0.62–0.67]
4 11.86 [11.76–12.28] 0.52 [0.51–0.62] 0.62 [0.62–0.62]
6 12.07 [11.82–12.27] 0.67 [0.67–0.82] 0.72 [0.67–1.02]
TS-Bun (tsc --noEmit) 1 0.27 [0.21–0.27] 0.22 [0.21–0.27] 0.27 [0.21–0.27]
4 0.26 [0.21–0.37] 0.22 [0.21–0.22] 0.27 [0.21–0.42]
6 0.26 [0.21–0.27] 0.27 [0.27–0.32] 0.27 [0.22–0.27]
TS-Bun (bun build --compile) 1 0.21 [0.21–0.36] 0.26 [0.26–0.26] 0.26 [0.26–0.26]
4 0.21 [0.21–0.27] 0.27 [0.26–0.32] 0.26 [0.26–0.42]
6 0.21 [0.21–0.21] 0.27 [0.26–0.31] 0.26 [0.26–0.27]

Sandbox overhead (a timed true in the same sandbox): 0.01 s for every stack and task.

How to read it

  • Clean build mostly measures dependencies, not the project: Rust tasks 1 and 6 compile serde/serde_json (task 4 has no dependencies, hence 0.47 s); Go starts from an empty build cache, so it compiles the standard-library packages it uses (on a developer machine that happens once); Elixir compiles jason plus the dev-only credo and dialyxir dependencies.
  • Project-only rebuild (every project source file changed, dependency caches warm) measures each stack's own project build command, which differ in what they produce (Rust cargo build --release and Go go build emit binaries, Elixir mix compile emits BEAM files, TypeScript tsc --noEmit only type-checks); read it as a per-stack rebuild time, not a like-for-like compiler comparison: Go ~0.1 s, TS ~0.2–0.3 s, Elixir ~0.5–0.7 s, Rust ~0.4–1.0 s.
  • Incremental: Rust release builds recompile the whole crate, so incremental equals the project rebuild. Elixir recompiles only the changed module and its dependents. Go's one-file rebuild (0.22–0.27 s) measured above its project rebuild (0.06–0.11 s); both are under 0.3 s, and the difference is within this harness's timing resolution (next point).
  • Resolution: commands run via Python subprocess.run(timeout=…), which polls with a backoff capped at 50 ms. Values above ~0.1 s are therefore accurate to about ±0.05 s; the sub-second differences between Go, TS and Elixir are real in order of magnitude but not to the hundredth.
  • Scale caveat: these are small task projects (2–5 source files, 8–19 KB). They don't show how compile time grows with a large codebase, where Rust's whole-crate recompilation and Elixir's dependency-graph recompiles behave very differently from Go's per-package caching. A Kogen-sized project would need its own measurement.

Implementations compiled: one complete agent-written solution per task × stack from the Round 70 implementation-stack cells (public deliveries; no sealed reference code). Their 40-cell as-graded RvE outcomes are a separate result. Compile time depends on the code compiled, so another implementation would give somewhat different numbers.

Supersedes the v1 compile/check table. The fixed-skeleton result report is in r70-rve-rerun/RESULTS.md.

Native full-check and hidden-wrapper diagnostic

This is a separate diagnostic, not a compile-only ranking. The table reports n=3 per task and stack, median [min–max], in seconds. Full make check includes each stack's native lint, typecheck, and tests; Rust uses a separate debug profile and Elixir includes Credo/Dialyzer, so check scopes differ. Hidden-wrapper plus pytest timing invokes the wrapper without a build and is not agent wall. The rows are recomputed from sanitized per-repetition timings in the public reproduction inputs.

Task Stack n Full make check, median [min–max] s Hidden wrapper + pytest, median [min–max] s
1 Rust 3 8.344 [8.293–8.449] 1.518 [1.517–1.567]
1 Go 3 16.292 [15.983–20.921] 1.567 [1.517–1.617]
1 TypeScript/Bun 3 0.266 [0.265–0.268] 2.169 [2.069–2.169]
1 Elixir 3 55.380 [55.017–59.774] 13.964 [13.959–14.017]
1 Gleam 3 0.917 [0.917–0.918] 10.553 [10.400–12.506]
4 Rust 3 0.516 [0.467–0.568] 0.966 [0.966–1.016]
4 Go 3 15.396 [15.239–15.441] 1.016 [0.966–1.017]
4 TypeScript/Bun 3 0.265 [0.265–0.265] 1.066 [1.066–1.067]
4 Elixir 3 59.474 [57.792–59.894] 5.685 [5.684–5.739]
4 Gleam 3 0.969 [0.919–1.172] 4.429 [4.378–6.170]
6 Rust 3 7.899 [7.894–12.277] 0.465 [0.465–0.466]
6 Go 3 20.654 [20.537–26.045] 0.566 [0.565–0.566]
6 TypeScript/Bun 3 0.265 [0.265–0.265] 0.665 [0.665–0.666]
6 Elixir 3 59.576 [54.471–60.031] 6.788 [6.786–6.789]
6 Gleam 3 0.917 [0.917–1.275] 5.540 [4.835–5.653]

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
unavailable
Script
No per-round analysis script is published.
Inputs
No identified public input bundle.
Full execution replay
Unavailable
Scope
No public per-round analysis script with a complete, identified input bundle is published. See the round README, MEASURED.md, and MISSING.md for recorded scope and gaps; historical execution inputs/toolchains/grading are not bundled.

View the complete per-round reproduction inventory · Repository reproduction guide