Round builder-low-1
Builder effort at low vs medium: correctness held, total input fell 4%
Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.
Round source README · rounds/builder-low-1/README.md
A descriptive screen with two repetitions per task. The round is approximate: Lab verified the code in the builder's workspace, not in a clean copy of the candidate as Kogen now does.
Question and scope
Does a low-effort builder keep the default medium-effort builder's correctness on harder tasks with less input? The screen compares Kogen's default medium-effort builder with the same model at low effort. Both settings use prose plus Gherkin specifications and four public development tasks, with two repetitions per task and setting.
Results
| Task | Arm | Correct submissions |
|---|---|---|
| r70-1-go | sol-medium | 2/2 |
| r70-1-go | sol-low | 2/2 |
| r70-4-elixir-2 | sol-medium | 2/2 |
| r70-4-elixir-2 | sol-low | 2/2 |
| r70-4-go | sol-medium | 2/2 |
| r70-4-go | sol-low | 2/2 |
| r70-6-elixir-2 | sol-medium | 2/2 |
| r70-6-elixir-2 | sol-low | 2/2 |
| Effort setting | Scored attempts | Correct submissions | Wrong submissions | Repairs | Builder input | Reviewer input | Total input | Build elapsed time | Review elapsed time |
|---|---|---|---|---|---|---|---|---|---|
| sol-medium (default) | 8 | 8 | 0 | 0 | 0.65M | 1.77M | 2.42M | 1765 s | 2855 s |
| sol-low | 8 | 8 | 0 | 0 | 0.41M | 1.92M | 2.33M | 1441 s | 3088 s |
Both settings scored 8 of 8, with no repairs and no wrong submissions. Builder input fell about 38% (651,878 to 407,155 tokens), and build elapsed time fell from 1765 s to 1441 s. Reviewer input rose from 1.77M to 1.92M, limiting the total input reduction to about 4% (2.42M to 2.33M). Reviewer input was about 73% of total input for sol-medium and about 83% for sol-low.
Limits
This is a small screen with two repetitions per task. It does not establish that low effort keeps correctness on other tasks or in other runs.
Required reproduction metadata
- The settings and resolved configurations are in the public bundle. They use the same prose and Gherkin specifications; builder effort differs.
- The task identifiers in the table are public development tasks. Lab hidden-test outcomes are reported; no official benchmark grade was assigned.
- Records contains one Standard 1.2 record per started attempt (16). Measured fields, declared gaps, and missing-field data describe record coverage.
- Public exports contain the frozen settings and per-attempt outcomes. Candidate code is not included.
Metadata
Round date: 2026-10-20 Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Recomputation status: OBSERVED SOURCE ONLY Why not VALID: DESCRIPTIVE_ONLY, PROTOCOL_DEVIATION, INCOMPLETE_CAPTURE — exploratory screen with two repetitions per task, small counts, approximate execution, and no official grades; Standard capture fields are declared missing. STATUS: DESCRIPTIVE
Execution record
- Kogen commit: b45ba32bacf26e2b17cb0bb81015ba56d27a4b0b.
- Harness commit: not recorded as a public Git commit; the execution build is identified below.
- Model and effort: Builder gpt-6.1-sol at medium or low effort; reviewer-fixer gpt-6.1-sol at high effort.
- Task IDs: r70-1-go, r70-4-elixir-2, r70-4-go, and r70-6-elixir-2.
- The experiment was run and exported with Kogen Lab build 3bf295ef, set to Kogen commit b45ba32b.
- Command: The experiment was run and exported in Kogen Lab before publication.
- The run scored 16 of 16 attempts. Recorded input usage was 4.76M tokens.
- Raw records and the public bundles are linked above.
What's missing and why
circumstances.cap_end(16 missing): The Lab export does not carry this Standard field. Boundary telemetry not retained. Reason: Boundary telemetry not retained (m08).circumstances.cap_start(16 missing): The Lab export does not carry this Standard field. Boundary telemetry not retained. Reason: Boundary telemetry not retained (m08).circumstances.concurrent_cells_end(16 missing): The Lab export does not carry this Standard field. Host-wide concurrent cell count was not captured. Reason: Host-wide concurrent cell count was not captured (m66).circumstances.concurrent_cells_start(16 missing): The Lab export does not carry this Standard field. Host-wide concurrent cell count was not captured. Reason: Host-wide concurrent cell count was not captured (m66).circumstances.dispatcher_id(16 missing): The Lab export does not carry this Standard field. Exact dispatcher revision is not present in the public receipt extract. Reason: Exact dispatcher revision is not present in the public receipt extract (m63).circumstances.incidents(16 missing): The Lab export does not carry this Standard field. Per-cell incident receipt was not captured. Reason: Per-cell incident receipt was not captured (m78).circumstances.load1_end(16 missing): The Lab export does not carry this Standard field. Host boundary load was not captured in the public round data. Reason: Host boundary load was not captured in the public round data (m65).circumstances.load1_start(16 missing): The Lab export does not carry this Standard field. Host boundary load was not captured in the public round data. Reason: Host boundary load was not captured in the public round data (m65).circumstances.load_samples(16 missing): The Lab export does not carry this Standard field. Timestamped host-wide load samples were not captured. Reason: Timestamped host-wide load samples were not captured (m93).circumstances.queue(16 missing): The Lab export does not carry this Standard field. Queue receipt was not captured. Reason: Queue receipt was not captured (m91).cost.accounting(16 missing): The Lab export does not carry this Standard field. Per-cell cost and pricing inputs are not present in the public round data. Reason: Per-cell cost and pricing inputs are not present in the public round data (m70).cost.calculator_version(16 missing): The Lab export does not carry this Standard field. Cost calculator version not recorded. Reason: Cost calculator version not recorded (m61).cost.long_context_reconciled(16 missing): The Lab export does not carry this Standard field. Boolean receipt not recorded. Reason: Boolean receipt not recorded (m07).cost.price_table_version(16 missing): The Lab export does not carry this Standard field. Price table version not recorded. Reason: Price table version not recorded (m89).cost.usd(16 missing): The Lab export does not carry this Standard field. Per-cell cost and pricing inputs are not present in the public round data. Reason: Per-cell cost and pricing inputs are not present in the public round data (m70).effort.effective(16 missing): The Lab export does not carry this Standard field. Per-cell effective-effort receipt not published. Reason: Per-cell effective-effort receipt not published (m73).effort.requested(16 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).effort.runner_requested(16 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).environment.account_class(16 missing): The Lab export does not carry this Standard field. No owner-approved class-only account receipt is available for the cell. Reason: No owner-approved class-only account receipt is available for the cell (m100).environment.network.allowlist_hosts(16 missing): The Lab export does not carry this Standard field. Per-cell network allowlist not present in the public round data. Reason: Per-cell network allowlist not present in the public round data (m79).environment.network.profile(16 missing): The Lab export does not carry this Standard field. Per-cell network profile not present in the public round data. Reason: Per-cell network profile not present in the public round data (m80).environment.toolchains.erlang(16 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).environment.toolchains.other_inventory(16 missing): The Lab export does not carry this Standard field. Full per-cell toolchain inventory not recorded. Reason: Full per-cell toolchain inventory not recorded (m64).environment.toolchains.ruby(16 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).environment.toolchains.rust(16 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).grade.grader(16 missing): The Lab export does not carry this Standard field. Per-cell grader software version is not in the public data. Reason: Per-cell grader software version is not in the public data (m77).grade.timestamp(16 missing): The Lab export does not carry this Standard field. No official grade in the public snapshot. Reason: No official grade in the public snapshot (m32).host.id(16 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).itt.class(16 missing): The Lab export does not carry this Standard field. No evidence-backed ITT classification. Reason: No evidence-backed ITT classification (m28).itt.cohort(16 missing): The Lab export does not carry this Standard field. No evidence-backed ITT classification. Reason: No evidence-backed ITT classification (m28).outcome(16 missing): The Lab export does not carry this Standard field. Not available for ungraded delivery. Reason: Not available for ungraded delivery (m38).provenance.ledger_sha256(16 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).sandbox.egress_allow(16 missing): The Lab export does not carry this Standard field. Per-cell egress allowlist is not present in the public round data. Reason: Per-cell egress allowlist is not present in the public round data (m75).sandbox.egress_profile(16 missing): The Lab export does not carry this Standard field. Per-cell egress profile is not present in the public round data. Reason: Per-cell egress profile is not present in the public round data (m76).sandbox.profile(16 missing): The Lab export does not carry this Standard field. Per-cell sandbox profile is not present in the public round data. Reason: Per-cell sandbox profile is not present in the public round data (m87).sandbox.profile_sha256(16 missing): The Lab export does not carry this Standard field. Per-cell sandbox profile fingerprint is not present in the public round data. Reason: Per-cell sandbox profile fingerprint is not present in the public round data (m86).setup.adapter_harness_sha(16 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).setup.deps_source(16 missing): The Lab export does not carry this Standard field. Per-cell dependency source is not recorded. Reason: Per-cell dependency source is not recorded (m71).setup.sandbox_mode(16 missing): The Lab export does not carry this Standard field. Per-cell sandbox mode is not present in the public round data. Reason: Per-cell sandbox mode is not present in the public round data (m85).setup.sandbox_profile_sha256(16 missing): The Lab export does not carry this Standard field. Per-cell sandbox fingerprint is not present in the public round data. Reason: Per-cell sandbox fingerprint is not present in the public round data (m84).setup.task_base.hash(16 missing): The Lab export does not carry this Standard field. Original base commit/tree hash absent; fresh_base_commit is not a base hash. Reason: Original base commit/tree hash absent; fresh_base_commit is not a base hash (m42).setup.task_base.kind(16 missing): The Lab export does not carry this Standard field. Original base revision type not recorded per cell. Reason: Original base revision type not recorded per cell (m43).task.base_repo(16 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).task.base_revision.hash(16 missing): The Lab export does not carry this Standard field. Original base commit/tree hash absent; fresh_base_commit is not a base hash. Reason: Original base commit/tree hash absent; fresh_base_commit is not a base hash (m42).task.base_revision.kind(16 missing): The Lab export does not carry this Standard field. Original base revision type not recorded per cell. Reason: Original base revision type not recorded per cell (m43).timestamps.phases.develop.end_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.develop.start_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.gate.end_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.gate.start_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.grade.end_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.grade.start_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.plan.end_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.plan.start_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.review.end_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.review.start_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.setup.end_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.setup.start_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.shape.end_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.shape.start_utc(16 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timing.phases_s.develop(16 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).timing.phases_s.gate(16 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).timing.phases_s.setup(16 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).timing.timeout_cap_s(16 missing): The Lab export does not carry this Standard field. Timeout receipt unavailable. Reason: Timeout receipt unavailable (m53).tools.grader(16 missing): The Lab export does not carry this Standard field. Per-cell grader software version is not in the public data. Reason: Per-cell grader software version is not in the public data (m77).tools.runner(16 missing): The Lab export does not carry this Standard field. Per-cell runner version not recorded in the public round data. Reason: Per-cell runner version not recorded in the public round data (m83).tools.toolchains.erlang(16 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).tools.toolchains.other_inventory(16 missing): The Lab export does not carry this Standard field. Full per-cell toolchain inventory not recorded. Reason: Full per-cell toolchain inventory not recorded (m64).tools.toolchains.ruby(16 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).tools.toolchains.rust(16 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).
Captured run records
Standard-schema records assigned to this round in the indexed snapshot.
No per-round file is listed. Check the run-record index for the complete partition list.
Reproduction inventory
Published analysis replay and full experiment replay are reported separately.
- Replay type
analysis-only- Script
- No per-round analysis script is published.
- Inputs
rounds/builder-low-1/MEASURED.mdrounds/builder-low-1/MISSING-DECLARED.jsonrounds/builder-low-1/MISSING.mdrounds/builder-low-1/PLAN.jsonrounds/builder-low-1/README.mdrounds/builder-low-1/bundles/bl1/EXCLUDED.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-1-go/r0/sol-low/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-1-go/r0/sol-low/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-1-go/r0/sol-medium/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-1-go/r0/sol-medium/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-1-go/r1/sol-low/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-1-go/r1/sol-low/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-1-go/r1/sol-medium/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-1-go/r1/sol-medium/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-elixir-2/r0/sol-low/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-elixir-2/r0/sol-low/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-elixir-2/r0/sol-medium/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-elixir-2/r0/sol-medium/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-elixir-2/r1/sol-low/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-elixir-2/r1/sol-low/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-elixir-2/r1/sol-medium/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-elixir-2/r1/sol-medium/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-go/r0/sol-low/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-go/r0/sol-low/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-go/r0/sol-medium/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-go/r0/sol-medium/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-go/r1/sol-low/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-go/r1/sol-low/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-go/r1/sol-medium/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-4-go/r1/sol-medium/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-6-elixir-2/r0/sol-low/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-6-elixir-2/r0/sol-low/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-6-elixir-2/r0/sol-medium/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-6-elixir-2/r0/sol-medium/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-6-elixir-2/r1/sol-low/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-6-elixir-2/r1/sol-low/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-6-elixir-2/r1/sol-medium/a1/attempt.jsonrounds/builder-low-1/bundles/bl1/attempts/r70-6-elixir-2/r1/sol-medium/a1/evaluation.jsonrounds/builder-low-1/bundles/bl1/frozen/experiment.tomlrounds/builder-low-1/bundles/bl1/frozen/manifest.jsonrounds/builder-low-1/bundles/bl1/frozen/resolved/sol-low.tomlrounds/builder-low-1/bundles/bl1/frozen/resolved/sol-medium.tomlrounds/builder-low-1/bundles/bl1/manifest.jsonrounds/builder-low-1/bundles/bl1/report.mdrounds/builder-low-1/records.jsonl
- Full execution replay
- Unavailable
- Scope
- Published Standard records and the Lab public exports (frozen configuration, per-attempt outcomes, public test states, stage times and usage) are available; raw conversations, tool output and candidate code are not included.
View the complete per-round reproduction inventory · Repository reproduction guide