Round repair-stop-1
Repair limit: the six-repair stop did not hide successes on the Elixir task
Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.
Round source README · rounds/repair-stop-1/README.md
Exploratory run in Kogen Lab, reported descriptively. In the builder effort confirmation, all six attempts at the Elixir task env-ex were ended by the experimental six-repair stop. In two of them, the review kept asking for changes while the code passed most or all of the hidden tests. This run asked whether allowing twelve repairs instead of six turns such attempts into correct submissions, or whether the review keeps rejecting.
With twelve repairs allowed, all 3 attempts were correct; with six, all 3 were stopped. But none of the correct attempts needed more than six repairs (they used 5, 6 and 5). The six-repair stop ends an attempt only when a seventh repair would be needed, so each of these attempts would also have finished under it. The extra repairs do not explain the difference; with three pairs we cannot say what does. In this run the review did not block correct code. Each stopped attempt ended with the review asking for changes on code that passed its own checks but failed some hidden tests (it passed 19, 19 and 24 of 26), and the review approved the code in all three correct attempts. In the earlier round, by contrast, one stopped attempt left code passing all 26 hidden tests.
Question and scope
On env-ex, does raising the experimental stop from 6 to 12 repairs turn stopped attempts into correct submissions? The rule was fixed before any attempt. "The stop hides successes" would need a twelve-repair attempt to succeed where the six-repair attempt in the same block stopped. "The review does not converge" would need every twelve-repair attempt to be stopped too, or to end with requests for changes on code that passes its checks. The run used Kogen Lab, which reproduces Kogen's build loop with its own engine; its equivalence to Kogen is not measured.
- Task:
env-ex, a strict multi-file environment-file merger with interpolation in Elixir, written for the builder effort confirmation. Its public parts are in that round's `tasks/env-ex`; its hidden tests, reference solution and wrong variants stay private. - Configuration: GPT-6 Luna at medium effort as planner and builder, GPT-6.1 Sol at low effort as reviewer after every passing verification, with the supplied specification.
- Three repetitions; each repetition ran both repair limits on one host. The twelve-repair attempt started first in repetitions 1 and 3, the six-repair attempt in repetition 2.
- Every attempt also carried the experimental stop at 100,000 output tokens.
- The repair limit is not shown to the models.
- The hidden tests grade each attempt independently of the workflow's own checks and review. For a stopped attempt they grade the code it left behind, which was never submitted.
Results
"Repairs" counts every repair round, whether it followed a failed check or a review asking for changes. A review runs only after the checks pass. Totals are summed from unrounded values.
| Repetition | Six repairs allowed | Twelve repairs allowed |
|---|---|---|
| 1 | stopped after 6 repairs; 3 reviews, all asking for changes; code left passes 19 of 26 hidden tests; 0.68M input | correct after 5 repairs; 3 reviews asking for changes, then approval; 26 of 26; 1.17M input |
| 2 | stopped after 6 repairs; 1 review, asking for changes; 19 of 26; 0.44M input | correct after 6 repairs; 4 reviews asking for changes, then approval; 26 of 26; 1.20M input |
| 3 | stopped after 6 repairs; 5 reviews, all asking for changes; 24 of 26; 0.65M input | correct after 5 repairs; 2 reviews asking for changes, then approval; 26 of 26; 1.02M input |
| Total | 0 of 3 correct; 1.76M input | 3 of 3 correct; 3.39M input |
The run recorded 6 attempts, all with outcomes, and used about 5.15M input and 76k output tokens. No attempt claimed success with failing hidden tests.
Pooling this run's six attempts with the three medium-effort env-ex attempts of the builder effort confirmation (all stopped), 3 of 9 were correct, all of them with twelve repairs allowed. In this run, all three stopped attempts left code that failed some hidden tests.
Limits
- Three pairs on one task. The result is descriptive and carries no significance claim.
- One earlier stopped attempt at
env-exleft code passing all 26 hidden tests while the review asked for changes; this run did not reproduce that case. - The stops are experimental and not part of Kogen.
- Results describe Kogen Lab's reproduction of Kogen's loop; equivalence with Kogen itself is not measured.
- Lab's evaluation is not an official kogen-bench grade, so the Standard records declare the official outcome as missing (see "What's missing and why").
Required reproduction metadata
- Kogen commit: fd0cdcfb422f6edef5ec69146cdecdb3e3c1317a, as reproduced by Kogen Lab.
- Harness commit: not recorded as a public Git commit. Kogen Lab's source is not published yet; its build fingerprint is in the bundle's
manifest.json. - Model and effort: planner and builder
gpt-6-lunaatmedium; reviewergpt-6.1-solatlow. - Task IDs:
env-ex. - Command:
kogen-lab freeze, thenkogen-lab runon the frozen experiment, thenkogen-lab exportand Lab'sexport_bench_round.pyto write these records. - Raw records: `records.jsonl` and the Lab export in `bundles/`.
Metadata
Round date: 2026-10-18
Publication badge: DESCRIPTIVE; KEPT FOR AUDIT
Recomputation status: OBSERVED SOURCE ONLY
Why not VALID: DESCRIPTIVE_ONLY, PROTOCOL_DEVIATION, INCOMPLETE_CAPTURE — exploratory Lab run on one task, three pairs, no official kogen-bench grade and declared missing Standard fields.
STATUS: DESCRIPTIVE
Execution record
- One experiment, frozen before its first attempt and run from a frozen Lab development build over three host slots (two work folders on one Ubuntu 24.04 server, one Ubuntu 24.04 laptop).
- Records: `records.jsonl` holds one Standard 1.2 record per attempt (6). `MISSING.md` and `MEASURED.md` declare what the records cannot carry.
- `bundles/` holds Lab's public export; its
EXCLUDED.jsonlists what was left out and why.
What's missing and why
circumstances.cap_end(6 missing): The Lab export does not carry this Standard field. Boundary telemetry not retained. Reason: Boundary telemetry not retained (m08).circumstances.cap_start(6 missing): The Lab export does not carry this Standard field. Boundary telemetry not retained. Reason: Boundary telemetry not retained (m08).circumstances.concurrent_cells_end(6 missing): The Lab export does not carry this Standard field. Host-wide concurrent cell count was not captured. Reason: Host-wide concurrent cell count was not captured (m66).circumstances.concurrent_cells_start(6 missing): The Lab export does not carry this Standard field. Host-wide concurrent cell count was not captured. Reason: Host-wide concurrent cell count was not captured (m66).circumstances.dispatcher_id(6 missing): The Lab export does not carry this Standard field. Exact dispatcher revision is not present in the public receipt extract. Reason: Exact dispatcher revision is not present in the public receipt extract (m63).circumstances.incidents(6 missing): The Lab export does not carry this Standard field. Per-cell incident receipt was not captured. Reason: Per-cell incident receipt was not captured (m78).circumstances.load1_end(6 missing): The Lab export does not carry this Standard field. Host boundary load was not captured in the public round data. Reason: Host boundary load was not captured in the public round data (m65).circumstances.load1_start(6 missing): The Lab export does not carry this Standard field. Host boundary load was not captured in the public round data. Reason: Host boundary load was not captured in the public round data (m65).circumstances.load_samples(6 missing): The Lab export does not carry this Standard field. Timestamped host-wide load samples were not captured. Reason: Timestamped host-wide load samples were not captured (m93).circumstances.queue(6 missing): The Lab export does not carry this Standard field. Queue receipt was not captured. Reason: Queue receipt was not captured (m91).cost.accounting(6 missing): The Lab export does not carry this Standard field. Per-cell cost and pricing inputs are not present in the public round data. Reason: Per-cell cost and pricing inputs are not present in the public round data (m70).cost.calculator_version(6 missing): The Lab export does not carry this Standard field. Cost calculator version not recorded. Reason: Cost calculator version not recorded (m61).cost.long_context_reconciled(6 missing): The Lab export does not carry this Standard field. Boolean receipt not recorded. Reason: Boolean receipt not recorded (m07).cost.price_table_version(6 missing): The Lab export does not carry this Standard field. Price table version not recorded. Reason: Price table version not recorded (m89).cost.usd(6 missing): The Lab export does not carry this Standard field. Per-cell cost and pricing inputs are not present in the public round data. Reason: Per-cell cost and pricing inputs are not present in the public round data (m70).effort.effective(6 missing): The Lab export does not carry this Standard field. Per-cell effective-effort receipt not published. Reason: Per-cell effective-effort receipt not published (m73).effort.requested(6 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).effort.runner_requested(6 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).environment.account_class(6 missing): The Lab export does not carry this Standard field. No owner-approved class-only account receipt is available for the cell. Reason: No owner-approved class-only account receipt is available for the cell (m100).environment.network.allowlist_hosts(6 missing): The Lab export does not carry this Standard field. Per-cell network allowlist not present in the public round data. Reason: Per-cell network allowlist not present in the public round data (m79).environment.network.profile(6 missing): The Lab export does not carry this Standard field. Per-cell network profile not present in the public round data. Reason: Per-cell network profile not present in the public round data (m80).environment.toolchains.elixir(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).environment.toolchains.erlang(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).environment.toolchains.other_inventory(6 missing): The Lab export does not carry this Standard field. Full per-cell toolchain inventory not recorded. Reason: Full per-cell toolchain inventory not recorded (m64).environment.toolchains.python(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).environment.toolchains.ruby(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).environment.toolchains.rust(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).grade.grader(6 missing): The Lab export does not carry this Standard field. Per-cell grader software version is not in the public data. Reason: Per-cell grader software version is not in the public data (m77).grade.timestamp(6 missing): The Lab export does not carry this Standard field. No official grade in the public snapshot. Reason: No official grade in the public snapshot (m32).host.id(6 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).itt.class(6 missing): The Lab export does not carry this Standard field. No evidence-backed ITT classification. Reason: No evidence-backed ITT classification (m28).itt.cohort(6 missing): The Lab export does not carry this Standard field. No evidence-backed ITT classification. Reason: No evidence-backed ITT classification (m28).model.effective(6 missing): The Lab export does not carry this Standard field. Per-cell effective-model receipt not published. Reason: Per-cell effective-model receipt not published (m74).model.requested(6 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).outcome(6 missing): The Lab export does not carry this Standard field. Not available for ungraded delivery. Reason: Not available for ungraded delivery (m38).sandbox.egress_allow(6 missing): The Lab export does not carry this Standard field. Per-cell egress allowlist is not present in the public round data. Reason: Per-cell egress allowlist is not present in the public round data (m75).sandbox.egress_profile(6 missing): The Lab export does not carry this Standard field. Per-cell egress profile is not present in the public round data. Reason: Per-cell egress profile is not present in the public round data (m76).sandbox.profile(6 missing): The Lab export does not carry this Standard field. Per-cell sandbox profile is not present in the public round data. Reason: Per-cell sandbox profile is not present in the public round data (m87).sandbox.profile_sha256(6 missing): The Lab export does not carry this Standard field. Per-cell sandbox profile fingerprint is not present in the public round data. Reason: Per-cell sandbox profile fingerprint is not present in the public round data (m86).setup.adapter_harness_sha(6 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).setup.deps_source(6 missing): The Lab export does not carry this Standard field. Per-cell dependency source is not recorded. Reason: Per-cell dependency source is not recorded (m71).setup.sandbox_mode(6 missing): The Lab export does not carry this Standard field. Per-cell sandbox mode is not present in the public round data. Reason: Per-cell sandbox mode is not present in the public round data (m85).setup.sandbox_profile_sha256(6 missing): The Lab export does not carry this Standard field. Per-cell sandbox fingerprint is not present in the public round data. Reason: Per-cell sandbox fingerprint is not present in the public round data (m84).setup.task_base.hash(6 missing): The Lab export does not carry this Standard field. Original base commit/tree hash absent; fresh_base_commit is not a base hash. Reason: Original base commit/tree hash absent; fresh_base_commit is not a base hash (m42).setup.task_base.kind(6 missing): The Lab export does not carry this Standard field. Original base revision type not recorded per cell. Reason: Original base revision type not recorded per cell (m43).task.base_repo(6 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).task.base_revision.hash(6 missing): The Lab export does not carry this Standard field. Original base commit/tree hash absent; fresh_base_commit is not a base hash. Reason: Original base commit/tree hash absent; fresh_base_commit is not a base hash (m42).task.base_revision.kind(6 missing): The Lab export does not carry this Standard field. Original base revision type not recorded per cell. Reason: Original base revision type not recorded per cell (m43).timestamps.phases.develop.end_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.develop.start_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.gate.end_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.gate.start_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.grade.end_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.grade.start_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.plan.end_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.plan.start_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.review.end_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.review.start_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.setup.end_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.setup.start_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.shape.end_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timestamps.phases.shape.start_utc(6 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).timing.phases_s.develop(6 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).timing.phases_s.gate(6 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).timing.phases_s.setup(6 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).timing.phases_s.shape(6 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).timing.timeout_cap_s(6 missing): The Lab export does not carry this Standard field. Timeout receipt unavailable. Reason: Timeout receipt unavailable (m53).tools.grader(6 missing): The Lab export does not carry this Standard field. Per-cell grader software version is not in the public data. Reason: Per-cell grader software version is not in the public data (m77).tools.runner(6 missing): The Lab export does not carry this Standard field. Per-cell runner version not recorded in the public round data. Reason: Per-cell runner version not recorded in the public round data (m83).tools.toolchains.elixir(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).tools.toolchains.erlang(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).tools.toolchains.other_inventory(6 missing): The Lab export does not carry this Standard field. Full per-cell toolchain inventory not recorded. Reason: Full per-cell toolchain inventory not recorded (m64).tools.toolchains.python(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).tools.toolchains.ruby(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).tools.toolchains.rust(6 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).
Captured run records
Standard-schema records assigned to this round in the indexed snapshot.
No per-round file is listed. Check the run-record index for the complete partition list.
Reproduction inventory
Published analysis replay and full experiment replay are reported separately.
- Replay type
analysis-only- Script
- No per-round analysis script is published.
- Inputs
rounds/repair-stop-1/MEASURED.mdrounds/repair-stop-1/MISSING-DECLARED.jsonrounds/repair-stop-1/MISSING.mdrounds/repair-stop-1/PLAN.jsonrounds/repair-stop-1/README.mdrounds/repair-stop-1/bundles/rs1/EXCLUDED.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r0/stop12/a1/attempt.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r0/stop12/a1/evaluation.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r0/stop6/a1/attempt.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r0/stop6/a1/evaluation.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r1/stop12/a1/attempt.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r1/stop12/a1/evaluation.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r1/stop6/a1/attempt.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r1/stop6/a1/evaluation.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r2/stop12/a1/attempt.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r2/stop12/a1/evaluation.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r2/stop6/a1/attempt.jsonrounds/repair-stop-1/bundles/rs1/attempts/env-ex/r2/stop6/a1/evaluation.jsonrounds/repair-stop-1/bundles/rs1/frozen/experiment.tomlrounds/repair-stop-1/bundles/rs1/frozen/manifest.jsonrounds/repair-stop-1/bundles/rs1/frozen/resolved/stop12.tomlrounds/repair-stop-1/bundles/rs1/frozen/resolved/stop6.tomlrounds/repair-stop-1/bundles/rs1/manifest.jsonrounds/repair-stop-1/bundles/rs1/report.mdrounds/repair-stop-1/records.jsonl
- Full execution replay
- Unavailable
- Scope
- Published Standard records and the Lab public exports (frozen configuration, per-attempt outcomes, public test states, stage times and usage) are available; raw conversations, tool output and candidate code are not included.
View the complete per-round reproduction inventory · Repository reproduction guide