Round shaping-2

Shaper helpers and generated specifications

Status · DESCRIPTIVEReviewed status is transcribed from rounds/STATUS.md. The original narrative and its qualifications remain below.

Source-reported narrativeRound README and results text are source material. Numeric values that are not tied to a claim ledger record remain source-reported.

Round source README · rounds/shaping-2/README.md

Exploratory Kogen Lab screen, reported descriptively. One attempt per task and setting was planned. The round is approximate: Lab verified the code in the builder's workspace, not in a clean copy of the candidate as Kogen now does.

Question and scope

Does generated shaping produce correct submissions as often as a supplied specification, and do helper settings change the shaper's output? The four settings compare generated shaping, a supplied Gherkin specification, a helper offered as optional guidance, and a helper call required by the setting. The tasks are development tasks from the task catalogue: r70-1-go, r70-4-elixir-2, r70-4-go, and r70-6-elixir-2.

This is a small screen, with one attempt per task and setting. Helper-call counts come before outcome counts because the helper needs to be used for a helper effect to be possible.

Results

Shaper helpers: called in 2 of 4 scored hint attempts, and also in the infrastructure attempt that was re-run; called in all 4 required attempts. The helper settings shipped 3 wrong submissions (hint 1, required 2) against none without a helper (one attempt per task).

Setting Helper calls by task (scored attempts, in task order above) Scored attempts Correct submissions Wrong submissions Infrastructure attempts
Generated — 4 4/4 0 0
Supplied specification — 3 3/3 0 1, not re-run
Helper hint 0, 1, 1, 0 4 3/4 1 1, re-run
Required helper 2, 1, 1, 1 4 2/4 2 0

The two dashes mean no helper was available in that setting. Generated and supplied settings had no wrong submissions. The counts are small and do not establish a general effect. The hint-setting wrong submission followed an attempt with no helper call. The two wrong required-helper submissions followed two and one helper calls, respectively.

Limits

The run's attempt accounting is in the execution record and source bundle.

Required reproduction metadata

  • The settings and their resolved configurations are in the public bundle. The experimental command and wall-time caps differ from Kogen's normal profile.
  • The run used four public development tasks and one attempt per task and setting. No official kogen-bench grade was assigned; the reported correctness counts come from Lab's hidden-test evaluation.
  • `records.jsonl` contains one Standard 1.2 record per started attempt. `MEASURED.md` and `MISSING.md` describe recorded measures and gaps. `bundles/` contains the public exports; candidate code is not included.

Metadata

Round date: 2026-10-20 Publication badge: DESCRIPTIVE; KEPT FOR AUDIT Recomputation status: OBSERVED SOURCE ONLY Why not VALID: DESCRIPTIVE_ONLY, PROTOCOL_DEVIATION, INCOMPLETE_CAPTURE — exploratory screen with one attempt per task and setting, one infrastructure attempt not repeated, and no official kogen-bench grades; Standard capture fields are declared missing. STATUS: DESCRIPTIVE

Execution record

  • Kogen commit: b45ba32bacf26e2b17cb0bb81015ba56d27a4b0b.

  • Harness commit: not recorded as a public Git commit; the execution build is identified above.

  • Model and effort: Builder gpt-6.1-sol at medium effort; shaper and reviewer gpt-6.1-sol at high effort; helper gpt-6-luna at low effort.

  • Task IDs: r70-1-go, r70-4-elixir-2, r70-4-go, r70-6-elixir-2.

  • Command: The experiment was run and exported in Kogen Lab before publication.

  • Raw records: `records.jsonl` and the public bundles in `bundles/`.

  • The experiment ran on Kogen Lab build e1e0dc34, set to Kogen commit b45ba32b. The public bundle was exported with Lab build 3bf295ef. In the published copy, comments were removed from the frozen experiment file; its TOML settings are unchanged.

  • The experiment used one local work folder. The run covered 04:19–10:55Z on 20 October 2026, with a provider usage-limit pause.

Planned cells Started attempts Scored attempts Infrastructure attempts Input tokens
16 17 15 2; one re-run, one not re-run 8.50M
  • The 15 scored attempts include 4 generated, 3 supplied, 4 hint, and 4 required-helper attempts. Of the 2 infrastructure attempts, one was replaced by a successful re-run included among the 15 scored attempts; the other was not repeated.
  • `records.jsonl` keeps one Standard 1.2 record per started attempt (17). `MISSING.md` and `MEASURED.md` state what those records carry. The public bundle's EXCLUDED.json lists excluded material.

The list below names what the records cannot carry; reason codes are explained in `MISSING-DECLARED.json`.

What's missing and why

  • circumstances.cap_end (17 missing): The Lab export does not carry this Standard field. Boundary telemetry not retained. Reason: Boundary telemetry not retained (m08).
  • circumstances.cap_start (17 missing): The Lab export does not carry this Standard field. Boundary telemetry not retained. Reason: Boundary telemetry not retained (m08).
  • circumstances.concurrent_cells_end (17 missing): The Lab export does not carry this Standard field. Host-wide concurrent cell count was not captured. Reason: Host-wide concurrent cell count was not captured (m66).
  • circumstances.concurrent_cells_start (17 missing): The Lab export does not carry this Standard field. Host-wide concurrent cell count was not captured. Reason: Host-wide concurrent cell count was not captured (m66).
  • circumstances.dispatcher_id (17 missing): The Lab export does not carry this Standard field. Exact dispatcher revision is not present in the public receipt extract. Reason: Exact dispatcher revision is not present in the public receipt extract (m63).
  • circumstances.incidents (17 missing): The Lab export does not carry this Standard field. Per-cell incident receipt was not captured. Reason: Per-cell incident receipt was not captured (m78).
  • circumstances.load1_end (17 missing): The Lab export does not carry this Standard field. Host boundary load was not captured in the public round data. Reason: Host boundary load was not captured in the public round data (m65).
  • circumstances.load1_start (17 missing): The Lab export does not carry this Standard field. Host boundary load was not captured in the public round data. Reason: Host boundary load was not captured in the public round data (m65).
  • circumstances.load_samples (17 missing): The Lab export does not carry this Standard field. Timestamped host-wide load samples were not captured. Reason: Timestamped host-wide load samples were not captured (m93).
  • circumstances.queue (17 missing): The Lab export does not carry this Standard field. Queue receipt was not captured. Reason: Queue receipt was not captured (m91).
  • cost.accounting (17 missing): The Lab export does not carry this Standard field. Per-cell cost and pricing inputs are not present in the public round data. Reason: Per-cell cost and pricing inputs are not present in the public round data (m70).
  • cost.calculator_version (17 missing): The Lab export does not carry this Standard field. Cost calculator version not recorded. Reason: Cost calculator version not recorded (m61).
  • cost.long_context_reconciled (17 missing): The Lab export does not carry this Standard field. Boolean receipt not recorded. Reason: Boolean receipt not recorded (m07).
  • cost.price_table_version (17 missing): The Lab export does not carry this Standard field. Price table version not recorded. Reason: Price table version not recorded (m89).
  • cost.usd (17 missing): The Lab export does not carry this Standard field. Per-cell cost and pricing inputs are not present in the public round data. Reason: Per-cell cost and pricing inputs are not present in the public round data (m70).
  • effort.effective (17 missing): The Lab export does not carry this Standard field. Per-cell effective-effort receipt not published. Reason: Per-cell effective-effort receipt not published (m73).
  • effort.requested (17 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).
  • effort.runner_requested (17 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).
  • environment.account_class (17 missing): The Lab export does not carry this Standard field. No owner-approved class-only account receipt is available for the cell. Reason: No owner-approved class-only account receipt is available for the cell (m100).
  • environment.network.allowlist_hosts (17 missing): The Lab export does not carry this Standard field. Per-cell network allowlist not present in the public round data. Reason: Per-cell network allowlist not present in the public round data (m79).
  • environment.network.profile (17 missing): The Lab export does not carry this Standard field. Per-cell network profile not present in the public round data. Reason: Per-cell network profile not present in the public round data (m80).
  • environment.toolchains.erlang (17 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).
  • environment.toolchains.other_inventory (17 missing): The Lab export does not carry this Standard field. Full per-cell toolchain inventory not recorded. Reason: Full per-cell toolchain inventory not recorded (m64).
  • environment.toolchains.ruby (17 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).
  • environment.toolchains.rust (17 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).
  • grade.grader (17 missing): The Lab export does not carry this Standard field. Per-cell grader software version is not in the public data. Reason: Per-cell grader software version is not in the public data (m77).
  • grade.timestamp (17 missing): The Lab export does not carry this Standard field. No official grade in the public snapshot. Reason: No official grade in the public snapshot (m32).
  • host.id (17 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).
  • itt.class (17 missing): The Lab export does not carry this Standard field. No evidence-backed ITT classification. Reason: No evidence-backed ITT classification (m28).
  • itt.cohort (17 missing): The Lab export does not carry this Standard field. No evidence-backed ITT classification. Reason: No evidence-backed ITT classification (m28).
  • model.effective (1 missing): The Lab export does not carry this Standard field. Per-cell effective-model receipt not published. Reason: Per-cell effective-model receipt not published (m74).
  • outcome (17 missing): The Lab export does not carry this Standard field. Not available for ungraded delivery. Reason: Not available for ungraded delivery (m38).
  • provenance.ledger_sha256 (17 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).
  • sandbox.egress_allow (17 missing): The Lab export does not carry this Standard field. Per-cell egress allowlist is not present in the public round data. Reason: Per-cell egress allowlist is not present in the public round data (m75).
  • sandbox.egress_profile (17 missing): The Lab export does not carry this Standard field. Per-cell egress profile is not present in the public round data. Reason: Per-cell egress profile is not present in the public round data (m76).
  • sandbox.profile (17 missing): The Lab export does not carry this Standard field. Per-cell sandbox profile is not present in the public round data. Reason: Per-cell sandbox profile is not present in the public round data (m87).
  • sandbox.profile_sha256 (17 missing): The Lab export does not carry this Standard field. Per-cell sandbox profile fingerprint is not present in the public round data. Reason: Per-cell sandbox profile fingerprint is not present in the public round data (m86).
  • setup.adapter_harness_sha (17 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).
  • setup.deps_source (17 missing): The Lab export does not carry this Standard field. Per-cell dependency source is not recorded. Reason: Per-cell dependency source is not recorded (m71).
  • setup.sandbox_mode (17 missing): The Lab export does not carry this Standard field. Per-cell sandbox mode is not present in the public round data. Reason: Per-cell sandbox mode is not present in the public round data (m85).
  • setup.sandbox_profile_sha256 (17 missing): The Lab export does not carry this Standard field. Per-cell sandbox fingerprint is not present in the public round data. Reason: Per-cell sandbox fingerprint is not present in the public round data (m84).
  • setup.task_base.hash (17 missing): The Lab export does not carry this Standard field. Original base commit/tree hash absent; fresh_base_commit is not a base hash. Reason: Original base commit/tree hash absent; fresh_base_commit is not a base hash (m42).
  • setup.task_base.kind (17 missing): The Lab export does not carry this Standard field. Original base revision type not recorded per cell. Reason: Original base revision type not recorded per cell (m43).
  • task.base_repo (17 missing): The Lab export does not carry this Standard field. Not recorded in available public metadata. Reason: Not recorded in available public metadata (m39).
  • task.base_revision.hash (17 missing): The Lab export does not carry this Standard field. Original base commit/tree hash absent; fresh_base_commit is not a base hash. Reason: Original base commit/tree hash absent; fresh_base_commit is not a base hash (m42).
  • task.base_revision.kind (17 missing): The Lab export does not carry this Standard field. Original base revision type not recorded per cell. Reason: Original base revision type not recorded per cell (m43).
  • timestamps.phases.develop.end_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.develop.start_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.gate.end_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.gate.start_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.grade.end_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.grade.start_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.plan.end_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.plan.start_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.review.end_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.review.start_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.setup.end_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.setup.start_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.shape.end_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timestamps.phases.shape.start_utc (17 missing): The Lab export does not carry this Standard field. Phase timestamp not present in the public per-cell data. Reason: Phase timestamp not present in the public per-cell data (m88).
  • timing.phases_s.develop (17 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).
  • timing.phases_s.gate (17 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).
  • timing.phases_s.setup (17 missing): The Lab export does not carry this Standard field. Phase wall not emitted or not separable. Reason: Phase wall not emitted or not separable (m45).
  • timing.timeout_cap_s (17 missing): The Lab export does not carry this Standard field. Timeout receipt unavailable. Reason: Timeout receipt unavailable (m53).
  • tools.grader (17 missing): The Lab export does not carry this Standard field. Per-cell grader software version is not in the public data. Reason: Per-cell grader software version is not in the public data (m77).
  • tools.runner (17 missing): The Lab export does not carry this Standard field. Per-cell runner version not recorded in the public round data. Reason: Per-cell runner version not recorded in the public round data (m83).
  • tools.toolchains.erlang (17 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).
  • tools.toolchains.other_inventory (17 missing): The Lab export does not carry this Standard field. Full per-cell toolchain inventory not recorded. Reason: Full per-cell toolchain inventory not recorded (m64).
  • tools.toolchains.ruby (17 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).
  • tools.toolchains.rust (17 missing): The Lab export does not carry this Standard field. Toolchain version not recorded. Reason: Toolchain version not recorded (m95).

Captured run records

Standard-schema records assigned to this round in the indexed snapshot.

No per-round file is listed. Check the run-record index for the complete partition list.

Reproduction inventory

Published analysis replay and full experiment replay are reported separately.

Replay type
analysis-only
Script
No per-round analysis script is published.
Inputs
Full execution replay
Unavailable
Scope
Published Standard records and the Lab public exports (frozen configuration, per-attempt outcomes, public test states, stage times and usage) are available; raw conversations, tool output and candidate code are not included.

View the complete per-round reproduction inventory · Repository reproduction guide