v0.42.11.0 feat(skillopt): held-out eval gate, honest receipts, ENFORCE + ablation opts (#1759)

* feat(skillopt): wire held-out gate, honest receipts, ENFORCE + ablation opts Wire the F11 held-out gate into the orchestrator at checkpoint acceptance (runHeldOutGate was dead code); parse + thread --held-out through CLI, batch, fleet, background job, and the run_skillopt MCP op. Populate the real receipt.baseline_sel_score (was hardcoded 0) and add a final-test eval (test_score + baseline_test_score) via a shared scoreSkillOnTasks primitive. Fix the --no-mutate proposed.md write (was a stub) and enforce maxRuntimeMin. D16 ENFORCE in core mutation policy (assertBundledMutationHeldOut): mutating a bundled skill in place requires a non-empty (>=5), benchmark-disjoint held-out set or hard-refuses. Add three eval-internal ablation opts (reflectMode, disableValidationGate, optimizerMode='one-shot-rewrite') recorded in the receipt + audit; ROLLOUT_SUCCESS_THRESHOLD named constant. Security: run_skillopt MCP op validates skill_name (kebab-only) and confines caller-supplied benchmark/held-out paths to the skills dir for remote callers. * test(skillopt): held-out gate, ENFORCE, one-shot rewrite, runtime + receipt honesty New test/skillopt/rollout.test.ts (rollout had zero coverage). Held-out ENFORCE unit cases + one-shot-rewrite fence handling (whole-response unwrap, embedded-fence preserved, error path). E2E: F11 held-out BLOCKS/ALLOWS, bundled no-mutate write, reflectMode/disableValidationGate/optimizerMode, maxRuntimeMin abort, receipt baseline/test-score honesty, held-out/benchmark disjointness, D2 no-DB-pollution. * chore: bump version and changelog (v0.42.9.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: document skillopt held-out gate + bundled mutation requirement for v0.42.9.0 Wire --held-out into the skill-optimizer SKILL.md, guide flags/safety tables, and the tutorial's bundled-skill step: mutating a bundled skill in place now requires --allow-mutate-bundled AND --held-out (>=5 benchmark-disjoint tasks) or it hard-refuses. Add the --held-out flag row + F11 held-out gate to the guide; update the receipt contract to the honest baseline/test-score fields. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): AI SDK v6 toolLoop compat — multi-turn tool calls work again The ai@6.x bump tightened ModelMessage + tool-schema validation, which silently broke every multi-turn tool loop. Both `gbrain skillopt` rollouts and production background `subagent` jobs route through `chat()`/`toolLoop` and crashed the moment the model called a tool ("messages do not match the ModelMessage[] schema" / "schema is not a function"). Surfaced end-to-end by the SkillOpt real-LLM eval. Three fixes: - chat(): wrap tool defs with the SDK's `jsonSchema()` helper instead of a bare `{jsonSchema}` object (v6 asSchema() treated the bare object as a thunk and threw). - chat(): new exported pure `toModelMessages()` converts gbrain's provider-neutral ChatMessage[] into v6 ModelMessage[] — tool results ride a dedicated `role:'tool'` message with structured `{type,value}` output; null output preserved as json null. Load-bearing for the production subagent path, not just skillopt. - rollout.ts: replace the inline params→schema mapper (dropped `items` on array params) with the shared `paramDefToSchema` single source of truth. Pinned by test/gateway-model-messages.test.ts (8 cases). Folds into the open v0.42.9.0 PR (#1759) — these complete the eval-readiness wave by making skillopt actually run against a live model. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(skillopt): budget no-pricing for Haiku silently scored every rollout 0 Surfaced by the SkillOpt real-LLM eval (Track B). Two coupled bugs that made a budget-capped Haiku run report a vacuous "0/N" measurement in ~2ms with zero LLM calls — indistinguishable from a real deficient-skill score: 1. Claude Haiku 4.5's canonical dateless id (`claude-haiku-4-5`) was missing from anthropic-pricing.ts (only the dated `-20251001` was present). With `--max-cost` set, BudgetTracker.reserve() threw no_pricing on the FIRST chat() of every rollout. Added the dateless entry (sonnet already had its dateless form). 2. runValidationGate swallowed that BUDGET_EXHAUSTED error — runWithLimit settled it as {ok:false}, which the gate turned into median:0. A pricing/cap crash became a fake score. The gate now scans settled results for isMustAbortError() and re-throws so the caller aborts loudly; ordinary (non-abort) rollout errors still fail-open to 0 (judge-hiccup posture kept). Pinned by test/skillopt/validate-gate-abort.test.ts (3 cases). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): llms-full.txt over size budget — drop what-schemas-unlock from full bundle The toolLoop + budget bug-fix annotations grew CLAUDE.md, pushing llms-full.txt to 756KB over the 750KB FULL_SIZE_BUDGET (the `build-llms > size budget` test failed, failing the `test` CI job). CLAUDE.md stays inlined by design (it's the point of the one-fetch bundle), so per the budget comment's own guidance ("ship with includeInFull=false exclusions") this excludes docs/what-schemas-unlock.md (15.4KB value-explainer, not load-bearing operational reference) from llms-full.txt; it stays linked in llms.txt. Bundle now 740KB with ~9KB headroom. No budget bump — 750KB is near the ~190k-token-context fit ceiling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(ci): re-admit policy docs into ci-cache-hash before doc relocation docs/**/*.md is deny-listed from the CI cache hash (test-irrelevant). The CLAUDE.md restructure moves test/release POLICY into docs/TESTING.md + docs/RELEASING.md, which DO carry contracts the test suite reads. Without re-admitting them, a policy-only edit would produce the same cache hash and skip the test shard that runs the build-llms + doc-history guards (false-pass). Adds an ALLOW_PATTERNS re-admit step after the deny, scoped to the named policy docs (not a blanket docs un-deny). Lands FIRST, before any doc moves. Pinned by 3 new cases in test/scripts/ci-cache-hash.test.ts: TESTING.md + RELEASING.md edits MUST change the hash; docs/guide.md still must not. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate Key files / thin-client / Testing out of CLAUDE.md (verbatim) CLAUDE.md had grown to 592KB / ~147k tokens auto-loaded every session (~77% of the llms-full.txt single-fetch bundle). The per-file index was append-only by mandate. This is the exact thin-dispatcher-vs-fat-blob anti-pattern gbrain exists to fix, so CLAUDE.md becomes a thin orientation + resolver that points at on-demand docs. This commit is the VERBATIM move (content-preserving — the next commit compresses): - docs/architecture/KEY_FILES.md <- ## Key files + the calibration key-files cluster + Schema Cathedral v3 impl detail - docs/architecture/thin-client.md <- ## Thin-client routing - docs/TESTING.md <- ## Testing - ## Commands DROPPED (18 'added in vX.Y' history blocks; current surface is gbrain 0.41.38.0 -- personal knowledge brain USAGE gbrain <command> [options] SETUP init [--pglite|--supabase|--url] Create brain (PGLite default, no server) migrate --to <supabase|pglite> Transfer brain between engines upgrade Self-update check-update [--json] Check for new versions doctor [--json] [--fast] Health check (resolver, skills, pgvector, RLS, embeddings) integrations [subcommand] Manage integration recipes (senses + reflexes) PAGES get <slug> Read a page put <slug> [< file.md] Write/update a page delete <slug> Delete a page list [--type T] [--tag T] [-n N] List pages SEARCH search <query> Keyword search (tsvector) query <question> [--no-expand] Hybrid search (RRF + expansion) ask <question> [--no-expand] Alias for query IMPORT/EXPORT import <dir> [--no-embed] Import markdown directory sync [--repo <path>] [flags] Git-to-brain incremental sync sync --watch [--interval N] Continuous sync (loops until stopped) sync --install-cron Install persistent sync daemon export [--dir ./out/] Export to markdown export --restore-only [--repo <p>] Restore missing supabase-only files [--type T] [--slug-prefix S] With optional filters FILES files list [slug] List stored files files upload <file> --page <slug> Upload file to storage files upload-raw <file> --page <s> Smart upload (size routing + .redirect.yaml) files signed-url <path> Generate signed URL (1-hour) files sync <dir> Bulk upload directory files verify Verify all uploads EMBEDDINGS embed [<slug>|--all|--stale] Generate/refresh embeddings LINKS link <from> <to> [--type T] Create typed link unlink <from> <to> Remove link backlinks <slug> Incoming links graph <slug> [--depth N] Traverse link graph (returns nodes) graph-query <slug> [--type T] Edge-based traversal with type/direction filters [--depth N] [--direction in|out|both] TAGS tags <slug> List tags tag <slug> <tag> Add tag untag <slug> <tag> Remove tag TIMELINE timeline [<slug>] View timeline timeline-add <slug> <date> <text> Add timeline entry TOOLS extract <links|timeline|all> Extract links/timeline (idempotent) [--source fs|db] fs (default) walks .md files; db iterates engine pages [--dir <brain>] brain dir for fs source [--type T] [--since DATE] filters (db source) [--dry-run] [--json] publish <page.md> [--password] Shareable HTML (strips private data, optional AES-256) check-backlinks <check|fix> [dir] Find/fix missing back-links across brain lint <dir|file> [--fix] Catch LLM artifacts, placeholder dates, bad frontmatter orphans [--json] [--count] Find pages with no inbound wikilinks salience [--days N] [--kind P] v0.29: pages ranked by emotional + activity salience anomalies [--since D] [--sigma N] v0.29: cohort-based statistical anomalies (tag, type) transcripts recent [--days N] v0.29: recent raw .txt transcripts (local-only) dream [--dry-run] [--json] Run the overnight maintenance cycle once (cron-friendly). See also: autopilot --install (continuous daemon). check-resolvable [--json] [--fix] Validate skill tree (reachability/MECE/DRY) report --type <name> --content ... Save timestamped report to brain/reports/ BRAIN (capture / ideate / explore — v0.37/v0.38) capture [content] [--file PATH] Single entrypoint for getting content into the brain [--stdin] [--slug s] [--type t] Inline content / file / stdin; writes to inbox/ by default [--source ID] [--quiet|--json] Multi-source brains: route to a non-default source brainstorm <question> [--json] Bisociation idea generator (hybrid search + far-set + judge) [--save|--no-save] [--limit N] lsd <question> [--json] Lateral Synaptic Drift: inverted-judge brainstorm [--save|--no-save] [--limit N] rewarding far-from-obvious + axiomatic inversions SOURCES (multi-repo / multi-brain) sources list Show registered sources sources add <id> --path <p> Register a source (id = short name, e.g. 'wiki') sources remove <id> Remove a source + its pages sync --all Sync all sources with a local_path sync --source <id> Sync one specific source repos ... DEPRECATED alias for 'sources' (v0.19.0) CODE INDEXING (v0.19.0 / v0.20.0 Cathedral II) code-def <symbol> [--lang l] Find the definition of a symbol across code pages code-refs <symbol> [--lang l] Find all references to a symbol (JSON-first) code-callers <symbol> Who calls this symbol? (v0.20.0 A1) code-callees <symbol> What does this symbol call? (v0.20.0 A1) query <q> --lang <l> Filter hybrid search to one language (v0.20.0) query <q> --symbol-kind <k> Filter to symbol type (function|class|method|...) (v0.20.0) reconcile-links [--dry-run] Batch-recompute doc↔impl edges (v0.20.0) reindex-code [--source id] [--yes] Explicit code-page reindex (v0.20.0) sync --strategy code Sync code files into the brain JOBS (Minions) jobs submit <name> [--params JSON] Submit background job [--follow] [--dry-run] jobs list [--status S] [--limit N] List jobs jobs get <id> Job details + history jobs cancel <id> Cancel job jobs retry <id> Re-queue failed/dead job jobs prune [--older-than 30d] Clean old jobs jobs stats Job health dashboard jobs work [--queue Q] Start worker daemon (Postgres only) ADMIN stats Brain statistics health Brain health dashboard history <slug> Page version history revert <slug> <version-id> Revert to version features [--json] [--auto-fix] Scan usage + recommend unused features autopilot [--repo] [--interval N] Self-maintaining brain daemon config [show|get|set] <key> [val] Brain config storage status [--repo <path>] Storage tier status and health [--json] (git-tracked vs supabase-only) serve MCP server (stdio) serve --http [--port N] HTTP MCP server with OAuth 2.1 --token-ttl N Access token TTL in seconds (default: 3600) --enable-dcr Enable Dynamic Client Registration --public-url URL Public issuer URL (required behind proxy/tunnel) call <tool> '<json>' Raw tool invocation version Version info --tools-json Tool discovery (JSON) Run gbrain <command> --help for command-specific help. + the per-command KEY_FILES entries; content stays in git) CLAUDE.md gains: a Reference map (resolver), a Maintaining section (the anti-disease rule), and a Cross-cutting invariants subsection under Architecture so the must-never-violate rules (trust fail-closed, sourceScopeOpts isolation, JSONB trap, engine parity, contract-first, migrations, multi-source) still auto-load after the index moved out. Result: CLAUDE.md 592KB -> 61KB; llms-full.txt 740KB -> 210KB (new docs link-only until compressed). build-llms drift + budget test green; verify 29/29 green. The pre-move content is recoverable at git show <this^>:CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): compress relocated docs to current-state + add recurrence guard Compresses the verbatim-relocated reference docs from append-only release-history to current-state-only (the disease cure), then makes recurrence structurally impossible via a CI guard. Compression (fan-out subagents + adversarial verify, audited mechanically): - KEY_FILES.md 453KB -> 356KB; TESTING.md 42KB -> 38KB; thin-client.md already clean. - 393/393 entries preserved; every src/test/scripts path from the verbatim original survives (mechanical comm-check); zero bolded **v0. markers remain. - Conservative ratio (~22%) because the content is invariant-dense — correctness over brevity. Dropped: **vX.Y.Z (#NNN):** clauses, codex/review tags, contributor credits, PR-numbers-as-ids, pre-fix/then/was-now history deltas. Kept: every exported symbol, invariant, and Pinned-by reference. Verbatim original recoverable at git show <relocation-commit>:docs/architecture/KEY_FILES.md. Recurrence guard (scripts/check-key-files-current-state.sh, wired into verify + check:all): - HARD: bans the bolded **v0.<digit> marker in the reference docs (scoped — plain 'as of pgvector 0.7' prose is fine, no false positives). - HARD: CLAUDE.md size cap (90KB; currently 61KB) — the structural backstop. - Pinned by test/scripts/check-key-files-current-state.test.ts (7 cases). Content contracts (test/build-llms.test.ts, +5 cases per codex outside-voice): CLAUDE.md keeps inline ship IRON RULES (version format, document-release, never-hand-roll); AGENTS.md keeps its boot order; llms indexes the new docs; KEY_FILES stays link-only (not inlined). Privacy: scrubbed the relocated 'wintermute/chat/' source-boost examples + the literal harvest-lint regex to generic placeholders (legitimate in allowlisted CLAUDE.md; genericized for the new public docs per the privacy rule). Reverts the 284c50a4 band-aid: re-inlines docs/what-schemas-unlock.md now that the restructure freed ~530KB of bundle headroom (llms-full.txt 740KB -> 225KB). verify 30/30 green (incl. new check:doc-history). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(docs): relocate verbose release process to docs/RELEASING.md The highest-/ship-risk commit (isolated so it can revert alone). Moves the verbose release + contributor procedure out of CLAUDE.md, keeping every ship-critical IRON RULE inline so /ship + /document-release (which read CLAUDE.md) cannot regress. Moved to docs/RELEASING.md: pre-ship test requirements; the CHANGELOG-branch-scoped + CHANGELOG voice + release-summary template; the 'To take advantage of vX' block spec; version migrations + migration-is-canonical; schema state tracking; GitHub Actions SHA maintenance; PR-descriptions-cover-the-branch; community-PR-wave; checking-out-PRs-from-garrytan-agents. Kept INLINE in CLAUDE.md (ship-critical IRON RULES — do NOT move): - the Version-locations table (5-file sync) + the 3-line consistency audit - Conductor branch=workspace - Post-ship /document-release (MANDATORY) - Privacy + Responsible-disclosure rules (Privacy also anchors the check-privacy allowlist — the only place allowed to name the fork) - PR-title-version-first - never-hand-roll-ship (Skill routing) Plus a new ## Releasing pointer ('Before any ship, read docs/RELEASING.md in full') and a resolver row. CLAUDE.md 61KB -> 39KB (592KB -> 39KB overall, 93% cut; ~9k tokens auto-loaded vs ~147k). CLAUDE.md size-gate tightened 90KB -> 60KB. The content-contract tests pin that the inline IRON RULES (MAJOR.MINOR.PATCH.MICRO, document-release, hand-roll ship) did NOT move out. The moved ranges carry no banned fork name, so RELEASING.md needs no privacy allowlist entry. verify 30/30; bundle 225KB -> 204KB. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(changelog): note CLAUDE.md restructure in v0.42.9.0 The CLAUDE.md thin-resolver restructure (592KB → 39KB) rides in this release; record it under the existing v0.42.9.0 For-contributors section. No version bump — v0.42.9.0 is unreleased and already allocated to this PR. * fix(ci): ci-cache-hash re-admit matched a literal \t, a no-op on GNU grep The policy-doc re-admit (75992b77) put `\t` inline in the ALLOW patterns passed to `grep -E`. BSD grep (macOS local) treats `\t` as a tab so it worked locally; GNU grep (Ubuntu CI) treats it as literal `t`, so nothing re-admitted and docs/TESTING.md / docs/RELEASING.md stayed deny-listed — the two policy-doc tests failed on CI shard 6 (1097 pass / 2 fail). Build ALLOW_RE with `printf '\t(%s)'` so the tab is a real byte, identical in construction to DENY_RE (line 117), which the CI log shows matches correctly on GNU grep. End-to-end: editing docs/TESTING.md now flips the hash; a normal docs/*.md add still does not (deny stays scoped). * fix(skillopt): feed the scorer's success criteria to the optimizer Surfaced by the SkillOpt real-LLM eval (Track B). The reflect step was shown only a pass/fail score and the agent transcript — never WHAT the benchmark judge rewards. On a skill judged by structure (e.g. "must include a Confidence: line") the optimizer proposed plausible-but-off edits ("close with a synthesis") that never satisfied the literal check; every candidate scored 0 on D_sel, the validation gate rejected them all, and the skill text never changed (optimized === baseline === 0). Fix: render each benchmark Judge (rule checks / llm rubric / qrels) into plain-English criteria via new exported describeJudge / describeJudges, and thread them into the reflect prompt (a SUCCESS CRITERIA block) for both the loop reflect calls and the one-shot-rewrite path. The orchestrator computes the distinct criteria across train+sel+test once. The optimizer system prompt now instructs it to satisfy the criteria through genuine content, never empty keywords — reward-hacking stays defended by the independent held-out gate (cat32 confirms the gate catches a keyword-stuffing hack). End-to-end this took a deficient skill from 0.00 to 1.00 on a held-out set it never trained on. Pinned by test/skillopt/reflect.test.ts (describeJudge per kind, describeJudges dedup, criteria present/absent in the prompt). Folds into the open v0.42.9.0 PR (#1759). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-27 22:15:33 +00:00 · 2026-06-03 05:50:21 -07:00
parent f09f9177a9
commit d4211f4176
44 changed files with 3472 additions and 3022 deletions
@@ -96,3 +96,52 @@ describe("build-llms generator", () => {
    ).toBeLessThan(FULL_SIZE_BUDGET);
  });
 });
+
+// Content contracts for the CLAUDE.md resolver restructure. The restructure moved
+// the per-file index + testing discipline into on-demand docs and (per codex
+// outside-voice) keeps the ship-critical IRON RULES inline. These pin that the
+// safety-relevant content did NOT silently move out of CLAUDE.md, and that the
+// new docs are wired into the bundle the way intended (KEY_FILES link-only,
+// not inlined).
+describe("CLAUDE.md restructure content contracts", () => {
+  const claude = () => readFileSync(join(repoRoot, "CLAUDE.md"), "utf8");
+
+  test("CLAUDE.md keeps the inline ship IRON RULES (must NOT move to a doc)", () => {
+    const c = claude();
+    // Version format — the table stays inline (CI version-gate depends on it).
+    expect(c).toContain("MAJOR.MINOR.PATCH.MICRO");
+    // Post-ship discipline — /document-release stays referenced inline.
+    expect(c.toLowerCase()).toContain("document-release");
+    // Never hand-roll ship.
+    expect(c.toLowerCase()).toMatch(/hand-roll ship/);
+  });
+
+  test("CLAUDE.md carries the resolver + cross-cutting invariants (orientation survived)", () => {
+    const c = claude();
+    expect(c).toContain("## Reference map");
+    expect(c).toContain("docs/architecture/KEY_FILES.md");
+    // Invariants that used to live in the per-file index now load here.
+    expect(c).toContain("sourceScopeOpts");
+    expect(c).toContain("JSON.stringify"); // the JSONB trap rule
+  });
+
+  test("AGENTS.md keeps its boot order + points at the new docs (not reduced to a pointer)", () => {
+    const agents = readFileSync(join(repoRoot, "AGENTS.md"), "utf8");
+    expect(agents).toContain("Read this order");
+    expect(agents).toContain("docs/architecture/KEY_FILES.md");
+    expect(agents).toContain("docs/TESTING.md");
+  });
+
+  test("llms.txt indexes the relocated docs", () => {
+    const { llmsTxt } = buildLlmsFiles();
+    expect(llmsTxt).toContain("docs/architecture/KEY_FILES.md");
+    expect(llmsTxt).toContain("docs/TESTING.md");
+    expect(llmsTxt).toContain("docs/architecture/thin-client.md");
+  });
+
+  test("llms-full.txt does NOT inline KEY_FILES.md (link-only; keeps the bundle lean)", () => {
+    const { llmsFullTxt } = buildLlmsFiles();
+    // This H1 appears only in KEY_FILES.md; if it were inlined the bundle would carry it.
+    expect(llmsFullTxt).not.toContain("# Key files — per-file index (gbrain repo)");
+  });
+});
@@ -134,7 +134,30 @@ interface Fixture {
  cleanup: () => void;
 }

-function setupFixture(skillBody: string = SKILL_PEOPLE_ONLY): Fixture {
+/** 50 tasks all checking `contains: Citations` — baseline People-only fails them all. */
+const CITATIONS_BENCHMARK = Array.from({ length: 50 }, (_, i) => {
+  const n = String(i + 1).padStart(3, '0');
+  return {
+    task_id: `cit-${n}`,
+    task: `Process task ${i + 1}`,
+    judge: { kind: 'rule' as const, checks: [{ op: 'contains' as const, arg: 'Citations' }] },
+  };
+});
+
+/** Held-out set checking `contains: People` — baseline passes, a People-dropping candidate fails. */
+const PEOPLE_HELDOUT = Array.from({ length: 6 }, (_, i) => {
+  const n = String(i + 1).padStart(3, '0');
+  return {
+    task_id: `ho-${n}`,
+    task: `Held-out task ${i + 1}`,
+    judge: { kind: 'rule' as const, checks: [{ op: 'contains' as const, arg: 'People' }] },
+  };
+});
+
+function setupFixture(
+  skillBody: string = SKILL_PEOPLE_ONLY,
+  benchmark: ReadonlyArray<{ task_id: string; task: string; judge: unknown }> = SAMPLE_BENCHMARK,
+): Fixture {
  const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'skillopt-loop-e2e-'));
  const skillDir = path.join(tmp, SKILL);
  fs.mkdirSync(skillDir, { recursive: true });
@@ -142,7 +165,7 @@ function setupFixture(skillBody: string = SKILL_PEOPLE_ONLY): Fixture {
  const benchmarkPath = path.join(skillDir, 'skillopt-benchmark.jsonl');
  fs.writeFileSync(
    benchmarkPath,
-    SAMPLE_BENCHMARK.map((t) => JSON.stringify(t)).join('\n') + '\n',
+    benchmark.map((t) => JSON.stringify(t)).join('\n') + '\n',
  );
  return {
    skillsDir: tmp,
@@ -153,6 +176,13 @@ function setupFixture(skillBody: string = SKILL_PEOPLE_ONLY): Fixture {
  };
 }

+/** Write a held-out JSONL into the fixture's skill dir; return its path. */
+function writeHeldOut(fixture: Fixture, tasks: ReadonlyArray<{ task_id: string; task: string; judge: unknown }>): string {
+  const p = path.join(fixture.skillsDir, SKILL, 'held-out.jsonl');
+  fs.writeFileSync(p, tasks.map((t) => JSON.stringify(t)).join('\n') + '\n');
+  return p;
+}
+
 // ─── Stub builder ───────────────────────────────────────────────────────────

 interface StubOpts {
@@ -176,10 +206,17 @@ interface StubOpts {
   * fast against a tight cap).
   */
  perCallUsage?: { input: number; output: number };
+  /** Raw body returned for ONE-SHOT REWRITE optimizer calls (optimizerMode test). */
+  oneShotBody?: string;
+  /** Counters incremented as the stub observes each call kind (ablation tests). */
+  stats?: { successReflectCalls: number; oneShotCalls: number };
 }

-const REFLECT_OPTIMIZER_PREFIX = "You are SkillOpt's optimizer.";
+// No trailing period: matches the FAILURE/SUCCESS reflect systems ("...optimizer.")
+// AND the one-shot system ("...optimizer in ONE-SHOT REWRITE mode.").
+const REFLECT_OPTIMIZER_PREFIX = "You are SkillOpt's optimizer";
 const FAILURE_REFLECT_MARKER = 'FAILURE TRAJECTORIES';
+const ONE_SHOT_MARKER = 'ONE-SHOT REWRITE';

 function defaultTargetText(skillText: string): string {
  // Faithful agent: read the skill's body, emit sections that exist there.
@@ -219,10 +256,16 @@ function installStub(opts: StubOpts): void {

    if (isOptimizerCall) {
      const model = chatOpts.model ?? 'anthropic:claude-opus-4-7';
+      // ONE-SHOT REWRITE mode returns a raw body, not edits JSON.
+      if (sys.includes(ONE_SHOT_MARKER)) {
+        if (opts.stats) opts.stats.oneShotCalls += 1;
+        return makeChatResult(opts.oneShotBody ?? '', model, usage);
+      }
      if (opts.optimizerRaw !== undefined) {
        return makeChatResult(opts.optimizerRaw, model, usage);
      }
      const isFailureMode = sys.includes(FAILURE_REFLECT_MARKER);
+      if (!isFailureMode && opts.stats) opts.stats.successReflectCalls += 1;
      const edit = isFailureMode ? opts.failureEdit : opts.successEdit;
      const text = JSON.stringify({ edits: edit ? [edit] : [] });
      return makeChatResult(text, model, usage);
@@ -245,6 +288,12 @@ interface RunOptsOverride {
  maxCostUsd?: number;
  epochs?: number;
  batchSize?: number;
+  noMutate?: boolean;
+  heldOutPath?: string;
+  optimizerMode?: 'reflect' | 'one-shot-rewrite';
+  reflectMode?: 'both' | 'failure-only';
+  disableValidationGate?: boolean;
+  maxRuntimeMin?: number;
 }

 async function runOnce(fixture: Fixture, over: RunOptsOverride = {}) {
@@ -263,13 +312,17 @@ async function runOnce(fixture: Fixture, over: RunOptsOverride = {}) {
    judgeModel: 'anthropic:claude-sonnet-4-6',
    mode: 'patch',
    dryRun: false,
-    noMutate: false,
+    noMutate: over.noMutate ?? false,
    allowMutateBundled: true,
    bootstrapReviewed: false,
    json: true,
    maxCostUsd: over.maxCostUsd ?? 100,
-    maxRuntimeMin: 1,
+    maxRuntimeMin: over.maxRuntimeMin ?? 1,
    force: true, // bypass dirty-tree (tempdir isn't a git repo)
+    ...(over.heldOutPath ? { heldOutPath: over.heldOutPath } : {}),
+    ...(over.optimizerMode ? { optimizerMode: over.optimizerMode } : {}),
+    ...(over.reflectMode ? { reflectMode: over.reflectMode } : {}),
+    ...(over.disableValidationGate ? { disableValidationGate: over.disableValidationGate } : {}),
  });
 }

@@ -629,3 +682,267 @@ describe('skillopt full-loop E2E (happy path + broken cases)', () => {
    }
  });
 });
+
+// ─── T3: held-out gate + ablation opts + no-DB-pollution ─────────────────────
+
+describe('skillopt T3 — F11 held-out gate, ablation opts, no-DB-pollution', () => {
+  test('F11 held-out BLOCKS: candidate passes D_sel but regresses held-out → no commit', async () => {
+    // baseline People-only fails the Citations benchmark; the failure edit
+    // REPLACES People with Citations → candidate passes D_sel (Citations) but
+    // tanks the held-out (People) → held-out gate refuses the commit.
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    const heldOutPath = writeHeldOut(fixture, PEOPLE_HELDOUT);
+    try {
+      installStub({
+        failureEdit: {
+          op: 'replace',
+          target: '## People\nList people mentioned.',
+          replacement: '## Citations\nCite the source.',
+          reason: 'swap People for Citations to pass the benchmark',
+        },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const result = await runOnce(fixture, { heldOutPath });
+          // Held-out gate blocked the promotion.
+          expect(result.outcome).toBe('no_improvement');
+          // SKILL.md unchanged: still People-only, never swapped to Citations.
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).toContain('## People');
+          expect(skill).not.toContain('## Citations');
+          // No committed history row.
+          expect(loadHistory(fixture.skillsDir, SKILL).filter((r) => r.status === 'committed')).toHaveLength(0);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('F11 held-out ALLOWS: candidate improves D_sel AND holds held-out → commit', async () => {
+    // ADD Citations (keep People): passes D_sel (Citations) and keeps held-out
+    // (People) at baseline → held-out gate allows → commit.
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    const heldOutPath = writeHeldOut(fixture, PEOPLE_HELDOUT);
+    try {
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite the source.', reason: 'add Citations' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const result = await runOnce(fixture, { heldOutPath });
+          expect(result.outcome).toBe('accepted');
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).toContain('## People');
+          expect(skill).toContain('## Citations');
+          expect(loadHistory(fixture.skillsDir, SKILL).filter((r) => r.status === 'committed')).toHaveLength(1);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('--no-mutate writes proposed.md (best.md), leaves SKILL.md untouched', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    try {
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite the source.', reason: 'add Citations' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const result = await runOnce(fixture, { noMutate: true });
+          expect(result.outcome).toBe('accepted');
+          expect(result.mutatedSkillFile).toBe(false);
+          expect(result.proposedPath).toBeDefined();
+          // proposed.md (best.md) exists and carries the improvement.
+          expect(fs.existsSync(result.proposedPath!)).toBe(true);
+          expect(fs.readFileSync(result.proposedPath!, 'utf8')).toContain('## Citations');
+          // SKILL.md on disk is UNCHANGED (still People-only).
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).not.toContain('## Citations');
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('optimizerMode one-shot-rewrite: single rewrite, no epoch loop, receipt records mode', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    const stats = { successReflectCalls: 0, oneShotCalls: 0 };
+    try {
+      installStub({
+        stats,
+        // Body-only rewrite (frontmatter re-attached by the orchestrator).
+        oneShotBody: '# E2E Loop Test Skill\n\nProduce a structured output.\n\n## People\nList people.\n\n## Citations\nCite the source.\n',
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const result = await runOnce(fixture, { optimizerMode: 'one-shot-rewrite' });
+          expect(result.outcome).toBe('accepted');
+          expect(result.receipt.optimizer_mode).toBe('one-shot-rewrite');
+          // Exactly ONE optimizer rewrite call — no epoch loop.
+          expect(stats.oneShotCalls).toBe(1);
+          expect(result.receipt.total_steps).toBe(1);
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).toContain('## Citations');
+          // Frontmatter preserved (D5).
+          expect(skill).toContain('name: e2e-loop-skill');
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('reflectMode failure-only SKIPS the success reflect call; default fires it', async () => {
+    // Mixed benchmark → baseline has both successes (People tasks) and failures
+    // (Citations tasks), so default mode WOULD fire the success reflect.
+    const failOnly = { successReflectCalls: 0, oneShotCalls: 0 };
+    const fixtureA = setupFixture(SKILL_PEOPLE_ONLY, SAMPLE_BENCHMARK);
+    try {
+      installStub({
+        stats: failOnly,
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite.', reason: 'add' },
+        successEdit: { op: 'add', anchor: 'People', content: '<!-- success note -->', reason: 'note' },
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixtureA.skillsDir }, async () => {
+          await runOnce(fixtureA, { reflectMode: 'failure-only' });
+          expect(failOnly.successReflectCalls).toBe(0);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixtureA.cleanup(); }
+
+    const both = { successReflectCalls: 0, oneShotCalls: 0 };
+    const fixtureB = setupFixture(SKILL_PEOPLE_ONLY, SAMPLE_BENCHMARK);
+    try {
+      installStub({
+        stats: both,
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite.', reason: 'add' },
+        successEdit: { op: 'add', anchor: 'People', content: '<!-- success note -->', reason: 'note' },
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixtureB.skillsDir }, async () => {
+          await runOnce(fixtureB);
+          expect(both.successReflectCalls).toBeGreaterThan(0);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixtureB.cleanup(); }
+  });
+
+  test('disableValidationGate greedy-accepts a no-improvement edit the gate would reject', async () => {
+    // BOTH_SECTIONS already scores 1.0; a benign success edit yields delta 0,
+    // which the D12 gate rejects — unless disableValidationGate greedy-accepts.
+    const benignEdit = { op: 'add' as const, anchor: 'Citations', content: 'Extra note.', reason: 'benign' };
+
+    const fixtureGated = setupFixture(SKILL_BOTH_SECTIONS, SAMPLE_BENCHMARK);
+    try {
+      installStub({ successEdit: benignEdit, failureEdit: null });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixtureGated.skillsDir }, async () => {
+          const result = await runOnce(fixtureGated);
+          expect(result.outcome).toBe('no_improvement'); // gate rejected
+        });
+      } finally { uninstallStub(); }
+    } finally { fixtureGated.cleanup(); }
+
+    const fixtureGreedy = setupFixture(SKILL_BOTH_SECTIONS, SAMPLE_BENCHMARK);
+    try {
+      installStub({ successEdit: benignEdit, failureEdit: null });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixtureGreedy.skillsDir }, async () => {
+          const result = await runOnce(fixtureGreedy, { disableValidationGate: true });
+          expect(result.outcome).toBe('accepted'); // greedy
+          expect(result.receipt.validation_gate_disabled).toBe(true);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixtureGreedy.cleanup(); }
+  });
+
+  test('D2 no-DB-pollution: subagent_messages count unchanged across a full run', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    const countMessages = async (): Promise<number> => {
+      const r = await engine.executeRaw('SELECT COUNT(*) AS c FROM subagent_messages', []);
+      const rows = Array.isArray(r) ? r : ((r as { rows?: unknown[] }).rows ?? []);
+      return Number((rows[0] as { c?: number | string })?.c ?? 0);
+    };
+    try {
+      const before = await countMessages();
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite the source.', reason: 'add' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          await runOnce(fixture);
+        });
+      } finally { uninstallStub(); }
+      const after = await countMessages();
+      // Rollouts use gateway.toolLoop with no-op persistence (D2) → zero rows written.
+      expect(after).toBe(before);
+    } finally { fixture.cleanup(); }
+  });
+});
+
+// ─── T3 (review follow-ups): maxRuntimeMin abort + receipt honesty ───────────
+
+describe('skillopt T3 — runtime deadline + receipt score honesty', () => {
+  test('maxRuntimeMin deadline aborts cleanly with no commit', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    try {
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite the source.', reason: 'add' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          // maxRuntimeMin:0 → deadline == run start; the first step's deadline
+          // check fires after the baseline eval has already elapsed → abort.
+          const result = await runOnce(fixture, { maxRuntimeMin: 0 });
+          expect(result.outcome).toBe('aborted');
+          // No commit: SKILL.md unchanged, no committed history row.
+          const skill = fs.readFileSync(skillPath(fixture.skillsDir, SKILL), 'utf8');
+          expect(skill).not.toContain('## Citations');
+          expect(loadHistory(fixture.skillsDir, SKILL).filter((r) => r.status === 'committed')).toHaveLength(0);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+
+  test('receipt records the REAL baseline_sel_score + final-test scores (regression: was hardcoded 0)', async () => {
+    // BOTH_SECTIONS already scores ~1.0 on the alternating benchmark, so a real
+    // baseline read is ~1.0 — a hardcoded-0 receipt would fail this immediately.
+    const fixture = setupFixture(SKILL_BOTH_SECTIONS, SAMPLE_BENCHMARK);
+    try {
+      installStub({ successEdit: null, failureEdit: null }); // baseline already perfect; no edits
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          const r = (await runOnce(fixture)).receipt;
+          // Real baseline, not the old hardcoded 0.
+          expect(r.baseline_sel_score).toBeGreaterThan(0.9);
+          // Final-test eval populated both test scores (D_test non-empty under 4:1:5).
+          expect(typeof r.test_score).toBe('number');
+          expect(typeof r.baseline_test_score).toBe('number');
+          expect(r.baseline_test_score!).toBeGreaterThan(0.9);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+});
+
+describe('skillopt T3 — held-out independence guard', () => {
+  test('held-out sharing task_ids with the benchmark is rejected (gaming defense)', async () => {
+    const fixture = setupFixture(SKILL_PEOPLE_ONLY, CITATIONS_BENCHMARK);
+    // Held-out file reuses benchmark task_ids (cit-001..006) → must be rejected
+    // before any optimization, since an overlapping held-out can't catch overfit.
+    const heldOutPath = writeHeldOut(fixture, CITATIONS_BENCHMARK.slice(0, 6));
+    try {
+      installStub({
+        failureEdit: { op: 'add', anchor: 'People', content: '## Citations\nCite.', reason: 'add' },
+        successEdit: null,
+      });
+      try {
+        await withEnv({ GBRAIN_AUDIT_DIR: fixture.skillsDir }, async () => {
+          await expect(runOnce(fixture, { heldOutPath })).rejects.toThrow(/independent|shares .* task_id/i);
+        });
+      } finally { uninstallStub(); }
+    } finally { fixture.cleanup(); }
+  });
+});
@@ -0,0 +1,123 @@
+/**
+ * Pins `toModelMessages` — the gbrain ChatMessage[] → AI SDK v6 ModelMessage[]
+ * converter. v6 tightened ModelMessage validation: tool results must ride on a
+ * dedicated `role:'tool'` message with a structured `{type,value}` output part,
+ * not a `role:'user'` message with a bare-value tool-result block (which is how
+ * gbrain's toolLoop pushes them). Without this conversion every multi-turn tool
+ * loop — skillopt rollouts AND production subagent jobs — throws "messages do
+ * not match the ModelMessage[] schema" the moment the model calls a tool.
+ *
+ * Surfaced by the SkillOpt real-LLM eval (Track B). These cases pin the exact
+ * v6 shapes that `generateText` accepts (verified against AI SDK 6.0.174).
+ */
+import { describe, test, expect } from 'bun:test';
+import { toModelMessages, type ChatMessage } from '../src/core/ai/gateway.ts';
+
+describe('toModelMessages — v6 ModelMessage shape', () => {
+  test('string content passes through unchanged', () => {
+    const msgs: ChatMessage[] = [{ role: 'user', content: 'hello' }];
+    expect(toModelMessages(msgs)).toEqual([{ role: 'user', content: 'hello' }]);
+  });
+
+  test('assistant text block maps to {type:text,text}', () => {
+    const msgs: ChatMessage[] = [
+      { role: 'assistant', content: [{ type: 'text', text: 'hi' }] },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      { role: 'assistant', content: [{ type: 'text', text: 'hi' }] },
+    ]);
+  });
+
+  test('assistant tool-call block keeps {toolCallId,toolName,input}', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'assistant',
+        content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: { query: 'x' } }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'assistant',
+        content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: { query: 'x' } }],
+      },
+    ]);
+  });
+
+  test('tool-result on a user-role message becomes role:tool with json output', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'user',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { hits: 0 } }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'tool',
+        content: [
+          { type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { type: 'json', value: { hits: 0 } } },
+        ],
+      },
+    ]);
+  });
+
+  test('string tool-result output becomes {type:text,value}', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'user',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'echo', output: 'done' }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'tool',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'echo', output: { type: 'text', value: 'done' } }],
+      },
+    ]);
+  });
+
+  test('errored tool-result becomes {type:error-text,value}', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'user',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { msg: 'boom' }, isError: true }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'tool',
+        content: [
+          { type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { type: 'error-text', value: '{"msg":"boom"}' } },
+        ],
+      },
+    ]);
+  });
+
+  test('null tool-result output is preserved as json null (not dropped)', () => {
+    const msgs: ChatMessage[] = [
+      {
+        role: 'user',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'noop', output: null }],
+      },
+    ];
+    expect(toModelMessages(msgs)).toEqual([
+      {
+        role: 'tool',
+        content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'noop', output: { type: 'json', value: null } }],
+      },
+    ]);
+  });
+
+  test('full multi-turn conversation: user → assistant(tool-call) → tool(result)', () => {
+    const msgs: ChatMessage[] = [
+      { role: 'user', content: 'find widget' },
+      { role: 'assistant', content: [{ type: 'tool-call', toolCallId: 'c1', toolName: 'search', input: { query: 'widget' } }] },
+      { role: 'user', content: [{ type: 'tool-result', toolCallId: 'c1', toolName: 'search', output: { hits: 0 } }] },
+    ];
+    const out = toModelMessages(msgs);
+    expect(out).toHaveLength(3);
+    expect((out[0] as any).role).toBe('user');
+    expect((out[1] as any).role).toBe('assistant');
+    expect((out[2] as any).role).toBe('tool');
+    expect((out[2] as any).content[0].output).toEqual({ type: 'json', value: { hits: 0 } });
+  });
+});
@@ -0,0 +1,113 @@
+/**
+ * check-key-files-current-state.test.ts — coverage of the anti-disease guard.
+ *
+ * The guard is the structural backstop that keeps CLAUDE.md from re-bloating:
+ * it bans bolded `**v0.<digit>` release markers in the reference docs and caps
+ * CLAUDE.md size. If the guard is broken, the append-only-history disease can
+ * silently return. This suite pins its contract against fixtures.
+ */
+
+import { afterEach, beforeEach, describe, expect, it } from "bun:test";
+import { spawnSync } from "node:child_process";
+import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+
+const SCRIPT = resolve(import.meta.dir, "..", "..", "scripts/check-key-files-current-state.sh");
+
+let root: string;
+
+beforeEach(() => {
+  root = mkdtempSync(join(tmpdir(), "doc-guard-"));
+  mkdirSync(join(root, "docs/architecture"), { recursive: true });
+});
+afterEach(() => {
+  rmSync(root, { recursive: true, force: true });
+});
+
+function writeDoc(rel: string, content: string) {
+  const abs = join(root, rel);
+  mkdirSync(resolve(abs, ".."), { recursive: true });
+  writeFileSync(abs, content);
+}
+
+function run(extraEnv: Record<string, string> = {}) {
+  return spawnSync("bash", [SCRIPT], {
+    encoding: "utf8",
+    env: { ...process.env, GBRAIN_DOC_GUARD_ROOT: root, ...extraEnv },
+  });
+}
+
+// A minimal clean repo shape the guard is happy with.
+function seedClean() {
+  writeDoc("CLAUDE.md", "# CLAUDE.md\n\norientation only\n");
+  writeDoc(
+    "docs/architecture/KEY_FILES.md",
+    "# Key files\n\n- `src/core/db.ts` — connection management. Pinned by `test/db.test.ts`.\n",
+  );
+  writeDoc("docs/architecture/thin-client.md", "# Thin-client\n\nrouting seam\n");
+  writeDoc("docs/TESTING.md", "# Testing\n\ntiers\n");
+}
+
+describe("check-key-files-current-state.sh", () => {
+  it("passes on a clean current-state repo", () => {
+    seedClean();
+    const r = run();
+    expect(r.status).toBe(0);
+    expect(r.stdout).toContain("ok");
+  });
+
+  it("FAILS when a reference doc carries a bolded release-clause marker", () => {
+    seedClean();
+    writeDoc(
+      "docs/architecture/KEY_FILES.md",
+      "# Key files\n\n- `src/core/db.ts` — connection mgmt. **v0.41.2 (#9):** added pool reconnect.\n",
+    );
+    const r = run();
+    expect(r.status).toBe(1);
+    expect(r.stderr).toContain("bolded release-clause");
+    expect(r.stderr).toContain("KEY_FILES.md");
+  });
+
+  it("PASSES on a legitimate non-bolded version mention (no false positive)", () => {
+    seedClean();
+    writeDoc(
+      "docs/architecture/KEY_FILES.md",
+      "# Key files\n\n- `src/core/db.ts` — requires pgvector 0.7; on Postgres 11+ the ADD COLUMN is metadata-only.\n",
+    );
+    const r = run();
+    expect(r.status).toBe(0);
+  });
+
+  it("FAILS when CLAUDE.md exceeds the size cap", () => {
+    seedClean();
+    writeDoc("CLAUDE.md", "x".repeat(200_000));
+    const r = run({ GBRAIN_CLAUDE_MD_MAX_BYTES: "90000" });
+    expect(r.status).toBe(1);
+    expect(r.stderr).toContain("over the");
+  });
+
+  it("size cap is configurable via env", () => {
+    seedClean();
+    writeDoc("CLAUDE.md", "x".repeat(5_000));
+    expect(run({ GBRAIN_CLAUDE_MD_MAX_BYTES: "1000" }).status).toBe(1);
+    expect(run({ GBRAIN_CLAUDE_MD_MAX_BYTES: "10000" }).status).toBe(0);
+  });
+
+  it("soft-warns (non-fatal) on prose history markers", () => {
+    seedClean();
+    writeDoc(
+      "docs/TESTING.md",
+      "# Testing\n\nThe tier set, then v0.26.7 added the parallel loop (pre-fix it was serial).\n",
+    );
+    const r = run();
+    expect(r.status).toBe(0); // warn, not fail
+    expect(r.stderr).toContain("WARN");
+  });
+
+  it("catches the marker in any of the three reference docs (thin-client)", () => {
+    seedClean();
+    writeDoc("docs/architecture/thin-client.md", "# Thin-client\n\n**v0.36.3:** added cross-modal.\n");
+    expect(run().status).toBe(1);
+  });
+});
@@ -400,6 +400,50 @@ describe("ci-cache-hash.sh — edge cases", () => {
  });
 });

+describe("ci-cache-hash.sh — policy-doc re-admit (test-affecting docs under docs/)", () => {
+  // These docs live under docs/ (normally deny-listed) but carry CI/release/
+  // test contracts the suite reads. The re-admit must make edits to them
+  // invalidate the hash, WITHOUT un-denying ordinary docs.
+  const POLICY_FILES: Record<string, string> = {
+    ...BASELINE_FILES,
+    "docs/TESTING.md": "# Testing\n\ntest tiers + isolation lint\n",
+    "docs/RELEASING.md": "# Releasing\n\nversion locations + ship process\n",
+  };
+
+  function withSandbox(test: (sb: Sandbox) => void) {
+    const sb = makeSandbox(POLICY_FILES);
+    try {
+      test(sb);
+    } finally {
+      rmSync(sb.dir, { recursive: true, force: true });
+    }
+  }
+
+  it("docs/TESTING.md edit MUST change hash (re-admitted policy doc)", () => {
+    withSandbox((sb) => {
+      const before = hash(sb);
+      modify(sb, "docs/TESTING.md", "# Testing\n\nNEW POLICY\n");
+      expect(hash(sb)).not.toBe(before);
+    });
+  });
+
+  it("docs/RELEASING.md edit MUST change hash (re-admitted policy doc)", () => {
+    withSandbox((sb) => {
+      const before = hash(sb);
+      modify(sb, "docs/RELEASING.md", "# Releasing\n\nNEW SHIP RULE\n");
+      expect(hash(sb)).not.toBe(before);
+    });
+  });
+
+  it("docs/guide.md edit STILL produces same hash (re-admit is scoped, not a blanket docs un-deny)", () => {
+    withSandbox((sb) => {
+      const before = hash(sb);
+      modify(sb, "docs/guide.md", "# Guide v3\n");
+      expect(hash(sb)).toBe(before);
+    });
+  });
+});
+
 describe("ci-cache-hash.sh — usage errors", () => {
  it("--bogus arg exits 2", () => {
    const r = spawnSync("bash", [SCRIPT_SRC, "--bogus"], {
@@ -15,7 +15,9 @@ import {
  capturesDir,
  loadHeldOut,
  runHeldOutGate,
+  MIN_HELD_OUT_SIZE,
 } from '../../src/core/skillopt/held-out.ts';
+import { assertBundledMutationHeldOut } from '../../src/core/skillopt/bundled-skill-gate.ts';

 let tmp: string;

@@ -109,3 +111,35 @@ describe('F11 runHeldOutGate vacuous case', () => {
    expect(result.candidateScore).toBe(0);
  });
 });
+
+describe('D16 assertBundledMutationHeldOut (core ENFORCE)', () => {
+  test('bundled + mutate + too-small held-out → throws (hard refuse)', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: true, willMutate: true, heldOutCount: MIN_HELD_OUT_SIZE - 1, skillName: 'brain-ops',
+    })).toThrow(/held-out/i);
+  });
+
+  test('bundled + mutate + empty held-out → throws (vacuous-pass hole closed)', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: true, willMutate: true, heldOutCount: 0, skillName: 'brain-ops',
+    })).toThrow(/non-empty held-out/i);
+  });
+
+  test('bundled + mutate + sufficient held-out → no throw', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: true, willMutate: true, heldOutCount: MIN_HELD_OUT_SIZE, skillName: 'brain-ops',
+    })).not.toThrow();
+  });
+
+  test('not bundled → no throw even with zero held-out (user skills are free)', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: false, willMutate: true, heldOutCount: 0, skillName: 'my-skill',
+    })).not.toThrow();
+  });
+
+  test('bundled but NOT mutating (no-mutate / proposed.md path) → no throw', () => {
+    expect(() => assertBundledMutationHeldOut({
+      isBundled: true, willMutate: false, heldOutCount: 0, skillName: 'brain-ops',
+    })).not.toThrow();
+  });
+});
@@ -18,7 +18,7 @@
 */

 import { describe, expect, test } from 'bun:test';
-import { parseEditsResponse, runReflect } from '../../src/core/skillopt/reflect.ts';
+import { parseEditsResponse, runReflect, runOneShotRewrite, describeJudge, describeJudges } from '../../src/core/skillopt/reflect.ts';
 import type { ChatOpts, ChatResult } from '../../src/core/ai/gateway.ts';
 import type { ScoredRollout, Trajectory } from '../../src/core/skillopt/types.ts';

@@ -287,3 +287,113 @@ describe('runReflect (D7 two-call contract)', () => {
    expect(observedUserMsg).toContain('validation_gate_below_baseline');
  });
 });
+
+// ─── runOneShotRewrite (cat31 config C baseline) ────────────────────────────
+
+function oneShotChatStub(text: string): NonNullable<Parameters<typeof runOneShotRewrite>[0]['chatFn']> {
+  return (async (_opts: ChatOpts): Promise<ChatResult> => ({
+    text,
+    blocks: [{ type: 'text', text }],
+    stopReason: 'end',
+    usage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
+    model: 'm',
+    providerId: 'anthropic',
+  })) as NonNullable<Parameters<typeof runOneShotRewrite>[0]['chatFn']>;
+}
+
+const ONE_SHOT_BASE = { skillBodyText: '# Skill', successes: [] as ScoredRollout[], failures: [] as ScoredRollout[], rejected: [], optimizerModel: 'm' };
+
+describe('runOneShotRewrite', () => {
+  test('unwraps a whole-response code fence', async () => {
+    const body = '# Skill\n\n## People\nList people.';
+    const r = await runOneShotRewrite({ ...ONE_SHOT_BASE, chatFn: oneShotChatStub('```markdown\n' + body + '\n```') });
+    expect(r.newBody).toBe(body);
+    expect(r.error).toBeUndefined();
+  });
+
+  test('preserves an EMBEDDED code fence — does not truncate to the first block', async () => {
+    const body = '# Skill\n\nRun this:\n```bash\nls -la\n```\n\n## People\nList people.';
+    const r = await runOneShotRewrite({ ...ONE_SHOT_BASE, chatFn: oneShotChatStub(body) });
+    expect(r.newBody).toBe(body);
+    expect(r.newBody).toContain('## People'); // regression: old non-anchored regex truncated to the bash block
+  });
+
+  test('chat error returns empty newBody + error (caller treats as no-change)', async () => {
+    const r = await runOneShotRewrite({
+      ...ONE_SHOT_BASE,
+      chatFn: (async () => { throw new Error('boom'); }) as NonNullable<Parameters<typeof runOneShotRewrite>[0]['chatFn']>,
+    });
+    expect(r.newBody).toBe('');
+    expect(r.error).toContain('one_shot_rewrite_failed');
+  });
+});
+
+// ─── Success criteria threading (v0.42.9.0) ──────────────────────────────────
+// The optimizer must be TOLD how its output is scored, or it optimizes blind:
+// on a rule-judged benchmark it proposes plausible-but-off edits, every
+// candidate scores 0, the gate rejects them, and the skill never changes. These
+// pin that the judge criteria render to plain English AND reach the reflect prompt.
+describe('describeJudge / criteria threading', () => {
+  test('describeJudge renders each rule check as a plain requirement', () => {
+    const d = describeJudge({ kind: 'rule', checks: [
+      { op: 'section_present', arg: 'Key Risks' },
+      { op: 'regex', arg: '[Cc]onfidence\\s*[:=]' },
+      { op: 'max_chars', arg: 1200 },
+      { op: 'tool_called', arg: 'search' },
+    ] });
+    expect(d).toContain('Key Risks');
+    expect(d).toContain('[Cc]onfidence');
+    expect(d).toContain('1200');
+    expect(d).toContain('search');
+  });
+
+  test('describeJudge renders an llm rubric', () => {
+    expect(describeJudge({ kind: 'llm', rubric: 'reward genuine substance' }))
+      .toContain('reward genuine substance');
+  });
+
+  test('describeJudges dedupes identical judge shapes across tasks', () => {
+    const j = { kind: 'rule' as const, checks: [{ op: 'contains' as const, arg: 'X' }] };
+    const out = describeJudges([{ judge: j }, { judge: j }, { judge: j }]);
+    // One distinct shape → one block, not three.
+    expect(out.match(/contain the exact text/g)).toHaveLength(1);
+  });
+
+  test('runReflect injects the criteria block into the optimizer prompt', async () => {
+    let seenUser = '';
+    const chatFn = async (opts: ChatOpts): Promise<ChatResult> => {
+      const u = opts.messages[0]?.content;
+      seenUser = typeof u === 'string' ? u : '';
+      return makeChatResult(JSON.stringify({ edits: [] }));
+    };
+    await runReflect({
+      skillBodyText: '# Test',
+      successes: [],
+      failures: [makeScored('f-1', 0.0)],
+      rejected: [],
+      criteria: 'CRITERIA: must include a Confidence: line',
+      optimizerModel: 'anthropic:claude-opus-4-7',
+      chatFn,
+    });
+    expect(seenUser).toContain('SUCCESS CRITERIA');
+    expect(seenUser).toContain('must include a Confidence: line');
+  });
+
+  test('runReflect omits the criteria block when none is given', async () => {
+    let seenUser = '';
+    const chatFn = async (opts: ChatOpts): Promise<ChatResult> => {
+      const u = opts.messages[0]?.content;
+      seenUser = typeof u === 'string' ? u : '';
+      return makeChatResult(JSON.stringify({ edits: [] }));
+    };
+    await runReflect({
+      skillBodyText: '# Test',
+      successes: [],
+      failures: [makeScored('f-1', 0.0)],
+      rejected: [],
+      optimizerModel: 'anthropic:claude-opus-4-7',
+      chatFn,
+    });
+    expect(seenUser).not.toContain('SUCCESS CRITERIA');
+  });
+});
@@ -0,0 +1,123 @@
+/**
+ * SkillOpt rollout tests (the `toolLoopFn` DI seam — previously zero coverage).
+ *
+ * Covers: trajectory shape capture, READ_ONLY_BRAIN_TOOLS zero-write invariant,
+ * default read-only tool registry (no put_page/submit_job), --write-capture
+ * routing to the virtual write registry, and tool-call ordering via the
+ * onToolCallStart callback. Hermetic: the toolLoop transport is stubbed, so no
+ * LLM calls, no DB, no API keys. Handlers are never executed (the stub returns
+ * messages directly), so a `{}`-shaped engine is sufficient.
+ */
+
+import { describe, expect, test } from 'bun:test';
+import { runRollout, READ_ONLY_BRAIN_TOOLS } from '../../src/core/skillopt/rollout.ts';
+import type { BenchmarkTask } from '../../src/core/skillopt/types.ts';
+
+const TASK: BenchmarkTask = {
+  task_id: 't1',
+  task: 'do the thing',
+  judge: { kind: 'rule', checks: [] },
+};
+
+/** Build a stubbed toolLoop that records the tool defs it was handed. */
+function makeStubLoop(opts: { capturedTools?: Array<{ name: string }>; finalText?: string }) {
+  // eslint-disable-next-line @typescript-eslint/no-explicit-any
+  return async (loopOpts: any) => {
+    if (opts.capturedTools) opts.capturedTools.push(...loopOpts.tools);
+    return {
+      messages: [
+        { role: 'user', content: 'do the thing' },
+        { role: 'assistant', content: opts.finalText ?? 'done' },
+      ],
+      totalUsage: { input_tokens: 10, output_tokens: 5, cache_read_tokens: 0, cache_creation_tokens: 0 },
+      totalTurns: 2,
+      stopReason: 'end',
+    };
+  };
+}
+
+describe('rollout — READ_ONLY_BRAIN_TOOLS zero-write invariant', () => {
+  test('excludes put_page / submit_job / file_upload, includes a read op', () => {
+    expect(READ_ONLY_BRAIN_TOOLS.has('put_page')).toBe(false);
+    expect(READ_ONLY_BRAIN_TOOLS.has('submit_job')).toBe(false);
+    expect(READ_ONLY_BRAIN_TOOLS.has('file_upload')).toBe(false);
+    expect(READ_ONLY_BRAIN_TOOLS.has('search')).toBe(true);
+  });
+});
+
+describe('rollout — trajectory capture', () => {
+  test('returns a Trajectory with the expected shape + threaded usage/stop_reason', async () => {
+    const traj = await runRollout({
+      engine: {} as never,
+      skillText: 'skill body',
+      task: TASK,
+      targetModel: 'anthropic:claude-sonnet-4-6',
+      toolLoopFn: makeStubLoop({ finalText: 'the answer' }) as never,
+    });
+    expect(traj.task_id).toBe('t1');
+    expect(traj.task).toBe('do the thing');
+    expect(traj.final_text).toBe('the answer');
+    expect(traj.stop_reason).toBe('end');
+    expect(traj.turns).toBe(2);
+    expect(traj.usage.input_tokens).toBe(10);
+    expect(traj.usage.output_tokens).toBe(5);
+    expect(typeof traj.duration_ms).toBe('number');
+    expect(Array.isArray(traj.tool_calls)).toBe(true);
+  });
+});
+
+describe('rollout — tool registry routing', () => {
+  test('default rollout passes ONLY read-only tools (no write defs)', async () => {
+    const capturedTools: Array<{ name: string }> = [];
+    await runRollout({
+      engine: {} as never,
+      skillText: 's',
+      task: TASK,
+      targetModel: 'm',
+      toolLoopFn: makeStubLoop({ capturedTools }) as never,
+    });
+    const names = capturedTools.map((d) => d.name);
+    expect(names).toContain('brain_search');
+    expect(names).not.toContain('brain_put_page');
+    expect(names).not.toContain('brain_submit_job');
+    expect(names).not.toContain('brain_file_upload');
+  });
+
+  test('--write-capture routes to the virtual write registry (put_page present, captured not real)', async () => {
+    const capturedTools: Array<{ name: string }> = [];
+    await runRollout({
+      engine: {} as never,
+      skillText: 's',
+      task: TASK,
+      targetModel: 'm',
+      writeCapture: true,
+      toolLoopFn: makeStubLoop({ capturedTools }) as never,
+    });
+    const names = capturedTools.map((d) => d.name);
+    expect(names).toContain('brain_put_page');
+    expect(names).toContain('brain_submit_job');
+    expect(names).toContain('brain_file_upload');
+  });
+});
+
+describe('rollout — tool-call ordering', () => {
+  test('onToolCallStart records calls in order with the brain_ prefix stripped', async () => {
+    // eslint-disable-next-line @typescript-eslint/no-explicit-any
+    const loop = async (loopOpts: any) => {
+      await loopOpts.onToolCallStart?.(0, 0, 0, 'brain_search', { q: 'x' }, 'pc1');
+      await loopOpts.onToolCallStart?.(0, 1, 1, 'brain_get_page', { slug: 'y' }, 'pc2');
+      return {
+        messages: [{ role: 'assistant', content: 'ok' }],
+        totalUsage: { input_tokens: 1, output_tokens: 1, cache_read_tokens: 0, cache_creation_tokens: 0 },
+        totalTurns: 1,
+        stopReason: 'end',
+      };
+    };
+    const traj = await runRollout({
+      engine: {} as never, skillText: 's', task: TASK, targetModel: 'm', toolLoopFn: loop as never,
+    });
+    expect(traj.tool_calls).toHaveLength(2);
+    expect(traj.tool_calls[0]!.name).toBe('search');
+    expect(traj.tool_calls[1]!.name).toBe('get_page');
+  });
+});
@@ -0,0 +1,78 @@
+/**
+ * Regression: `runValidationGate` must SURFACE must-abort errors (budget
+ * exhaustion / no-pricing), not swallow them as score=0.
+ *
+ * The bug the SkillOpt real-LLM eval surfaced: a Haiku run with `--max-cost`
+ * hit `BudgetTracker` no_pricing on the FIRST chat() of every rollout, the
+ * error threw before any network call, and `runWithLimit` caught it as a
+ * `{ok:false}` settled item which the gate turned into `median:0`. Result: the
+ * whole gate reported a vacuous `selScore:0` in milliseconds with zero LLM
+ * calls — a pricing crash masquerading as a real "0/N" measurement. The fix
+ * re-throws any MUST_ABORT-class error so the caller aborts loudly.
+ */
+import { describe, test, expect } from 'bun:test';
+import { runValidationGate, scoreSkillOnTasks } from '../../src/core/skillopt/validate-gate.ts';
+import type { BenchmarkTask } from '../../src/core/skillopt/types.ts';
+
+const TASKS: BenchmarkTask[] = [
+  { task_id: 't1', task: 'do a thing', judge: { kind: 'rule', checks: [{ op: 'contains', arg: 'x' }] } } as never,
+  { task_id: 't2', task: 'do another', judge: { kind: 'rule', checks: [{ op: 'contains', arg: 'y' }] } } as never,
+];
+
+function budgetExhausted(): Error {
+  const e = new Error('no pricing entry for model "anthropic:claude-haiku-4-5" (kind=chat)');
+  (e as { tag?: string }).tag = 'BUDGET_EXHAUSTED';
+  return e;
+}
+
+describe('runValidationGate — must-abort errors surface', () => {
+  test('a BUDGET_EXHAUSTED rollout error is re-thrown, not scored 0', async () => {
+    const throwingRollout = (async () => {
+      throw budgetExhausted();
+    }) as never;
+    await expect(
+      runValidationGate({
+        engine: {} as never,
+        candidateSkillText: 'skill',
+        selSet: TASKS,
+        bestScore: -1,
+        targetModel: 'anthropic:claude-haiku-4-5',
+        runsPerTask: 1,
+        rolloutFn: throwingRollout,
+      }),
+    ).rejects.toThrow(/no pricing entry/);
+  });
+
+  test('scoreSkillOnTasks propagates the abort too (does not return a vacuous 0)', async () => {
+    const throwingRollout = (async () => {
+      throw budgetExhausted();
+    }) as never;
+    await expect(
+      scoreSkillOnTasks({
+        engine: {} as never,
+        skillText: 'skill',
+        tasks: TASKS,
+        targetModel: 'anthropic:claude-haiku-4-5',
+        runsPerTask: 1,
+        rolloutFn: throwingRollout,
+      }),
+    ).rejects.toThrow(/no pricing entry/);
+  });
+
+  test('an ordinary (non-abort) rollout error still scores 0 — fail-open preserved', async () => {
+    const flakyRollout = (async () => {
+      throw new Error('transient judge hiccup'); // no .tag → not must-abort
+    }) as never;
+    const gate = await runValidationGate({
+      engine: {} as never,
+      candidateSkillText: 'skill',
+      selSet: TASKS,
+      bestScore: -1,
+      targetModel: 'anthropic:claude-haiku-4-5',
+      runsPerTask: 1,
+      rolloutFn: flakyRollout,
+    });
+    expect(gate.selScore).toBe(0);
+    expect(gate.scoredRollouts).toHaveLength(0);
+  });
+});