docs: document skillopt held-out gate + bundled mutation requirement for v0.42.9.0

Wire --held-out into the skill-optimizer SKILL.md, guide flags/safety tables, and
the tutorial's bundled-skill step: mutating a bundled skill in place now requires
--allow-mutate-bundled AND --held-out (>=5 benchmark-disjoint tasks) or it
hard-refuses. Add the --held-out flag row + F11 held-out gate to the guide; update
the receipt contract to the honest baseline/test-score fields.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs/guides/skillopt.md

@@ -87,8 +87,9 @@ proposal (this lives in v0.42 follow-up; v1 emits the audit event).
 | `--judge-model MODEL` | tier.reasoning | Scores rollouts |
 | `--patch \| --rewrite` | patch | Edit ops only vs. full rewrites |
 | `--dry-run` | off | Cost preview, no LLM calls |
-| `--no-mutate` | off | Write proposed.md, don't replace SKILL.md |
-| `--allow-mutate-bundled` | off | Required to mutate gbrain-bundled skills |
+| `--no-mutate` | off | Write proposed.md, don't replace SKILL.md (no held-out needed) |
+| `--allow-mutate-bundled` | off | Required to mutate gbrain-bundled skills in place — ALSO requires `--held-out` (>=5 rows) or the run hard-refuses |
+| `--held-out <path>` | — | Independent test set (same JSONL shape as the benchmark, task IDs disjoint from it). A candidate that beats the benchmark but regresses on the held-out set is refused. Required for in-place bundled mutation. |
 | `--max-cost-usd N` | 5.00 | Hard cap; preflight refuses if exceeded |
 | `--max-runtime-min N` | 30 | Wall-clock cap |
 | `--force` | off | Bypass dirty-working-tree refusal |
@@ -123,7 +124,8 @@ refuses to start when the estimate exceeds `--max-cost-usd`.
 | Validation gate is mandatory | D12 (paper) | Accepting LLM judge noise as improvement |
 | Frontmatter mutation forbidden | D5 | Routing surface drift (`check-resolvable` regression) |
 | Per-skill DB lock | D14 | Two concurrent runs corrupting history/versions |
-| Bundled-skill gate | D16 | Auto-mutating skills shipped with gbrain |
+| Bundled-skill gate | D16 | Auto-mutating skills shipped with gbrain (in-place mutation requires `--allow-mutate-bundled` + a `--held-out` set of >=5 benchmark-disjoint tasks; else hard-refuse + proposed.md) |
+| Held-out gate | F11 | Accepting a candidate that overfits its own benchmark — `--held-out` refuses a candidate whose held-out score regresses below baseline |
 | Bootstrap review sentinel | D15 | Self-referential benchmark gaming |
 | Read-only tool sandbox in rollouts | D13 | Optimization runs writing junk pages to your brain |
 | History-intent-first atomic commit | D8 | Half-written SKILL.md on crash |

docs/tutorials/improving-skills-with-skillopt.md

@@ -239,14 +239,23 @@ silently mutate a skill other people depend on. Two ways to handle that:
 ```bash
 # See the proposed improvement without touching SKILL.md (works for ANY skill):
 gbrain skillopt meeting-prep --split 1:1:1 --no-mutate
-# → writes skills/meeting-prep/skillopt/best.md, prints its path. Copy what you want.
+# → writes skills/meeting-prep/skillopt/best.md (the proposed rewrite), prints its path. Copy what you want.
 
-# Actually rewrite a bundled skill (explicit opt-in):
-gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled
+# Actually rewrite a bundled skill (explicit opt-in + an independent held-out set):
+gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled \
+  --held-out skills/brain-ops/held-out.jsonl
 ```
 
-Rule of thumb: `--no-mutate` when you want to read the diff before trusting it;
-`--allow-mutate-bundled` only when you intend to commit a change to a shared skill.
+Rewriting a bundled skill in place now requires BOTH `--allow-mutate-bundled` AND
+`--held-out <path>` (a JSONL with the same shape as your benchmark, but at least 5
+tasks whose IDs don't appear in the benchmark). The held-out set is how the run
+proves the edit didn't just learn the benchmark: a candidate that climbs the
+benchmark but slips on the held-out tasks is refused. Drop `--held-out` and the
+run hard-refuses and points you at `proposed.md` instead.
+
+Rule of thumb: `--no-mutate` when you want to read the diff before trusting it
+(no held-out needed); `--allow-mutate-bundled --held-out` only when you intend to
+commit a proven change to a shared skill.
 
 ## Step 6: Iterate
 

skills/skill-optimizer/SKILL.md

@@ -32,10 +32,13 @@ The user wants to:
   + epsilon=0.05 margin against the sel-set before SKILL.md gets rewritten.
 - **Frontmatter mutation is FORBIDDEN.** The optimizer only edits the body.
   Routing surface (`triggers:`, `brain_first:`) stays invariant.
-- **Bundled skills require explicit opt-in.** Skills shipping with gbrain
-  cannot be auto-mutated; user passes `--allow-mutate-bundled` or
-  `--no-mutate` (default for the dream-cycle phase) writes proposed.md
-  for review.
+- **Bundled skills require explicit opt-in AND an independent held-out set.**
+  Skills shipping with gbrain cannot be auto-mutated. To rewrite one in place
+  the user passes BOTH `--allow-mutate-bundled` AND `--held-out <path>` with
+  at least 5 benchmark-disjoint tasks; without the held-out set the run
+  hard-refuses (exit 2). Drop `--allow-mutate-bundled` (or pass `--no-mutate`,
+  the default for the dream-cycle phase) to write proposed.md for review
+  instead — no held-out needed for review-only output.
 - **Bootstrap output requires human review.** Both `--bootstrap-from-skill`
   and `--bootstrap-from-routing` write a sentinel; you must review + STRENGTHEN
   the generated judges, delete the sentinel, and re-run with
@@ -127,8 +130,9 @@ attach >=2 rule checks each, save to `skills/X/skillopt-benchmark.jsonl`, run wi
 | Skill has a `routing-eval.jsonl` and you want a head start | `gbrain skillopt foo --bootstrap-from-routing` → review the generated tasks → `--bootstrap-reviewed` (routing tasks test dispatch; tighten them into quality tasks before trusting) |
 | Iterating on an existing skill | `gbrain skillopt foo --benchmark skills/foo/skillopt-benchmark.jsonl` |
 | Costly run, want preview | Add `--dry-run` |
-| Bundled skill (skills/ in gbrain repo) | Default writes proposed.md; add `--allow-mutate-bundled` to commit |
-| Want to review changes before applying | Add `--no-mutate` |
+| Bundled skill (skills/ in gbrain repo) | Default writes proposed.md; to commit in place add `--allow-mutate-bundled` AND `--held-out <path>` (>=5 benchmark-disjoint tasks) — else it hard-refuses |
+| Want to review changes before applying | Add `--no-mutate` (writes proposed.md, no held-out needed) |
+| Guard against benchmark overfitting | Add `--held-out <path>` — a candidate that beats the benchmark but regresses on the held-out set is refused |
 | Mid-run crash | `gbrain skillopt foo --resume <run-id>` |
 
 ## Output Format
@@ -146,8 +150,11 @@ When invoked, this skill produces:
 
 - **Don't bypass the validation gate.** The median-of-3 + epsilon=0.05 is
   load-bearing; without it, the optimizer accepts noise as improvement.
-- **Don't optimize bundled skills without `--allow-mutate-bundled`.** They
-  ship with gbrain and are load-bearing for downstream agents.
+- **Don't optimize bundled skills without `--allow-mutate-bundled` AND
+  `--held-out`.** They ship with gbrain and are load-bearing for downstream
+  agents. In-place mutation requires both flags (held-out >=5 benchmark-disjoint
+  tasks); without the held-out set the run hard-refuses and points you at
+  proposed.md.
 - **Don't use bootstrap output without strengthening it.** Both
   `--bootstrap-from-skill` and `--bootstrap-from-routing` have the optimizer
   model invent success criteria — generic and weak by default. Review and
@@ -163,7 +170,11 @@ When invoked, this skill produces:
 ```
 {
   outcome: 'accepted' | 'no_improvement' | 'aborted' | 'errored',
-  receipt: { run_id, skill_sha8, benchmark_sha8, models, scores, cost },
+  receipt: {
+    run_id, skill_sha8, benchmark_sha8, models, cost,
+    baseline_sel_score, best_sel_score,   // real measured baseline (no longer hardcoded 0)
+    baseline_test_score, test_score,      // final held-out test-split eval
+  },
   finalText: string,
   mutatedSkillFile: boolean,
   proposedPath?: string