From d6c7ac740eac158878fc1c08810fbe37d86f1cfc Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Mon, 1 Jun 2026 20:40:08 -0700 Subject: [PATCH] docs: document skillopt held-out gate + bundled mutation requirement for v0.42.9.0 Wire --held-out into the skill-optimizer SKILL.md, guide flags/safety tables, and the tutorial's bundled-skill step: mutating a bundled skill in place now requires --allow-mutate-bundled AND --held-out (>=5 benchmark-disjoint tasks) or it hard-refuses. Add the --held-out flag row + F11 held-out gate to the guide; update the receipt contract to the honest baseline/test-score fields. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/guides/skillopt.md | 8 +++-- .../improving-skills-with-skillopt.md | 19 ++++++++---- skills/skill-optimizer/SKILL.md | 29 +++++++++++++------ 3 files changed, 39 insertions(+), 17 deletions(-) diff --git a/docs/guides/skillopt.md b/docs/guides/skillopt.md index 32a76b861..2ce805ab7 100644 --- a/docs/guides/skillopt.md +++ b/docs/guides/skillopt.md @@ -87,8 +87,9 @@ proposal (this lives in v0.42 follow-up; v1 emits the audit event). | `--judge-model MODEL` | tier.reasoning | Scores rollouts | | `--patch \| --rewrite` | patch | Edit ops only vs. full rewrites | | `--dry-run` | off | Cost preview, no LLM calls | -| `--no-mutate` | off | Write proposed.md, don't replace SKILL.md | -| `--allow-mutate-bundled` | off | Required to mutate gbrain-bundled skills | +| `--no-mutate` | off | Write proposed.md, don't replace SKILL.md (no held-out needed) | +| `--allow-mutate-bundled` | off | Required to mutate gbrain-bundled skills in place — ALSO requires `--held-out` (>=5 rows) or the run hard-refuses | +| `--held-out ` | — | Independent test set (same JSONL shape as the benchmark, task IDs disjoint from it). A candidate that beats the benchmark but regresses on the held-out set is refused. Required for in-place bundled mutation. | | `--max-cost-usd N` | 5.00 | Hard cap; preflight refuses if exceeded | | `--max-runtime-min N` | 30 | Wall-clock cap | | `--force` | off | Bypass dirty-working-tree refusal | @@ -123,7 +124,8 @@ refuses to start when the estimate exceeds `--max-cost-usd`. | Validation gate is mandatory | D12 (paper) | Accepting LLM judge noise as improvement | | Frontmatter mutation forbidden | D5 | Routing surface drift (`check-resolvable` regression) | | Per-skill DB lock | D14 | Two concurrent runs corrupting history/versions | -| Bundled-skill gate | D16 | Auto-mutating skills shipped with gbrain | +| Bundled-skill gate | D16 | Auto-mutating skills shipped with gbrain (in-place mutation requires `--allow-mutate-bundled` + a `--held-out` set of >=5 benchmark-disjoint tasks; else hard-refuse + proposed.md) | +| Held-out gate | F11 | Accepting a candidate that overfits its own benchmark — `--held-out` refuses a candidate whose held-out score regresses below baseline | | Bootstrap review sentinel | D15 | Self-referential benchmark gaming | | Read-only tool sandbox in rollouts | D13 | Optimization runs writing junk pages to your brain | | History-intent-first atomic commit | D8 | Half-written SKILL.md on crash | diff --git a/docs/tutorials/improving-skills-with-skillopt.md b/docs/tutorials/improving-skills-with-skillopt.md index aa5df60b5..6c2a7fd53 100644 --- a/docs/tutorials/improving-skills-with-skillopt.md +++ b/docs/tutorials/improving-skills-with-skillopt.md @@ -239,14 +239,23 @@ silently mutate a skill other people depend on. Two ways to handle that: ```bash # See the proposed improvement without touching SKILL.md (works for ANY skill): gbrain skillopt meeting-prep --split 1:1:1 --no-mutate -# → writes skills/meeting-prep/skillopt/best.md, prints its path. Copy what you want. +# → writes skills/meeting-prep/skillopt/best.md (the proposed rewrite), prints its path. Copy what you want. -# Actually rewrite a bundled skill (explicit opt-in): -gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled +# Actually rewrite a bundled skill (explicit opt-in + an independent held-out set): +gbrain skillopt brain-ops --split 1:1:1 --allow-mutate-bundled \ + --held-out skills/brain-ops/held-out.jsonl ``` -Rule of thumb: `--no-mutate` when you want to read the diff before trusting it; -`--allow-mutate-bundled` only when you intend to commit a change to a shared skill. +Rewriting a bundled skill in place now requires BOTH `--allow-mutate-bundled` AND +`--held-out ` (a JSONL with the same shape as your benchmark, but at least 5 +tasks whose IDs don't appear in the benchmark). The held-out set is how the run +proves the edit didn't just learn the benchmark: a candidate that climbs the +benchmark but slips on the held-out tasks is refused. Drop `--held-out` and the +run hard-refuses and points you at `proposed.md` instead. + +Rule of thumb: `--no-mutate` when you want to read the diff before trusting it +(no held-out needed); `--allow-mutate-bundled --held-out` only when you intend to +commit a proven change to a shared skill. ## Step 6: Iterate diff --git a/skills/skill-optimizer/SKILL.md b/skills/skill-optimizer/SKILL.md index f7597a93e..ee4db2541 100644 --- a/skills/skill-optimizer/SKILL.md +++ b/skills/skill-optimizer/SKILL.md @@ -32,10 +32,13 @@ The user wants to: + epsilon=0.05 margin against the sel-set before SKILL.md gets rewritten. - **Frontmatter mutation is FORBIDDEN.** The optimizer only edits the body. Routing surface (`triggers:`, `brain_first:`) stays invariant. -- **Bundled skills require explicit opt-in.** Skills shipping with gbrain - cannot be auto-mutated; user passes `--allow-mutate-bundled` or - `--no-mutate` (default for the dream-cycle phase) writes proposed.md - for review. +- **Bundled skills require explicit opt-in AND an independent held-out set.** + Skills shipping with gbrain cannot be auto-mutated. To rewrite one in place + the user passes BOTH `--allow-mutate-bundled` AND `--held-out ` with + at least 5 benchmark-disjoint tasks; without the held-out set the run + hard-refuses (exit 2). Drop `--allow-mutate-bundled` (or pass `--no-mutate`, + the default for the dream-cycle phase) to write proposed.md for review + instead — no held-out needed for review-only output. - **Bootstrap output requires human review.** Both `--bootstrap-from-skill` and `--bootstrap-from-routing` write a sentinel; you must review + STRENGTHEN the generated judges, delete the sentinel, and re-run with @@ -127,8 +130,9 @@ attach >=2 rule checks each, save to `skills/X/skillopt-benchmark.jsonl`, run wi | Skill has a `routing-eval.jsonl` and you want a head start | `gbrain skillopt foo --bootstrap-from-routing` → review the generated tasks → `--bootstrap-reviewed` (routing tasks test dispatch; tighten them into quality tasks before trusting) | | Iterating on an existing skill | `gbrain skillopt foo --benchmark skills/foo/skillopt-benchmark.jsonl` | | Costly run, want preview | Add `--dry-run` | -| Bundled skill (skills/ in gbrain repo) | Default writes proposed.md; add `--allow-mutate-bundled` to commit | -| Want to review changes before applying | Add `--no-mutate` | +| Bundled skill (skills/ in gbrain repo) | Default writes proposed.md; to commit in place add `--allow-mutate-bundled` AND `--held-out ` (>=5 benchmark-disjoint tasks) — else it hard-refuses | +| Want to review changes before applying | Add `--no-mutate` (writes proposed.md, no held-out needed) | +| Guard against benchmark overfitting | Add `--held-out ` — a candidate that beats the benchmark but regresses on the held-out set is refused | | Mid-run crash | `gbrain skillopt foo --resume ` | ## Output Format @@ -146,8 +150,11 @@ When invoked, this skill produces: - **Don't bypass the validation gate.** The median-of-3 + epsilon=0.05 is load-bearing; without it, the optimizer accepts noise as improvement. -- **Don't optimize bundled skills without `--allow-mutate-bundled`.** They - ship with gbrain and are load-bearing for downstream agents. +- **Don't optimize bundled skills without `--allow-mutate-bundled` AND + `--held-out`.** They ship with gbrain and are load-bearing for downstream + agents. In-place mutation requires both flags (held-out >=5 benchmark-disjoint + tasks); without the held-out set the run hard-refuses and points you at + proposed.md. - **Don't use bootstrap output without strengthening it.** Both `--bootstrap-from-skill` and `--bootstrap-from-routing` have the optimizer model invent success criteria — generic and weak by default. Review and @@ -163,7 +170,11 @@ When invoked, this skill produces: ``` { outcome: 'accepted' | 'no_improvement' | 'aborted' | 'errored', - receipt: { run_id, skill_sha8, benchmark_sha8, models, scores, cost }, + receipt: { + run_id, skill_sha8, benchmark_sha8, models, cost, + baseline_sel_score, best_sel_score, // real measured baseline (no longer hardcoded 0) + baseline_test_score, test_score, // final held-out test-split eval + }, finalText: string, mutatedSkillFile: boolean, proposedPath?: string