diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b37fcba1..be939f2f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,96 @@ All notable changes to GBrain will be documented in this file. +## [0.42.9.0] - 2026-06-01 + +**Self-improving skills can no longer cheat. When you run `gbrain skillopt` to let +a skill rewrite itself, it now has to prove the change actually helps on a set of +tasks it wasn't optimized against — and for the skills gbrain ships, it won't +overwrite them in place unless you hand it that independent check.** + +Here is the problem. `gbrain skillopt` treats a skill's SKILL.md as something it +can edit and re-score against a benchmark, keeping edits that score higher. The +trap: an edit can score higher on its own benchmark while quietly getting worse at +the real job (classic "teaching to the test"). Until now the safety net for that — +a held-out check — was documented but never actually wired in, the run's report +showed a fake baseline score of 0, and a "final test" score was never computed. So +you couldn't tell from the receipt whether a skill genuinely improved. + +This release makes the loop honest. Pass `--held-out ` (a set of tasks +with different IDs than your benchmark) and a candidate that climbs the benchmark +but slips on the held-out set is refused. The run report now records the real +baseline score and a real test-set score, so "did this skill get better" is a +number you can read. `--no-mutate` finally writes the proposed rewrite to disk for +review (it was a stub), and `--max-runtime-min` is actually enforced. + +For the ~47 skills gbrain ships, the bar is higher: mutating one in place now +*requires* `--held-out` with at least 5 independent tasks. Without it you get a +`proposed.md` to review instead of a silent overwrite. The held-out file must use +task IDs disjoint from the benchmark — point it at a copy of the benchmark and the +run refuses, because an overlapping check can't catch overfitting. + +The `run_skillopt` MCP tool got a security tighten in the same pass: it validates +the skill name and confines benchmark/held-out paths to the skills directory for +remote callers, so an admin token can't read arbitrary host files through it. + +## To take advantage of v0.42.9.0 + +`gbrain upgrade` is all you need — these are behavior changes to an existing +command, no migration. + +1. **Optimize a user skill with the new safety net:** + ```bash + gbrain skillopt my-skill --held-out skills/my-skill/held-out.jsonl + ``` + The held-out file is the same JSONL shape as the benchmark, with task IDs that + do NOT appear in the benchmark. +2. **Optimize a bundled (shipped) skill in place** — now requires the held-out + check; otherwise it writes `proposed.md` for review: + ```bash + gbrain skillopt brain-ops --allow-mutate-bundled --held-out skills/brain-ops/held-out.jsonl + ``` +3. **Read the honest receipt:** `gbrain skillopt ... --json` now reports + `baseline_sel_score`, `best_sel_score`, `baseline_test_score`, and `test_score`. + +### Itemized changes + +#### Added +- **Held-out validation gate (F11) is now wired into the optimizer loop.** `--held-out ` + (CLI), `held_out_path` (background job + `run_skillopt` MCP op), and `heldOutPath` + (batch/fleet) load an independent task set; the gate runs at checkpoint acceptance + and blocks any candidate whose held-out score regresses below baseline. Previously + `runHeldOutGate` existed but nothing called it. +- **Final-test eval.** After optimization, the best skill and the baseline are scored + on the held-out test split; receipts now carry `test_score` + `baseline_test_score`. +- **Shared `scoreSkillOnTasks` primitive** (`validate-gate.ts`) used by the baseline + eval, final-test, held-out gate, and external eval harnesses so they can't drift. + +#### Changed +- **Bundled-skill mutation requires a non-empty held-out set (>=5 tasks).** Enforced in + core mutation policy (`assertBundledMutationHeldOut`), so it fires for every entry + point — CLI, batch, fleet, background job, and the `run_skillopt` MCP op. Without it, + the run hard-refuses (exit 2) and points you at `proposed.md`. +- **Held-out must be independent of the benchmark.** A held-out file sharing task IDs + with the benchmark is rejected (an overlapping check can't catch overfitting). +- **Honest receipts.** `baseline_sel_score` is the real measured baseline (was hardcoded + to 0). +- **`run_skillopt` MCP op hardening:** validates `skill_name` is kebab-case and confines + caller-supplied benchmark/held-out paths to the skills directory for remote callers. + +#### Fixed +- **`--no-mutate` now writes `proposed.md`** with the winning rewrite (was a stub that + wrote nothing). +- **`--max-runtime-min` is enforced** via a wall-clock deadline between optimization steps. + +### For contributors +- Eval-internal ablation knobs on `runSkillOpt` (not exposed on the CLI): `reflectMode` + (`'both'`/`'failure-only'`), `disableValidationGate`, and `optimizerMode` + (`'reflect'`/`'one-shot-rewrite'`), recorded in the receipt + audit for replayability. + These drive the SkillOpt benchmark suite in the sibling `gbrain-evals` repo. +- New tests: `test/skillopt/rollout.test.ts`, held-out + one-shot-rewrite unit cases, and + e2e coverage for the held-out gate (block/allow), bundled enforcement, no-mutate write, + runtime deadline, receipt honesty, and no-DB-pollution. + ## [0.42.3.0] - 2026-05-30 **Search now returns the *confident handful* instead of a fixed wall of results diff --git a/CLAUDE.md b/CLAUDE.md index 0ce8f1529..97c4f05a3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -7,6 +7,17 @@ suggests Supabase for 1000+ files. GStack teaches agents how to code. GBrain tea agents everything else: brain ops, signal detection, content ingestion, enrichment, cron scheduling, reports, identity, and access control. +## North Star + +gbrain aims to be the **next Postgres for memory**: the most well-tested, widest-coverage, +best-for-the-most-at-the-least retrieval + agent memory system for company brains and +personal AI, built to serve a billion people. Every feature and every eval is judged +against this bar. "gbrain is best" is a WHOLE-SYSTEM claim — proven across the full +BrainBench suite (retrieval, longmemeval, calibration, …) — not by any single feature. +When scoping an eval, prove the FEATURE delivers value to gbrain users; do not waste it +proving that gbrain's particular algorithm beats some other algorithm (a research +bake-off, off-mission). + ## Two organizational axes (read this first) GBrain knowledge is organized along two orthogonal axes. Users AND agents must @@ -270,7 +281,7 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` (v0.42.0.0 SkillOpt wave, closes #1481) — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904, Microsoft Research, May 2026). Top-level CLI `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent; validation-gated (D12 median-of-3 + epsilon=0.05), budget-capped (D3 preflight estimator), per-skill DB-locked (D14 `tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (D8 history-intent-first 5-step commit), body-only mutations (D5 — frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (D2 — zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload` (D13). Two reflect calls per step (D7 paper-faithful). Rejected-edit buffer LRU-bounded to 100 entries. Bundled-skill gate (D16). Bootstrap workflow (D15 sentinel + `--bootstrap-reviewed`). D17 D_sel floor (>=5 with `--split` override). Audit JSONL via shared `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Full cathedral ships: F1 dream-cycle phase wrapper, F2 41-case adversarial test suite (6 files), F3 E2E PGLite serial test (3 cases), F4 `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`), F5 `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`), F6 MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly), F7 Minion `skillopt` handler + `--background` CLI flag with `allowProtectedSubmit: true`, F8 reflect-prompt quality eval at `evals/skillopt-reflect/`, F9 judge LLM accuracy eval at `evals/skillopt-judge/`, F10 write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag), F11 held-out real-user test set scaffold via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Pinned by 152 tests across 18 files (88 foundation + 41 adversarial + 23 v2 surface + 3 E2E PGLite serial); hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`, R2-compliant). Plan + 17 review decisions + outside-voice codex absorption at `~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md`. v0.42+ remaining: admin UI Calibration-style dashboard tab for optimizer history; sweep all 47 bundled skills with their own `skillopt-benchmark.jsonl` fixtures (now unblocked by `--bootstrap-from-skill`, below). **v0.42.1.0 (`--bootstrap-from-skill`):** `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts` is the second bootstrap generator — reads `SKILL.md` directly (no `routing-eval.jsonl`) and makes ONE LLM call that emits a full starter benchmark (tasks + rule judges) as **JSONL**, parsed line-by-line with skip-bad-line salvage (D5 — a truncated final line drops, the rest survive) and a min-2-valid-checks-per-task drop (D6 — never a task judged by one weak check). Provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`) so the CLI surfaces the real failure. `--bootstrap-tasks N` (default 15, CLI-capped at 50) tunes the count; `maxTokens` scales `min(8000, max(4000, N*220))`. The generator's stderr REVIEW line prints the literal paste-ready `gbrain skillopt --bootstrap-reviewed --split 1:1:1` next command — load-bearing because the optimizer's default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark is unusable without `--split 1:1:1`. Both bootstrap generators now share extracted helpers `assertBenchmarkAbsent` + `readSkillBodyOrThrow` (D1); `runBootstrap`'s routing-eval parse hardened to skip malformed lines. CLI: `--bootstrap-from-skill` short-circuits after the routing-bootstrap block, mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume` (`--background`/`--follow` already rejected by the unknown-flag guard — a pre-existing CLI gap NOT fixed here: those two flags are never parsed so the documented `--background` path is currently unreachable; filed for a separate fix). `parseFlags` exported for CLI unit tests. Generated rule judges are explicitly WEAK DRAFTS — `skill-optimizer/SKILL.md` + `docs/tutorials/improving-skills-with-skillopt.md` reposition from-skill as the PRIMARY no-benchmark path and tell the agent to STRENGTHEN the judges during review (closes the codex "trains toward benchmark artifacts" concern via the D15 review gate). Pinned by 20 cases in `test/skillopt/bootstrap-from-skill.test.ts` (happy-path + deterministic task_ids, round-trip into `loadBenchmark` + `splitBench(1:1:1)`, JSONL salvage, min-2-checks drop, provider-error propagation, `bootstrap_empty`, `benchmark_exists`/`--force`, `no_skill_md`, fenced output, maxTokens scaling, sub-15 warning + REVIEW line, `parseFlags` cap + 6-way mutual exclusion). Plan + eng-review + codex absorption at `~/.claude/plans/system-instruction-you-are-working-recursive-sutton.md`. +- `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` (v0.42.0.0 SkillOpt wave, closes #1481) — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904, Microsoft Research, May 2026). Top-level CLI `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent; validation-gated (D12 median-of-3 + epsilon=0.05), budget-capped (D3 preflight estimator), per-skill DB-locked (D14 `tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (D8 history-intent-first 5-step commit), body-only mutations (D5 — frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (D2 — zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload` (D13). Two reflect calls per step (D7 paper-faithful). Rejected-edit buffer LRU-bounded to 100 entries. Bundled-skill gate (D16). Bootstrap workflow (D15 sentinel + `--bootstrap-reviewed`). D17 D_sel floor (>=5 with `--split` override). Audit JSONL via shared `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Full cathedral ships: F1 dream-cycle phase wrapper, F2 41-case adversarial test suite (6 files), F3 E2E PGLite serial test (3 cases), F4 `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`), F5 `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`), F6 MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly), F7 Minion `skillopt` handler + `--background` CLI flag with `allowProtectedSubmit: true`, F8 reflect-prompt quality eval at `evals/skillopt-reflect/`, F9 judge LLM accuracy eval at `evals/skillopt-judge/`, F10 write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag), F11 held-out real-user test set scaffold via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Pinned by 152 tests across 18 files (88 foundation + 41 adversarial + 23 v2 surface + 3 E2E PGLite serial); hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`, R2-compliant). Plan + 17 review decisions + outside-voice codex absorption at `~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md`. v0.42+ remaining: admin UI Calibration-style dashboard tab for optimizer history; sweep all 47 bundled skills with their own `skillopt-benchmark.jsonl` fixtures (now unblocked by `--bootstrap-from-skill`, below). **v0.42.1.0 (`--bootstrap-from-skill`):** `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts` is the second bootstrap generator — reads `SKILL.md` directly (no `routing-eval.jsonl`) and makes ONE LLM call that emits a full starter benchmark (tasks + rule judges) as **JSONL**, parsed line-by-line with skip-bad-line salvage (D5 — a truncated final line drops, the rest survive) and a min-2-valid-checks-per-task drop (D6 — never a task judged by one weak check). Provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`) so the CLI surfaces the real failure. `--bootstrap-tasks N` (default 15, CLI-capped at 50) tunes the count; `maxTokens` scales `min(8000, max(4000, N*220))`. The generator's stderr REVIEW line prints the literal paste-ready `gbrain skillopt --bootstrap-reviewed --split 1:1:1` next command — load-bearing because the optimizer's default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark is unusable without `--split 1:1:1`. Both bootstrap generators now share extracted helpers `assertBenchmarkAbsent` + `readSkillBodyOrThrow` (D1); `runBootstrap`'s routing-eval parse hardened to skip malformed lines. CLI: `--bootstrap-from-skill` short-circuits after the routing-bootstrap block, mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume` (`--background`/`--follow` already rejected by the unknown-flag guard — a pre-existing CLI gap NOT fixed here: those two flags are never parsed so the documented `--background` path is currently unreachable; filed for a separate fix). `parseFlags` exported for CLI unit tests. Generated rule judges are explicitly WEAK DRAFTS — `skill-optimizer/SKILL.md` + `docs/tutorials/improving-skills-with-skillopt.md` reposition from-skill as the PRIMARY no-benchmark path and tell the agent to STRENGTHEN the judges during review (closes the codex "trains toward benchmark artifacts" concern via the D15 review gate). Pinned by 20 cases in `test/skillopt/bootstrap-from-skill.test.ts` (happy-path + deterministic task_ids, round-trip into `loadBenchmark` + `splitBench(1:1:1)`, JSONL salvage, min-2-checks drop, provider-error propagation, `bootstrap_empty`, `benchmark_exists`/`--force`, `no_skill_md`, fenced output, maxTokens scaling, sub-15 warning + REVIEW line, `parseFlags` cap + 6-way mutual exclusion). Plan + eng-review + codex absorption at `~/.claude/plans/system-instruction-you-are-working-recursive-sutton.md`. **v0.42.9.0 (eval-readiness wave):** the F11 held-out gate is now actually WIRED — `runHeldOutGate` (held-out.ts) was dead code (orchestrator never imported it; `--held-out` was documented but never parsed/threaded). Now `--held-out ` is parsed (skillopt.ts), threaded through every caller (CLI main + `--background` job-data `held_out_path` + batch/fleet `heldOutPath` + the `run_skillopt` MCP op `held_out_path` param), and the gate runs at CHECKPOINT ACCEPTANCE (not just file mutation) so no-mutate/fleet paths can't promote a held-out-failing candidate either. **D16 ENFORCE moved to CORE mutation policy** (`assertBundledMutationHeldOut` in bundled-skill-gate.ts): bundled + `--allow-mutate-bundled` requires a NON-EMPTY held-out (`MIN_HELD_OUT_SIZE = D_SEL_MIN_SIZE` = 5; was an independent literal-5, now derived so they can't desync) or hard-refuses (exit 2) — fires for ALL callers since they funnel through `runSkillOpt`. Held-out must be task_id-DISJOINT from the benchmark (orchestrator rejects overlap — an overlapping held-out can't catch overfitting). **Receipt honesty:** `receipt.baseline_sel_score` was hardcoded `0` (real value discarded via `void baselineSelScore`); now populated, plus a real **final-test eval** (`test_score` + `baseline_test_score`) scoring best + baseline on `split.test`. Shared `scoreSkillOnTasks` primitive (validate-gate.ts) backs the baseline/final-test/held-out scoring (held-out.ts refactored onto it). `--no-mutate` proposed.md write FIXED (was a comment stub → `writeProposed` in version-store.ts). `maxRuntimeMin` ENFORCED (wall-clock deadline between steps → `skillopt_runtime_exceeded` → outcome aborted). Three eval-internal ablation opts on `SkillOptOpts` (NOT on CLI): `reflectMode` (`'both'`/`'failure-only'`), `disableValidationGate` (greedy-accept, skips D12), `optimizerMode` (`'reflect'`/`'one-shot-rewrite'` — one rewrite, no loop; replaced a random-edit strawman), all recorded in `RunReceipt` + audit `run_start` for cat31 replayability; `ROLLOUT_SUCCESS_THRESHOLD = 0.5` named constant for the D7 partition. one-shot fence-strip is anchored (`^```...```$`) so a body with an embedded code sample isn't truncated. `run_skillopt` MCP op hardened: validates `skill_name` kebab-only + confines caller-supplied benchmark/held-out paths to skillsDir for remote callers (closes an arbitrary-file-read/existence-oracle for admin OAuth tokens). Pinned by `test/skillopt/rollout.test.ts` (new), held-out ENFORCE + one-shot-rewrite unit cases, and e2e (F11 block/allow, bundled no-mutate, runtime deadline, receipt honesty, held-out disjointness, D2 no-DB-pollution). Drives the Track B SkillOpt benchmark suite (cat30 improvement / cat31 ablation / cat32 reward-hacking defense / cat33 transfer) in the sibling `gbrain-evals` repo. v0.42+ remaining: `promoteCandidate` extraction (one-shot/loop DRY); bundled-detection hardening (realpath vs canonical skills dir regardless of detection source); preflight awareness of ablation opts. - `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. **v0.41.30.0:** `--save` for both `brainstorm` and `lsd` rewritten to persist through the canonical ingestion path instead of the lightweight DB-only write that never produced the advertised `.md` file. New exported `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so `gbrain search` finds the idea; no embedding cost at save time) THEN renders the saved row to disk via the shared `writePageThrough` helper (see `src/core/write-through.ts`) — file rendered FROM the row so the two sinks can't diverge and a later `gbrain sync` doesn't churn it. New exported pure `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message: both-sinks success, DB-only (no `sync.repo_path` / repo-not-a-dir → file write skipped, exit 0), DB-saved-but-file-errored (exit 0, sync reconciles), and the total-failure case (neither sink → loud `save FAILED … NOT persisted` on stderr + nonzero exit). Closes the silent-false-success bug class where `--save` printed "Saved" unconditionally even when the PgBouncer-transaction-mode DB write failed and `gbrain get ` then returned `page_not_found`. `buildIdeaSlug(question, label, nonce?)` gains a random nonce suffix (injectable for deterministic tests) so two same-day runs whose questions share the first 60 slug chars no longer clobber each other's page + file. `--json` callers stay DB-only (unchanged). New exported `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (the string `buildBrainstormFrontmatter` is left untouched). Pinned by `test/brainstorm/save.test.ts` (covers `persistSavedIdea` both-sinks / DB-only / total-failure paths, `formatSaveOutcome` per-branch messages + exit codes, and `buildIdeaSlug` nonce collision-resistance). - `src/core/write-through.ts` (v0.41.30.0, NEW) — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` reads `sync.repo_path`, re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown` + `resolvePageFilePath`, and writes the `.md` under the repo so the brain has a committable artifact that round-trips through `gbrain sync`. Rendering FROM the row means the file and the DB row cannot diverge. Extracted from the v0.38 `put_page` write-through in `src/core/operations.ts` (which the op now calls instead of hand-rolling its own copy) AND upgraded to ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync` into place, cleaning up the temp on any failure, so a crash or a concurrently-running `gbrain sync`/autopilot walking the live git tree can never read a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op (write-through behavior preserved, now atomic) and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts` (no-repo / repo-not-found / page-not-found-after-write skips, atomic-rename happy path + no stray `.tmp` on write failure, frontmatter-override stamping). - `src/core/model-id.ts` (v0.41.21.0, NEW) — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface no longer has 5 parallel re-implementations of `provider:model` splitting. Closes the bug class where slash-form ids (`anthropic/claude-sonnet-4-6` — the form CLI flags accept and OpenRouter recipes emit) silently fell through to "unknown model" at every site. Distinct from the existing gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`: that one throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by 16 cases in `test/model-id.test.ts`. diff --git a/TODOS.md b/TODOS.md index 3c55df82a..b01f9e768 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,35 @@ # TODOS +## v0.42.9.0 SkillOpt eval-readiness follow-ups (v0.42+) + +Deferred from the v0.42.9.0 wave (held-out gate wiring + ENFORCE + ablation opts). +Adversarial-review findings that are real but not blockers — the shipped fixes are +complete and tested; these are hardening/cleanup. + +- [ ] **P2 — Extract `promoteCandidate` helper (DRY).** The candidate-promotion + sequence (optional `runHeldOutGate` → branch on `mutateDecision.mutate` → + `acceptCandidate` else `writeProposed` → set outcome/finalText) is duplicated between + the one-shot-rewrite block and the main loop accept branch in + `src/core/skillopt/orchestrator.ts`. A future change to the held-out gate or promotion + policy must be applied in two places. Extract a shared `promoteCandidate({...})`. Deferred + this wave to avoid a >20-line refactor of freshly-tested accept-path code. +- [ ] **P2 — Harden bundled-skill detection.** `getBundledSkillContext` + (`src/core/skillopt/bundled-skill-gate.ts`) only sets `isBundled` when the skills dir was + resolved via the `install_path` tier. If the same bundled `skills/` is found via + `cwd_walk_up` / `repo_root` / `$GBRAIN_SKILLS_DIR`, `isBundled=false` and the D16 ENFORCE + never fires (same weakness governs `--allow-mutate-bundled` itself — pre-existing, not a + v0.42.9.0 regression). Fix: compare realpaths against the canonical bundled skills dir + independent of detection source. +- [ ] **P3 — Preflight cost estimate is blind to ablation opts.** `preflight.ts:estimateCost` + doesn't know `optimizerMode`/`disableValidationGate`/`reflectMode`, so `--dry-run` + over-counts for `one-shot-rewrite` / `failure-only`. Low impact (eval-internal knobs; + runtime BudgetTracker enforcement is correct, no overspend) — just a lying preview. +- [ ] **P3 — `maxRuntimeMin` is enforced only between optimization steps.** The baseline + eval, per-step held-out gate, one-shot rewrite, and final-test `scoreSkillOnTasks` calls + run unbounded LLM rollouts with no deadline check. BudgetTracker still caps spend; the + runtime guarantee is best-effort. Thread the deadline + abortSignal into those phases, or + document runtime as best-effort. + ## v0.42.2.0 gbrain connect follow-ups (v0.42+) - [ ] **T6 (P3): `gbrain connect --env-token` form.** Ship the env-var-indirection diff --git a/VERSION b/VERSION index 9ece4afec..eeebd1773 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.3.0 \ No newline at end of file +0.42.9.0 diff --git a/llms-full.txt b/llms-full.txt index cb090f936..d483bf08f 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -149,6 +149,17 @@ suggests Supabase for 1000+ files. GStack teaches agents how to code. GBrain tea agents everything else: brain ops, signal detection, content ingestion, enrichment, cron scheduling, reports, identity, and access control. +## North Star + +gbrain aims to be the **next Postgres for memory**: the most well-tested, widest-coverage, +best-for-the-most-at-the-least retrieval + agent memory system for company brains and +personal AI, built to serve a billion people. Every feature and every eval is judged +against this bar. "gbrain is best" is a WHOLE-SYSTEM claim — proven across the full +BrainBench suite (retrieval, longmemeval, calibration, …) — not by any single feature. +When scoping an eval, prove the FEATURE delivers value to gbrain users; do not waste it +proving that gbrain's particular algorithm beats some other algorithm (a research +bake-off, off-mission). + ## Two organizational axes (read this first) GBrain knowledge is organized along two orthogonal axes. Users AND agents must @@ -412,7 +423,7 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` (v0.42.0.0 SkillOpt wave, closes #1481) — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904, Microsoft Research, May 2026). Top-level CLI `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent; validation-gated (D12 median-of-3 + epsilon=0.05), budget-capped (D3 preflight estimator), per-skill DB-locked (D14 `tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (D8 history-intent-first 5-step commit), body-only mutations (D5 — frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (D2 — zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload` (D13). Two reflect calls per step (D7 paper-faithful). Rejected-edit buffer LRU-bounded to 100 entries. Bundled-skill gate (D16). Bootstrap workflow (D15 sentinel + `--bootstrap-reviewed`). D17 D_sel floor (>=5 with `--split` override). Audit JSONL via shared `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Full cathedral ships: F1 dream-cycle phase wrapper, F2 41-case adversarial test suite (6 files), F3 E2E PGLite serial test (3 cases), F4 `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`), F5 `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`), F6 MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly), F7 Minion `skillopt` handler + `--background` CLI flag with `allowProtectedSubmit: true`, F8 reflect-prompt quality eval at `evals/skillopt-reflect/`, F9 judge LLM accuracy eval at `evals/skillopt-judge/`, F10 write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag), F11 held-out real-user test set scaffold via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Pinned by 152 tests across 18 files (88 foundation + 41 adversarial + 23 v2 surface + 3 E2E PGLite serial); hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`, R2-compliant). Plan + 17 review decisions + outside-voice codex absorption at `~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md`. v0.42+ remaining: admin UI Calibration-style dashboard tab for optimizer history; sweep all 47 bundled skills with their own `skillopt-benchmark.jsonl` fixtures (now unblocked by `--bootstrap-from-skill`, below). **v0.42.1.0 (`--bootstrap-from-skill`):** `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts` is the second bootstrap generator — reads `SKILL.md` directly (no `routing-eval.jsonl`) and makes ONE LLM call that emits a full starter benchmark (tasks + rule judges) as **JSONL**, parsed line-by-line with skip-bad-line salvage (D5 — a truncated final line drops, the rest survive) and a min-2-valid-checks-per-task drop (D6 — never a task judged by one weak check). Provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`) so the CLI surfaces the real failure. `--bootstrap-tasks N` (default 15, CLI-capped at 50) tunes the count; `maxTokens` scales `min(8000, max(4000, N*220))`. The generator's stderr REVIEW line prints the literal paste-ready `gbrain skillopt --bootstrap-reviewed --split 1:1:1` next command — load-bearing because the optimizer's default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark is unusable without `--split 1:1:1`. Both bootstrap generators now share extracted helpers `assertBenchmarkAbsent` + `readSkillBodyOrThrow` (D1); `runBootstrap`'s routing-eval parse hardened to skip malformed lines. CLI: `--bootstrap-from-skill` short-circuits after the routing-bootstrap block, mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume` (`--background`/`--follow` already rejected by the unknown-flag guard — a pre-existing CLI gap NOT fixed here: those two flags are never parsed so the documented `--background` path is currently unreachable; filed for a separate fix). `parseFlags` exported for CLI unit tests. Generated rule judges are explicitly WEAK DRAFTS — `skill-optimizer/SKILL.md` + `docs/tutorials/improving-skills-with-skillopt.md` reposition from-skill as the PRIMARY no-benchmark path and tell the agent to STRENGTHEN the judges during review (closes the codex "trains toward benchmark artifacts" concern via the D15 review gate). Pinned by 20 cases in `test/skillopt/bootstrap-from-skill.test.ts` (happy-path + deterministic task_ids, round-trip into `loadBenchmark` + `splitBench(1:1:1)`, JSONL salvage, min-2-checks drop, provider-error propagation, `bootstrap_empty`, `benchmark_exists`/`--force`, `no_skill_md`, fenced output, maxTokens scaling, sub-15 warning + REVIEW line, `parseFlags` cap + 6-way mutual exclusion). Plan + eng-review + codex absorption at `~/.claude/plans/system-instruction-you-are-working-recursive-sutton.md`. +- `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` (v0.42.0.0 SkillOpt wave, closes #1481) — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904, Microsoft Research, May 2026). Top-level CLI `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent; validation-gated (D12 median-of-3 + epsilon=0.05), budget-capped (D3 preflight estimator), per-skill DB-locked (D14 `tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (D8 history-intent-first 5-step commit), body-only mutations (D5 — frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (D2 — zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload` (D13). Two reflect calls per step (D7 paper-faithful). Rejected-edit buffer LRU-bounded to 100 entries. Bundled-skill gate (D16). Bootstrap workflow (D15 sentinel + `--bootstrap-reviewed`). D17 D_sel floor (>=5 with `--split` override). Audit JSONL via shared `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Full cathedral ships: F1 dream-cycle phase wrapper, F2 41-case adversarial test suite (6 files), F3 E2E PGLite serial test (3 cases), F4 `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`), F5 `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`), F6 MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly), F7 Minion `skillopt` handler + `--background` CLI flag with `allowProtectedSubmit: true`, F8 reflect-prompt quality eval at `evals/skillopt-reflect/`, F9 judge LLM accuracy eval at `evals/skillopt-judge/`, F10 write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag), F11 held-out real-user test set scaffold via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Pinned by 152 tests across 18 files (88 foundation + 41 adversarial + 23 v2 surface + 3 E2E PGLite serial); hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`, R2-compliant). Plan + 17 review decisions + outside-voice codex absorption at `~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md`. v0.42+ remaining: admin UI Calibration-style dashboard tab for optimizer history; sweep all 47 bundled skills with their own `skillopt-benchmark.jsonl` fixtures (now unblocked by `--bootstrap-from-skill`, below). **v0.42.1.0 (`--bootstrap-from-skill`):** `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts` is the second bootstrap generator — reads `SKILL.md` directly (no `routing-eval.jsonl`) and makes ONE LLM call that emits a full starter benchmark (tasks + rule judges) as **JSONL**, parsed line-by-line with skip-bad-line salvage (D5 — a truncated final line drops, the rest survive) and a min-2-valid-checks-per-task drop (D6 — never a task judged by one weak check). Provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`) so the CLI surfaces the real failure. `--bootstrap-tasks N` (default 15, CLI-capped at 50) tunes the count; `maxTokens` scales `min(8000, max(4000, N*220))`. The generator's stderr REVIEW line prints the literal paste-ready `gbrain skillopt --bootstrap-reviewed --split 1:1:1` next command — load-bearing because the optimizer's default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark is unusable without `--split 1:1:1`. Both bootstrap generators now share extracted helpers `assertBenchmarkAbsent` + `readSkillBodyOrThrow` (D1); `runBootstrap`'s routing-eval parse hardened to skip malformed lines. CLI: `--bootstrap-from-skill` short-circuits after the routing-bootstrap block, mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume` (`--background`/`--follow` already rejected by the unknown-flag guard — a pre-existing CLI gap NOT fixed here: those two flags are never parsed so the documented `--background` path is currently unreachable; filed for a separate fix). `parseFlags` exported for CLI unit tests. Generated rule judges are explicitly WEAK DRAFTS — `skill-optimizer/SKILL.md` + `docs/tutorials/improving-skills-with-skillopt.md` reposition from-skill as the PRIMARY no-benchmark path and tell the agent to STRENGTHEN the judges during review (closes the codex "trains toward benchmark artifacts" concern via the D15 review gate). Pinned by 20 cases in `test/skillopt/bootstrap-from-skill.test.ts` (happy-path + deterministic task_ids, round-trip into `loadBenchmark` + `splitBench(1:1:1)`, JSONL salvage, min-2-checks drop, provider-error propagation, `bootstrap_empty`, `benchmark_exists`/`--force`, `no_skill_md`, fenced output, maxTokens scaling, sub-15 warning + REVIEW line, `parseFlags` cap + 6-way mutual exclusion). Plan + eng-review + codex absorption at `~/.claude/plans/system-instruction-you-are-working-recursive-sutton.md`. **v0.42.9.0 (eval-readiness wave):** the F11 held-out gate is now actually WIRED — `runHeldOutGate` (held-out.ts) was dead code (orchestrator never imported it; `--held-out` was documented but never parsed/threaded). Now `--held-out ` is parsed (skillopt.ts), threaded through every caller (CLI main + `--background` job-data `held_out_path` + batch/fleet `heldOutPath` + the `run_skillopt` MCP op `held_out_path` param), and the gate runs at CHECKPOINT ACCEPTANCE (not just file mutation) so no-mutate/fleet paths can't promote a held-out-failing candidate either. **D16 ENFORCE moved to CORE mutation policy** (`assertBundledMutationHeldOut` in bundled-skill-gate.ts): bundled + `--allow-mutate-bundled` requires a NON-EMPTY held-out (`MIN_HELD_OUT_SIZE = D_SEL_MIN_SIZE` = 5; was an independent literal-5, now derived so they can't desync) or hard-refuses (exit 2) — fires for ALL callers since they funnel through `runSkillOpt`. Held-out must be task_id-DISJOINT from the benchmark (orchestrator rejects overlap — an overlapping held-out can't catch overfitting). **Receipt honesty:** `receipt.baseline_sel_score` was hardcoded `0` (real value discarded via `void baselineSelScore`); now populated, plus a real **final-test eval** (`test_score` + `baseline_test_score`) scoring best + baseline on `split.test`. Shared `scoreSkillOnTasks` primitive (validate-gate.ts) backs the baseline/final-test/held-out scoring (held-out.ts refactored onto it). `--no-mutate` proposed.md write FIXED (was a comment stub → `writeProposed` in version-store.ts). `maxRuntimeMin` ENFORCED (wall-clock deadline between steps → `skillopt_runtime_exceeded` → outcome aborted). Three eval-internal ablation opts on `SkillOptOpts` (NOT on CLI): `reflectMode` (`'both'`/`'failure-only'`), `disableValidationGate` (greedy-accept, skips D12), `optimizerMode` (`'reflect'`/`'one-shot-rewrite'` — one rewrite, no loop; replaced a random-edit strawman), all recorded in `RunReceipt` + audit `run_start` for cat31 replayability; `ROLLOUT_SUCCESS_THRESHOLD = 0.5` named constant for the D7 partition. one-shot fence-strip is anchored (`^```...```$`) so a body with an embedded code sample isn't truncated. `run_skillopt` MCP op hardened: validates `skill_name` kebab-only + confines caller-supplied benchmark/held-out paths to skillsDir for remote callers (closes an arbitrary-file-read/existence-oracle for admin OAuth tokens). Pinned by `test/skillopt/rollout.test.ts` (new), held-out ENFORCE + one-shot-rewrite unit cases, and e2e (F11 block/allow, bundled no-mutate, runtime deadline, receipt honesty, held-out disjointness, D2 no-DB-pollution). Drives the Track B SkillOpt benchmark suite (cat30 improvement / cat31 ablation / cat32 reward-hacking defense / cat33 transfer) in the sibling `gbrain-evals` repo. v0.42+ remaining: `promoteCandidate` extraction (one-shot/loop DRY); bundled-detection hardening (realpath vs canonical skills dir regardless of detection source); preflight awareness of ablation opts. - `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. **v0.41.30.0:** `--save` for both `brainstorm` and `lsd` rewritten to persist through the canonical ingestion path instead of the lightweight DB-only write that never produced the advertised `.md` file. New exported `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so `gbrain search` finds the idea; no embedding cost at save time) THEN renders the saved row to disk via the shared `writePageThrough` helper (see `src/core/write-through.ts`) — file rendered FROM the row so the two sinks can't diverge and a later `gbrain sync` doesn't churn it. New exported pure `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message: both-sinks success, DB-only (no `sync.repo_path` / repo-not-a-dir → file write skipped, exit 0), DB-saved-but-file-errored (exit 0, sync reconciles), and the total-failure case (neither sink → loud `save FAILED … NOT persisted` on stderr + nonzero exit). Closes the silent-false-success bug class where `--save` printed "Saved" unconditionally even when the PgBouncer-transaction-mode DB write failed and `gbrain get ` then returned `page_not_found`. `buildIdeaSlug(question, label, nonce?)` gains a random nonce suffix (injectable for deterministic tests) so two same-day runs whose questions share the first 60 slug chars no longer clobber each other's page + file. `--json` callers stay DB-only (unchanged). New exported `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (the string `buildBrainstormFrontmatter` is left untouched). Pinned by `test/brainstorm/save.test.ts` (covers `persistSavedIdea` both-sinks / DB-only / total-failure paths, `formatSaveOutcome` per-branch messages + exit codes, and `buildIdeaSlug` nonce collision-resistance). - `src/core/write-through.ts` (v0.41.30.0, NEW) — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` reads `sync.repo_path`, re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown` + `resolvePageFilePath`, and writes the `.md` under the repo so the brain has a committable artifact that round-trips through `gbrain sync`. Rendering FROM the row means the file and the DB row cannot diverge. Extracted from the v0.38 `put_page` write-through in `src/core/operations.ts` (which the op now calls instead of hand-rolling its own copy) AND upgraded to ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync` into place, cleaning up the temp on any failure, so a crash or a concurrently-running `gbrain sync`/autopilot walking the live git tree can never read a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op (write-through behavior preserved, now atomic) and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts` (no-repo / repo-not-found / page-not-found-after-write skips, atomic-rename happy path + no stray `.tmp` on write failure, frontmatter-override stamping). - `src/core/model-id.ts` (v0.41.21.0, NEW) — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface no longer has 5 parallel re-implementations of `provider:model` splitting. Closes the bug class where slash-form ids (`anthropic/claude-sonnet-4-6` — the form CLI flags accept and OpenRouter recipes emit) silently fell through to "unknown model" at every site. Distinct from the existing gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`: that one throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by 16 cases in `test/model-id.test.ts`. diff --git a/package.json b/package.json index 464f99044..049f77414 100644 --- a/package.json +++ b/package.json @@ -142,5 +142,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.3.0" + "version": "0.42.9.0" }