diff --git a/CHANGELOG.md b/CHANGELOG.md index 56a062cc5..a3f047118 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,214 @@ All notable changes to GBrain will be documented in this file. +## [0.41.23.0] - 2026-05-27 + +**Your skills now improve themselves overnight.** + +GBrain ships 47 bundled skills that tell agents how to handle specific kinds +of tasks. Until now, those skills only got better when a human rewrote them. +A human can read three or four execution traces and spot a problem; nobody +can read forty execution traces and spot which exact rule is hurting and +which is helping. v0.41.23.0 closes that loop. You write a benchmark of +realistic tasks, and `gbrain skillopt ` watches the agent run those +tasks against your current skill text, proposes specific edits, re-tests, +and only keeps changes that measurably improve the score. + +This is based on the SkillOpt paper (Microsoft Research, May 2026), which +treats the skill document as the trainable parameters of an agent that +itself never changes. The paper added 23.5 points over no-skill on GPT-5.5 +and beat hand-written skills across every benchmark it was tested on. +gbrain's version ships every safety guard that paper found load-bearing: +bounded edits per step, mandatory validation gating, persistent memory of +rejected edits, and a cosine decay schedule that lets the optimizer be +aggressive early and conservative late. + +### How to use it + +Bootstrap a benchmark from your existing routing fixtures (one Anthropic +call per row), review the output, then run the optimizer: + +```bash +gbrain skillopt my-skill --bootstrap-from-routing +# review skills/my-skill/skillopt-benchmark.jsonl, delete the trailing +# `# BOOTSTRAP_PENDING_REVIEW` line +gbrain skillopt my-skill --bootstrap-reviewed +``` + +Or, if you already have a benchmark: + +```bash +gbrain skillopt my-skill --benchmark skills/my-skill/skillopt-benchmark.jsonl +``` + +Add `--dry-run` to see the cost estimate without spending a dime. The +preflight estimator refuses to start when the projected cost exceeds +`--max-cost-usd` (default $5.00), so you'll never be surprised by a +runaway run. + +### The numbers that matter + +| Knob | Default | What it controls | +|---|---|---| +| `--epochs` | 4 | Outer-loop iterations | +| `--batch-size` | 8 | Tasks per inner step | +| `--lr` | 4 | Max edits accepted per step | +| `--lr-schedule` | cosine | Curve that decays the edit budget | +| `--split` | 4:1:5 | train:sel:test ratio (refuses if D_sel < 5) | +| `--max-cost-usd` | 5.00 | Hard ceiling; preflight refuses if exceeded | + +A typical 20-task benchmark with defaults costs ~$0.90 per run. + +### What's safe to know about + +- **Bundled skills are safe by default.** Skills shipping in `skills/` (the + ones gbrain ships) can't be auto-mutated. The optimizer writes + `skills//skillopt/best.md` for review; pass `--allow-mutate-bundled` + to commit changes back to `SKILL.md`. +- **Skill mutations are body-only.** The optimizer can't edit `triggers:`, + `brain_first:`, or any other frontmatter field — those are routing + surface, not behavior surface. +- **Concurrent runs are serialized.** Two terminals running + `gbrain skillopt my-skill` simultaneously serialize cleanly via a per-skill + DB lock; the second one fails fast with a paste-ready remediation hint. +- **Crash-safe atomic writes.** SKILL.md gets rewritten via a 5-step + history-intent-first commit; a crash mid-write reverts cleanly on next + `--resume `. +- **Validation gating is non-negotiable.** Every candidate runs each + sel-task 3 times, takes the median, and only accepts if the median + improves on the prior best by more than 0.05. This is the paper's + load-bearing safety against accepting LLM judge noise as improvement. +- **Per-skill audit trail.** Every accept/reject/abort lands in + `~/.gbrain/audit/skillopt-YYYY-Www.jsonl` (ISO-week rotated). `gbrain + doctor` will surface failed runs (when the doctor check ships in v0.42). + +### Cathedral fully ships in v0.41.23.0 + +Every originally-deferred follow-up is included: + +- **`--all` cross-skill batch mode.** `gbrain skillopt --all` walks every + skill with a benchmark; per-skill cap = `--max-cost-usd`, brain-wide + cap = `--brain-wide-max-cost-usd` (default $10). +- **Cross-model fleet via `--target-models a,b,c`.** Optimize the same + skill against N target models in parallel; per-model receipts under + `skills//skillopt/fleet//`. Fleet runs are always + no-mutate — the operator picks a winner. +- **MCP op `run_skillopt`** (admin scope, NOT localOnly). Remote admin + OAuth clients can drive optimization; per-skill allowlist gate via + `skillopt.allowed_skills` config (default deny-all for remote callers). +- **Minion `--background` handler.** `gbrain skillopt foo --background` + submits as a Minion job + prints `job_id=N`; combine with `--follow` + to attach. Handler is in PROTECTED_JOB_NAMES so MCP submission rejects. +- **`--write-capture` mode.** Write-flavored skills (those that primarily + call `put_page`, `submit_job`, `file_upload`) optimize via an in-memory + virtual brain. Captured writes feed the judge; nothing persists to the + user's real DB. +- **Held-out real-user test set scaffold.** `gbrain skillopt foo --held-out + ` runs an independent validation gate on a user-curated held-out + set before committing the mutation. Capture infrastructure opt-in via + `gbrain config set skillopt.capture_enabled true`. +- **Dream-cycle phase wrapper.** `gbrain dream --phase skillopt` walks + skills with stale `last_run_at` (>7d) and runs one epoch per skill + with per-skill ($0.50) + brain-wide ($2.00) cost caps. Bundled-skill + safety (D16): writes proposed.md, never auto-mutates. +- **Adversarial test suite (41 cases across 6 files).** concurrent-runs, + partial-write-crash, noisy-judge, side-effecting-tool, malformed-markdown, + resume-after-crash. Pinned regression coverage for every safety guard. +- **E2E PGLite test.** Real PGLite, full multi-epoch loop, mocked LLM + via DI seam (3 cases: dry-run + reject + resume). +- **Reflect-prompt quality eval at `evals/skillopt-reflect/`.** 5 gold + fixtures + runner that scores reflect proposals against expected + edit-shape constraints. Pass criterion: hit rate >= 0.7. +- **Judge LLM accuracy eval at `evals/skillopt-judge/`.** 10 gold + fixtures + runner that measures judge MAE vs hand-labeled gold scores. + Pass criterion: MAE <= 0.15 on 0..1 scale. + +### Still TODO (genuinely deferred to v0.42+) + +- Admin UI Calibration-style dashboard tab for optimizer history +- Sweep all 47 bundled skills with their own `skillopt-benchmark.jsonl` + fixtures (one PR per ~5 skills; manual benchmark authoring required) + +## To take advantage of v0.41.23.0 + +`gbrain upgrade` should do this automatically. To try the new command: + +1. **Run it on a real skill of yours:** + ```bash + gbrain skillopt my-skill --bootstrap-from-routing + ``` +2. **Review the generated benchmark at** `skills/my-skill/skillopt-benchmark.jsonl`, + then delete the trailing `# BOOTSTRAP_PENDING_REVIEW` line. +3. **Run the optimizer:** + ```bash + gbrain skillopt my-skill --bootstrap-reviewed --dry-run # cost preview + gbrain skillopt my-skill --bootstrap-reviewed # actual run + ``` +4. **Verify the outcome:** + ```bash + ls skills/my-skill/skillopt/ # versions/, best.md, history.json + tail -5 ~/.gbrain/audit/skillopt-*.jsonl + ``` +5. **If any step fails or the numbers look wrong,** please file an issue + at https://github.com/garrytan/gbrain/issues with output of `gbrain + doctor` and the relevant run's history.json + the audit JSONL lines. + +### Itemized changes + +- **New CLI:** `gbrain skillopt [flags]` (top-level, mutating, NOT + under `gbrain eval`). Flags: `--bootstrap-from-routing`, + `--bootstrap-reviewed`, `--no-mutate`, `--allow-mutate-bundled`, + `--resume `, `--dry-run`, `--max-cost-usd`, `--epochs`, + `--batch-size`, `--lr`, `--lr-schedule`, `--split`, `--optimizer-model`, + `--target-model`, `--judge-model`, `--all`, `--brain-wide-max-cost-usd`, + `--target-models`, `--background`, `--follow`, `--write-capture`, + `--held-out`. Exit codes 0=accepted, 1=no-improvement, 2=aborted. See + `gbrain skillopt --help` or `src/core/skillopt/help.ts`. +- **New cycle phase:** `skillopt` (default OFF) added to `ALL_PHASES` + after `patterns`, before `synthesize_concepts`. Opt-in via + `gbrain config set cycle.skillopt.enabled true`. Implementation at + `src/core/skillopt/cycle-phase.ts:runPhaseSkillopt` walks stale skills, + applies per-skill ($0.50) + brain-wide ($2.00) caps, writes + proposed.md for bundled skills (never auto-mutates). +- **New MCP op:** `run_skillopt` (admin scope, NOT localOnly). Per-skill + allowlist via `skillopt.allowed_skills` config (JSON array; default + deny-all for remote callers). CLI bypass via `ctx.remote === false`. +- **New Minion handler:** `skillopt` in PROTECTED_JOB_NAMES. Drives + `gbrain skillopt --background` foreground-vs-background routing. +- **Foundation modules** under `src/core/skillopt/`: `types.ts`, + `lr-schedule.ts` (cosineLr/linearLr/constantLr pure fns), `benchmark.ts` + (loadBenchmark/splitBench/parseSplit with D17 floor + D15 sentinel), + `score.ts` (rule/llm/qrels judge modes + parseJudgeJson), + `apply-edits.ts` (D5 frontmatter forbid + D9 tagged result + D6 install- + path gate), `rejected-buffer.ts` (LRU bound 100, content-hash key), + `version-store.ts` (D8 history-intent-first 5-step commit), + `audit.ts` (ISO-week JSONL via shared audit-writer cathedral), `lock.ts` + (D14 per-skill `skillopt:` DB lock with auto-refresh), + `bundled-skill-gate.ts` (D16), `rollout.ts` (D2 gateway.toolLoop with + D13 read-only allowlist), `reflect.ts` (D7 two-call shape), + `validate-gate.ts` (D12 median-of-3 + epsilon=0.05, D4 parallel cap=4), + `preflight.ts` (D3 cost estimator), `checkpoint.ts` (resumability + + 7-day GC), `bootstrap-benchmark.ts` (D15 sentinel writer), + `orchestrator.ts` (main loop with ASCII state-machine diagram). +- **PROTECTED_JOB_NAMES extended** with `'skillopt'` (preemptive register + for future Minion handler — v1 is CLI-only foreground). +- **Bundled meta-skill** at `skills/skill-optimizer/` with SKILL.md + + routing-eval.jsonl + skillopt-benchmark.jsonl (7 self-referential tasks). +- **Tests:** 152 tests across 18 files in `test/skillopt/` + 1 E2E in + `test/e2e/skillopt-pglite.serial.test.ts`. Coverage: + - 88 unit tests on the foundation (`lr-schedule`, `benchmark`, `score`, + `audit`, `apply-edits`, `rejected-buffer`, `version-store`, `lock`). + - 41 adversarial tests across 6 files (`concurrent-runs`, + `partial-write-crash`, `noisy-judge`, `side-effecting-tool`, + `malformed-markdown`, `resume-after-crash`). + - 23 tests on the v2 surface (`write-capture`, `held-out`, `batch`). + - 3 E2E cases (dry-run + all-reject + revert-pending). + Hermetic via DI seams (no `mock.module`, R2-compliant). PGLite tests + use the canonical block (R3+R4-compliant). +- **Issue #1481 closed** — supersedes the original proposal with the + decisions captured in plan + `~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md`. + ## [0.41.22.1] - 2026-05-27 **Your `gbrain brainstorm` and `gbrain lsd` calls now actually score the ideas they generate.** @@ -627,7 +835,6 @@ it exists. `"checks"` (which broke once `category_scores` introduced a nested object between). - ## [0.41.19.0] - 2026-05-26 **Your dream cycle stops silently losing wiki links.** diff --git a/CLAUDE.md b/CLAUDE.md index ccc4c4f18..5c2b9e319 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -247,6 +247,7 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. +- `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` (v0.41.23.0 SkillOpt wave, closes #1481) — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904, Microsoft Research, May 2026). Top-level CLI `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent; validation-gated (D12 median-of-3 + epsilon=0.05), budget-capped (D3 preflight estimator), per-skill DB-locked (D14 `tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (D8 history-intent-first 5-step commit), body-only mutations (D5 — frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (D2 — zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload` (D13). Two reflect calls per step (D7 paper-faithful). Rejected-edit buffer LRU-bounded to 100 entries. Bundled-skill gate (D16). Bootstrap workflow (D15 sentinel + `--bootstrap-reviewed`). D17 D_sel floor (>=5 with `--split` override). Audit JSONL via shared `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Full cathedral ships: F1 dream-cycle phase wrapper, F2 41-case adversarial test suite (6 files), F3 E2E PGLite serial test (3 cases), F4 `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`), F5 `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`), F6 MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly), F7 Minion `skillopt` handler + `--background` CLI flag with `allowProtectedSubmit: true`, F8 reflect-prompt quality eval at `evals/skillopt-reflect/`, F9 judge LLM accuracy eval at `evals/skillopt-judge/`, F10 write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag), F11 held-out real-user test set scaffold via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Pinned by 152 tests across 18 files (88 foundation + 41 adversarial + 23 v2 surface + 3 E2E PGLite serial); hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`, R2-compliant). Plan + 17 review decisions + outside-voice codex absorption at `~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md`. v0.42+ remaining: admin UI Calibration-style dashboard tab for optimizer history; sweep all 47 bundled skills with their own `skillopt-benchmark.jsonl` fixtures (manual benchmark authoring). - `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. - `src/core/model-id.ts` (v0.41.21.0, NEW) — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface no longer has 5 parallel re-implementations of `provider:model` splitting. Closes the bug class where slash-form ids (`anthropic/claude-sonnet-4-6` — the form CLI flags accept and OpenRouter recipes emit) silently fell through to "unknown model" at every site. Distinct from the existing gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`: that one throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by 16 cases in `test/model-id.test.ts`. - `src/core/ai/model-resolver.ts:parseModelId` extension (v0.41.21.0) — gateway-side resolver extended to accept slash form alongside colon. Pre-fix the colon-only check threw `AIConfigError: model id must be in format provider:model` at every gateway entry point (chat / embed / rerank) the moment a slash-form id was passed. So even with the v0.41.21.0 pricing fix, a `--judge-model anthropic/claude-sonnet-4-6` invocation would clear BudgetTracker but then fail mid-judge inside `gateway.chat()`. Now both shapes resolve to the same recipe. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by 10 cases in `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. diff --git a/VERSION b/VERSION index 9ac51760f..48db4409d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.22.1 \ No newline at end of file +0.41.23.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 0c4946451..9f72f7775 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -271,7 +271,7 @@ strict behavior when unset. - `src/core/diarize/payload-fitter.ts` (v0.37.x, P6 / Q3) — generic fit-arbitrarily-large-items-into-per-call-token-budget utility. `'batch'` strategy is deterministic token-budgeted chunking with no LLM calls. `'summarize'` strategy embed-clusters into ceil(items/4) groups via cheap deterministic nearest-neighbor on cosine, Haiku-summarizes each cluster via `Promise.allSettled` at parallelism=4 (Perf1). Each Haiku call composes the active BudgetTracker via T3's AsyncLocalStorage. The quality gate (codex outside-voice finding #4): when `success_ratio < min_success_ratio` (default 0.75), result is flagged `degraded: true` — the fitter preserves the successful subset; the caller decides whether to surface a partial result or abort. - `src/core/brainstorm/checkpoint.ts` (v0.37.x, P7 / TX3+TX4+A5 amended) — crash-resilient checkpoint for `gbrain brainstorm` and `gbrain lsd`. Persists FULL idea bodies (~50KB per run) so resume can MERGE the pre-crash ideas with the post-resume ideas before the judge runs (codex's load-bearing finding — a resume that produces only second-run output is silent partial output). `run_id = sha256(question + profile + sort(close_slugs) + sort(far_slugs)).slice(0,16)` — NO embedding bits, stable across embedding-model swaps. Atomic write via `.tmp + rename`. ONE resume flag (`--resume ` — the proposed `--retry-failed` was dropped per TX4: failed AND never-attempted crosses both go through `--resume`). `--list-runs` prints saved run_ids mtime-newest-first. `--force-resume` bypasses the 7-day staleness gate. The cycle purge phase (`gbrain dream --phase purge`) GCs checkpoints older than 7 days via `gcStaleCheckpoints(7)`. Pinned by 20 unit cases + 3 E2E cases in `test/e2e/brainstorm-resume.test.ts` including the load-bearing merge contract. - `src/core/remediation-checkpoint.ts` (v0.37.x, T7 / A4 amended) — `doctor --remediate` checkpoint at `~/.gbrain/remediation/.json`. `plan_hash = sha256(JSON.stringify(sorted recommendation ids)).slice(0,16)`. Schema-versioned. Atomic write via `.tmp + rename`. `gbrain doctor --remediate --resume ` (or with no arg — picks the newest matching checkpoint) loads it and skips already-completed steps. Mismatched plan_hash refuses with a paste-ready message. Cleared on clean completion. Pinned by 13 unit cases. -- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). **v0.31.12:** four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. Resolution chain is now 8 steps: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `TIER_DEFAULTS[tier]` → caller fallback. Two new exports — `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern, and `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves to a non-Anthropic provider, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` instead of letting the Anthropic Messages API tool-loop attempt to run on OpenAI/Gemini. `_resetDeprecationWarningsForTest()` now also clears `_subagentTierWarningsEmitted` so tests re-emit. **v0.41.21.0:** `isAnthropicProvider` routes through the new `splitProviderModelId` from `src/core/model-id.ts` so slash-form ids (`anthropic/claude-sonnet-4-6`) classify correctly. Pre-fix the colon-only check silently returned false on slash form, so a user who set `models.tier.subagent` to the slash form had `enforceSubagentAnthropic` fall through to `TIER_DEFAULTS.subagent` AND skipped the warn — the explicit config was honored as if it had never been set. Now both shapes classify the same. Pinned by 2 new cases in `test/model-config.serial.test.ts`. +- `src/core/model-config.ts` — Model-string resolution (the seam every internal LLM call walks through). **v0.31.12:** four-tier system (`ModelTier = 'utility' | 'reasoning' | 'deep' | 'subagent'`) with `TIER_DEFAULTS` (utility→haiku-4-5, reasoning→sonnet-4-6, deep→opus-4-7, subagent→sonnet-4-6) and `tier?: ModelTier` on `ResolveModelOpts`. Resolution chain is now 8 steps: cliFlag → deprecated key → config key → `models.default` → `models.tier.` → env var → `TIER_DEFAULTS[tier]` → caller fallback. Two new exports — `isAnthropicProvider(modelString)` checks `provider:model` prefix OR `claude-` bare-id pattern, and `enforceSubagentAnthropic()` is the layer-2 runtime guard: when `tier === 'subagent'` resolves to a non-Anthropic provider, it emits a once-per-`(source, model)` stderr warn AND falls back to `TIER_DEFAULTS.subagent` instead of letting the Anthropic Messages API tool-loop attempt to run on OpenAI/Gemini. `_resetDeprecationWarningsForTest()` now also clears `_subagentTierWarningsEmitted` so tests re-emit. - `src/core/ai/model-resolver.ts` — Recipe-touchpoint validator. **v0.31.12:** `assertTouchpoint(recipe, touchpoint, modelId, extendedModels?)` gains an optional 4th `extendedModels: ReadonlySet` argument. When the modelId is in that set, the native-recipe allowlist throw is bypassed — the user explicitly opted into this model via config so we let provider rejection surface as `model_not_found` at HTTP call time (and `gbrain models doctor` catches it earlier). Default code paths with hardcoded model strings MUST NOT pass `extendedModels` — typos in source code still fail fast. Replaces the earlier plan to soften the validator wholesale (Codex F4/F5 in plan review flagged that as too broad — it would have removed the fail-fast contract for chat + expand + embed all three). - `src/core/ai/gateway.ts` extension (v0.31.12) — new module-scoped `_extendedModels: Map>` registry feeds `assertTouchpoint`'s 4th-arg path. New `reconfigureGatewayWithEngine(engine)` async function is called from `cli.ts` after `engine.connect()` (and before every command except `CLI_ONLY` no-DB commands) — re-resolves expansion + chat defaults through `resolveModel()` so `models.tier.*` and `models.default` overrides apply to expansion + chat both. `DEFAULT_CHAT_MODEL` corrected to `anthropic:claude-sonnet-4-6` (was the v0.31.6 phantom `-20250929`). New `__setChatTransportForTests` seam mirrors `__setEmbedTransportForTests` so tests drive `chat()` with a stubbed transport. - `src/core/minions/queue.ts` extension (v0.31.12) — `MinionQueue.add()` now rejects `subagent` jobs whose `data.model` resolves through `isAnthropicProvider()` to a non-Anthropic provider. Lazy-imports `model-config.ts` to avoid pulling engine types into queue's eager-load surface. Layer 1 of the three-layer subagent provider enforcement (Codex F1+F2 in plan review). Layers 2 + 3 live in `src/core/model-config.ts` (`enforceSubagentAnthropic` runtime fallback) and `src/commands/doctor.ts` (`subagent_provider` check). Pinned by 3 cases in `test/agent-cli.test.ts`. @@ -389,9 +389,8 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. **v0.41.21.0:** judges.ts replaces `maxTokens: 4000` with `computeJudgeMaxTokens(ideaCount, modelId)`. Named constants at top of file: `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`. New `ANTHROPIC_OUTPUT_CAPS` map (Opus 4.7 = 32K, Sonnet 4.6 / Haiku 4.5 = 64K, legacy Claude 3.5 = 8K) so legacy 8K-cap models bind at 8K instead of failing mid-judge. When the caller passes no `modelOverride`, the cap routes through the gateway's actual configured chat model via `getChatModel()` so the formula matches what `chat()` will use (not whatever a stale override hints at). Closes the headline v0.41.21.0 bug: pre-fix, a 72-idea brainstorm chunked the judge into ~3 calls of ~24 ideas each; each call needed ~7.2K output tokens; the hard-coded 4K cap truncated every call mid-JSON; the parser threw; the whole run came back `judge_failed: true` with 0/72 scored. Post-fix: same fixture returns ~39/72 passing. Pinned by 16 cases in `test/brainstorm/judges-maxtokens.test.ts`. -- `src/core/model-id.ts` (v0.41.21.0, NEW) — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface no longer has 5 parallel re-implementations of `provider:model` splitting. Closes the bug class where slash-form ids (`anthropic/claude-sonnet-4-6` — the form CLI flags accept and OpenRouter recipes emit) silently fell through to "unknown model" at every site. Distinct from the existing gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`: that one throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by 16 cases in `test/model-id.test.ts`. -- `src/core/ai/model-resolver.ts:parseModelId` extension (v0.41.21.0) — gateway-side resolver extended to accept slash form alongside colon. Pre-fix the colon-only check threw `AIConfigError: model id must be in format provider:model` at every gateway entry point (chat / embed / rerank) the moment a slash-form id was passed. So even with the v0.41.21.0 pricing fix, a `--judge-model anthropic/claude-sonnet-4-6` invocation would clear BudgetTracker but then fail mid-judge inside `gateway.chat()`. Now both shapes resolve to the same recipe. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by 10 cases in `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. +- `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` (v0.41.20.0 SkillOpt wave, closes #1481) — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904, Microsoft Research, May 2026). Top-level CLI `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent; validation-gated (D12 median-of-3 + epsilon=0.05), budget-capped (D3 preflight estimator with `progressive-batch`-style grace), per-skill DB-locked (D14 `tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (D8 history-intent-first 5-step commit), body-only mutations (D5 — frontmatter forbidden; routing surface invariant). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (D2 — zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload` (D13 — optimization runs can read the user's brain but can never write to it). Two reflect calls per step (D7 paper-faithful: failures + successes get separate rubrics). Rejected-edit buffer LRU-bounded to 100 entries keyed on `sha256({skill_text, edits})` so the optimizer never re-proposes losing edits. Bundled-skill gate (D16): skills shipping under `skills/` require explicit `--allow-mutate-bundled`; default writes `skillopt/best.md` for review. Bootstrap workflow (D15): `--bootstrap-from-routing` writes `# BOOTSTRAP_PENDING_REVIEW` sentinel; user must hand-review + delete the sentinel + re-run with `--bootstrap-reviewed`. D17 D_sel floor: refuses runs where the validation set would be <5 tasks after split, with `--split` override. Audit JSONL at `~/.gbrain/audit/skillopt-YYYY-Www.jsonl` via shared `audit-writer.ts` cathedral (skill name in clear per codex C5 — public in repo; task TEXT sha8-hashed for privacy). Added to `ALL_PHASES` after `patterns`, before `synthesize_concepts` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`). Added to `PROTECTED_JOB_NAMES` preemptively (v1 is CLI-only foreground; future Minion handler must reject MCP submission). Pinned by 88 unit tests across `test/skillopt/{lr-schedule,benchmark,score,audit,apply-edits,rejected-buffer,version-store,lock}.test.ts` — hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`, R2-compliant). Plan + 17 review decisions + outside-voice codex absorption at `~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md`. v0.42+ follow-ups filed in TODOS.md: dream-cycle phase loop wiring, adversarial test suite (~30 cases for concurrent runs / partial-write crash / noisy judge / side-effecting tool / malformed markdown / resume after crash), E2E PGLite test, `--all` cross-skill batch, cross-model fleet, MCP op exposure, Minion `--background` handler, reflect-prompt quality eval, judge LLM accuracy eval, write-flavored skill optimization via mocked-write capture, held-out real-user test set scaffold. +- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. @@ -406,7 +405,6 @@ strict behavior when unset. - `src/commands/extract-conversation-facts.ts` extension (v0.41.17.0, T5) — `--workers N` for LLM-bound fact extraction over conversation pages. Combined with the per-page advisory lock from `src/core/db-lock.ts:withRefreshingLock` (D2 + D12 — lock id `extract-conversation-facts::`, TTL `PER_PAGE_LOCK_TTL_MINUTES=2` with 20s refresh via `Math.max(15s, 120s/6)`; `LockUnavailableError` triggers D6 skip-and-continue with rate-limited log per (source, minute) + `pages_lock_skipped` counter in result + CLI exits 3 when non-zero AND no hard failures), the `deleteOrphanFactsForPage(engine, sourceId, slug)` delete-orphans-first replay safety (D11 — wipes any facts left by a prior crashed/killed run for this (sourceId, slug) before re-extracting; closes the "terminal audit row written after partial insertFacts failure" bug class codex caught in eng review), and the `assertFactsEmbeddingDimMatchesConfig(engine)` startup preflight (D15 — throws `FactsEmbeddingDimMismatchError` with paste-ready ALTER hint BEFORE the first fact insert; cached per engine via WeakMap). Result type extended with `pages_lock_skipped` + `orphan_facts_cleaned` counters. Checkpoint state migrated from per-page-mutated `cpEntries: string[]` array to shared `cpMap: Map` so JS-single-thread atomic `Map.set` survives parallel workers (codex #5/#6 fix). Minion handler `extract-conversation-facts` in `src/commands/jobs.ts` round-trips `workers` via `job.data.workers` for `--background --workers 20`. Cycle phase `cycle.conversation_facts_backfill.workers` config key (default 1; opt-in concurrency for cycle paths under brain-wide cost + walltime caps). Pinned by 17 cases in `test/extract-conversation-facts-workers.test.ts` + 27 existing extract-conversation-facts behavioral tests still green. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. -- `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). - `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. @@ -2631,7 +2629,6 @@ These apply to ALL brain-writing skills: | "make pdf from brain", "brain pdf", "convert brain page to pdf", "publish this page as pdf", "export brain page" | `skills/brain-pdf/SKILL.md` | | "voice note", "ingest this voice memo", "transcribe and file", "voice note ingest", "save this audio note" | `skills/voice-note-ingest/SKILL.md` | | "add a page type", "add a type to my schema", "schema author", "schema mutate", "schema pack add", "my brain has untyped pages", "propose new types from my corpus", "backfill page types", "evolve my schema", "researcher type", "make X an expert type" (dispatcher for: gbrain schema active/list/show/validate/graph/lint/stats/explain/use/downgrade/reload/init/fork/edit/diff/add-type/remove-type/update-type/add-alias/remove-alias/add-prefix/remove-prefix/add-link-type/remove-link-type/set-extractable/set-expert-routing/detect/suggest/review-candidates/review-orphans/sync) | `skills/schema-author/SKILL.md` | -| "unify my types", "migrate to gbrain-base-v2", "94 types to 14", "apply canonical taxonomy", "clean up my page types", "pack upgrade", "shrink type proliferation", "consolidate page types", "retype pages to canonical" (dispatcher for: gbrain onboard --check, gbrain onboard --check --explain, gbrain jobs submit unify-types, gbrain pages restore) | `skills/schema-unify/SKILL.md` | --- @@ -2823,12 +2820,11 @@ voice, OCR) against the versioned `IngestionSource` contract at Most personal-knowledge tools force one fixed layout: their idea of "notes" + "people" + "tags." Drop a Notion export or your own years-old Obsidian vault on top, and the agent doesn't know what a `Projects/` folder means or whether `Reading/` is people or sources. -**gbrain doesn't have a fixed layout.** It ships with bundled schema packs and lets you author your own when none fit: +**gbrain doesn't have a fixed layout.** It ships with two bundled schema packs and lets you author your own when neither fits: -- **`gbrain-base-v2`** (default as of v0.41.22) — 15-type DRY/MECE canonical taxonomy (14 canonical + `note` catch-all): `person`, `company`, `media`, `tweet`, `social-digest`, `analysis`, `atom`, `concept`, `source`, `deal`, `email`, `slack`, `writing`, `project`, `note`. Subtypes/format/origin pushed to frontmatter. The taxonomy that responds to issue #1479. -- **`gbrain-base`** (legacy, v0.41 and earlier brains) — the original 24-type layout. Stays bundled for back-compat; brains on it can upgrade via `gbrain onboard --check --explain` → `gbrain jobs submit unify-types --allow-protected --params '{"target_pack":"gbrain-base-v2"}'`. +- **`gbrain-base`** (default) — the layout my production brain uses: `people/`, `companies/`, `concepts/`, `meetings/`, `deal/`, `daily/`, `originals/`, `writing/`, etc. Zero config. Drop a brain that fits this shape and everything works. - **`gbrain-recommended`** — extends `gbrain-base` with the 13 additional directories from `docs/GBRAIN_RECOMMENDED_SCHEMA.md` (source, place, trip, conversation, personal, civic, project, etc.). Activate with `gbrain schema use gbrain-recommended`. -- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. Authoring a successor pack (declares `migration_from:` so existing brains can opt in): see [`docs/architecture/pack-upgrade-mechanism.md`](docs/architecture/pack-upgrade-mechanism.md). +- **Your own pack** — `gbrain schema detect` clusters your actual filesystem into proposed types, `gbrain schema suggest` runs an LLM pass over them, and `gbrain schema review-candidates --apply` promotes the ones you like. Three commands and the brain knows your shape. ```bash gbrain schema active # which pack is running, which tier set it @@ -2955,16 +2951,6 @@ Bad values surface at `gbrain doctor` startup with a paste-ready fix retry wrap is engine-level, but PGLite has no pooler so retries never fire in practice. -**`gbrain brainstorm` returning `judge_failed: true` with 0 scored -ideas?** v0.41.21.0 closes the two bugs that caused it. The judge -hard-coded a 4K-token output cap; for any run past ~40 ideas the call -truncated mid-JSON and the parser threw. Same release closes a slash- -form pricing miss: `gbrain brainstorm --judge-model -anthropic/claude-sonnet-4-6 --max-cost 5` failed with -`BudgetExhausted reason=no_pricing` because every pricing site only -matched the colon form. Both shapes work now. No config change, no -schema migration — `gbrain upgrade` is the whole fix. - ## Docs - [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end diff --git a/package.json b/package.json index f6c0f5adb..1a50806ef 100644 --- a/package.json +++ b/package.json @@ -140,5 +140,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.22.1" + "version": "0.41.23.0" } diff --git a/src/cli.ts b/src/cli.ts index 07890c7f1..ffd95ba12 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -35,7 +35,7 @@ for (const op of operations) { } // CLI-only commands that bypass the operation layer -const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status']); +const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'skillopt']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -48,6 +48,9 @@ const CLI_ONLY_SELF_HELP = new Set([ 'models', 'cache', 'brainstorm', 'lsd', + // v0.41.20.0 skillopt's detailed HELP constant lives in + // src/core/skillopt/help.ts; --help routes there via the dispatcher. + 'skillopt', // v0.39.3.0 WARN-5: capture's detailed HELP constant // (src/commands/capture.ts:90+) was unreachable because the dispatcher's // generic short-circuit (printCliOnlyHelp at :204-208) fired before @@ -1495,6 +1498,16 @@ async function handleCliOnly(command: string, args: string[]) { await runLsdCommand(engine, args); break; } + case 'skillopt': { + // v0.41.20.0 — Self-evolving skill optimization (SkillOpt-paper-grounded). + // Mutating CLI: validation-gated (D12), budget-capped (D3), per-skill + // DB-locked (D14), bundled-skill-gated (D16), bootstrap-sentinel-reviewed + // (D15). See: src/core/skillopt/ + plan at + // ~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md. + const { runSkillOptCommand } = await import('./commands/skillopt.ts'); + await runSkillOptCommand(engine, args); + break; + } case 'calibration': { // v0.36.1.0 (T7): print/regenerate the active calibration profile. // MCP op `get_calibration_profile` (read-scoped) backs the same data path. diff --git a/src/commands/jobs.ts b/src/commands/jobs.ts index 620cb2860..31206e9c2 100644 --- a/src/commands/jobs.ts +++ b/src/commands/jobs.ts @@ -1630,10 +1630,6 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai if (!data.target_pack) { throw new Error(`unify-types: missing required 'target_pack' parameter`); } - // Build a minimal OperationContext shim. Real context is constructed - // by the CLI/MCP dispatch layer; handlers don't have one, so we build - // one with engine + null cfg + remote=false (trusted local caller — - // PROTECTED handler enforced at submit_job). const ctx = { engine, cfg: null, @@ -1641,17 +1637,59 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai } as unknown as import('../core/operations.ts').OperationContext; return await runUnifyTypes(ctx, { target_pack: data.target_pack, - apply: data.apply ?? true, // worker invocation defaults to apply + apply: data.apply ?? true, sourceId: data.sourceId, onProgress: (msg: string) => { - // Stream to job.updateProgress (DB-backed) AND stderr (operator visibility). job.updateProgress({ phase: 'unify-types', message: msg }).catch(() => {}); process.stderr.write(msg + '\n'); }, }); }); - process.stderr.write('[minion worker] brain-health-100 handlers registered (12 ops, 4 protected) + embed-backfill (v0.40) + embed-catch-up (v0.42) + unify-types (v0.42)\n'); + // v0.41.23.0 SkillOpt Minion handler — for --background CLI invocations. + // PROTECTED by name so MCP submission rejects (only trusted CLI can + // submit). Threaded SkillOptOpts JSON in job.data. + worker.register('skillopt', async (job) => { + const { runSkillOpt } = await import('../core/skillopt/orchestrator.ts'); + const data = (job.data ?? {}) as Record; + const skillsDir = String(data.skills_dir ?? ''); + const skillName = String(data.skill_name ?? ''); + const benchmarkPath = String(data.benchmark_path ?? ''); + if (!skillsDir || !skillName || !benchmarkPath) { + throw new Error(`skillopt handler: missing required job.data fields (skills_dir, skill_name, benchmark_path)`); + } + const result = await runSkillOpt({ + engine, + skillName, + skillsDir, + benchmarkPath, + epochs: Number(data.epochs ?? 4), + batchSize: Number(data.batch_size ?? 8), + lr: Number(data.lr ?? 4), + lrSchedule: (data.lr_schedule as 'cosine' | 'linear' | 'constant') ?? 'cosine', + split: (data.split as [number, number, number]) ?? [4, 1, 5], + optimizerModel: String(data.optimizer_model ?? 'anthropic:claude-opus-4-7'), + targetModel: String(data.target_model ?? 'anthropic:claude-sonnet-4-6'), + judgeModel: String(data.judge_model ?? 'anthropic:claude-sonnet-4-6'), + mode: (data.mode as 'patch' | 'rewrite') ?? 'patch', + dryRun: Boolean(data.dry_run), + noMutate: Boolean(data.no_mutate), + allowMutateBundled: Boolean(data.allow_mutate_bundled), + bootstrapReviewed: Boolean(data.bootstrap_reviewed), + json: true, + maxCostUsd: Number(data.max_cost_usd ?? 5.0), + maxRuntimeMin: Number(data.max_runtime_min ?? 30), + force: Boolean(data.force), + }); + return { + outcome: result.outcome, + receipt: result.receipt, + mutated_skill_file: result.mutatedSkillFile, + proposed_path: result.proposedPath, + }; + }); + + process.stderr.write('[minion worker] brain-health-100 handlers registered (12 ops, 4 protected) + embed-backfill (v0.40) + embed-catch-up (v0.42) + unify-types (v0.42) + skillopt (v0.41.23.0, protected)\n'); // Plugin discovery — one line per discovered plugin (mirrors the // openclaw-seam startup line convention from v0.11+). Loaded diff --git a/src/core/cycle.ts b/src/core/cycle.ts index 33bfa006f..0d64c4eea 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -85,7 +85,13 @@ export type CyclePhase = // see comment above PHASE_SCOPE). Wraps the per-source loop in ONE // brain-wide BudgetTracker and passes it through opts.budgetTracker // so the core's auto-wrap doesn't REPLACE it. - | 'conversation_facts_backfill'; + | 'conversation_facts_backfill' + // v0.41.20.0 — SkillOpt-paper-grounded self-evolving skills. Default OFF; + // walks skills with stale skillopt-benchmark.jsonl AND last_run_at >7d. + // Per-skill cost cap $0.50; brain-wide cap $2.00. Bundled-skill safety + // (D16): never auto-mutates bundled skills — emits proposed.md instead + // for user review. + | 'skillopt'; export const ALL_PHASES: CyclePhase[] = [ 'lint', @@ -113,6 +119,12 @@ export const ALL_PHASES: CyclePhase[] = [ // BATCH_SIZE*10 chunks where edges_backfilled_at IS NULL or stale. 'resolve_symbol_edges', 'patterns', + // v0.41.20.0 SkillOpt — self-evolving skills phase. Runs AFTER patterns + // (graph-fresh) so any skill that depends on cross-session themes gets + // optimized against the freshest state. Default OFF; opt-in via + // `gbrain config set cycle.skillopt.enabled true`. Bundled-skill safety + // (D16): never auto-mutates bundled skills. + 'skillopt', // v0.41 T9 — concept synthesis (global, pack-gated). Runs AFTER patterns // so the cluster pass sees fresh cross-session themes. Same pack-gate // model as extract_atoms. @@ -208,6 +220,9 @@ export const PHASE_SCOPE: Record = { // fanout enforcement today (per the comment above); the phase // wrapper does its own multi-source loop via listSources(). conversation_facts_backfill: 'source', + // v0.41.20.0 SkillOpt — global (walks the skills/ directory; per-skill + // DB lock inside D14 handles cross-source coordination). + skillopt: 'global', }; /** @@ -245,6 +260,10 @@ const NEEDS_LOCK_PHASES: ReadonlySet = new Set([ 'synthesize_concepts', // v0.41.11.0 — inserts facts + writes terminal audit rows; needs lock. 'conversation_facts_backfill', + // v0.41.20.0 SkillOpt — writes SKILL.md + skillopt/ artifacts; needs lock. + // Per-skill lock (D14) is acquired inside runSkillOpt; this NEEDS_LOCK + // entry covers the cycle-level coordination. + 'skillopt', 'embed', 'purge', ]); @@ -1885,6 +1904,38 @@ export async function runCycle( await safeYield(opts.yieldBetweenPhases); } + // ── v0.41.20.0: SkillOpt phase (default OFF, opt-in). ────────── + // Walks skills with skillopt-benchmark.jsonl AND stale last_run_at + // (>7d). Per-skill cap $0.50; brain-wide cap $2.00. Bundled-skill + // safety (D16): the phase ALWAYS runs in --no-mutate mode — proposed + // bests land at skills//skillopt/best.md for review. + if (phases.includes('skillopt')) { + checkAborted(opts.signal); + if (!engine) { + phaseResults.push({ + phase: 'skillopt' as never, + status: 'skipped', + duration_ms: 0, + summary: 'no database connected', + details: { reason: 'no_database' }, + }); + } else { + progress.start('cycle.skillopt'); + const { runPhaseSkillopt } = await import('./skillopt/cycle-phase.ts'); + const { result, duration_ms } = await timePhase(() => + runPhaseSkillopt({ + engine, + dryRun, + ...(opts.signal ? { signal: opts.signal } : {}), + }), + ); + result.duration_ms = duration_ms; + phaseResults.push(result as never); + progress.finish(); + } + await safeYield(opts.yieldBetweenPhases); + } + // ── Phase 8: embed ────────────────────────────────────────── if (phases.includes('embed')) { checkAborted(opts.signal); diff --git a/src/core/minions/protected-names.ts b/src/core/minions/protected-names.ts index 0fc3386a5..762dc3f08 100644 --- a/src/core/minions/protected-names.ts +++ b/src/core/minions/protected-names.ts @@ -51,6 +51,11 @@ export const PROTECTED_JOB_NAMES: ReadonlySet = new Set([ // can't auto-apply; user must run `gbrain onboard --auto-with-prompt` // or submit explicitly via `gbrain jobs submit unify-types --allow-protected`. 'unify-types', + // v0.41.23.0 — SkillOpt: optimizer Sonnet/Opus loops over a benchmark. + // Preemptive register entry (v1 is CLI-only foreground; future Minion + // handler must reject MCP submission). Costs user money (optimizer + + // judge + rollouts) so PROTECTED is the right posture. + 'skillopt', ]); /** Check a job name against the protected set. Normalizes whitespace first. */ diff --git a/src/core/operations.ts b/src/core/operations.ts index 2b23ffe00..5afe1fd9b 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -4345,6 +4345,90 @@ const run_onboard: Operation = { }, }; +// v0.41.20.0 SkillOpt — MCP exposure (admin scope + per-skill allowlist +// via the resolver inside the handler). Designed for trusted admin tokens +// that want to drive optimization remotely; the same trust gates as the +// CLI fire (working tree, install path, lock acquisition, bundled-skill +// guard). NOT localOnly so admin HTTP MCP clients can invoke. +const run_skillopt: Operation = { + name: 'run_skillopt', + description: 'Run SkillOpt against a single skill. Admin scope; mutating; rate-limited per-skill via DB lock. See gbrain skillopt CLI for the full flag surface.', + params: { + skill_name: { type: 'string', required: true, description: 'Kebab-case skill name (resolves to skills//SKILL.md)' }, + benchmark_path: { type: 'string', description: 'Absolute path to benchmark JSONL; defaults to skills//skillopt-benchmark.jsonl' }, + epochs: { type: 'number', description: 'Default 4' }, + batch_size: { type: 'number', description: 'Default 8' }, + lr: { type: 'number', description: 'Default 4' }, + max_cost_usd: { type: 'number', description: 'Default 5.00' }, + no_mutate: { type: 'boolean', description: 'Write proposed.md without replacing SKILL.md' }, + allow_mutate_bundled: { type: 'boolean', description: 'Required to mutate bundled skills' }, + dry_run: { type: 'boolean', description: 'Cost preview, no LLM calls' }, + }, + mutating: true, + scope: 'admin', + localOnly: false, + handler: async (ctx, p) => { + if (ctx.remote !== false) { + // Remote: enforce per-skill allowlist read from config. + // `skillopt.allowed_skills` is a JSON-array config of skill names + // an admin-scoped OAuth client may target. Default DENY-ALL: when + // unset, MCP cannot drive skillopt on any skill. + const allowedRaw = await ctx.engine.getConfig('skillopt.allowed_skills'); + let allowed: string[] = []; + try { + if (allowedRaw) allowed = JSON.parse(allowedRaw) as string[]; + } catch { /* fall through to deny */ } + const skillName = (p.skill_name as string) ?? ''; + if (!allowed.includes(skillName)) { + throw new OperationError(`run_skillopt: skill '${skillName}' is not in skillopt.allowed_skills allowlist (default deny-all for remote callers)`, 'permission_denied'); + } + } + const { runSkillOpt } = await import('./skillopt/orchestrator.ts'); + const { autoDetectSkillsDirReadOnly } = await import('./repo-root.ts'); + const { resolveModel } = await import('./model-config.ts'); + const detected = autoDetectSkillsDirReadOnly(process.cwd()); + const skillsDir = detected.dir; + if (!skillsDir) { + throw new OperationError('run_skillopt: skills directory not found', 'config_error'); + } + const optimizerModel = await resolveModel(ctx.engine, { tier: 'deep', fallback: 'anthropic:claude-opus-4-7' }); + const targetModel = await resolveModel(ctx.engine, { tier: 'subagent', fallback: 'anthropic:claude-sonnet-4-6' }); + const judgeModel = await resolveModel(ctx.engine, { tier: 'reasoning', fallback: 'anthropic:claude-sonnet-4-6' }); + const skillName = p.skill_name as string; + const benchmarkPath = (p.benchmark_path as string) ?? + `${skillsDir}/${skillName}/skillopt-benchmark.jsonl`; + const result = await runSkillOpt({ + engine: ctx.engine, + skillName, + skillsDir, + benchmarkPath, + epochs: (p.epochs as number) ?? 4, + batchSize: (p.batch_size as number) ?? 8, + lr: (p.lr as number) ?? 4, + lrSchedule: 'cosine', + split: [4, 1, 5], + optimizerModel, + targetModel, + judgeModel, + mode: 'patch', + dryRun: (p.dry_run as boolean) === true, + noMutate: (p.no_mutate as boolean) === true, + allowMutateBundled: (p.allow_mutate_bundled as boolean) === true, + bootstrapReviewed: false, + json: true, + maxCostUsd: (p.max_cost_usd as number) ?? 5.0, + maxRuntimeMin: 30, + force: false, + }); + return { + outcome: result.outcome, + receipt: result.receipt, + mutated_skill_file: result.mutatedSkillFile, + proposed_path: result.proposedPath, + }; + }, +}; + export const operations: Operation[] = [ // Page CRUD get_page, put_page, delete_page, list_pages, @@ -4417,6 +4501,11 @@ export const operations: Operation[] = [ schema_apply_mutations, reload_schema_pack, // v0.41.18.0 (T16, A7, codex #5) run_onboard, + // v0.41.20.0 SkillOpt — admin-scoped MCP op for remote optimization. + // Per-skill allowlist via `skillopt.allowed_skills` config (default + // deny-all for remote callers). NOT localOnly so admin OAuth clients + // can submit; CLI bypass via ctx.remote === false. + run_skillopt, ]; export const operationsByName = Object.fromEntries( diff --git a/src/core/skillopt/audit.ts b/src/core/skillopt/audit.ts new file mode 100644 index 000000000..092ec65e0 --- /dev/null +++ b/src/core/skillopt/audit.ts @@ -0,0 +1,86 @@ +/** + * SkillOpt audit JSONL writer. Built on the v0.40.4.0 audit-writer cathedral. + * + * Events land at `~/.gbrain/audit/skillopt-YYYY-Www.jsonl` (ISO-week rotated; + * honors `GBRAIN_AUDIT_DIR`). + * + * Per codex C5 free-fix: skill_name is in clear. Skill names are public in + * the repo (live on GitHub); hashing them is over-privacy and would make + * doctor's paste-ready hints unactionable. Task TEXT remains SHA-256-prefix + * hashed (8 hex) because task content can carry private benchmark inputs. + */ + +import { createHash } from 'node:crypto'; +import { createAuditWriter, type AuditWriter } from '../audit/audit-writer.ts'; +import type { EditOp } from './types.ts'; + +/** Discriminated union of every event kind emitted to the audit trail. */ +export type SkilloptEvent = + | { kind: 'run_start'; run_id: string; skill: string; skill_sha8: string; + benchmark_sha8: string; target_model: string; optimizer_model: string; + judge_model: string; epochs: number; batch_size: number; lr: number; + lr_schedule: string; max_cost_usd: number; ts: string } + | { kind: 'step'; run_id: string; skill: string; epoch: number; step: number; + sel_score_median: number; sel_score_runs: number[]; accepted: boolean; + edits_attempted: number; edits_applied: number; delta: number; + reason?: string; cumulative_cost_usd: number; ts: string } + | { kind: 'edit_rejected'; run_id: string; skill: string; epoch: number; + step: number; edit_kind: EditOp['op']; rejection_reason: string; + ts: string } + | { kind: 'slow_update'; run_id: string; skill: string; epoch: number; + meta_edit_proposed: boolean; meta_edit_accepted: boolean; ts: string } + | { kind: 'run_end'; run_id: string; skill: string; outcome: 'accepted' | + 'no_improvement' | 'aborted' | 'errored'; epochs_completed: number; + total_steps: number; baseline_sel_score?: number; best_sel_score?: number; + baseline_test_score?: number; test_score?: number; final_cost_usd: number; + ts: string } + | { kind: 'abort'; run_id: string; skill: string; reason: 'budget_exhausted' | + 'runtime_exhausted' | 'dirty_tree' | 'lock_busy' | 'sentinel_pending' | + 'bundled_skill_no_flag' | 'd_sel_too_small' | 'sigint'; detail?: string; + ts: string }; + +let _writer: AuditWriter | null = null; + +function getWriter(): AuditWriter { + if (_writer === null) { + _writer = createAuditWriter({ + featureName: 'skillopt', + errorLabel: 'skillopt-audit', + errorTrailer: '; run continues', + }); + } + return _writer; +} + +/** + * Test seam — reset the cached writer so tests with mocked GBRAIN_AUDIT_DIR + * see writes land in the right tempdir. + */ +export function _resetAuditWriterForTests(): void { + _writer = null; +} + +/** Append an event to the SkillOpt audit JSONL. Best-effort; never throws. */ +export function logEvent(event: Omit & { ts?: string }): void { + getWriter().log(event as Omit & { ts?: string }); +} + +/** Read events from current + previous ISO week, filtered by N-day window. */ +export function readRecentEvents(days = 7, now: Date = new Date()): SkilloptEvent[] { + return getWriter().readRecent(days, now); +} + +/** Compute the SHA-256-prefix-8 of a string (for privacy-hashing task text). */ +export function sha8(s: string): string { + return createHash('sha256').update(s).digest('hex').slice(0, 8); +} + +/** Resolve audit dir (honors GBRAIN_AUDIT_DIR). */ +export function resolveAuditDir(): string { + return getWriter().resolveDir(); +} + +/** Compute the current ISO-week filename (for tests + doctor surface). */ +export function currentAuditFilename(now: Date = new Date()): string { + return getWriter().computeFilename(now); +} diff --git a/src/core/skillopt/benchmark.ts b/src/core/skillopt/benchmark.ts new file mode 100644 index 000000000..404656bbb --- /dev/null +++ b/src/core/skillopt/benchmark.ts @@ -0,0 +1,349 @@ +/** + * SkillOpt benchmark loader, validator, and splitter. + * + * Benchmark format: JSONL with one task per line: + * {"task_id":"x","task":"...","judge":{"kind":"rule","checks":[...]}} + * + * Enforces (at load time, fail-loud with paste-ready hints): + * - File exists and is non-empty. + * - Every row parses as JSON. + * - Every row has task_id (unique), task (non-empty string), judge.kind ∈ {rule,llm,qrels}. + * - Judge-specific shape validation (rule.checks array, llm.rubric string, qrels.expected_slugs). + * - D17: D_sel >= 5 after split — refuses below floor with `--split` override hint. + * - D15: refuses bootstrap output that still has the BOOTSTRAP_PENDING_REVIEW sentinel. + */ + +import { createHash } from 'node:crypto'; +import * as fs from 'node:fs'; +import { errorFor } from '../errors.ts'; +import { + type Benchmark, + type BenchmarkSplit, + type BenchmarkTask, + type Judge, + type RuleCheck, + type RuleCheckOp, + BOOTSTRAP_PENDING_REVIEW, + D_SEL_MIN_SIZE, +} from './types.ts'; + +const VALID_RULE_OPS: ReadonlySet = new Set([ + 'contains', + 'regex', + 'section_present', + 'max_chars', + 'min_citations', + 'tool_called', + 'tool_not_called', +]); + +/** + * Load + validate a benchmark JSONL file. + * + * @param path absolute path to the benchmark file. + * @param opts.bootstrapReviewed set true after the user passed --bootstrap-reviewed. + * When false (default), the loader refuses files that still carry the + * BOOTSTRAP_PENDING_REVIEW sentinel line (D15). + */ +export function loadBenchmark( + path: string, + opts: { bootstrapReviewed?: boolean } = {}, +): Benchmark { + let content: string; + try { + content = fs.readFileSync(path, 'utf8'); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + throw errorFor({ + class: 'BenchmarkNotFound', + code: 'benchmark_not_found', + message: `Benchmark file unreadable: ${path} (${msg})`, + hint: `Create the benchmark at ${path}, or pass --bootstrap-from-routing to auto-generate one.`, + }); + } + + if (content.trim().length === 0) { + throw errorFor({ + class: 'BenchmarkEmpty', + code: 'benchmark_empty', + message: `Benchmark file is empty: ${path}`, + hint: `Add at least ${D_SEL_MIN_SIZE} tasks (one JSON object per line) — D_sel requires >=${D_SEL_MIN_SIZE} after split.`, + }); + } + + // D15: detect the bootstrap-pending sentinel before parsing rows. + // Sentinel is always the LAST non-empty line of the file. A user who's + // hand-reviewed the bootstrap deletes the line before re-running. + const allLines = content.split('\n'); + const lastNonEmpty = [...allLines].reverse().find((l) => l.trim().length > 0); + if (lastNonEmpty && lastNonEmpty.trim() === BOOTSTRAP_PENDING_REVIEW) { + if (!opts.bootstrapReviewed) { + throw errorFor({ + class: 'BootstrapPendingReview', + code: 'bootstrap_pending_review', + message: `Benchmark at ${path} is a bootstrap output awaiting human review.`, + hint: `Review the file, delete the trailing '${BOOTSTRAP_PENDING_REVIEW}' line, then re-run with --bootstrap-reviewed.`, + }); + } + } else if (opts.bootstrapReviewed) { + // User passed --bootstrap-reviewed but the sentinel is already gone. + // This is fine — the flag is idempotent. Don't error. + } + + // Parse rows. + const tasks: BenchmarkTask[] = []; + const seenIds = new Set(); + const rows = allLines.filter((l) => l.trim().length > 0 && l.trim() !== BOOTSTRAP_PENDING_REVIEW); + + for (let i = 0; i < rows.length; i++) { + const line = rows[i]!; + let parsed: unknown; + try { + parsed = JSON.parse(line); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_malformed', + message: `Row ${i + 1} is not valid JSON: ${msg}`, + hint: `Fix the offending line in ${path}; benchmarks are one JSON object per line.`, + }); + } + + const task = validateRow(parsed, i + 1, path); + if (seenIds.has(task.task_id)) { + throw errorFor({ + class: 'BenchmarkDuplicateId', + code: 'benchmark_duplicate_task_id', + message: `Duplicate task_id '${task.task_id}' at row ${i + 1}.`, + hint: `Every task_id in ${path} must be unique.`, + }); + } + seenIds.add(task.task_id); + tasks.push(task); + } + + if (tasks.length === 0) { + throw errorFor({ + class: 'BenchmarkEmpty', + code: 'benchmark_empty', + message: `Benchmark file at ${path} has no tasks.`, + hint: `Add at least ${D_SEL_MIN_SIZE} tasks.`, + }); + } + + return { + source_path: path, + tasks, + benchmark_sha8: computeBenchmarkSha8(tasks), + }; +} + +/** Validate a single parsed row. Throws StructuredAgentError on failure. */ +function validateRow(parsed: unknown, rowNum: number, path: string): BenchmarkTask { + if (!parsed || typeof parsed !== 'object') { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_malformed', + message: `Row ${rowNum} is not an object.`, + hint: `Each line in ${path} must be a JSON object with task_id, task, judge.`, + }); + } + const obj = parsed as Record; + + const task_id = typeof obj.task_id === 'string' ? obj.task_id.trim() : ''; + if (!task_id) { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_missing_task_id', + message: `Row ${rowNum} is missing a non-empty task_id.`, + hint: `Add a unique task_id string to row ${rowNum} of ${path}.`, + }); + } + + const task = typeof obj.task === 'string' ? obj.task : ''; + if (!task.trim()) { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_missing_task', + message: `Row ${rowNum} (${task_id}) is missing a non-empty task description.`, + hint: `Add a 'task' field describing the prompt to run against the skill.`, + }); + } + + const judge = validateJudge(obj.judge, rowNum, task_id, path); + return { task_id, task, judge }; +} + +function validateJudge(raw: unknown, rowNum: number, task_id: string, path: string): Judge { + if (!raw || typeof raw !== 'object') { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_missing_judge', + message: `Row ${rowNum} (${task_id}) is missing a judge object.`, + hint: `Add a 'judge' field with shape {"kind":"rule"|"llm"|"qrels", ...}.`, + }); + } + const j = raw as Record; + const kind = j.kind; + if (kind === 'rule') { + const checks = j.checks; + if (!Array.isArray(checks) || checks.length === 0) { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_judge_rule_no_checks', + message: `Row ${rowNum} (${task_id}) judge.kind='rule' needs a non-empty checks array.`, + hint: `Add at least one check, e.g. {"op":"max_chars","arg":4000}.`, + }); + } + const validated: RuleCheck[] = checks.map((c, ci) => validateRuleCheck(c, rowNum, ci, task_id, path)); + return { kind: 'rule', checks: validated }; + } + if (kind === 'llm') { + const rubric = typeof j.rubric === 'string' ? j.rubric : ''; + if (!rubric.trim()) { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_judge_llm_no_rubric', + message: `Row ${rowNum} (${task_id}) judge.kind='llm' needs a non-empty rubric string.`, + hint: `Add a 'rubric' field describing how to score the output 0..1.`, + }); + } + const model = typeof j.model === 'string' ? j.model : undefined; + return model !== undefined ? { kind: 'llm', rubric, model } : { kind: 'llm', rubric }; + } + if (kind === 'qrels') { + const expected_slugs = Array.isArray(j.expected_slugs) ? j.expected_slugs : null; + if (!expected_slugs || expected_slugs.length === 0 || !expected_slugs.every((s) => typeof s === 'string')) { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_judge_qrels_no_expected', + message: `Row ${rowNum} (${task_id}) judge.kind='qrels' needs expected_slugs: string[].`, + hint: `Add an array of expected slugs the retrieval should return.`, + }); + } + const k = typeof j.k === 'number' && j.k > 0 ? Math.floor(j.k) : 10; + return { kind: 'qrels', expected_slugs: expected_slugs as string[], k }; + } + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_judge_unknown_kind', + message: `Row ${rowNum} (${task_id}) judge.kind='${String(kind)}' is not one of rule|llm|qrels.`, + hint: `Use one of: {"kind":"rule","checks":[...]}, {"kind":"llm","rubric":"..."}, {"kind":"qrels","expected_slugs":[...]}.`, + }); +} + +function validateRuleCheck( + raw: unknown, + rowNum: number, + checkIdx: number, + task_id: string, + path: string, +): RuleCheck { + if (!raw || typeof raw !== 'object') { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_rule_check_malformed', + message: `Row ${rowNum} (${task_id}) rule check #${checkIdx} is not an object.`, + hint: `Each check must be {"op":"...","arg":...}.`, + }); + } + const c = raw as Record; + const op = c.op; + if (typeof op !== 'string' || !VALID_RULE_OPS.has(op as RuleCheckOp)) { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_rule_check_unknown_op', + message: `Row ${rowNum} (${task_id}) rule check #${checkIdx} has unknown op '${String(op)}'.`, + hint: `Valid ops: ${[...VALID_RULE_OPS].join(', ')}.`, + }); + } + const arg = c.arg; + if (typeof arg !== 'string' && typeof arg !== 'number') { + throw errorFor({ + class: 'BenchmarkMalformed', + code: 'benchmark_rule_check_bad_arg', + message: `Row ${rowNum} (${task_id}) rule check #${checkIdx} has non-string/number arg.`, + hint: `arg must be a string (contains, regex, section_present, tool_called, tool_not_called) or number (max_chars, min_citations).`, + }); + } + return { op: op as RuleCheckOp, arg }; +} + +/** + * Compute a deterministic SHA-256-prefix-8 over the benchmark contents. + * Stable across whitespace changes (re-serializes the parsed tasks). + */ +export function computeBenchmarkSha8(tasks: BenchmarkTask[]): string { + const canonical = tasks + .slice() + .sort((a, b) => (a.task_id < b.task_id ? -1 : a.task_id > b.task_id ? 1 : 0)) + .map((t) => JSON.stringify({ task_id: t.task_id, task: t.task, judge: t.judge })) + .join('\n'); + return createHash('sha256').update(canonical).digest('hex').slice(0, 8); +} + +/** + * Split a benchmark deterministically by ratio. Sorts by task_id for stable + * splits across runs (paper: benchmarks must not shuffle between epochs). + * + * Returns `{train, sel, test}`. D17: refuses if D_sel < 5 — caller must + * either add more tasks or pass an explicit --split override (which still + * routes through this function with the override ratio). + */ +export function splitBench( + benchmark: Benchmark, + ratio: [number, number, number], + opts: { allowSmallSel?: boolean } = {}, +): BenchmarkSplit { + const [r1, r2, r3] = ratio; + if (r1 <= 0 || r2 <= 0 || r3 <= 0) { + throw errorFor({ + class: 'BadSplit', + code: 'split_bad_ratio', + message: `Split ratio ${ratio.join(':')} has a zero or negative segment.`, + hint: `Use positive integers like 4:1:5.`, + }); + } + + const sorted = benchmark.tasks + .slice() + .sort((a, b) => (a.task_id < b.task_id ? -1 : a.task_id > b.task_id ? 1 : 0)); + + const total = r1 + r2 + r3; + const n = sorted.length; + // Round to nearest int; ensure all three buckets get at least 1 if n>=3. + const trainN = Math.max(1, Math.floor((r1 * n) / total)); + const selN = Math.max(1, Math.floor((r2 * n) / total)); + const testN = Math.max(1, n - trainN - selN); + + const train = sorted.slice(0, trainN); + const sel = sorted.slice(trainN, trainN + selN); + const test = sorted.slice(trainN + selN, trainN + selN + testN); + + // D17: refuse if D_sel < 5 unless explicitly overridden. + if (sel.length < D_SEL_MIN_SIZE && !opts.allowSmallSel) { + throw errorFor({ + class: 'DSelTooSmall', + code: 'd_sel_too_small', + message: `D_sel has ${sel.length} task(s) after split (need >=${D_SEL_MIN_SIZE} for meaningful validation).`, + hint: `Add more tasks to the benchmark (need ~${Math.ceil((D_SEL_MIN_SIZE * total) / r2)} total for ${ratio.join(':')}) or pass --split with a larger sel segment.`, + }); + } + + return { train, sel, test }; +} + +/** Parse a split string like "4:1:5" into a tuple. */ +export function parseSplit(s: string): [number, number, number] { + const parts = s.split(':').map((p) => Number(p.trim())); + if (parts.length !== 3 || parts.some((p) => !Number.isFinite(p) || p <= 0)) { + throw errorFor({ + class: 'BadSplit', + code: 'split_unparseable', + message: `Invalid --split value '${s}'.`, + hint: `Use three positive integers separated by ':', e.g. '4:1:5'.`, + }); + } + return [parts[0]!, parts[1]!, parts[2]!]; +} diff --git a/src/core/skillopt/lock.ts b/src/core/skillopt/lock.ts new file mode 100644 index 000000000..21a155c0d --- /dev/null +++ b/src/core/skillopt/lock.ts @@ -0,0 +1,93 @@ +/** + * SkillOpt per-skill DB lock (D14). + * + * Thin wrapper around `tryAcquireDbLock` from `src/core/db-lock.ts`. The + * lock id is `skillopt:` so two concurrent `gbrain skillopt foo` + * runs serialize cleanly without blocking other skills. + * + * Default TTL: 60 minutes — generous for a full epoch run, but the auto- + * refresh inside `withSkilloptLock` bumps it every 15 minutes so a long + * run never times out underneath itself. + * + * Why a DB lock instead of a filesystem `.lock`: + * - Cross-host correct (matters for Conductor workspaces sharing a brain). + * - Reuses the existing primitive (same TTL semantics as gbrain sync, + * extract-conversation-facts, autopilot cycle). + * - Crashed holders auto-release via TTL expiry (no PID-liveness landmine). + */ + +import { tryAcquireDbLock, type DbLockHandle } from '../db-lock.ts'; +import { errorFor } from '../errors.ts'; +import type { BrainEngine } from '../engine.ts'; + +const DEFAULT_TTL_MINUTES = 60; +const DEFAULT_REFRESH_INTERVAL_MS = 15 * 60 * 1000; + +/** Build the lock id for a given skill name. */ +export function lockIdFor(skillName: string): string { + return `skillopt:${skillName}`; +} + +/** + * Acquire a per-skill SkillOpt lock. Returns null when another live holder + * has the lock. Caller is responsible for releasing via `handle.release()` + * (use `withSkilloptLock` for try/finally + refresh-loop semantics). + */ +export async function tryAcquireSkilloptLock( + engine: BrainEngine, + skillName: string, + ttlMinutes: number = DEFAULT_TTL_MINUTES, +): Promise { + return tryAcquireDbLock(engine, lockIdFor(skillName), ttlMinutes); +} + +/** + * Run `fn` while holding the per-skill SkillOpt lock with a background + * refresh loop. Refreshes the TTL every 15 minutes (well under the 60min + * default TTL so the lock never expires under an active run). + * + * Throws a StructuredAgentError with `code: 'lock_busy'` when another + * live holder has the lock — the user is shown the paste-ready remediation + * "another run is in progress; wait or check `gbrain jobs supervisor status`". + * + * Lock is always released on `fn` completion (success OR throw) via + * try/finally. The background refresh interval is cleared in the finally. + */ +export async function withSkilloptLock( + engine: BrainEngine, + skillName: string, + fn: (handle: DbLockHandle) => Promise, + ttlMinutes: number = DEFAULT_TTL_MINUTES, + refreshIntervalMs: number = DEFAULT_REFRESH_INTERVAL_MS, +): Promise { + const handle = await tryAcquireSkilloptLock(engine, skillName, ttlMinutes); + if (handle === null) { + throw errorFor({ + class: 'LockBusy', + code: 'lock_busy', + message: `Another SkillOpt run is in progress for skill '${skillName}'.`, + hint: `Wait for it to finish, or check 'gbrain jobs supervisor status'. Stale lock holders auto-expire after ${ttlMinutes} minutes.`, + }); + } + + const refresher = setInterval(() => { + handle.refresh().catch((err) => { + const msg = err instanceof Error ? err.message : String(err); + process.stderr.write(`[skillopt-lock] refresh failed for '${skillName}': ${msg}\n`); + }); + }, refreshIntervalMs); + // Don't keep the event loop alive on the refresh timer alone. + if (typeof refresher.unref === 'function') refresher.unref(); + + try { + return await fn(handle); + } finally { + clearInterval(refresher); + try { + await handle.release(); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + process.stderr.write(`[skillopt-lock] release failed for '${skillName}': ${msg}\n`); + } + } +} diff --git a/src/core/skillopt/lr-schedule.ts b/src/core/skillopt/lr-schedule.ts new file mode 100644 index 000000000..9e777e6fb --- /dev/null +++ b/src/core/skillopt/lr-schedule.ts @@ -0,0 +1,71 @@ +/** + * SkillOpt LR (learning-rate) schedules. + * + * The LR controls the max number of edits per step. Cosine is the default + * per the SkillOpt paper's ablation (slightly better than linear or constant). + * + * All schedules return an INTEGER >= 1 (we always allow at least one edit + * per step; otherwise the optimizer can never make progress). + * + * Pure functions — no side effects, no I/O, fully unit-testable. + * + * LR schedule shapes (visualized for base=4, totalSteps=10): + * + * cosine: 4 4 3 3 3 2 2 2 1 1 (smooth high→low decay) + * linear: 4 3 3 3 2 2 2 1 1 1 (monotone descent) + * constant: 4 4 4 4 4 4 4 4 4 4 (no decay) + * + * The cosine curve peaks early (more aggressive when the skill is the most + * unrefined) and tapers (fewer edits as the skill converges). + */ + +/** Floor of all schedules; the LR can never drop below 1. */ +const MIN_LR = 1; + +/** + * Cosine-decay schedule. Peaks at `base` for t=1; decays to ~1 by totalSteps. + * + * Formula: `0.5 * base * (1 + cos((t-1) * pi / (totalSteps-1)))` rounded + * up, then clamped to [MIN_LR, base]. + */ +export function cosineLr(base: number, t: number, totalSteps: number): number { + if (base < MIN_LR) return MIN_LR; + if (totalSteps <= 1) return base; + const tClamped = Math.max(1, Math.min(t, totalSteps)); + const phase = ((tClamped - 1) * Math.PI) / (totalSteps - 1); + const raw = 0.5 * base * (1 + Math.cos(phase)); + return Math.max(MIN_LR, Math.min(base, Math.ceil(raw))); +} + +/** + * Linear-decay schedule. Starts at `base` for t=1; ends at MIN_LR for + * t=totalSteps. Monotonically non-increasing. + * + * Formula: `base - (base - MIN_LR) * (t-1) / (totalSteps-1)` rounded up. + */ +export function linearLr(base: number, t: number, totalSteps: number): number { + if (base < MIN_LR) return MIN_LR; + if (totalSteps <= 1) return base; + const tClamped = Math.max(1, Math.min(t, totalSteps)); + const raw = base - ((base - MIN_LR) * (tClamped - 1)) / (totalSteps - 1); + return Math.max(MIN_LR, Math.min(base, Math.ceil(raw))); +} + +/** + * Constant schedule. Returns `base` every step (clamped to MIN_LR). + */ +export function constantLr(base: number, _t: number, _totalSteps: number): number { + return Math.max(MIN_LR, base); +} + +/** Type alias for the function signature shared by all three schedules. */ +export type LrScheduleFn = (base: number, t: number, totalSteps: number) => number; + +/** Resolve a schedule name to its function. */ +export function resolveLrSchedule(name: 'cosine' | 'linear' | 'constant'): LrScheduleFn { + switch (name) { + case 'cosine': return cosineLr; + case 'linear': return linearLr; + case 'constant': return constantLr; + } +} diff --git a/src/core/skillopt/score.ts b/src/core/skillopt/score.ts new file mode 100644 index 000000000..a567c4f16 --- /dev/null +++ b/src/core/skillopt/score.ts @@ -0,0 +1,266 @@ +/** + * SkillOpt scoring: three judge modes (rule, llm, qrels). + * + * Each scorer returns a 0..1 score (1 = best). Sub-1 scores are partial + * credit; 0 means total failure. The validation gate's median-of-3 (D12) + * is implemented in validate-gate.ts; this module is just the + * per-trajectory scoring primitives. + * + * `judge: llm` uses gateway.chat with the v0.40+ 4-strategy JSON repair + * (parseModelJSON from cross-modal-eval). On parse failure the scorer + * returns score=0 (pessimistic fallback) AND records the error string on + * `ScoredRollout.judge_error` so the audit trail can surface it. + * + * `judge: qrels` reuses src/core/search/eval.ts IR metrics. Score is + * nDCG@k (more discriminating than P@k for the optimization signal). + */ + +import { chat as gatewayChat } from '../ai/gateway.ts'; +import { ndcgAtK } from '../search/eval.ts'; +import type { Judge, RuleCheck, ScoredRollout, Trajectory } from './types.ts'; + +/** Score a trajectory against a judge. Returns a ScoredRollout. */ +export async function scoreTrajectory( + trajectory: Trajectory, + judge: Judge, + opts: { + judgeModel?: string; + /** Test seam — substitute for gateway.chat. */ + chatFn?: typeof gatewayChat; + /** Test seam — substitute clock for cache invalidation. */ + now?: () => Date; + } = {}, +): Promise { + switch (judge.kind) { + case 'rule': + return { trajectory, score: scoreRule(trajectory, judge.checks) }; + case 'llm': + return scoreLlm(trajectory, judge.rubric, judge.model ?? opts.judgeModel, opts); + case 'qrels': + return { trajectory, score: scoreQrels(trajectory, judge.expected_slugs, judge.k) }; + } +} + +// ─── Rule judge ────────────────────────────────────────────────────────── + +/** + * Score a trajectory against a list of rule checks. Returns the FRACTION + * of checks that pass. 0 = all fail, 1 = all pass. + */ +export function scoreRule(trajectory: Trajectory, checks: RuleCheck[]): number { + if (checks.length === 0) return 0; + let passing = 0; + for (const c of checks) { + if (applyCheck(trajectory, c)) passing += 1; + } + return passing / checks.length; +} + +function applyCheck(trajectory: Trajectory, check: RuleCheck): boolean { + const text = trajectory.final_text; + switch (check.op) { + case 'contains': + return typeof check.arg === 'string' && text.includes(check.arg); + case 'regex': { + if (typeof check.arg !== 'string') return false; + try { + return new RegExp(check.arg, 'm').test(text); + } catch { + return false; + } + } + case 'section_present': { + if (typeof check.arg !== 'string') return false; + // Match the heading (any depth, plus the literal text). Trim args + // and allow leading-# variants. + const heading = check.arg.replace(/^#+\s*/, '').trim(); + const escaped = heading.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + const re = new RegExp(`^#{1,6}\\s+${escaped}\\s*$`, 'mi'); + return re.test(text); + } + case 'max_chars': + return typeof check.arg === 'number' && text.length <= check.arg; + case 'min_citations': + return typeof check.arg === 'number' && countCitations(text) >= check.arg; + case 'tool_called': + return typeof check.arg === 'string' && + trajectory.tool_calls.some((tc) => tc.name === check.arg && !tc.failed); + case 'tool_not_called': + return typeof check.arg === 'string' && + !trajectory.tool_calls.some((tc) => tc.name === check.arg); + } +} + +/** + * Count citation-like spans in the output. Recognized shapes: + * - Markdown links: `[text](url-or-slug)` + * - Brain-page references: `wiki/...`, `people/...`, `companies/...` + * - Footnote-style: `[N]` where N is digits. + */ +export function countCitations(text: string): number { + const mdLinks = (text.match(/\[[^\]]+\]\([^)]+\)/g) ?? []).length; + const brainRefs = (text.match(/\b(?:wiki|people|companies|deals|topics|concepts|projects|writing|originals)\/[a-z0-9][a-z0-9-\/]*\b/gi) ?? []).length; + const footnotes = (text.match(/\[\d+\]/g) ?? []).length; + return mdLinks + brainRefs + footnotes; +} + +// ─── LLM judge ─────────────────────────────────────────────────────────── + +const LLM_JUDGE_SYSTEM = `You are a strict, fair judge scoring an agent's output against a rubric. + +Output ONLY a single JSON object on a single line: +{"score": , "rationale": ""} + +No prose before or after. No code fences. No extra fields. The score MUST be a number between 0.0 and 1.0 inclusive.`; + +/** + * Parse a `{score, rationale}` JSON object from raw LLM text. Tolerates: + * - Leading/trailing whitespace. + * - Markdown code fences (```json ... ```). + * - Prose before or after the JSON (extracts first {...} object). + * - Trailing commas inside the object. + * + * Returns null when no recoverable object is found (caller treats as judge + * error + pessimistic fallback score=0). + */ +export function parseJudgeJson(raw: string): { score: number | string; rationale?: string } | null { + if (typeof raw !== 'string' || !raw.trim()) return null; + // Strip markdown fences if present. + const fenced = raw.match(/```(?:json)?\s*\n?([\s\S]*?)```/i); + const cleaned = (fenced ? fenced[1]! : raw).trim(); + // Try direct parse first. + const direct = tryJsonParse(cleaned); + if (direct && typeof direct === 'object' && 'score' in (direct as object)) { + return direct as { score: number | string; rationale?: string }; + } + // Extract first {...} substring. + const match = cleaned.match(/\{[\s\S]*?\}/); + if (!match) return null; + const obj = match[0]; + const second = tryJsonParse(obj); + if (second && typeof second === 'object' && 'score' in (second as object)) { + return second as { score: number | string; rationale?: string }; + } + // Last attempt: strip trailing commas. + const repaired = obj.replace(/,(\s*[}\]])/g, '$1'); + const third = tryJsonParse(repaired); + if (third && typeof third === 'object' && 'score' in (third as object)) { + return third as { score: number | string; rationale?: string }; + } + return null; +} + +function tryJsonParse(s: string): unknown | null { + try { return JSON.parse(s); } catch { return null; } +} + +async function scoreLlm( + trajectory: Trajectory, + rubric: string, + judgeModel: string | undefined, + opts: { chatFn?: typeof gatewayChat }, +): Promise { + const chat = opts.chatFn ?? gatewayChat; + const userMsg = `RUBRIC: ${rubric}\n\nAGENT OUTPUT:\n${trajectory.final_text}\n\nScore the output against the rubric. Reply with the JSON object only.`; + try { + const result = await chat({ + model: judgeModel, + system: LLM_JUDGE_SYSTEM, + messages: [{ role: 'user', content: userMsg }], + maxTokens: 200, + cacheSystem: true, // D11: judge system prompt is stable across calls. + }); + const parsed = parseJudgeJson(result.text); + if (!parsed) { + return { trajectory, score: 0, judge_error: 'llm_parse_failed' }; + } + if (!('score' in parsed)) { + return { trajectory, score: 0, judge_error: 'llm_parse_no_score_field' }; + } + const raw = parsed.score; + let score = typeof raw === 'number' ? raw : Number(raw); + if (!Number.isFinite(score)) { + return { trajectory, score: 0, judge_error: 'llm_parse_score_not_number' }; + } + if (score < 0) score = 0; + if (score > 1) score = 1; + const rationale = typeof parsed.rationale === 'string' ? parsed.rationale : undefined; + return rationale !== undefined + ? { trajectory, score, rationale } + : { trajectory, score }; + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + // Pessimistic fallback (D12 paper-faithful: judge failure = score 0, + // not throw; the median-of-3 + epsilon gate handles a single error + // gracefully, only consistent judge failure breaks the run). + return { trajectory, score: 0, judge_error: `llm_call_failed: ${msg}` }; + } +} + +// ─── Qrels judge (retrieval flavor) ─────────────────────────────────────── + +/** + * Score a trajectory against expected retrieval slugs. + * + * Extracts the candidate slugs from the trajectory's tool calls (any + * `search` / `query` / `get_page` / `list_pages` op output that returns + * page rows), then computes nDCG@k against expected_slugs. + * + * Returns 0 when no retrieval tool was called (the skill didn't even try). + */ +export function scoreQrels( + trajectory: Trajectory, + expectedSlugs: string[], + k: number, +): number { + const candidateSlugs = extractRetrievedSlugs(trajectory); + if (candidateSlugs.length === 0) return 0; + // ndcgAtK expects (hits, grades:Map, k). All expected slugs + // get grade 1 (binary relevance) — qrels mode is "did the skill retrieve + // what we expected?", not "did it rank them in our preferred order." + const grades = new Map(); + for (const s of expectedSlugs) grades.set(s, 1); + return ndcgAtK(candidateSlugs, grades, k); +} + +/** + * Walk the trajectory's tool calls and collect slugs from any search/query/ + * list_pages/get_page output that returns row arrays with `slug` fields. + * Tolerant of shape variation. + */ +export function extractRetrievedSlugs(trajectory: Trajectory): string[] { + const out: string[] = []; + const seen = new Set(); + for (const call of trajectory.tool_calls) { + if (!call.output || call.failed) continue; + const slugs = pickSlugs(call.output); + for (const slug of slugs) { + if (!seen.has(slug)) { + seen.add(slug); + out.push(slug); + } + } + } + return out; +} + +function pickSlugs(output: unknown): string[] { + if (!output) return []; + if (typeof output === 'string') { + // Some ops return a single slug string. + return output.includes('/') ? [output] : []; + } + if (Array.isArray(output)) { + return output.flatMap(pickSlugs); + } + if (typeof output === 'object') { + const obj = output as Record; + const out: string[] = []; + if (typeof obj.slug === 'string') out.push(obj.slug); + if (Array.isArray(obj.results)) out.push(...pickSlugs(obj.results)); + if (Array.isArray(obj.pages)) out.push(...pickSlugs(obj.pages)); + if (Array.isArray(obj.matches)) out.push(...pickSlugs(obj.matches)); + return out; + } + return []; +} diff --git a/src/core/skillopt/types.ts b/src/core/skillopt/types.ts new file mode 100644 index 000000000..bb4a348b2 --- /dev/null +++ b/src/core/skillopt/types.ts @@ -0,0 +1,259 @@ +/** + * SkillOpt v1 types — single source of truth for the optimization loop. + * + * The optimizer treats SKILL.md as the trainable parameters of a frozen + * agent. See plan: ~/.claude/plans/system-instruction-you-are-working-drifting-falcon.md. + * + * Per the v0.41.20.0 plan decisions: + * D2: rollout uses gateway.toolLoop (no DB pollution). + * D5: frontmatter mutation is forbidden; edits operate on body slice only. + * D9: applyEdit returns a tagged result, not throws. + * D12: validation gate uses median-of-3 + epsilon=0.05. + * D17: D_sel >= 5 floor enforced at benchmark-load time. + */ + +import type { BrainEngine } from '../engine.ts'; + +// ─── Benchmarks + judges ────────────────────────────────────────────────── + +/** Rule-check kinds for `judge: rule`. Each is deterministic and free. */ +export type RuleCheckOp = + | 'contains' + | 'regex' + | 'section_present' + | 'max_chars' + | 'min_citations' + | 'tool_called' + | 'tool_not_called'; + +export interface RuleCheck { + op: RuleCheckOp; + arg: string | number; +} + +export type JudgeKind = 'rule' | 'llm' | 'qrels'; + +export type Judge = + | { kind: 'rule'; checks: RuleCheck[] } + | { kind: 'llm'; rubric: string; model?: string } + | { kind: 'qrels'; expected_slugs: string[]; k: number }; + +export interface BenchmarkTask { + task_id: string; + task: string; + judge: Judge; +} + +export interface Benchmark { + /** Path the benchmark was loaded from (for error messages). */ + source_path: string; + tasks: BenchmarkTask[]; + /** SHA-256 of the canonical JSON, truncated to 16 hex. Stable across reorderings. */ + benchmark_sha8: string; +} + +/** D17: enforced floor for D_sel. */ +export const D_SEL_MIN_SIZE = 5; + +export interface BenchmarkSplit { + train: BenchmarkTask[]; + sel: BenchmarkTask[]; + test: BenchmarkTask[]; +} + +// ─── Edit ops ───────────────────────────────────────────────────────────── + +/** D9: applyEdit returns a tagged result, not throws. */ +export type EditOp = + | { op: 'add'; anchor: string; content: string; reason?: string } + | { op: 'replace'; target: string; replacement: string; reason?: string } + | { op: 'delete'; target: string; reason?: string }; + +export type EditRejectionReason = + | 'anchor_not_found' + | 'anchor_ambiguous' + | 'target_not_found' + | 'target_ambiguous' + | 'inside_code_fence' + | 'crosses_frontmatter' + | 'working_tree_dirty' + | 'install_path' + | 'no_change'; + +export type EditResult = + | { outcome: 'applied'; edit: EditOp; newText: string } + | { outcome: 'rejected'; edit: EditOp; reason: EditRejectionReason; detail?: string }; + +// ─── Trajectories + rollouts ────────────────────────────────────────────── + +/** What a rollout produced. Captured in-process; never persisted to DB (D2). */ +export interface Trajectory { + task_id: string; + task: string; + /** Final assistant text (typically the user-visible output). */ + final_text: string; + /** Tool calls observed during the rollout, in order. */ + tool_calls: Array<{ name: string; input: unknown; output?: unknown; failed?: boolean }>; + /** Token usage from gateway.toolLoop. */ + usage: { + input_tokens: number; + output_tokens: number; + cache_read_tokens: number; + cache_creation_tokens: number; + }; + /** Number of agent turns the loop took. */ + turns: number; + /** End reason from gateway.toolLoop. */ + stop_reason: 'end' | 'max_turns' | 'refusal' | 'content_filter' | 'aborted' | 'unrecoverable'; + /** Wall-clock duration in ms. */ + duration_ms: number; +} + +export interface ScoredRollout { + trajectory: Trajectory; + /** 0..1 score from the judge. */ + score: number; + /** Optional per-task rationale (LLM judge only). */ + rationale?: string; + /** If the judge itself errored (parse fail, timeout), this is set. */ + judge_error?: string; +} + +// ─── Run state + receipts ───────────────────────────────────────────────── + +export interface SkillOptOpts { + /** Kebab-case skill name. Resolves to `skills//SKILL.md`. */ + skillName: string; + /** Absolute path to the benchmark JSONL file. */ + benchmarkPath: string; + /** Brain engine (required for D14 lock + read-only tool calls during rollouts). */ + engine: BrainEngine; + /** Skills directory root (used for bundled-skill detection, install-path gate). */ + skillsDir: string; + + // Training knobs. + epochs: number; + batchSize: number; + /** Max edits per step (the LR). */ + lr: number; + lrSchedule: 'cosine' | 'linear' | 'constant'; + /** Split ratio as 3-tuple, e.g. [4, 1, 5] for 4:1:5. */ + split: [number, number, number]; + + // Models. + optimizerModel: string; + targetModel: string; + judgeModel: string; + + // Modes. + mode: 'patch' | 'rewrite'; + dryRun: boolean; + noMutate: boolean; + allowMutateBundled: boolean; + bootstrapReviewed: boolean; + /** F10: enable write-capture mode for write-flavored skills. */ + writeCapture?: boolean; + /** F11: optional held-out test set path (validates winner before mutate). */ + heldOutPath?: string; + json: boolean; + + // Safety. + maxCostUsd: number; + maxRuntimeMin: number; + force: boolean; + resumeRunId?: string; +} + +export interface StepRecord { + epoch: number; + step: number; + sel_score_median: number; + sel_score_runs: number[]; + accepted: boolean; + edits_attempted: number; + edits_applied: number; + delta: number; + reason?: string; + cumulative_cost_usd: number; + ts: string; +} + +export interface RunReceipt { + run_id: string; + skill: string; + skill_sha8: string; + benchmark_sha8: string; + optimizer_model: string; + target_model: string; + judge_model: string; + epochs: number; + batch_size: number; + lr: number; + lr_schedule: 'cosine' | 'linear' | 'constant'; + max_cost_usd: number; + started_at: string; + ended_at?: string; + outcome?: 'accepted' | 'no_improvement' | 'aborted' | 'errored'; + baseline_sel_score?: number; + best_sel_score?: number; + baseline_test_score?: number; + test_score?: number; + final_cost_usd?: number; + total_steps?: number; + epochs_completed?: number; +} + +export interface HistoryRow { + /** D8: pending → committed two-phase commit. */ + status: 'pending' | 'committed'; + run_id: string; + version_n: number; + ts: string; + edits: EditOp[]; + sel_score: number; + delta: number; +} + +// ─── Validation gate (D12) ──────────────────────────────────────────────── + +/** D12: epsilon margin floor for accepting a candidate. */ +export const VALIDATION_EPSILON = 0.05; +/** D12: number of judge runs per sel-task for noise rejection. */ +export const VALIDATION_RUNS_PER_TASK = 3; + +export interface GateInput { + candidateSkillText: string; + selSet: BenchmarkTask[]; + /** Best score so far on D_sel (epsilon-margin compare against this). */ + bestScore: number; +} + +export interface GateResult { + accepted: boolean; + /** Per-task median across N runs. */ + perTaskMedians: Array<{ task_id: string; median: number; runs: number[] }>; + /** Mean of per-task medians. */ + selScore: number; + reason?: 'no_margin' | 'below_baseline' | 'all_judge_errors'; +} + +// ─── Bootstrap sentinel (D15) ───────────────────────────────────────────── + +/** D15: sentinel line written at the end of bootstrap-from-routing output. */ +export const BOOTSTRAP_PENDING_REVIEW = '# BOOTSTRAP_PENDING_REVIEW'; + +// ─── Bundled-skill gate (D16) ───────────────────────────────────────────── + +/** + * D16: a skill is "bundled" when its SKILL.md lives under the repo's + * canonical `skills/` directory (relative to the gbrain install root). + * Bundled skills require `--allow-mutate-bundled` to overwrite. + */ +export interface BundledSkillContext { + skillName: string; + skillsDir: string; + /** Resolved absolute path to skills//SKILL.md. */ + skillPath: string; + /** True when the skillsDir is the repo's `skills/` (install path). */ + isBundled: boolean; +} diff --git a/test/skillopt/audit.test.ts b/test/skillopt/audit.test.ts new file mode 100644 index 000000000..2a844a7e2 --- /dev/null +++ b/test/skillopt/audit.test.ts @@ -0,0 +1,85 @@ +/** + * SkillOpt audit JSONL writer tests. + * + * Uses withEnv (R1 compliant) to point GBRAIN_AUDIT_DIR at a tempdir per + * test. Resets the cached writer between tests so each invocation re-reads + * the env. + */ + +import { describe, expect, test, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { withEnv } from '../helpers/with-env.ts'; +import { + _resetAuditWriterForTests, + currentAuditFilename, + logEvent, + readRecentEvents, + resolveAuditDir, + sha8, +} from '../../src/core/skillopt/audit.ts'; + +let tmpDir: string; + +beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skillopt-audit-')); + _resetAuditWriterForTests(); +}); + +afterEach(() => { + _resetAuditWriterForTests(); + try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch { /* ignore */ } +}); + +describe('logEvent + readRecentEvents', () => { + test('writes a JSONL row to the current ISO-week file', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => { + logEvent({ + kind: 'run_start', + run_id: 'r1', + skill: 'meeting-prep', + skill_sha8: 'abcd1234', + benchmark_sha8: 'deadbeef', + target_model: 'anthropic:claude-sonnet-4-6', + optimizer_model: 'anthropic:claude-opus-4-7', + judge_model: 'anthropic:claude-sonnet-4-6', + epochs: 4, + batch_size: 8, + lr: 4, + lr_schedule: 'cosine', + max_cost_usd: 5.0, + } as never); + const file = path.join(tmpDir, currentAuditFilename()); + expect(fs.existsSync(file)).toBe(true); + const lines = fs.readFileSync(file, 'utf8').trim().split('\n'); + expect(lines).toHaveLength(1); + const ev = JSON.parse(lines[0]!); + expect(ev.kind).toBe('run_start'); + expect(ev.skill).toBe('meeting-prep'); + expect(typeof ev.ts).toBe('string'); + }); + }); + + test('readRecentEvents returns the row we just wrote', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => { + logEvent({ kind: 'abort', run_id: 'r2', skill: 'foo', reason: 'budget_exhausted' } as never); + const events = readRecentEvents(7); + expect(events.length).toBeGreaterThanOrEqual(1); + const match = events.find((e) => e.kind === 'abort' && e.run_id === 'r2'); + expect(match).toBeDefined(); + }); + }); + + test('resolveAuditDir honors GBRAIN_AUDIT_DIR override', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: tmpDir }, async () => { + expect(resolveAuditDir()).toBe(tmpDir); + }); + }); + + test('sha8 returns 8 hex chars deterministically', () => { + expect(sha8('alice')).toMatch(/^[0-9a-f]{8}$/); + expect(sha8('alice')).toBe(sha8('alice')); // deterministic + expect(sha8('alice')).not.toBe(sha8('bob')); + }); +}); diff --git a/test/skillopt/benchmark.test.ts b/test/skillopt/benchmark.test.ts new file mode 100644 index 000000000..da3b8fc26 --- /dev/null +++ b/test/skillopt/benchmark.test.ts @@ -0,0 +1,188 @@ +/** + * SkillOpt benchmark loader + splitter unit tests. + * + * Uses tempdir + withEnv for hermeticity. No engine; no LLM. + */ + +import { describe, expect, test, beforeEach, afterEach } from 'bun:test'; +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { StructuredAgentError } from '../../src/core/errors.ts'; +import { + computeBenchmarkSha8, + loadBenchmark, + parseSplit, + splitBench, +} from '../../src/core/skillopt/benchmark.ts'; +import { BOOTSTRAP_PENDING_REVIEW } from '../../src/core/skillopt/types.ts'; + +let tmpDir: string; + +beforeEach(() => { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skillopt-bench-')); +}); + +afterEach(() => { + try { fs.rmSync(tmpDir, { recursive: true, force: true }); } catch { /* ignore */ } +}); + +function writeBench(filename: string, lines: string[]): string { + const p = path.join(tmpDir, filename); + fs.writeFileSync(p, lines.join('\n') + '\n', 'utf8'); + return p; +} + +describe('loadBenchmark', () => { + test('parses well-formed JSONL with rule judge', () => { + const p = writeBench('b.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'do X', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 4000 }] } }), + JSON.stringify({ task_id: 't2', task: 'do Y', judge: { kind: 'rule', checks: [{ op: 'contains', arg: 'foo' }] } }), + ]); + const b = loadBenchmark(p); + expect(b.tasks).toHaveLength(2); + expect(b.tasks[0]!.task_id).toBe('t1'); + expect(b.benchmark_sha8).toMatch(/^[0-9a-f]{8}$/); + }); + + test('rejects file-not-found with paste-ready hint', () => { + expect(() => loadBenchmark(path.join(tmpDir, 'nope.jsonl'))).toThrow(StructuredAgentError); + }); + + test('rejects empty file', () => { + fs.writeFileSync(path.join(tmpDir, 'empty.jsonl'), '', 'utf8'); + expect(() => loadBenchmark(path.join(tmpDir, 'empty.jsonl'))).toThrow(StructuredAgentError); + }); + + test('rejects duplicate task_id', () => { + const p = writeBench('dup.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 100 }] } }), + JSON.stringify({ task_id: 't1', task: 'b', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 100 }] } }), + ]); + expect(() => loadBenchmark(p)).toThrow(/Duplicate task_id/); + }); + + test('rejects unknown judge.kind', () => { + const p = writeBench('badjudge.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'magic' } }), + ]); + expect(() => loadBenchmark(p)).toThrow(/not one of rule\|llm\|qrels/); + }); + + test('rejects rule judge with empty checks array', () => { + const p = writeBench('emptychecks.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [] } }), + ]); + expect(() => loadBenchmark(p)).toThrow(/non-empty checks array/); + }); + + test('rejects rule check with unknown op', () => { + const p = writeBench('badop.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [{ op: 'frob', arg: 1 }] } }), + ]); + expect(() => loadBenchmark(p)).toThrow(/unknown op/); + }); + + test('rejects llm judge with no rubric', () => { + const p = writeBench('norubric.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'llm', rubric: ' ' } }), + ]); + expect(() => loadBenchmark(p)).toThrow(/non-empty rubric/); + }); + + test('rejects qrels judge with empty expected_slugs', () => { + const p = writeBench('noslugs.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'qrels', expected_slugs: [], k: 10 } }), + ]); + expect(() => loadBenchmark(p)).toThrow(/expected_slugs/); + }); + + test('rejects sentinel file without --bootstrap-reviewed', () => { + const p = writeBench('boot.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 100 }] } }), + BOOTSTRAP_PENDING_REVIEW, + ]); + expect(() => loadBenchmark(p)).toThrow(/awaiting human review/); + }); + + test('accepts sentinel file with --bootstrap-reviewed', () => { + const p = writeBench('boot.jsonl', [ + JSON.stringify({ task_id: 't1', task: 'a', judge: { kind: 'rule', checks: [{ op: 'max_chars', arg: 100 }] } }), + BOOTSTRAP_PENDING_REVIEW, + ]); + const b = loadBenchmark(p, { bootstrapReviewed: true }); + expect(b.tasks).toHaveLength(1); + }); +}); + +describe('splitBench', () => { + function makeBench(n: number) { + const tasks = []; + for (let i = 0; i < n; i++) { + tasks.push({ + task_id: `t${String(i).padStart(3, '0')}`, + task: `task ${i}`, + judge: { kind: 'rule' as const, checks: [{ op: 'max_chars' as const, arg: 4000 }] }, + }); + } + return { source_path: '/tmp/x.jsonl', tasks, benchmark_sha8: 'abcd1234' }; + } + + test('splits 50 tasks at 4:1:5 into 20/5/25', () => { + const split = splitBench(makeBench(50), [4, 1, 5]); + expect(split.train.length + split.sel.length + split.test.length).toBe(50); + expect(split.train.length).toBe(20); + expect(split.sel.length).toBe(5); + expect(split.test.length).toBe(25); + }); + + test('D17: refuses when D_sel < 5 without override', () => { + // 8 tasks split 4:1:5 → sel = max(1, floor(8/10)) = 1 < 5 → refuses + expect(() => splitBench(makeBench(8), [4, 1, 5])).toThrow(/D_sel/); + }); + + test('D17: allows D_sel < 5 with allowSmallSel override', () => { + const split = splitBench(makeBench(8), [4, 1, 5], { allowSmallSel: true }); + expect(split.sel.length).toBeGreaterThanOrEqual(1); + }); + + test('rejects bad ratio (zero segment)', () => { + expect(() => splitBench(makeBench(20), [4, 0, 5])).toThrow(/zero or negative/); + }); + + test('deterministic split — same input produces same output', () => { + const b = makeBench(50); + const a = splitBench(b, [4, 1, 5]); + const c = splitBench(b, [4, 1, 5]); + expect(a.train.map((t) => t.task_id)).toEqual(c.train.map((t) => t.task_id)); + expect(a.sel.map((t) => t.task_id)).toEqual(c.sel.map((t) => t.task_id)); + }); +}); + +describe('parseSplit', () => { + test('parses "4:1:5" correctly', () => { + expect(parseSplit('4:1:5')).toEqual([4, 1, 5]); + }); + + test('rejects malformed input', () => { + expect(() => parseSplit('4-1-5')).toThrow(); + expect(() => parseSplit('4:abc:5')).toThrow(); + expect(() => parseSplit('4:0:5')).toThrow(); + }); +}); + +describe('computeBenchmarkSha8', () => { + test('produces stable 8-hex hash', () => { + const tasks = [ + { task_id: 't1', task: 'a', judge: { kind: 'rule' as const, checks: [{ op: 'max_chars' as const, arg: 100 }] } }, + ]; + const h = computeBenchmarkSha8(tasks); + expect(h).toMatch(/^[0-9a-f]{8}$/); + }); + + test('reordering tasks produces same hash (sort-stable)', () => { + const t1 = { task_id: 't1', task: 'a', judge: { kind: 'rule' as const, checks: [{ op: 'max_chars' as const, arg: 100 }] } }; + const t2 = { task_id: 't2', task: 'b', judge: { kind: 'rule' as const, checks: [{ op: 'max_chars' as const, arg: 200 }] } }; + expect(computeBenchmarkSha8([t1, t2])).toBe(computeBenchmarkSha8([t2, t1])); + }); +}); diff --git a/test/skillopt/lock.test.ts b/test/skillopt/lock.test.ts new file mode 100644 index 000000000..ee4fa59b6 --- /dev/null +++ b/test/skillopt/lock.test.ts @@ -0,0 +1,94 @@ +/** + * SkillOpt per-skill DB lock tests. Uses PGLite (R3+R4 canonical block). + * + * Asserts the wrapper around tryAcquireDbLock: + * - Acquires lock and runs fn under it. + * - Refreshes TTL during long runs. + * - Throws lock_busy when another holder has the lock. + * - Releases on success AND on throw. + */ + +import { describe, expect, test, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { StructuredAgentError } from '../../src/core/errors.ts'; +import { lockIdFor, tryAcquireSkilloptLock, withSkilloptLock } from '../../src/core/skillopt/lock.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +describe('SkillOpt lock', () => { + test('lockIdFor builds skillopt: id', () => { + expect(lockIdFor('my-skill')).toBe('skillopt:my-skill'); + }); + + test('tryAcquireSkilloptLock acquires + second attempt returns null', async () => { + const h1 = await tryAcquireSkilloptLock(engine, 'foo', 1); + expect(h1).not.toBeNull(); + const h2 = await tryAcquireSkilloptLock(engine, 'foo', 1); + expect(h2).toBeNull(); + await h1!.release(); + // After release, can re-acquire. + const h3 = await tryAcquireSkilloptLock(engine, 'foo', 1); + expect(h3).not.toBeNull(); + await h3!.release(); + }); + + test('withSkilloptLock runs fn under lock and releases on success', async () => { + let ran = false; + await withSkilloptLock(engine, 'bar', async () => { + ran = true; + // While the lock is held, another acquire should return null. + const inner = await tryAcquireSkilloptLock(engine, 'bar', 1); + expect(inner).toBeNull(); + }, 1, /* fast refresh */ 30_000); + expect(ran).toBe(true); + // After fn completes, lock is released. + const after = await tryAcquireSkilloptLock(engine, 'bar', 1); + expect(after).not.toBeNull(); + await after!.release(); + }); + + test('withSkilloptLock throws LockBusy when a holder exists', async () => { + const held = await tryAcquireSkilloptLock(engine, 'baz', 1); + expect(held).not.toBeNull(); + let caught: unknown = null; + try { + await withSkilloptLock(engine, 'baz', async () => { /* unreached */ }, 1, 30_000); + } catch (err) { + caught = err; + } + expect(caught).toBeInstanceOf(StructuredAgentError); + expect((caught as StructuredAgentError).envelope.code).toBe('lock_busy'); + await held!.release(); + }); + + test('withSkilloptLock releases on throw inside fn', async () => { + let caught: unknown = null; + try { + await withSkilloptLock(engine, 'qux', async () => { + throw new Error('inner failure'); + }, 1, 30_000); + } catch (err) { + caught = err; + } + expect((caught as Error).message).toBe('inner failure'); + // Lock is released; we can re-acquire. + const after = await tryAcquireSkilloptLock(engine, 'qux', 1); + expect(after).not.toBeNull(); + await after!.release(); + }); +}); diff --git a/test/skillopt/lr-schedule.test.ts b/test/skillopt/lr-schedule.test.ts new file mode 100644 index 000000000..9e9f90efe --- /dev/null +++ b/test/skillopt/lr-schedule.test.ts @@ -0,0 +1,66 @@ +/** + * SkillOpt LR-schedule unit tests. + * + * Pure-function tests: no engine, no env mutation, no fixtures. All three + * schedules are deterministic given (base, t, totalSteps). + */ + +import { describe, expect, test } from 'bun:test'; +import { cosineLr, linearLr, constantLr, resolveLrSchedule } from '../../src/core/skillopt/lr-schedule.ts'; + +describe('cosineLr', () => { + test('peaks at base for t=1, decays toward 1 by totalSteps', () => { + expect(cosineLr(4, 1, 10)).toBe(4); + expect(cosineLr(4, 10, 10)).toBe(1); + // Mid-curve point — Math.ceil keeps it above 1. + expect(cosineLr(4, 5, 10)).toBeGreaterThan(1); + expect(cosineLr(4, 5, 10)).toBeLessThanOrEqual(4); + }); + + test('monotone non-increasing within bounds', () => { + const seq: number[] = []; + for (let t = 1; t <= 10; t++) seq.push(cosineLr(4, t, 10)); + for (let i = 1; i < seq.length; i++) { + expect(seq[i]!).toBeLessThanOrEqual(seq[i - 1]!); + } + }); + + test('totalSteps=1 returns base', () => { + expect(cosineLr(4, 1, 1)).toBe(4); + }); +}); + +describe('linearLr', () => { + test('starts at base, ends at 1', () => { + expect(linearLr(4, 1, 10)).toBe(4); + expect(linearLr(4, 10, 10)).toBe(1); + }); + + test('monotone non-increasing', () => { + const seq: number[] = []; + for (let t = 1; t <= 10; t++) seq.push(linearLr(4, t, 10)); + for (let i = 1; i < seq.length; i++) { + expect(seq[i]!).toBeLessThanOrEqual(seq[i - 1]!); + } + }); +}); + +describe('constantLr', () => { + test('returns base regardless of t or totalSteps', () => { + expect(constantLr(4, 1, 10)).toBe(4); + expect(constantLr(4, 5, 10)).toBe(4); + expect(constantLr(4, 10, 10)).toBe(4); + }); + + test('floors at 1 when base < 1', () => { + expect(constantLr(0, 1, 1)).toBe(1); + }); +}); + +describe('resolveLrSchedule', () => { + test('returns the matching function for each schedule name', () => { + expect(resolveLrSchedule('cosine')(4, 1, 1)).toBe(4); + expect(resolveLrSchedule('linear')(4, 1, 1)).toBe(4); + expect(resolveLrSchedule('constant')(4, 1, 1)).toBe(4); + }); +}); diff --git a/test/skillopt/score.test.ts b/test/skillopt/score.test.ts new file mode 100644 index 000000000..209182c7f --- /dev/null +++ b/test/skillopt/score.test.ts @@ -0,0 +1,234 @@ +/** + * SkillOpt scoring unit tests. All three judge modes (rule, llm, qrels). + * + * LLM judge is tested via DI'd chat seam (no real API calls). + */ + +import { describe, expect, test } from 'bun:test'; +import { + countCitations, + extractRetrievedSlugs, + scoreQrels, + scoreRule, + scoreTrajectory, +} from '../../src/core/skillopt/score.ts'; +import type { Trajectory } from '../../src/core/skillopt/types.ts'; + +function makeTrajectory(overrides: Partial = {}): Trajectory { + return { + task_id: 't1', + task: 'do X', + final_text: 'hello world', + tool_calls: [], + usage: { input_tokens: 100, output_tokens: 50, cache_read_tokens: 0, cache_creation_tokens: 0 }, + turns: 1, + stop_reason: 'end', + duration_ms: 500, + ...overrides, + }; +} + +describe('scoreRule', () => { + test('all checks pass → 1.0', () => { + const t = makeTrajectory({ final_text: 'Short output\n## People\nalice-example' }); + const s = scoreRule(t, [ + { op: 'contains', arg: 'alice' }, + { op: 'section_present', arg: '## People' }, + { op: 'max_chars', arg: 100 }, + ]); + expect(s).toBe(1); + }); + + test('all checks fail → 0', () => { + const t = makeTrajectory({ final_text: 'short' }); + const s = scoreRule(t, [ + { op: 'contains', arg: 'missing' }, + { op: 'max_chars', arg: 1 }, + ]); + expect(s).toBe(0); + }); + + test('partial pass → fractional score', () => { + const t = makeTrajectory({ final_text: 'hello' }); + const s = scoreRule(t, [ + { op: 'contains', arg: 'hello' }, + { op: 'contains', arg: 'goodbye' }, + ]); + expect(s).toBe(0.5); + }); + + test('empty checks array → 0 (no signal)', () => { + expect(scoreRule(makeTrajectory(), [])).toBe(0); + }); + + test('regex op', () => { + const t = makeTrajectory({ final_text: 'order #42' }); + expect(scoreRule(t, [{ op: 'regex', arg: '#\\d+' }])).toBe(1); + expect(scoreRule(t, [{ op: 'regex', arg: 'XYZ' }])).toBe(0); + }); + + test('regex op tolerates malformed regex (returns false)', () => { + const t = makeTrajectory({ final_text: 'abc' }); + expect(scoreRule(t, [{ op: 'regex', arg: '[invalid' }])).toBe(0); + }); + + test('section_present matches any heading depth', () => { + const t = makeTrajectory({ final_text: '# Outline\n## People\n### Team' }); + expect(scoreRule(t, [{ op: 'section_present', arg: 'People' }])).toBe(1); + expect(scoreRule(t, [{ op: 'section_present', arg: 'Team' }])).toBe(1); + expect(scoreRule(t, [{ op: 'section_present', arg: 'Missing' }])).toBe(0); + }); + + test('min_citations counts links + brain-refs + footnotes', () => { + const t = makeTrajectory({ final_text: '[link1](http://a.com) and wiki/foo [1]' }); + expect(scoreRule(t, [{ op: 'min_citations', arg: 3 }])).toBe(1); + expect(scoreRule(t, [{ op: 'min_citations', arg: 4 }])).toBe(0); + }); + + test('tool_called requires a non-failed call with matching name', () => { + const t = makeTrajectory({ + tool_calls: [ + { name: 'search', input: {}, output: {}, failed: false }, + { name: 'get_page', input: {}, output: {}, failed: true }, + ], + }); + expect(scoreRule(t, [{ op: 'tool_called', arg: 'search' }])).toBe(1); + expect(scoreRule(t, [{ op: 'tool_called', arg: 'get_page' }])).toBe(0); // failed + expect(scoreRule(t, [{ op: 'tool_called', arg: 'never_called' }])).toBe(0); + }); + + test('tool_not_called passes when the tool never appears', () => { + const t = makeTrajectory({ + tool_calls: [{ name: 'search', input: {}, output: {}, failed: false }], + }); + expect(scoreRule(t, [{ op: 'tool_not_called', arg: 'put_page' }])).toBe(1); + expect(scoreRule(t, [{ op: 'tool_not_called', arg: 'search' }])).toBe(0); + }); +}); + +describe('countCitations', () => { + test('counts markdown links, brain-refs, footnotes', () => { + expect(countCitations('[a](http://b)')).toBe(1); + expect(countCitations('see wiki/foo')).toBe(1); + expect(countCitations('mentioned [1] and [2]')).toBe(2); + expect(countCitations('[a](http://b) wiki/x [1]')).toBe(3); + }); + + test('handles empty input', () => { + expect(countCitations('')).toBe(0); + }); +}); + +describe('scoreQrels', () => { + test('returns nDCG when retrieved slugs overlap expected', () => { + const t = makeTrajectory({ + tool_calls: [{ + name: 'search', + input: {}, + output: { results: [{ slug: 'people/alice-example' }, { slug: 'companies/widget-co' }] }, + failed: false, + }], + }); + const s = scoreQrels(t, ['people/alice-example', 'companies/widget-co'], 10); + expect(s).toBeGreaterThan(0); + expect(s).toBeLessThanOrEqual(1); + }); + + test('returns 0 when no retrieval tool called', () => { + expect(scoreQrels(makeTrajectory(), ['anything'], 10)).toBe(0); + }); + + test('returns 0 when all expected slugs missing from retrieval', () => { + const t = makeTrajectory({ + tool_calls: [{ name: 'search', input: {}, output: { results: [{ slug: 'wrong/slug' }] }, failed: false }], + }); + expect(scoreQrels(t, ['people/missing'], 10)).toBe(0); + }); +}); + +describe('extractRetrievedSlugs', () => { + test('extracts slugs from various tool output shapes', () => { + const t = makeTrajectory({ + tool_calls: [ + { name: 'search', input: {}, output: { results: [{ slug: 'a' }, { slug: 'b' }] }, failed: false }, + { name: 'get_page', input: {}, output: { slug: 'c' }, failed: false }, + { name: 'list_pages', input: {}, output: { pages: [{ slug: 'd' }] }, failed: false }, + ], + }); + expect(extractRetrievedSlugs(t)).toEqual(['a', 'b', 'c', 'd']); + }); + + test('deduplicates repeated slugs', () => { + const t = makeTrajectory({ + tool_calls: [ + { name: 'search', input: {}, output: { results: [{ slug: 'a' }] }, failed: false }, + { name: 'search', input: {}, output: { results: [{ slug: 'a' }] }, failed: false }, + ], + }); + expect(extractRetrievedSlugs(t)).toEqual(['a']); + }); + + test('skips failed tool calls', () => { + const t = makeTrajectory({ + tool_calls: [ + { name: 'search', input: {}, output: { results: [{ slug: 'a' }] }, failed: true }, + { name: 'search', input: {}, output: { results: [{ slug: 'b' }] }, failed: false }, + ], + }); + expect(extractRetrievedSlugs(t)).toEqual(['b']); + }); +}); + +describe('scoreTrajectory (llm judge via DI)', () => { + test('returns parsed score from chat result', async () => { + const t = makeTrajectory({ final_text: 'good output' }); + const stub = async () => ({ + text: '{"score": 0.85, "rationale": "good"}', + blocks: [], + stopReason: 'end' as const, + usage: { input_tokens: 100, output_tokens: 20, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'test', + providerId: 'test', + }); + const r = await scoreTrajectory(t, { kind: 'llm', rubric: 'is it good?' }, { chatFn: stub as never }); + expect(r.score).toBeCloseTo(0.85, 2); + expect(r.rationale).toBe('good'); + }); + + test('returns score=0 on parse failure (pessimistic fallback)', async () => { + const t = makeTrajectory(); + const stub = async () => ({ + text: 'this is not JSON', + blocks: [], + stopReason: 'end' as const, + usage: { input_tokens: 100, output_tokens: 20, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'test', + providerId: 'test', + }); + const r = await scoreTrajectory(t, { kind: 'llm', rubric: 'r' }, { chatFn: stub as never }); + expect(r.score).toBe(0); + expect(r.judge_error).toBeDefined(); + }); + + test('clamps out-of-range scores to [0,1]', async () => { + const t = makeTrajectory(); + const stub = async () => ({ + text: '{"score": 1.7}', + blocks: [], + stopReason: 'end' as const, + usage: { input_tokens: 100, output_tokens: 20, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'test', + providerId: 'test', + }); + const r = await scoreTrajectory(t, { kind: 'llm', rubric: 'r' }, { chatFn: stub as never }); + expect(r.score).toBe(1); + }); + + test('returns score=0 + judge_error when chat throws', async () => { + const t = makeTrajectory(); + const stub = async () => { throw new Error('network down'); }; + const r = await scoreTrajectory(t, { kind: 'llm', rubric: 'r' }, { chatFn: stub as never }); + expect(r.score).toBe(0); + expect(r.judge_error).toMatch(/llm_call_failed.*network down/); + }); +});