diff --git a/CHANGELOG.md b/CHANGELOG.md index 857b8e63f..8249ae377 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,83 @@ All notable changes to GBrain will be documented in this file. +## [0.41.36.0] - 2026-05-30 + +**Your MCP clients can now see and use your agent's skills. Point Codex desktop, +Claude Code, Claude Cowork, or Perplexity at your `gbrain serve` and they can +list every skill your brain knows, read one, and follow it — using the same +search/query/store tools the server already gives them. No copy-paste, no +re-explaining your workflows to each new tool.** + +Until now, a skill (the fat-markdown files that teach an agent how to do things: +brain ops, ingestion, enrichment, your custom workflows) only worked inside the +one agent that had them on disk. A thin MCP client connecting to your brain saw +the data tools but had no idea your skills existed. This release publishes them. + +Two new tools show up on your MCP server: + +- **`list_skills`** — a flat catalog of every skill: name, what it does, the + phrasings that should trigger it, and which tools it needs. Each entry also + says which of those tools *you* can actually call (given this server and your + access), so the catalog never points you at something that'll fail. It opens + with a short "here's what these are and how to use them" note, because a skill + is instructions to follow, not a program to run. +- **`get_skill `** — the full prose of one skill, plus a sanitized header + and a reminder of the tools you can call. You read it, then do what it says by + calling the brain tools this server exposes. + +### How to turn it on + +New installs are on by default. Existing installs stay off until you say yes — +upgrading never silently hands your already-issued tokens a new capability. On +`gbrain upgrade` you'll get a one-time prompt that shows the exact skills folder +being published and states plainly that the contents of those SKILL.md files +become readable by the remote MCP clients you've authorized. Say yes (strongly +recommended — it makes your MCP server dramatically more useful) and it's on. +Flip it anytime: + + gbrain config set mcp.publish_skills true # or false to stop publishing + +If autodetect picks the wrong folder under a daemon or container, pin it: + + gbrain config set mcp.skills_dir /path/to/your/skills + +### What's safe to know about + +This reads SKILL.md files off the serving machine and returns their text over +the network, so it's gated carefully: off-by-default for existing installs, +explicit owner opt-in, prose only (no source code), a size cap so a stray giant +file can't choke the server, and strict path confinement so a request can only +ever reach a real skill inside your skills folder. Private frontmatter fields +(your brain's filing taxonomy, any absolute paths a skill declares) are stripped +before anything goes out. A hosted brain with no agent repo on disk returns a +clear "no skills here" instead of accidentally serving gbrain's own bundled +dev skills. Locally, `gbrain skills` and `gbrain skill ` always work +regardless of the publish setting — the owner on the box owns the box. + +Downloadable, installable skillpacks (tarball any skill or the whole set and +pull it into your own setup) are the next step and ship separately. + +## To take advantage of v0.41.36.0 + +`gbrain upgrade` handles this. On upgrade you'll be asked once whether to publish +your skills to MCP clients — say yes to let Codex/Perplexity/Cowork discover them. + +1. **Confirm or flip the setting:** + ```bash + gbrain config set mcp.publish_skills true + ``` +2. **Check what's published** (works locally regardless of the gate): + ```bash + gbrain skills # the catalog your MCP clients will see + gbrain skill brain-ops # one skill's full instructions + ``` +3. **If the catalog is empty or wrong under a daemon/container,** pin the folder: + ```bash + gbrain config set mcp.skills_dir /path/to/your/skills + ``` +4. **If anything looks off,** file an issue at + https://github.com/garrytan/gbrain/issues with the output of `gbrain skills`. ## [0.41.35.0] - 2026-05-30 **GBrain now has five built-in spots where an outside content checker can watch diff --git a/CLAUDE.md b/CLAUDE.md index 859c50c86..82e31e249 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -149,6 +149,7 @@ strict behavior when unset. - `src/commands/models.ts` (v0.31.12) — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain to attribute properly), every per-task override (11 `PER_TASK_KEYS` entries — `models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map (defaults + user overrides), and a source-of-truth column showing `default` / `config: ` / `env: `. `gbrain models doctor [--skip=] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}` — the structural fix for the v0.31.6 silent-no-op bug class. Wired into `cli.ts` dispatch table + `CLI_ONLY` set. **v0.33.1.1 (#962, Codex P3 follow-up):** doctor gains a zero-token `embedding_config` probe that runs FIRST, before any chat/expansion probes spend money. `probeEmbeddingConfig()` reads `getEmbeddingModel()` + `getEmbeddingDimensions()` from the gateway, parses the model id, and (for Voyage flexible-dim models) checks `isValidVoyageOutputDim(dims)` against `VOYAGE_VALID_OUTPUT_DIMS`. New `ProbeStatus` variant `'config'` and optional `fix?: string` field on `ProbeResult` — surfaced in both human output (paste-ready `gbrain config set ...` line under the bad probe) and JSON output. New touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'` in the probe-row taxonomy. Closes the Voyage flexible-dim bug class at config time, not first-embed. - `src/commands/doctor.ts` extension (v0.31.12) — new `subagent_provider` check (layer 3 of 3 — Codex F13). Warns when `models.tier.subagent` is explicitly set to a non-Anthropic provider (fail-loud since the user clearly meant it — message names the bad value and prints the paste-ready fix command `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK status when subagent tier resolves to Anthropic. Tests cover all three paths in `test/doctor.test.ts`. - `src/core/skill-trigger-index.ts` (v0.41.14.0, closes #1451) — Shared loader that unions per-skill SKILL.md frontmatter `triggers:` with curated RESOLVER.md / AGENTS.md rows from `skillsDir` AND the parent directory (preserves the OpenClaw workspace-root layout). UNION semantics: explicit RESOLVER.md rows ADD to frontmatter triggers (don't replace). Dedup keyed on `(skillPath, trigger.trim().toLowerCase())` so case/whitespace drift collapses to one entry. Three consumers fold through this primitive — `checkResolvable`, `runRoutingEvalCli`, `mounts-cache.composeResolvers` — closing the v0.41 drift bug class (#1451) where per-consumer loaders meant fixing frontmatter for doctor wouldn't reach the routing-eval CLI or cross-brain composed dispatchers. Exports `loadSkillTriggerIndex(skillsDir): SkillTriggerEntry[]`, `entriesToResolverContent(entries): string` (synthesizes a markdown-table resolver-content string for `runRoutingEval`'s string-content API), `findPrimaryResolverPath(skillsDir): string | null`, plus the `FRONTMATTER_SECTION` constant and `_resetWarnedSkillsForTests` test seam. Skip rules: non-directory entries, `_*` / `.*` prefixes, `conventions/` + `migrations/` subdirs, skills with no `SKILL.md` (deprecated `install/` graceful-skipped), skills with no `triggers:` array, malformed YAML (warn-once + skip). Reuses `parseSkillFrontmatter` from `src/core/skill-frontmatter.ts` (regex-based, not full YAML — adequate for the uniform shape every bundled skill uses, follow-up TODO if drift surfaces). Pinned by 18 hermetic cases in `test/skill-trigger-index.test.ts` covering frontmatter auto-registration, RESOLVER.md/AGENTS.md merge, OpenClaw `../AGENTS.md` parent-dir scan, case-insensitive dedupe, deprecated `install` skip, missing-skillsDir resilience, malformed-frontmatter warn-once, and `entriesToResolverContent` round-trip through `parseResolverEntries`. CI gate: `bun run check:resolver` (= `bun src/cli.ts check-resolvable --strict --skills-dir skills/`) wired into `bun run verify` so future drift between frontmatter and routing-eval fixtures fails PR CI. +- `src/core/skill-catalog.ts` (v0.41.36.0, PR1) — host-repo skill catalog backing the MCP `list_skills` / `get_skill` ops. Lets a thin MCP client (Codex desktop, Claude Code, Claude Cowork, Perplexity) DISCOVER + FOLLOW the agent repo's fat-markdown skills over `gbrain serve` — a skill is prose, not code, so "using" one = fetching its body then calling the same gbrain MCP tools the server already exposes. Read-scope, NOT localOnly (defensible only via the full mitigation stack documented in the file's trust-boundary memo). Mitigations: (1) **publish gate** — `assertPublishEnabled(ctx, publishSkills)`; remote callers require `mcp.publish_skills === true`, default-OFF at runtime so an upgrade never silently grants existing read tokens host-skill read; local callers (`ctx.remote === false`) always pass. (2) **path confinement** — `assertSkillNameShape` rejects separators/`..`/null/space before any FS access; the client `name` is a manifest LOOKUP KEY (via `loadOrDeriveManifest`), never a raw path segment; `confineManifestPath` does realpath + relative-containment + `SKILL.md`-regular-file check on EVERY entry (defeats poisoned manifest.json `path`, symlink/`..` escape). (3) **frontmatter allowlist** — `GetSkillResult.frontmatter` projects a safe subset; private `writes_to` (brain taxonomy) + `sources` (absolute paths) dropped. (4) **prose-only + 256KB cap** (`MAX_SKILL_MD_BYTES`, env `GBRAIN_MAX_SKILL_MD_BYTES`) — no source code (tarballs deferred to PR2), size-checked twice (statSync + UTF-8 byte length). (5) **no install_path serve for remote** — remote callers use `autoDetectSkillsDir` (no install-path tier) so a hosted gbrain with no agent repo returns `storage_error`, never gbrain's own bundled dev skills; local callers use `autoDetectSkillsDirReadOnly`. (6) MCP rate-limiter caps call rate. Config reads honor BOTH planes: `readMcpPublishSkills` / `readMcpSkillsDir` prefer the DB plane (`engine.getConfig`, what `gbrain config set` + `gbrain init` write) over the file plane (`ctx.config.mcp`). D7 tool-honesty: `crossReferenceTools(declared, ctx)` splits a skill's declared `tools:` into `usable_tools` (in this server's op set AND callable by the caller's scope) vs `unavailable_tools`, and `buildSkillCatalog`'s `instructions` envelope (`SKILL_CATALOG_INSTRUCTIONS` from operations-descriptions.ts) carries the load-bearing "these are prose, follow-then-call-tools" use protocol. Skills are host-filesystem repo-global — `sourceScopeOpts(ctx)` / `ctx.brainId` deliberately do NOT apply (one server host = one skills dir = one catalog). `buildSkillCatalog` is resilient (one malformed/escaping skill is skipped, never throws the call). Config keys land in `src/core/config.ts`: `GBrainConfig.mcp?: { publish_skills?, skills_dir? }` + `KNOWN_CONFIG_KEYS` entries `mcp.publish_skills` / `mcp.publish_skills_prompted` / `mcp.skills_dir` + the `mcp.` prefix in `KNOWN_CONFIG_KEY_PREFIXES`. `src/commands/init.ts` writes `config.mcp = { publish_skills: true, ... }` for new PGLite + Postgres installs (existing config wins on re-init so a prior opt-out survives). `src/commands/upgrade.ts:runPostUpgrade` adds a one-time consent prompt (gated by `mcp.publish_skills_prompted`; existing installs stay OFF until the owner opts in; TTY readline Y/n with non-TTY paste-ready hint). The two ops register in `src/core/operations.ts` (`list_skills` with optional `section` filter + `cliHints: { name: 'skills' }`; `get_skill` taking `name` + `cliHints: { name: 'skill', positional: ['name'] }`) and dynamically import this module to avoid the import cycle (skill-catalog statically imports the `operations` array for D7). Descriptions + envelopes in `src/core/operations-descriptions.ts` (`LIST_SKILLS_DESCRIPTION`, `GET_SKILL_DESCRIPTION`, `SKILL_CATALOG_INSTRUCTIONS`, `SKILL_CLIENT_GUIDANCE`), pinned by `test/operations-descriptions.test.ts`. CLI surfaces: `gbrain skills` / `gbrain skill ` (via cliHints). Pinned by `test/skill-catalog.test.ts`, `test/skill-catalog-security.test.ts` (path-confinement / poisoned-manifest / symlink-escape), `test/skill-catalog-transports.test.ts` (publish-gate + remote-vs-local resolution) over fixtures at `test/fixtures/skill-catalog/`. PR2 (downloadable/installable skillpack tarballs) is deferred and tracked in TODOS.md. - `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. **v0.41.7.0:** `parseResolverEntries` accepts a second compact list format alongside the original markdown table (`- **skill-name**: trigger1 | trigger2 | trigger3` or `- skill-name: trigger1 | trigger2`). Both shapes can mix in one file; the v0.31.7 multi-resolver merge folds them into one entry stream. Skill name MUST be kebab-lowercase (regex `[a-z][a-z0-9-]+`) so prose bullets like `- **Note**: …`, `- **Convention**: …`, `- **TODO**: …` in real-world AGENTS.md files don't false-match as skill rows (codex F2 / D4). `skillPath` is ALWAYS derived as `skills//SKILL.md`: an optional `→ \`skills/path\`` (or ASCII `->`) suffix is stripped from the trigger string but NOT honored as the path — two downstream consumers (`routing-eval.ts:skillSlugFromPath`, the manifest lookup at `:367`) assume the convention, and honoring the explicit path would silently break their coverage. For non-conventional paths, use the table format. Multi-trigger rows fan out to one entry per trigger sharing the same `skillPath`; `checkResolvable` dedupes downstream so the reachability count counts each skill once. Closes the OpenClaw bug class where a 306-skill agent's list-format resolver registered 238 FAILs ("skill not reachable from RESOLVER.md") on every `gbrain doctor` run — productionized from upstream PR #1370 by @garrytan-agents. Pinned by 11 new unit cases in `test/check-resolvable.test.ts` (bold + plain forms, Unicode + ASCII suffix strip, ellipsis filter, empty pipe segments, mixed shapes, prose-bullet rejection regression) and the 8-case integration suite in `test/check-resolvable-openclaw-compact.test.ts` over two fixtures: `test/fixtures/openclaw-compact-resolver/` (10 skills in pure compact form) + `test/fixtures/openclaw-mixed-merge/` (table-format `skills/RESOLVER.md` + compact `../AGENTS.md` for the v0.31.7 merge contract). User-facing tutorial: `docs/guides/scaling-skills.md` walks through when and why to switch to the compact format (the three-tier scaling architecture that gets a 300-skill agent down to ~4K tokens per turn from ~25K with zero capability loss). - `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic. **v0.31.7:** read-path / write-path split. `autoDetectSkillsDir` (shared, read+write-safe) gains tier-0 `$GBRAIN_SKILLS_DIR` explicit operator override (Docker mounts, CI, monorepo subdirs) ahead of the existing 4-tier chain. New `autoDetectSkillsDirReadOnly` wraps it with a tier-5 install-path fallback that walks up from `fileURLToPath(import.meta.url)` and gates on `isGbrainRepoRoot` so unrelated repos can't false-positive. Read-path callers (`doctor`, `check-resolvable`, `routing-eval`) use the read-only variant; write-path callers (`skillpack install`, `skillify scaffold`, `post-install-advisory`) deliberately stay on the shared function so `gbrain skillpack install` from `~` cannot silently retarget the bundled gbrain repo's `skills/` instead of the user's actual workspace. Two new `SkillsDirSource` variants: `'env_explicit'`, `'install_path'`. New `AUTO_DETECT_HINT_READ_ONLY` documents the extra tier. The D6 `--fix` safety gate in `doctor.ts` + `check-resolvable.ts` refuses auto-repair when `detected.source === 'install_path'` so `gbrain doctor --fix` from `~` cannot silently rewrite the bundled install tree. - `src/commands/check-resolvable.ts` — Standalone CLI wrapper (v0.16.4) over `checkResolvable()`. Exports `parseFlags`, `resolveSkillsDir`, `DEFERRED`, `runCheckResolvable`. Exit rule: **1 on any issue (warnings OR errors)**, stricter than doctor's `ok` flag — honors README:259. Stable JSON envelope `{ok, skillsDir, report, autoFix, deferred, error, message}` — same shape on success and error paths. `--fix` path runs `autoFixDryViolations` BEFORE `checkResolvable` (same ordering as doctor). `scripts/skillify-check.ts` subprocess-calls `gbrain check-resolvable --json` (cached per process) and fails loud on binary-missing — no silent false-pass. **v0.19:** AGENTS.md workspaces now resolve natively (see `src/core/resolver-filenames.ts`) — gbrain inspects the 107-skill OpenClaw deployment whether the routing file is `RESOLVER.md` or `AGENTS.md`. `DEFERRED[]` is empty — Checks 5 + 6 shipped as real code, not issue URLs. **v0.31.7:** the resolver lookup switched from first-match-wins to the multi-file merge in `src/core/check-resolvable.ts` — entries collected from every `RESOLVER.md` / `AGENTS.md` across the skills dir AND its parent, deduped by `skillPath` (first occurrence wins). Lifted reachable skills on the reference OpenClaw layout from 37/224 to 200/224 — the deployment ships a thin `skills/RESOLVER.md` (~40 entries from skillpack) plus a fat `../AGENTS.md` (200+ entries, the real dispatcher), and the previous code only saw the first one. The CLI also switched to `autoDetectSkillsDirReadOnly` so `cd ~ && gbrain check-resolvable` finds the bundled skills via the install-path fallback. `--fix` carries the same D6 safety gate as `gbrain doctor --fix`: refuses to write when `detected.source === 'install_path'`. diff --git a/TODOS.md b/TODOS.md index cac097918..9173c37a4 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,17 @@ # TODOS +## MCP skillpack distribution — PR2 (v0.41.37+) + +Filed from the v0.41.36.0 skill-catalog wave (`list_skills` / `get_skill`). +PR1 shipped the read-only catalog; PR2 is the download-and-install surface, +deferred per the plan's D1 + D8 because it stands up new HTTP/binary/token +infra and reaches into third-party packs that live outside the host skills dir. + +- [ ] **v0.41.37+: `build_skillpack` op + `GET /skillpack/download/:token` endpoint.** Build a deterministic `.tgz` on demand (named skillpack, ad-hoc skill subset, or whole repo) and deliver it both base64-inline (universal/stdio) and via an authenticated short-lived download URL when running under `gbrain serve --http`. **What:** new admin-or-write-scoped op + a token-store + cache-dir GC; reuse `packTarball` from `src/core/skillpack/tarball.ts` (already deterministic + symlink-rejecting + size-capped) and the magic-link nonce pattern in `serve-http.ts`. The tarball ships source CODE, so it needs its own trust decision separate from PR1's prose-only catalog. **Why:** lets a thin client install a skillpack into its own setup, not just follow one live. **Depends on:** PR1 (landed in v0.41.36.0). Priority: P2. +- [ ] **v0.41.37+: `include_skillpacks` merge in `list_skills`.** Fold pinned third-party packs (from `~/.gbrain/skillpack-state.json`) into the catalog. Deferred from PR1 (D8) because packs live OUTSIDE the host skills dir and need (a) a per-pack trusted-root realpath confinement and (b) `{name, skillpack_name?}` disambiguation when a pack skill and a host skill share a name. Lands naturally with PR2's pack machinery. Priority: P2. +- [ ] **v0.41.37+: TTL+mtime cache for the skill-catalog walk.** PR1 reads fresh every call (cold path, ~ms). If telemetry shows repeated `list_skills` calls, add a TTL+mtime-keyed cache shared by `list_skills` + `get_skill`. Priority: P3 (do-nothing was the deliberate PR1 call). +- [ ] **v0.41.37+: routing-eval for the `list_skills` instructional envelope + per-skill `tools:` version-skew validation.** The envelope is load-bearing prose with no eval gate yet; and a skill's declared `tools:` aren't validated against the serving gbrain's actual op set for version drift. Priority: P3. +- [ ] **v0.41.37+: fix malformed `~/.agents/skills/gbrain/.../install/SKILL.md` (missing frontmatter).** Surfaced by codex's own startup error during the v0.41.36.0 plan review — an unrelated stray skill in the agents tree has no `---` frontmatter fence. Not gbrain-repo code; flag/clean separately. Priority: P3. ## v0.41.34.0 retrieval-cathedral follow-ups (v0.42+) Deferred from the v0.41.34.0 wave (codex adversarial P1/P2 — documented tradeoffs, diff --git a/VERSION b/VERSION index e8eb02247..4c1a4ec65 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.35.0 \ No newline at end of file +0.41.36.0 diff --git a/llms-full.txt b/llms-full.txt index 9eb0f74b9..16c2f0cc7 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -291,6 +291,7 @@ strict behavior when unset. - `src/commands/models.ts` (v0.31.12) — `gbrain models [--json]` read-only routing dashboard: prints tier defaults (`utility`/`reasoning`/`deep`/`subagent`), the resolved value for each (re-walking the resolution chain to attribute properly), every per-task override (11 `PER_TASK_KEYS` entries — `models.dream.synthesize`, `models.dream.patterns`, `models.drift`, `models.auto_think`, `models.think`, `models.subagent`, `facts.extraction_model`, `models.eval.longmemeval`, `models.expansion`, `models.chat`, `models.dream.synthesize_verdict`), the alias map (defaults + user overrides), and a source-of-truth column showing `default` / `config: ` / `env: `. `gbrain models doctor [--skip=] [--json]` fires a 1-token `gateway.chat()` probe against each configured chat + expansion model and classifies failures into `{model_not_found, auth, rate_limit, network, unknown}` — the structural fix for the v0.31.6 silent-no-op bug class. Wired into `cli.ts` dispatch table + `CLI_ONLY` set. **v0.33.1.1 (#962, Codex P3 follow-up):** doctor gains a zero-token `embedding_config` probe that runs FIRST, before any chat/expansion probes spend money. `probeEmbeddingConfig()` reads `getEmbeddingModel()` + `getEmbeddingDimensions()` from the gateway, parses the model id, and (for Voyage flexible-dim models) checks `isValidVoyageOutputDim(dims)` against `VOYAGE_VALID_OUTPUT_DIMS`. New `ProbeStatus` variant `'config'` and optional `fix?: string` field on `ProbeResult` — surfaced in both human output (paste-ready `gbrain config set ...` line under the bad probe) and JSON output. New touchpoint label `'embedding_config'` joins `'chat'` and `'expansion'` in the probe-row taxonomy. Closes the Voyage flexible-dim bug class at config time, not first-embed. - `src/commands/doctor.ts` extension (v0.31.12) — new `subagent_provider` check (layer 3 of 3 — Codex F13). Warns when `models.tier.subagent` is explicitly set to a non-Anthropic provider (fail-loud since the user clearly meant it — message names the bad value and prints the paste-ready fix command `gbrain config set models.tier.subagent anthropic:claude-sonnet-4-6`); also warns when `models.default` would sneak `subagent` into a non-Anthropic provider via tier inheritance. OK status when subagent tier resolves to Anthropic. Tests cover all three paths in `test/doctor.test.ts`. - `src/core/skill-trigger-index.ts` (v0.41.14.0, closes #1451) — Shared loader that unions per-skill SKILL.md frontmatter `triggers:` with curated RESOLVER.md / AGENTS.md rows from `skillsDir` AND the parent directory (preserves the OpenClaw workspace-root layout). UNION semantics: explicit RESOLVER.md rows ADD to frontmatter triggers (don't replace). Dedup keyed on `(skillPath, trigger.trim().toLowerCase())` so case/whitespace drift collapses to one entry. Three consumers fold through this primitive — `checkResolvable`, `runRoutingEvalCli`, `mounts-cache.composeResolvers` — closing the v0.41 drift bug class (#1451) where per-consumer loaders meant fixing frontmatter for doctor wouldn't reach the routing-eval CLI or cross-brain composed dispatchers. Exports `loadSkillTriggerIndex(skillsDir): SkillTriggerEntry[]`, `entriesToResolverContent(entries): string` (synthesizes a markdown-table resolver-content string for `runRoutingEval`'s string-content API), `findPrimaryResolverPath(skillsDir): string | null`, plus the `FRONTMATTER_SECTION` constant and `_resetWarnedSkillsForTests` test seam. Skip rules: non-directory entries, `_*` / `.*` prefixes, `conventions/` + `migrations/` subdirs, skills with no `SKILL.md` (deprecated `install/` graceful-skipped), skills with no `triggers:` array, malformed YAML (warn-once + skip). Reuses `parseSkillFrontmatter` from `src/core/skill-frontmatter.ts` (regex-based, not full YAML — adequate for the uniform shape every bundled skill uses, follow-up TODO if drift surfaces). Pinned by 18 hermetic cases in `test/skill-trigger-index.test.ts` covering frontmatter auto-registration, RESOLVER.md/AGENTS.md merge, OpenClaw `../AGENTS.md` parent-dir scan, case-insensitive dedupe, deprecated `install` skip, missing-skillsDir resilience, malformed-frontmatter warn-once, and `entriesToResolverContent` round-trip through `parseResolverEntries`. CI gate: `bun run check:resolver` (= `bun src/cli.ts check-resolvable --strict --skills-dir skills/`) wired into `bun run verify` so future drift between frontmatter and routing-eval fixtures fails PR CI. +- `src/core/skill-catalog.ts` (v0.41.36.0, PR1) — host-repo skill catalog backing the MCP `list_skills` / `get_skill` ops. Lets a thin MCP client (Codex desktop, Claude Code, Claude Cowork, Perplexity) DISCOVER + FOLLOW the agent repo's fat-markdown skills over `gbrain serve` — a skill is prose, not code, so "using" one = fetching its body then calling the same gbrain MCP tools the server already exposes. Read-scope, NOT localOnly (defensible only via the full mitigation stack documented in the file's trust-boundary memo). Mitigations: (1) **publish gate** — `assertPublishEnabled(ctx, publishSkills)`; remote callers require `mcp.publish_skills === true`, default-OFF at runtime so an upgrade never silently grants existing read tokens host-skill read; local callers (`ctx.remote === false`) always pass. (2) **path confinement** — `assertSkillNameShape` rejects separators/`..`/null/space before any FS access; the client `name` is a manifest LOOKUP KEY (via `loadOrDeriveManifest`), never a raw path segment; `confineManifestPath` does realpath + relative-containment + `SKILL.md`-regular-file check on EVERY entry (defeats poisoned manifest.json `path`, symlink/`..` escape). (3) **frontmatter allowlist** — `GetSkillResult.frontmatter` projects a safe subset; private `writes_to` (brain taxonomy) + `sources` (absolute paths) dropped. (4) **prose-only + 256KB cap** (`MAX_SKILL_MD_BYTES`, env `GBRAIN_MAX_SKILL_MD_BYTES`) — no source code (tarballs deferred to PR2), size-checked twice (statSync + UTF-8 byte length). (5) **no install_path serve for remote** — remote callers use `autoDetectSkillsDir` (no install-path tier) so a hosted gbrain with no agent repo returns `storage_error`, never gbrain's own bundled dev skills; local callers use `autoDetectSkillsDirReadOnly`. (6) MCP rate-limiter caps call rate. Config reads honor BOTH planes: `readMcpPublishSkills` / `readMcpSkillsDir` prefer the DB plane (`engine.getConfig`, what `gbrain config set` + `gbrain init` write) over the file plane (`ctx.config.mcp`). D7 tool-honesty: `crossReferenceTools(declared, ctx)` splits a skill's declared `tools:` into `usable_tools` (in this server's op set AND callable by the caller's scope) vs `unavailable_tools`, and `buildSkillCatalog`'s `instructions` envelope (`SKILL_CATALOG_INSTRUCTIONS` from operations-descriptions.ts) carries the load-bearing "these are prose, follow-then-call-tools" use protocol. Skills are host-filesystem repo-global — `sourceScopeOpts(ctx)` / `ctx.brainId` deliberately do NOT apply (one server host = one skills dir = one catalog). `buildSkillCatalog` is resilient (one malformed/escaping skill is skipped, never throws the call). Config keys land in `src/core/config.ts`: `GBrainConfig.mcp?: { publish_skills?, skills_dir? }` + `KNOWN_CONFIG_KEYS` entries `mcp.publish_skills` / `mcp.publish_skills_prompted` / `mcp.skills_dir` + the `mcp.` prefix in `KNOWN_CONFIG_KEY_PREFIXES`. `src/commands/init.ts` writes `config.mcp = { publish_skills: true, ... }` for new PGLite + Postgres installs (existing config wins on re-init so a prior opt-out survives). `src/commands/upgrade.ts:runPostUpgrade` adds a one-time consent prompt (gated by `mcp.publish_skills_prompted`; existing installs stay OFF until the owner opts in; TTY readline Y/n with non-TTY paste-ready hint). The two ops register in `src/core/operations.ts` (`list_skills` with optional `section` filter + `cliHints: { name: 'skills' }`; `get_skill` taking `name` + `cliHints: { name: 'skill', positional: ['name'] }`) and dynamically import this module to avoid the import cycle (skill-catalog statically imports the `operations` array for D7). Descriptions + envelopes in `src/core/operations-descriptions.ts` (`LIST_SKILLS_DESCRIPTION`, `GET_SKILL_DESCRIPTION`, `SKILL_CATALOG_INSTRUCTIONS`, `SKILL_CLIENT_GUIDANCE`), pinned by `test/operations-descriptions.test.ts`. CLI surfaces: `gbrain skills` / `gbrain skill ` (via cliHints). Pinned by `test/skill-catalog.test.ts`, `test/skill-catalog-security.test.ts` (path-confinement / poisoned-manifest / symlink-escape), `test/skill-catalog-transports.test.ts` (publish-gate + remote-vs-local resolution) over fixtures at `test/fixtures/skill-catalog/`. PR2 (downloadable/installable skillpack tarballs) is deferred and tracked in TODOS.md. - `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. v0.14.1: `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts both `conventions/quality.md` and `_brain-filing-rules.md`). New `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. **v0.41.7.0:** `parseResolverEntries` accepts a second compact list format alongside the original markdown table (`- **skill-name**: trigger1 | trigger2 | trigger3` or `- skill-name: trigger1 | trigger2`). Both shapes can mix in one file; the v0.31.7 multi-resolver merge folds them into one entry stream. Skill name MUST be kebab-lowercase (regex `[a-z][a-z0-9-]+`) so prose bullets like `- **Note**: …`, `- **Convention**: …`, `- **TODO**: …` in real-world AGENTS.md files don't false-match as skill rows (codex F2 / D4). `skillPath` is ALWAYS derived as `skills//SKILL.md`: an optional `→ \`skills/path\`` (or ASCII `->`) suffix is stripped from the trigger string but NOT honored as the path — two downstream consumers (`routing-eval.ts:skillSlugFromPath`, the manifest lookup at `:367`) assume the convention, and honoring the explicit path would silently break their coverage. For non-conventional paths, use the table format. Multi-trigger rows fan out to one entry per trigger sharing the same `skillPath`; `checkResolvable` dedupes downstream so the reachability count counts each skill once. Closes the OpenClaw bug class where a 306-skill agent's list-format resolver registered 238 FAILs ("skill not reachable from RESOLVER.md") on every `gbrain doctor` run — productionized from upstream PR #1370 by @garrytan-agents. Pinned by 11 new unit cases in `test/check-resolvable.test.ts` (bold + plain forms, Unicode + ASCII suffix strip, ellipsis filter, empty pipe segments, mixed shapes, prose-bullet rejection regression) and the 8-case integration suite in `test/check-resolvable-openclaw-compact.test.ts` over two fixtures: `test/fixtures/openclaw-compact-resolver/` (10 skills in pure compact form) + `test/fixtures/openclaw-mixed-merge/` (table-format `skills/RESOLVER.md` + compact `../AGENTS.md` for the v0.31.7 merge contract). User-facing tutorial: `docs/guides/scaling-skills.md` walks through when and why to switch to the compact format (the three-tier scaling architecture that gets a 300-skill agent down to ~4K tokens per turn from ~25K with zero capability loss). - `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)` (v0.16.4): walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency module imported by both `doctor.ts` and `check-resolvable.ts`. Parameterized `startDir` makes tests hermetic. **v0.31.7:** read-path / write-path split. `autoDetectSkillsDir` (shared, read+write-safe) gains tier-0 `$GBRAIN_SKILLS_DIR` explicit operator override (Docker mounts, CI, monorepo subdirs) ahead of the existing 4-tier chain. New `autoDetectSkillsDirReadOnly` wraps it with a tier-5 install-path fallback that walks up from `fileURLToPath(import.meta.url)` and gates on `isGbrainRepoRoot` so unrelated repos can't false-positive. Read-path callers (`doctor`, `check-resolvable`, `routing-eval`) use the read-only variant; write-path callers (`skillpack install`, `skillify scaffold`, `post-install-advisory`) deliberately stay on the shared function so `gbrain skillpack install` from `~` cannot silently retarget the bundled gbrain repo's `skills/` instead of the user's actual workspace. Two new `SkillsDirSource` variants: `'env_explicit'`, `'install_path'`. New `AUTO_DETECT_HINT_READ_ONLY` documents the extra tier. The D6 `--fix` safety gate in `doctor.ts` + `check-resolvable.ts` refuses auto-repair when `detected.source === 'install_path'` so `gbrain doctor --fix` from `~` cannot silently rewrite the bundled install tree. - `src/commands/check-resolvable.ts` — Standalone CLI wrapper (v0.16.4) over `checkResolvable()`. Exports `parseFlags`, `resolveSkillsDir`, `DEFERRED`, `runCheckResolvable`. Exit rule: **1 on any issue (warnings OR errors)**, stricter than doctor's `ok` flag — honors README:259. Stable JSON envelope `{ok, skillsDir, report, autoFix, deferred, error, message}` — same shape on success and error paths. `--fix` path runs `autoFixDryViolations` BEFORE `checkResolvable` (same ordering as doctor). `scripts/skillify-check.ts` subprocess-calls `gbrain check-resolvable --json` (cached per process) and fails loud on binary-missing — no silent false-pass. **v0.19:** AGENTS.md workspaces now resolve natively (see `src/core/resolver-filenames.ts`) — gbrain inspects the 107-skill OpenClaw deployment whether the routing file is `RESOLVER.md` or `AGENTS.md`. `DEFERRED[]` is empty — Checks 5 + 6 shipped as real code, not issue URLs. **v0.31.7:** the resolver lookup switched from first-match-wins to the multi-file merge in `src/core/check-resolvable.ts` — entries collected from every `RESOLVER.md` / `AGENTS.md` across the skills dir AND its parent, deduped by `skillPath` (first occurrence wins). Lifted reachable skills on the reference OpenClaw layout from 37/224 to 200/224 — the deployment ships a thin `skills/RESOLVER.md` (~40 entries from skillpack) plus a fat `../AGENTS.md` (200+ entries, the real dispatcher), and the previous code only saw the first one. The CLI also switched to `autoDetectSkillsDirReadOnly` so `cd ~ && gbrain check-resolvable` finds the bundled skills via the install-path fallback. `--fix` carries the same D6 safety gate as `gbrain doctor --fix`: refuses to write when `detected.source === 'install_path'`. diff --git a/package.json b/package.json index 4b7aee086..12adeb4b0 100644 --- a/package.json +++ b/package.json @@ -141,5 +141,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.35.0" + "version": "0.41.36.0" } diff --git a/src/commands/init.ts b/src/commands/init.ts index 766dee8e1..e2d7e7daa 100644 --- a/src/commands/init.ts +++ b/src/commands/init.ts @@ -951,6 +951,9 @@ async function initPGLite(opts: { // unless explicitly overridden by --schema-pack on re-init. ...(opts.schemaPack ? { schema_pack: opts.schemaPack } : {}), }; + // PR1: new installs publish their skill catalog over MCP by default + // (existing config wins on re-init, so a prior opt-out is preserved). + config.mcp = { publish_skills: true, ...(config.mcp ?? {}) }; saveConfig(config); if (opts.schemaPack) { process.stderr.write( @@ -1188,6 +1191,9 @@ async function initPostgres(opts: { // v0.42 (T17): same schema_pack default as PGLite path. ...(opts.schemaPack ? { schema_pack: opts.schemaPack } : {}), }; + // PR1: new installs publish their skill catalog over MCP by default + // (existing config wins on re-init, so a prior opt-out is preserved). + config.mcp = { publish_skills: true, ...(config.mcp ?? {}) }; saveConfig(config); console.log('Config saved to ~/.gbrain/config.json'); if (opts.schemaPack) { diff --git a/src/commands/upgrade.ts b/src/commands/upgrade.ts index 48284b44d..c70ba5068 100644 --- a/src/commands/upgrade.ts +++ b/src/commands/upgrade.ts @@ -357,6 +357,59 @@ export async function runPostUpgrade(args: string[] = []): Promise { // Banner is cosmetic; never block the upgrade. } + // PR1: skill-catalog publish consent. New installs default ON at + // `gbrain init`; EXISTING installs stay OFF (default-OFF runtime = no + // silent capability grant on upgrade) until the owner opts in HERE. + // One-time, gated by `mcp.publish_skills_prompted`. Strongly recommended. + try { + const prompted = await engine.getConfig('mcp.publish_skills_prompted'); + const current = await engine.getConfig('mcp.publish_skills'); + if (prompted !== 'true' && current == null) { + const { autoDetectSkillsDir } = await import('../core/repo-root.ts'); + const det = autoDetectSkillsDir(); + const dirLine = det.dir + ? `Skills dir: ${det.dir} (source: ${det.source})` + : 'Skills dir: not auto-detected — set $GBRAIN_SKILLS_DIR or mcp.skills_dir before enabling.'; + console.log(''); + console.log('═══════════════════════════════════════════════════════════════'); + console.log('[gbrain] Publish your skills to MCP clients?'); + console.log('[gbrain] Codex desktop, Claude Code/Cowork, and Perplexity can then'); + console.log("[gbrain] DISCOVER and FOLLOW your agent's skills over `gbrain serve`."); + console.log('[gbrain] This makes your MCP server dramatically more useful.'); + console.log('[gbrain]'); + console.log(`[gbrain] ${dirLine}`); + console.log('[gbrain] Effect: the CONTENTS of your SKILL.md files become readable by'); + console.log('[gbrain] remote MCP callers you have authorized. Source code is NOT exposed.'); + console.log('═══════════════════════════════════════════════════════════════'); + const isTty = Boolean(process.stdin.isTTY && process.stdout.isTTY); + let enabled = false; + if (isTty) { + const { createInterface } = await import('readline'); + enabled = await new Promise((resolveAns) => { + const rl = createInterface({ input: process.stdin, output: process.stdout }); + rl.question('[gbrain] Enable skill publishing now? (recommended) [Y/n] ', (answer) => { + rl.close(); + const a = answer.trim().toLowerCase(); + resolveAns(a === '' || a === 'y' || a === 'yes'); + }); + rl.on('close', () => resolveAns(false)); + }); + } else { + console.log('[AGENT] Relay this to your operator. Recommended: enable it.'); + console.log('[AGENT] Enable with: gbrain config set mcp.publish_skills true'); + } + if (enabled) { + await engine.setConfig('mcp.publish_skills', 'true'); + console.log('[gbrain] Skill publishing ENABLED. Disable anytime: gbrain config set mcp.publish_skills false'); + } else if (isTty) { + console.log('[gbrain] Left disabled. Enable later: gbrain config set mcp.publish_skills true'); + } + await engine.setConfig('mcp.publish_skills_prompted', 'true'); + } + } catch { + // Consent prompt is best-effort; never block the upgrade. + } + // v0.32.7 CJK wave: chunker-version bump → re-embed sweep. // Idempotent — `runReindex` short-circuits when no pages are pending. try { diff --git a/src/core/config.ts b/src/core/config.ts index 711d3d0f0..2833e5b9d 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -240,6 +240,29 @@ export interface GBrainConfig { * operator escape hatch. */ schema_pack?: string; + + /** + * PR1 — MCP skill-catalog publishing. Lets a thin MCP client (Codex desktop, + * Claude Code, Perplexity) discover and follow this agent repo's skills over + * `gbrain serve`. See `src/core/skill-catalog.ts` for the trust-boundary memo. + */ + mcp?: { + /** + * Gate for `list_skills` / `get_skill` over a REMOTE transport. Runtime + * default is OFF (absent key → OFF) so an upgrade never silently grants + * existing read tokens host-skill read. `gbrain init` writes `true` for new + * installs; the upgrade migration prompts existing owners to enable it. + * Local CLI callers (`ctx.remote === false`) bypass the gate entirely. + */ + publish_skills?: boolean; + /** + * Explicit skills-dir override. Wins over autodetect — makes which skills + * get published deterministic across laptop / daemon / container launches. + * When unset, the ops autodetect (remote callers exclude the install-path + * tier so a hosted gbrain never serves its own bundled dev skills). + */ + skills_dir?: string; + }; } /** @@ -670,6 +693,10 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ 'content_sanity.bytes_block', 'content_sanity.junk_patterns_enabled', 'content_sanity.disabled', + // MCP skill-catalog publishing (PR1) + 'mcp.publish_skills', + 'mcp.publish_skills_prompted', + 'mcp.skills_dir', // Misc 'artifacts_sync_mode', 'cross_project_learnings', @@ -688,6 +715,7 @@ export const KNOWN_CONFIG_KEY_PREFIXES: readonly string[] = [ 'embedding_columns.', // per-column overrides 'provider_base_urls.', // per-provider base URL overrides 'content_sanity.', // v0.41 content-sanity tunables + 'mcp.', // mcp.publish_skills, mcp.skills_dir (PR1 skill catalog) ]; export function saveConfig(config: GBrainConfig): void { diff --git a/src/core/operations-descriptions.ts b/src/core/operations-descriptions.ts index e0d02b5c9..7cf77ae07 100644 --- a/src/core/operations-descriptions.ts +++ b/src/core/operations-descriptions.ts @@ -150,3 +150,75 @@ export const CODE_REFS_DESCRIPTION = "numbers, not symbol-qualified edges. Use this when planning a rename or " + "deprecation where you need to touch every literal mention. " + "Returns: `{symbol, count, refs: [{slug, file, language, line, context}]}`."; + +// ────────────────────────────────────────────────────────────────────────────── +// PR1 — skill catalog over MCP (list_skills / get_skill). The agent repo's +// fat-markdown skills, published so a thin MCP client (Codex desktop, Claude +// Code, Perplexity) can discover and FOLLOW them. Skills are prose instruction +// sets, not executable code — "using" one = fetching its prose and then calling +// the gbrain MCP tools the same server already exposes. The instructional +// envelope below is the load-bearing UX: it tells the pulling agent what these +// are and the use protocol. Pinned by test/operations-descriptions.test.ts. +// ────────────────────────────────────────────────────────────────────────────── + +export const LIST_SKILLS_DESCRIPTION = + "List the skills this agent's brain publishes. A skill is a named prose " + + "instruction set (NOT executable code) that teaches you how to do a task " + + "using this server's other tools. Returns a flat catalog — each entry has a " + + "name, one-line description, triggers (phrasings that should invoke it), and " + + "`usable_tools` / `unavailable_tools` (which tools the skill calls that you " + + "CAN vs CANNOT call given this server + your access). To actually use a skill, " + + "call get_skill with its name, read the returned prose, and follow it — calling " + + "the correspondingly-named tools on THIS server. The response also carries an " + + "`instructions` envelope explaining this protocol. Reflects the serving repo's " + + "skills even when the call targets a mounted brain. Read-scope; published only " + + "when the brain owner enabled mcp.publish_skills."; + +export const GET_SKILL_DESCRIPTION = + "Fetch one skill's full instructions by name. Returns `{name, frontmatter " + + "(sanitized), body, usable_tools, unavailable_tools, client_guidance}`. The " + + "`body` is prose — read it as your operating instructions for this task, and " + + "when it says to search / store / look something up, call the same-named MCP " + + "tool on THIS server. There is nothing to 'execute' — the value is the " + + "instructions plus your tool calls back to this server. Tools listed in " + + "`unavailable_tools` won't work for you (not exposed here, or beyond your " + + "access) — adapt accordingly. Size-capped; read-scope; requires the owner to " + + "have enabled mcp.publish_skills."; + +/** + * The load-bearing `instructions` envelope for list_skills. Pinned so the + * agent-facing protocol can't silently drift. `how_to_use` is the ordered + * protocol a thin client follows. + */ +export const SKILL_CATALOG_INSTRUCTIONS = { + summary: + "These are 'skills': named prose instruction sets, not executable tools. " + + "There is no skill to 'run' — a skill tells YOU how to accomplish a task " + + "using the MCP tools this same server already exposes.", + how_to_use: [ + "Pick a skill from this list whose triggers match the user's intent.", + "Call get_skill with its name to fetch the full prose (the `body`).", + "Follow that prose as your plan for the task.", + "When the prose says to search, store, link, or look something up, call the " + + "correspondingly-named MCP tool on THIS server (e.g. search, query, put_page).", + "Only call tools in this skill's `usable_tools`; tools in `unavailable_tools` " + + "are not callable by you on this server.", + ], +} as const; + +/** + * Per-skill `client_guidance` for get_skill. Same protocol, scoped to one skill. + */ +export const SKILL_CLIENT_GUIDANCE = { + nature: + "This is a fat-markdown instruction set, not code to execute. The `body` is " + + "your operating procedure; carry it out using this server's MCP tools.", + protocol: [ + "Read `body` as your operating instructions for this task.", + "When the prose names a brain operation (search, store, link, look up), call " + + "the MCP tool of that name on THIS server.", + "Do not invent tools — only the tools in `usable_tools` are callable by you.", + "If `mutating` is true, this skill writes to the brain; confirm before doing so " + + "if the user hasn't clearly asked for a write.", + ], +} as const; diff --git a/src/core/operations.ts b/src/core/operations.ts index 2ef0585c6..40871a70c 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -41,6 +41,8 @@ import { CODE_CALLEES_DESCRIPTION, CODE_DEF_DESCRIPTION, CODE_REFS_DESCRIPTION, + LIST_SKILLS_DESCRIPTION, + GET_SKILL_DESCRIPTION, } from './operations-descriptions.ts'; // --- Types --- @@ -1979,6 +1981,58 @@ const get_brain_identity: Operation = { // intentionally no cliHints — banner-only op }; +// --- PR1: skill catalog over MCP (host-repo skills for thin clients) --- +// Both ops dynamically import ./skill-catalog.ts to avoid an import cycle +// (skill-catalog statically imports the `operations` array for D7 tool honesty). +// Read-scope, non-localOnly: a thin client (Codex/Perplexity/Cowork) reaches +// these over HTTP. The host-filesystem read is gated by mcp.publish_skills + +// path confinement — see the trust-boundary memo in skill-catalog.ts. + +const list_skills: Operation = { + name: 'list_skills', + description: LIST_SKILLS_DESCRIPTION, + params: { + section: { + type: 'string', + description: 'Optional: only skills whose routing section matches this exactly.', + }, + }, + handler: async (ctx, p) => { + const sc = await import('./skill-catalog.ts'); + const publish = await sc.readMcpPublishSkills(ctx); + sc.assertPublishEnabled(ctx, publish); + const override = await sc.readMcpSkillsDir(ctx); + const { dir, source } = sc.resolveSkillsDir(ctx, override); + const section = typeof p.section === 'string' ? p.section : undefined; + return sc.buildSkillCatalog(ctx, dir, source, { section }); + }, + scope: 'read', + cliHints: { name: 'skills', positional: [] }, +}; + +const get_skill: Operation = { + name: 'get_skill', + description: GET_SKILL_DESCRIPTION, + params: { + name: { + type: 'string', + required: true, + description: 'Skill name exactly as returned by list_skills.', + }, + }, + handler: async (ctx, p) => { + const sc = await import('./skill-catalog.ts'); + const publish = await sc.readMcpPublishSkills(ctx); + sc.assertPublishEnabled(ctx, publish); + const override = await sc.readMcpSkillsDir(ctx); + const { dir } = sc.resolveSkillsDir(ctx, override); + const name = typeof p.name === 'string' ? p.name : ''; + return sc.getSkillDetail(ctx, dir, name); + }, + scope: 'read', + cliHints: { name: 'skill', positional: ['name'] }, +}; + /** * v0.41.19.0 — `gbrain status` thin-client surface. * @@ -4423,6 +4477,8 @@ export const operations: Operation[] = [ get_stats, get_health, run_doctor, get_versions, revert_version, // v0.31.1 (Issue #734): thin-client banner identity packet (read-scope, banner-only) get_brain_identity, + // PR1: skill catalog over MCP — discover + fetch host-repo skills (read-scope) + list_skills, get_skill, // v0.41.19.0: thin-client `gbrain status` payload (admin-scope, sync + cycle only) get_status_snapshot, // Sync diff --git a/src/core/skill-catalog.ts b/src/core/skill-catalog.ts new file mode 100644 index 000000000..237283521 --- /dev/null +++ b/src/core/skill-catalog.ts @@ -0,0 +1,539 @@ +/** + * skill-catalog.ts — host-repo skill catalog for the MCP `list_skills` / + * `get_skill` ops (PR1). + * + * WHAT THIS IS: the agent repo (OpenClaw/Hermes) ships fat-markdown "skills" — + * prose instruction sets, NOT executable code. This module reads them off the + * server host's filesystem and shapes them for a thin MCP client (Codex desktop, + * Claude Code, Perplexity) to discover and FOLLOW. "Using" a skill = fetching its + * prose and then calling the gbrain MCP tools the same server already exposes. + * + * TRUST-BOUNDARY MEMO (read before changing scope/localOnly on the ops): + * These ops read the server-host filesystem and return file CONTENTS over HTTP. + * gbrain treats `file_list`/`file_url` as `admin + localOnly` for exactly this + * reason. `read` + non-localOnly is defensible HERE, but ONLY because of the full + * mitigation stack — do not weaken one piece without re-reading the rest: + * 1. Explicit owner opt-in — the publish gate (`mcp.publish_skills`) defaults + * OFF at runtime; new installs turn it on at `gbrain init`, existing installs + * via a consenting migration. Unlike `file_list`, this is content the owner + * deliberately published. + * 2. Confinement — only the resolved skills dir is reachable. The client `name` + * is a manifest LOOKUP KEY, never a raw path segment; the manifest-derived + * path is realpath + relative-contained on every call (defeats symlink/`..` + * escape, including a poisoned manifest.json `path`); resolved path must be a + * regular file named `SKILL.md`. + * 3. Field allowlist — frontmatter is projected to a safe subset; `writes_to` + * (private brain taxonomy) and `sources` (absolute paths) are dropped. + * 4. Prose-only + size-capped — no source code (PR2 ships tarballs); `get_skill` + * caps the file size so a huge/binary file can't OOM the server. + * 5. No install_path serve for remote — a hosted gbrain with no agent repo + * returns `storage_error`, never gbrain's own bundled dev skills. + * 6. Bounded — the MCP rate limiter caps call rate; per-call memory is bounded + * by the size cap. + * + * Skills live on the HOST FILESYSTEM and are repo-global: `sourceScopeOpts(ctx)` / + * `ctx.sourceId` (in-DB tenancy) and `ctx.brainId` (mounts) deliberately do NOT + * apply. One server host = one skills dir = one catalog for all callers. + */ + +import { existsSync, readFileSync, realpathSync, statSync } from 'fs'; +import { basename, join, relative, resolve } from 'path'; +import { + autoDetectSkillsDir, + autoDetectSkillsDirReadOnly, + AUTO_DETECT_HINT, + AUTO_DETECT_HINT_READ_ONLY, + type SkillsDirSource, +} from './repo-root.ts'; +import { loadOrDeriveManifest, type ManifestEntry } from './skill-manifest.ts'; +import { loadSkillTriggerIndex, FRONTMATTER_SECTION } from './skill-trigger-index.ts'; +import { parseSkillFrontmatter } from './skill-frontmatter.ts'; +import { hasScope } from './scope.ts'; +import { operations, OperationError, type Operation, type OperationContext } from './operations.ts'; +import { + SKILL_CATALOG_INSTRUCTIONS, + SKILL_CLIENT_GUIDANCE, +} from './operations-descriptions.ts'; + +// --------------------------------------------------------------------------- +// Constants +// --------------------------------------------------------------------------- + +/** Max SKILL.md size `get_skill` will read/return. Env-overridable. Default + * 256 KB — generous for prose, small enough that a stray binary/generated file + * can't balloon the server's RSS (codex P1-c). */ +export const MAX_SKILL_MD_BYTES = (() => { + const raw = process.env.GBRAIN_MAX_SKILL_MD_BYTES; + if (raw) { + const n = parseInt(raw, 10); + if (Number.isFinite(n) && n > 0) return n; + } + return 256 * 1024; +})(); + +/** Max client-supplied skill name length before we even hit the manifest. */ +const MAX_SKILL_NAME_LEN = 128; + +/** Where auto-detect found the dir, plus the explicit-config variant. */ +export type ResolvedSkillsDirSource = SkillsDirSource | 'config'; + +// --------------------------------------------------------------------------- +// Types +// --------------------------------------------------------------------------- + +export interface SkillCatalogEntry { + name: string; + description: string; + section: string; + triggers: string[]; + tools: string[]; + /** declared tools ∩ (exposed by this server AND permitted to this caller). */ + usable_tools: string[]; + /** declared tools − usable (not on this server, or beyond caller scope). */ + unavailable_tools: string[]; + writes_pages: boolean; + mutating: boolean; +} + +export interface ListSkillsResult { + schema_version: 1; + skills_dir_source: ResolvedSkillsDirSource; + count: number; + skills: SkillCatalogEntry[]; + instructions: { + summary: string; + how_to_use: string[]; + available_brain_tools: string[]; + fetch_op: 'get_skill'; + }; +} + +export interface GetSkillResult { + schema_version: 1; + name: string; + /** Allowlisted projection — never the raw frontmatter object. */ + frontmatter: { + name?: string; + description?: string; + triggers?: string[]; + tools?: string[]; + writes_pages?: boolean; + mutating?: boolean; + }; + body: string; + usable_tools: string[]; + unavailable_tools: string[]; + client_guidance: { + nature: string; + protocol: string[]; + available_brain_tools: string[]; + mutating: boolean; + }; +} + +// --------------------------------------------------------------------------- +// Skills-dir resolution +// --------------------------------------------------------------------------- + +/** + * Read `mcp.publish_skills` honoring BOTH config planes. `gbrain config set` + * writes the DB plane (engine.setConfig) — so the DB value wins; `gbrain init` + * also writes the DB plane. The file plane (`config.json` → `ctx.config.mcp`) is + * a fallback for hand-edited configs. Absent in both → false (fail-safe). + */ +export async function readMcpPublishSkills(ctx: OperationContext): Promise { + let dbVal: string | null = null; + try { + dbVal = await ctx.engine.getConfig('mcp.publish_skills'); + } catch { + // Engine without a config table / transient error → fall back to file plane. + } + if (dbVal != null) return dbVal === 'true'; + return ctx.config?.mcp?.publish_skills === true; +} + +/** + * Read `mcp.skills_dir` from the DB plane (what `gbrain config set` writes), + * falling back to the file plane. Returns undefined when unset on both. + */ +export async function readMcpSkillsDir(ctx: OperationContext): Promise { + let dbVal: string | null = null; + try { + dbVal = await ctx.engine.getConfig('mcp.skills_dir'); + } catch { + // ignore — fall through to file plane + } + if (dbVal && dbVal.trim().length > 0) return dbVal; + const fileVal = ctx.config?.mcp?.skills_dir; + return fileVal && fileVal.trim().length > 0 ? fileVal : undefined; +} + +/** + * Resolve the skills dir for this call. An explicit `mcp.skills_dir` override + * (resolved from either config plane by the caller) wins; otherwise autodetect. + * REMOTE callers use `autoDetectSkillsDir` (no install_path tier — a hosted + * gbrain must never serve its own bundled dev skills); LOCAL callers + * (`ctx.remote === false`) use the read-only variant with the install_path + * fallback. Throws `storage_error` (with the search-path hint) when nothing + * resolves. Pure modulo filesystem state — `skillsDirOverride` keeps it testable + * without an engine. + */ +export function resolveSkillsDir( + ctx: OperationContext, + skillsDirOverride?: string, +): { + dir: string; + source: ResolvedSkillsDirSource; +} { + if (skillsDirOverride && skillsDirOverride.trim().length > 0) { + if (!existsSync(skillsDirOverride)) { + throw new OperationError( + 'storage_error', + `Configured mcp.skills_dir does not exist: ${skillsDirOverride}`, + 'Fix it with `gbrain config set mcp.skills_dir ` or unset it to autodetect.', + ); + } + return { dir: skillsDirOverride, source: 'config' }; + } + + const det = + ctx.remote === false ? autoDetectSkillsDirReadOnly() : autoDetectSkillsDir(); + if (!det.dir || !det.source) { + const hint = ctx.remote === false ? AUTO_DETECT_HINT_READ_ONLY : AUTO_DETECT_HINT; + throw new OperationError( + 'storage_error', + 'No skills directory found on the server host.', + `Set it explicitly: \`gbrain config set mcp.skills_dir \` (or $GBRAIN_SKILLS_DIR). Search order:\n${hint}`, + ); + } + return { dir: det.dir, source: det.source }; +} + +// --------------------------------------------------------------------------- +// Path confinement (the security boundary) +// --------------------------------------------------------------------------- + +/** Validate the client-supplied skill name BEFORE any filesystem access. */ +function assertSkillNameShape(name: unknown): asserts name is string { + if (typeof name !== 'string' || name.length === 0) { + throw new OperationError('invalid_params', 'skill name must be a non-empty string'); + } + if (name.length > MAX_SKILL_NAME_LEN) { + throw new OperationError('invalid_params', `skill name exceeds ${MAX_SKILL_NAME_LEN} characters`); + } + // No path separators, no traversal, no null byte. The name is a manifest + // LOOKUP KEY — it must never look like a path component. + if (/[/\\]|\.\.|/.test(name)) { + throw new OperationError('invalid_params', `Invalid skill name: ${name}`); + } +} + +/** + * Resolve `name` → the absolute, confined `SKILL.md` path. The name is resolved + * THROUGH the vetted manifest (never used as a raw path segment); the + * manifest-derived path is realpath + relative-contained on every call; the + * resolved target must be a regular file named `SKILL.md`. Throws on any miss. + */ +export function resolveSkillMdPath(skillsDir: string, name: string): string { + assertSkillNameShape(name); + const { skills } = loadOrDeriveManifest(skillsDir); + const entry = skills.find(s => s.name === name); + if (!entry) { + throw new OperationError( + 'page_not_found', + `Skill not found: ${name}`, + 'Call list_skills to see available skills.', + ); + } + return confineManifestPath(skillsDir, entry); +} + +/** + * Realpath + containment + file-type check for a manifest entry's `path`. Used + * by both `get_skill` (one entry) and `list_skills` (every entry — manifest.json + * `path` is taken verbatim, so a poisoned entry must not escape). Throws on + * escape / missing / non-`SKILL.md` / non-regular-file. + */ +export function confineManifestPath(skillsDir: string, entry: ManifestEntry): string { + let realRoot: string; + let realCandidate: string; + try { + realRoot = realpathSync(skillsDir); + } catch { + throw new OperationError('storage_error', `Cannot resolve skills dir: ${skillsDir}`); + } + const candidate = join(realRoot, entry.path); + try { + realCandidate = realpathSync(candidate); + } catch { + throw new OperationError('page_not_found', `Skill file not found: ${entry.name}`); + } + const rel = relative(realRoot, realCandidate); + if (rel.startsWith('..') || resolve(realRoot, rel) !== realCandidate) { + throw new OperationError('invalid_params', `Skill path escapes skills dir: ${entry.name}`); + } + let st; + try { + st = statSync(realCandidate); + } catch { + throw new OperationError('page_not_found', `Skill file not found: ${entry.name}`); + } + if (!st.isFile() || basename(realCandidate) !== 'SKILL.md') { + throw new OperationError( + 'invalid_params', + `Skill target is not a SKILL.md regular file: ${entry.name}`, + ); + } + return realCandidate; +} + +// --------------------------------------------------------------------------- +// Tool honesty (D7) +// --------------------------------------------------------------------------- + +/** Can THIS caller call THIS op on THIS server? Local owns everything. */ +function opCallableByCaller(op: Operation, ctx: OperationContext): boolean { + if (ctx.remote === false) return true; // local CLI — OS is the trust boundary + if (op.localOnly) return false; // not reachable over a remote transport + return hasScope(ctx.auth?.scopes ?? [], op.scope ?? 'read'); +} + +/** + * Split a skill's declared `tools:` into what the caller can vs cannot actually + * use, cross-referenced against this server's op set AND the caller's scope. + * A declared tool not in the op set is unavailable (not on this server). + */ +export function crossReferenceTools( + declared: string[], + ctx: OperationContext, +): { usable_tools: string[]; unavailable_tools: string[] } { + const usable: string[] = []; + const unavailable: string[] = []; + for (const tool of declared) { + const op = operations.find(o => o.name === tool); + if (op && opCallableByCaller(op, ctx)) usable.push(tool); + else unavailable.push(tool); + } + return { usable_tools: usable, unavailable_tools: unavailable }; +} + +/** Every server tool this caller can call — the envelope's "what you can use". */ +function availableBrainTools(ctx: OperationContext): string[] { + return operations.filter(op => opCallableByCaller(op, ctx)).map(op => op.name).sort(); +} + +// --------------------------------------------------------------------------- +// Frontmatter projection + description +// --------------------------------------------------------------------------- + +/** Parse a single-line `description:` from raw frontmatter (best-effort). */ +function parseDescriptionField(raw: string): string | undefined { + const m = raw.match(/^description:\s*["']?(.+?)["']?\s*$/m); + if (!m) return undefined; + const v = m[1].trim(); + return v.length > 0 ? v : undefined; +} + +/** Strip the leading `---\n...\n---` fence; return the prose body. */ +function stripFrontmatterFence(content: string): string { + const m = content.match(/^---\n[\s\S]*?\n---\n?/); + return m ? content.slice(m[0].length) : content; +} + +/** First non-empty, non-heading prose line of the body (description fallback). */ +function firstProseLine(body: string): string { + for (const line of body.split('\n')) { + const t = line.trim(); + if (t.length === 0) continue; + if (t.startsWith('#')) continue; + if (t.startsWith('