feat(doctor): tamper-evident skills manifest + freshness guard (#159)

skills/ ships fat-markdown files that agents execute as instructions, but
nothing detected silent edits to them (#159). This adds lightweight tamper
EVIDENCE (a committed sha256 manifest), not a signature system:

- skills/skills.lock.json — committed manifest mapping every bundled file
  under skills/ to its sha256 (sorted relative paths, deterministic JSON,
  trailing newline; excludes itself from its own hash set).
- src/core/skills-integrity.ts — pure, unit-testable core: compute /
  render / verify(dir, manifest) -> { modified, missing, extra }.
- scripts/generate-skills-manifest.ts — regenerator
  (bun run scripts/generate-skills-manifest.ts).
- gbrain doctor check `skills_manifest_integrity` (SKILL group) — WARNS on
  drift, never fails or blocks; fail-safe ok/skip when no manifest is
  present at the resolved skills dir (user workspaces, compiled-binary
  installs far from the repo).
- scripts/check-skills-manifest-fresh.sh — CI freshness guard modeled on
  check-eval-glossary-fresh.sh; wired as `check:skills-manifest` in
  package.json and into bun run verify (run-verify-parallel.sh).
- test/skills-integrity.test.ts — clean / modified / missing / extra /
  self-exclusion / deterministic-render, over tmp-dir fixtures.

Docs: regeneration workflow in docs/TESTING.md + a KEY_FILES.md entry.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-07-27 16:19:41 -07:00
co-authored by Claude Fable 5
parent 032af6e5f7
commit f489243159
11 changed files with 423 additions and 0 deletions
+9
View File
@@ -46,6 +46,15 @@ If a shard wedges (per-shard `GBRAIN_TEST_SHARD_TIMEOUT` cap, default 600s), the
- `tests/heavy/*.sh` → ops-shape shell scripts. Cost minutes per run; NOT in default `bun test`. Run via `bun run test:heavy` or scheduled nightly via `.github/workflows/heavy-tests.yml`. Examples: pg_upgrade matrix (boot legacy brain → walk to head), RSS budget gate (measure peak worker RSS vs committed baseline), read-latency-under-sync (p50/p95/p99 under concurrent writer load), sync lock regression (N concurrent syncs assert 1 winner + N-1 lock-busy + zero leaked `gbrain_cycle_locks` rows). See `tests/heavy/README.md` for when to add a script here vs `*.slow.test.ts`. Files prefixed with `_` (e.g. `tests/heavy/_build_legacy_fixtures.sh`) are helpers/libs invoked by sibling tests — the runner skips them.
- `test/fuzz/*.test.ts` → property-based fuzz harness. Pure-validator targets in `pure-validators.test.ts` are guarded by `scripts/check-fuzz-purity.sh` (in `bun run verify`), which `bun build --target=bun` bundles each target and greps the resulting bundle for banned transitive imports (`node:fs`, `node:child_process`, engine modules). Anything that fails the guard moves to `mixed-validators.test.ts` (still property-tested, but no purity guarantee) or `filesystem-validators.test.ts` (fs-backed, uses temp dirs). Fuzz tests run in the default `bun test` loop because they're fast (~3s for ~12 properties × 1000 runs each).
### Skills-manifest freshness guard
`skills/skills.lock.json` is a committed sha256 inventory of every bundled file under
`skills/` (tamper evidence, not signatures — see `src/core/skills-integrity.ts`).
Any change under `skills/` must regenerate it: `bun run scripts/generate-skills-manifest.ts`.
`scripts/check-skills-manifest-fresh.sh` (`bun run check:skills-manifest`, wired into
`bun run verify`) regenerates to a tmp file and diffs, failing CI on drift; at runtime
`gbrain doctor` reports the same drift as a warn-only `skills_manifest_integrity` check.
### Test-isolation lint and helpers
The cross-file flake class is enforced statically by `scripts/check-test-isolation.sh`, wired into `bun run verify` and `bun run check:all`. Rules (non-serial unit files only; `*.serial.test.ts` and `test/e2e/*` are skipped):
+1
View File
@@ -146,6 +146,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/core/skill-catalog.ts` — host-repo skill catalog backing the MCP `list_skills` / `get_skill` ops. Lets a thin MCP client (Codex desktop, Claude Code, Claude Cowork, Perplexity) DISCOVER + FOLLOW the agent repo's fat-markdown skills over `gbrain serve` — a skill is prose, so "using" one = fetching its body then calling the gbrain MCP tools the server already exposes. Read-scope, NOT localOnly (defensible only via the full mitigation stack): (1) **publish gate**`assertPublishEnabled(ctx, publishSkills)`; remote callers require `mcp.publish_skills === true`, default-OFF so an upgrade never silently grants existing read tokens host-skill read; local callers (`ctx.remote === false`) always pass. (2) **path confinement**`assertSkillNameShape` rejects separators/`..`/null/space before any FS access; the client `name` is a manifest LOOKUP KEY (via `loadOrDeriveManifest`), never a raw path segment; `confineManifestPath` does realpath + relative-containment + `SKILL.md`-regular-file check on EVERY entry (defeats poisoned manifest.json `path`, symlink/`..` escape). (3) **frontmatter allowlist**`GetSkillResult.frontmatter` projects a safe subset; private `writes_to` + `sources` dropped. (4) **prose-only + 256KB cap** (`MAX_SKILL_MD_BYTES`, env `GBRAIN_MAX_SKILL_MD_BYTES`), size-checked twice (statSync + UTF-8 byte length). (5) **no install_path serve for remote** — remote callers use `autoDetectSkillsDir` (no install-path tier) so a hosted gbrain with no agent repo returns `storage_error`; local callers use `autoDetectSkillsDirReadOnly`. (6) MCP rate-limiter caps call rate. Config reads honor BOTH planes: `readMcpPublishSkills` / `readMcpSkillsDir` prefer the DB plane (`engine.getConfig`) over the file plane (`ctx.config.mcp`). Tool-honesty: `crossReferenceTools(declared, ctx)` splits a skill's declared `tools:` into `usable_tools` vs `unavailable_tools`; `buildSkillCatalog`'s `instructions` envelope (`SKILL_CATALOG_INSTRUCTIONS`) carries the "these are prose, follow-then-call-tools" protocol. Skills are host-filesystem repo-global — `sourceScopeOpts(ctx)` / `ctx.brainId` deliberately do NOT apply. `buildSkillCatalog` is resilient (one malformed/escaping skill is skipped, never throws). Config keys in `src/core/config.ts`: `GBrainConfig.mcp?: { publish_skills?, skills_dir? }` + `KNOWN_CONFIG_KEYS` entries `mcp.publish_skills`/`mcp.publish_skills_prompted`/`mcp.skills_dir` + `mcp.` prefix in `KNOWN_CONFIG_KEY_PREFIXES`. `src/commands/init.ts` writes `config.mcp = { publish_skills: true, ... }` for new installs (existing config wins on re-init). `src/commands/upgrade.ts:runPostUpgrade` adds a one-time consent prompt (gated by `mcp.publish_skills_prompted`; existing installs stay OFF until owner opts in). Two ops register in `src/core/operations.ts` (`list_skills` with optional `section` filter + `cliHints:{name:'skills'}`; `get_skill` taking `name` + `cliHints:{name:'skill', positional:['name']}`) and dynamically import this module to avoid the import cycle (skill-catalog statically imports the `operations` array). Descriptions in `src/core/operations-descriptions.ts` (`LIST_SKILLS_DESCRIPTION`, `GET_SKILL_DESCRIPTION`, `SKILL_CATALOG_INSTRUCTIONS`, `SKILL_CLIENT_GUIDANCE`), pinned by `test/operations-descriptions.test.ts`. CLI: `gbrain skills` / `gbrain skill <name>`. Pinned by `test/skill-catalog.test.ts`, `test/skill-catalog-security.test.ts` (path-confinement / poisoned-manifest / symlink-escape), `test/skill-catalog-transports.test.ts` (publish-gate + remote-vs-local) over `test/fixtures/skill-catalog/`.
- `src/core/check-resolvable.ts` — Resolver validation: reachability, MECE overlap, DRY checks, structured fix objects. `CROSS_CUTTING_PATTERNS.conventions` is an array (notability gate accepts `conventions/quality.md` and `_brain-filing-rules.md`). `extractDelegationTargets()` parses `> **Convention:**`, `> **Filing rule:**`, and inline backtick references. DRY suppression is proximity-based via `DRY_PROXIMITY_LINES = 40`. `parseResolverEntries` accepts BOTH the markdown table AND a compact list format (`- **skill-name**: trigger1 | trigger2 | trigger3` or `- skill-name: trigger1 | trigger2`); shapes can mix in one file, folded by the multi-resolver merge. Skill name MUST be kebab-lowercase (regex `[a-z][a-z0-9-]+`) so prose bullets like `- **Note**:`/`- **Convention**:`/`- **TODO**:` don't false-match as skill rows. `skillPath` is ALWAYS derived as `skills/<name>/SKILL.md`: an optional `→ \`skills/path\`` (or ASCII `->`) suffix is stripped from the trigger but NOT honored as the path — two consumers (`routing-eval.ts:skillSlugFromPath`, the manifest lookup) assume the convention; use the table format for non-conventional paths. Multi-trigger rows fan out to one entry per trigger sharing the same `skillPath`; `checkResolvable` dedupes so the reachability count counts each skill once. Pinned by `test/check-resolvable.test.ts` (11 cases: bold+plain forms, Unicode+ASCII suffix strip, ellipsis filter, empty pipe segments, mixed shapes, prose-bullet rejection) + `test/check-resolvable-openclaw-compact.test.ts` (8 cases over `test/fixtures/openclaw-compact-resolver/` and `test/fixtures/openclaw-mixed-merge/`). Tutorial: `docs/guides/scaling-skills.md` (three-tier scaling: ~300-skill agent to ~4K tokens/turn from ~25K).
- `src/core/repo-root.ts` — Shared `findRepoRoot(startDir?)`: walks up from `startDir` (default `process.cwd()`) looking for `skills/RESOLVER.md`. Zero-dependency, imported by `doctor.ts` and `check-resolvable.ts`; parameterized `startDir` makes tests hermetic. Read-path / write-path split: `autoDetectSkillsDir` (shared, read+write-safe) has tier-0 `$GBRAIN_SKILLS_DIR` operator override ahead of the 4-tier chain. `autoDetectSkillsDirReadOnly` wraps it with a tier-5 install-path fallback that walks up from `fileURLToPath(import.meta.url)` and gates on `isGbrainRepoRoot` so unrelated repos can't false-positive. Read-path callers (`doctor`, `check-resolvable`, `routing-eval`) use the read-only variant; write-path callers (`skillpack install`, `skillify scaffold`, `post-install-advisory`) stay on the shared function so install-from-`~` can't retarget the bundled gbrain `skills/` instead of the user's workspace. `SkillsDirSource` variants `'env_explicit'`, `'install_path'`; `AUTO_DETECT_HINT_READ_ONLY` documents the extra tier. The `--fix` safety gate in `doctor.ts` + `check-resolvable.ts` refuses auto-repair when `detected.source === 'install_path'`.
- `src/core/skills-integrity.ts` — Tamper-evidence manifest for the bundled `skills/` tree (#159); NOT a signature system. Pure functions over `node:crypto` sha256: `computeSkillsManifest(dir)` (recursive, sorted '/'-relative paths, excludes the manifest itself, skips symlinks), `renderSkillsManifest(dir)` (2-space JSON + trailing newline, deterministic), `verifySkillsManifest(dir, manifest)``{modified, missing, extra}`. Committed manifest lives at `skills/skills.lock.json` (`SKILLS_MANIFEST_FILENAME`); regenerate via `bun run scripts/generate-skills-manifest.ts`. Consumers: the warn-only `skills_manifest_integrity` doctor check in `src/commands/doctor.ts` (ok/skip when no manifest is present — user workspaces and compiled-binary installs are not drift) and the CI freshness guard `scripts/check-skills-manifest-fresh.sh` (`bun run check:skills-manifest`, in `bun run verify`). Pinned by `test/skills-integrity.test.ts`.
- `src/commands/check-resolvable.ts` — Standalone CLI wrapper over `checkResolvable()`. Exports `parseFlags`, `resolveSkillsDir`, `DEFERRED`, `runCheckResolvable`. Exit rule: **1 on any issue (warnings OR errors)**, stricter than doctor's `ok` flag. Stable JSON envelope `{ok, skillsDir, report, autoFix, deferred, error, message}` — same shape on success and error. `--fix` runs `autoFixDryViolations` BEFORE `checkResolvable` (same ordering as doctor). `scripts/skillify-check.ts` subprocess-calls `gbrain check-resolvable --json` (cached per process) and fails loud on binary-missing. AGENTS.md workspaces resolve natively (see `src/core/resolver-filenames.ts`). `DEFERRED[]` is empty. Resolver lookup is the multi-file merge in `src/core/check-resolvable.ts` — entries collected from every `RESOLVER.md`/`AGENTS.md` across the skills dir AND its parent, deduped by `skillPath` (first occurrence wins). Uses `autoDetectSkillsDirReadOnly` so `cd ~ && gbrain check-resolvable` finds bundled skills via the install-path fallback; `--fix` carries the same install-path safety gate (refuses to write when `detected.source === 'install_path'`).
- `src/core/resolver-filenames.ts` — central list of accepted routing filenames (`RESOLVER.md`, `AGENTS.md`). Shared by `findRepoRoot`, `check-resolvable`, and skillpack install so every code path walks the same fallback chain.
- `src/commands/skillify.ts` + `src/core/skillify/{generator,templates}.ts``gbrain skillify scaffold <name>` creates all stubs for a new skill: SKILL.md, script, tests, routing-eval.jsonl, resolver entry, filing-rules pointer. `gbrain skillify check <script>` runs the 10-step checklist (LLM evals, routing evals, check-resolvable gate, filing audit) against a candidate skill before it lands.