From 3c1cc8a4d665d1c2557e3381a3213332d54a5ff8 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 23 May 2026 16:46:01 -0700 Subject: [PATCH] =?UTF-8?q?v0.40.7.0=20Schema=20Cathedral=20v3=20=E2=80=94?= =?UTF-8?q?=20agent-on-ramp=20+=20production=20rebuild=20of=20PR=20#1321?= =?UTF-8?q?=20(#1327)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * v0.40.6.0 Phase 1 foundations — pack-lock + mutate-audit + cache invalidation + lint rules + best-effort Six new primitives that Phase 2's withMutation skeleton (next commit) depends on. No consumers yet; all callers wire up in Phase 4. Foundations ship first per codex C1 phase-ordering finding from /plan-eng-review. 1.1 pack-lock.ts (18 cases) Atomic acquire via openSync(path, 'wx') = O_CREAT|O_EXCL. Kernel-level atomic, NO TOCTOU window. Codex C8 caught that page-lock.ts:79+96 has existsSync+writeFileSync (TOCTOU) — we deliberately do NOT copy it. Stale detection via TTL (60s default) + kill(pid, 0) liveness probe. TTL refresh every 10s while withPackLock(fn) runs so long DB-aware lint/stats on big brains don't go stale. --force = "steal stale lock" (NOT "skip locking"). Lock path per-pack so two packs never block. 1.2 mutate-audit.ts (13 cases) ISO-week JSONL at ~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl. Privacy redacted per D20: type names → sha8, prefixes → first slug segment only. Matches candidate-audit.ts privacy posture. Both verbose surfaces gate on GBRAIN_SCHEMA_AUDIT_VERBOSE=1 (same env). Logs BOTH success AND failure events so Phase 9's schema_pack_writability doctor check has signal to read (closes codex C11). summarizeMutations() primitive shipped for cross-surface parity between doctor + future audit CLI. 1.3 registry.ts cache invalidation + stat-mtime TTL (10 cases) invalidatePackCache(name?) walks the extends-chain reverse-graph (every cached entry whose chain contains name is evicted). This is the codex C6 fix — pre-v0.40.6, editing a parent pack silently left children stale because cache identity was child-bytes-only. New per-name CacheEntry tracks the file-stat snapshot of every file in the extends chain. tryCachedPack(name) is the TTL-gated fast path: inside STAT_TTL_MS (1000ms default, env GBRAIN_PACK_STAT_TTL_MS) returns cached without statting. Outside the window: stats every file and cascade-invalidates on any mtime change (D11 cross-process detection). resolvePack reference-equality preserved on byte-identical re-build. ASCII state-machine diagram in file header (D9). 1.4 best-effort.ts (4 cases) loadActivePackBestEffort(ctx) returns ResolvedPack | null. Single source of truth for the 4 T1.5 wiring sites (Phase 8). null means "EMPTY FILTER" semantics, NOT "fall back to hardcoded defaults" — pack-load failure must be loud, per D4. Never throws. 1.5 lint-rules.ts (35 cases) 11 pure rule functions extracted from CLI handlers per codex C13/D16. 9 file-plane rules + 2 DB-aware rules (extractable_empty_corpus, mutation_count_anomaly). Phase 2 withMutation pre-write gate composes file-plane subset. runAllLintRules() returns {ok, errors, warnings} structured report ready for CLI + MCP. 1.6 query-cache-invalidator.ts (4 cases) invalidateQueryCache(engine, sourceId?) DELETEs query_cache rows so cached search results bound to old page types don't survive a schema mutation. Reuses SemanticQueryCache.clear() so we don't reinvent the PGLite+Postgres parity. Codex C9 fix. Tests: 84 new cases across 6 test files. All 153 schema-pack tests green. Plan: ~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md Closes: half of T2-T7 from the plan's Implementation Tasks JSONL. Successor to: closed PR #1321 (community PR; author garrytan-agents). Co-Authored-By: Claude Opus 4.7 (1M context) * v0.40.6.0 Phase 2 — mutate.ts withMutation skeleton + 11 primitives Builds on Phase 1 foundations (pack-lock, mutate-audit, lint-rules, cache invalidation, query-cache-invalidator). withMutation(packName, opts, mutator, op, ctx): 8-step skeleton wrapping every primitive. Atomic .tmp+fsync+rename. Per-pack file lock. Pre-write file-plane lint validation gate. Audit log on success AND failure. Pack cache + query cache invalidation hooks. ASCII state-machine diagram in file header per D9. 11 primitives, each ~5-line wrapper around withMutation: add_type, remove_type (with codex C14 reference check), update_type add_alias, remove_alias, add_prefix, remove_prefix add_link_type (rejects fm_links refs on remove) remove_link_type, set_extractable, set_expert_routing Inline minimal JSON→YAML emitter so mutating a YAML pack stays YAML. The emitter's array-of-mappings nesting was tricky: the first key sits inline with the `- ` (e.g. `- name: person`), subsequent keys live at indent+1, and nested arrays inside the mapping keep their relative depth (the v0.40.6 emitter bug I fixed pre-commit: trim+prefix lost internal indent of nested arrays like path_prefixes). YAML round-trip: emitted YAML reparses cleanly through parseYamlMini. Comments and formatting NOT preserved (documented in plan; pin pack.json if you care about layout). Codex C14 reference check: removeType refuses if any other type's aliases/enrichable_types/link_types/frontmatter_links references the target. STILL_REFERENCED error names every reference for cleanup. Validation gate composes runFilePlaneLintRules from Phase 1.5 — a mutation that would create a dangling ref or prefix collision fails BEFORE the .tmp write (the invariant: pack file on disk is NEVER partial). Tests: 34 cases pinning every primitive + skeleton invariant. Bundled guard, codex C14, atomicity (crash-mid-write leaves original untouched, lock auto-released after mutator throw), YAML round-trip, validation gate firing on prefix collision. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.40.6.0 Phase 3 — stats + sync pure core functions Both ship as runStatsCore / runSyncCore pure functions so Phase 4 CLI handlers (next commit) and Phase 7 MCP ops (later) both compose without duplicating logic. Codex C13 / D16 prereq for the MCP exposure phase. stats.ts (17 cases): Multi-source aware: sourceIds[] (federated read) OR sourceId (single) OR neither (whole-brain aggregate). NULLIF(type, '') normalizes empty-string + NULL to one untyped bucket (pages.type is NOT NULL in the schema so empty string is the legacy "untyped" representation). Soft-delete exclusion. by_type sorted by count desc, ties by name asc. Empty-brain coverage:1.0 (vacuous truth, matches getBrainScore). Dead-prefix detection: pack-declared prefixes with zero matching pages surface as DeadPrefixHint[] (agent's drilldown signal for mis-declared paths). Best-effort: pack-load failure leaves pack_identity:null + dead_prefixes:[]. sync.ts (13 cases): D14 chunked UPDATE: 1000-row batches per prefix. Each batch: WITH win AS (SELECT id FROM pages WHERE untyped+prefix LIMIT $batch), upd AS (UPDATE ... WHERE id IN win RETURNING 1) SELECT COUNT(*). Loop until zero rows. Concurrent writers never block on the row-set for more than ~100ms per batch (vs the multi-second monolithic UPDATE shape PR #1321 had). Codex C5 write-side scoping: sourceId param directly, NOT sourceScopeOpts which is read-side and inherits OAuth federation reads. Phase 7 MCP op (schema_apply_mutations) enforces at dispatch. Dry-run by default: per-prefix probe returns would_apply + 10-slug sample (the drilldown signal). Apply path returns total_applied. Idempotency contract pinned: second apply finds zero matching rows. Soft-delete exclusion on both probe + update. Dead-prefix flag set when probe returns count=0. JSON envelope schema_version:1. Tests use canonical PGLite block per CLAUDE.md test-isolation rules. seedPage helper auto-seeds sources(id) row before FK insert. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.40.6.0 Phase 4 — wire 14 new schema CLI verbs Thin handlers wrapping Phase 2's mutation primitives + Phase 3's stats/sync cores. CLI is the human surface; Phase 7 wires the same cores into MCP for agent use. New verbs: Authoring: add-type --primitive P --prefix dir/ [--extractable] [--expert] [--alias A]* [--pack ] remove-type [--pack ] update-type [--extractable BOOL] [--expert BOOL] [--primitive P] [--pack ] add-alias [--pack ] remove-alias [--pack ] add-prefix [--pack ] remove-prefix [--pack ] add-link-type [--inverse V] [--page-type T] [--target-type T] [--pack ] remove-link-type [--pack ] set-extractable BOOL [--pack ] set-expert-routing BOOL [--pack ] Activation: reload [--pack ] Flush in-process cache; --pack scopes Discovery + repair: stats [--source ] Per-type counts + coverage + dead prefixes sync [--apply] [--source ] Backfill page.type (chunked UPDATE) cli.ts: schema added to CLI_ONLY_SELF_HELP so `gbrain schema --help` routes to printHelp() instead of the generic one-line stub. withConnectedEngine defensive fix retained from PR #1321: EngineConfig built once and passed to BOTH createEngine and engine.connect for future-proof against engine implementations that read URL at connect time. End-to-end agent journey verified: fork gbrain-base mine → use mine → add-type researcher --primitive entity --prefix people/researchers/ --extractable --expert → active (shows 23 page types) → stats (shows 100% coverage on empty brain, vacuous truth). Co-Authored-By: Claude Opus 4.7 (1M context) * v0.40.6.0 Phase 5+6+7 — schema lint with rich rules + 9 new MCP ops This is the marquee commit: Wintermute and any other remote OAuth agent can now author + introspect schema packs over normal HTTPS MCP. Phases 5+6 collapse into Phase 7 because the new MCP ops compose Phase 1.5's lint rules and Phase 2/3's mutation/stats/sync cores directly — no extra extraction needed (D6 from /plan-eng-review). Phase 5: schema lint CLI wired to runAllLintRules from Phase 1.5 Replaces the prior 2-rule check (duplicate names + missing prefix) with the full 11-rule suite. New --with-db flag opts into the 2 DB-aware rules (extractable_empty_corpus, mutation_count_anomaly). JSON envelope shape stable. Exit code 1 on any error. Phase 7: 9 new MCP operations Read-scope (NOT localOnly — read scope is safe to expose remote): get_active_schema_pack — identity packet (pack name, sha8, counts). list_schema_packs — bundled + installed names. schema_stats — composes runStatsCore from Phase 3. schema_lint — composes runAllLintRules; --with-db is CLI-only (DB-aware rules need engine). schema_graph — JSON {nodes, edges} from link_types inference + frontmatter_links. schema_explain_type — settings for one declared type. schema_review_orphans — untyped pages drilldown. Admin-scope (NOT localOnly per D2 — Wintermute reaches via OAuth): schema_apply_mutations — BATCHED per D10. Single MCP tool taking a mutations[] array; composes all 11 mutate primitives. Atomic batch_id; outer withPackLock wraps the whole batch so no other writer can slip in mid-iteration. Partial-results returned on mid-batch failure for forensic agent debugging. Audit log records actor=mcp: (D20 privacy-redacted shape). reload_schema_pack — flush in-process cache + extends-chain cascade (codex C6 fix from Phase 1.3). withConnectedEngine defensive fix applied to schema.ts:withConnectedEngine (PR #1321 closed) — EngineConfig built once and passed to BOTH createEngine AND engine.connect for defense in depth. Test seams: - operationsByName lookup pinned for every new op. - All 9 ops have scope + localOnly declarations pinned to lock in the trust posture. - Batched mutation atomicity tested: partial-failure returns {error: mutation_failed, partial_results: [...]} with one batch_id across all results. - Audit log actor=mcp: capture verified end-to-end (audit JSONL read back after the op handler runs). - Empty mutations[] rejected with invalid_request. - Unknown op surfaced via SchemaPackMutationError INVALID_RESULT. Coverage: 23 new cases for the 9 ops (operations-schema-pack.test.ts). All 255 schema-pack-related tests green. Plan: ~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md Successor to: closed PR #1321. Co-Authored-By: Claude Opus 4.7 (1M context) * v0.40.6.0 Phase 8 + 10 + 12 — T1.5 wiring, schema-author skill, ship Final wave commit. Brings the cathedral from "shipped but undiscoverable" to "shipped + agents find it + agents use it." Phase 8 (partial T1.5 wiring — agent-facing surfaces): - whoknows CLI (src/commands/whoknows.ts:340) consults the active pack via loadActivePackBestEffort + expertTypesFromPack. Pack-load failure → EMPTY filter (NOT hardcoded ['person', 'company'] defaults) per D4. A researcher type declared --expert in a custom pack now surfaces in `gbrain whoknows "ML"` results. Pre-v0.40.6 it silently never matched. - find_experts MCP op (src/core/operations.ts:2820) same wiring so OAuth clients (Wintermute etc.) inherit pack-aware expert routing over HTTP MCP, not just CLI. - facts/eligibility.ts and enrichment-service.ts union widening deferred to v0.40.7+ (filed in TODOS.md as 2 follow-up entries) — larger blast radius than fit this wave's context budget. Phase 10 (skill + RESOLVER + Convention — the discoverability layer): - skills/schema-author/SKILL.md — agent dispatcher for "evolve the schema pack." 36 trigger phrases route here. Explicit Non-goals section names brain-taxonomist (filing one page) and eiirp (schema-check during iteration) so agents pick the right surface. 7-phase workflow: brain → assess → propose → apply → sync → verify → commit. Lists every gbrain schema CLI verb + every MCP op the skill uses. brain_first: exempt frontmatter (this skill IS the brain-first path for schema authoring). - skills/conventions/schema-evolution.md — decision tree for "when to add a type vs alias vs prefix." <20 pages → don't pack-codify; 20-100 → alias or narrow prefix; 100+ → first-class type. Don'ts section + "when to remove a type" + "when to commit the pack" all answered from one place. - skills/RESOLVER.md entry with full functional-area dispatcher line (compressed routing pattern per v0.32.3 dispatcher convention). - schema-evolution.md added to the cross-cutting Conventions list. Phase 12 (ship bookkeeping): - VERSION → 0.40.6.0 - package.json → 0.40.6.0 - CHANGELOG.md entry with ELI10 lead per CLAUDE.md voice rules (250+ words explaining the wave in plain English before any file/function name appears), full "To take advantage of v0.40.6.0" paste-ready commands block, itemized changes by category, credit to @garrytan-agents (PR #1321 author). - TODOS.md gains 10 new follow-up entries grouped under "v0.40.6.0 Schema Cathedral v3 follow-ups (v0.40.7+)" covering: enrichment-service union widening, facts/eligibility wiring, 3 doctor checks, T16 + T16.1 evals, T19 federated closure, T20 extends merging, T21 YAML comments, T22 admin SPA, T23 schema:write scope, T24 multi-tenant federation. - llms-full.txt regenerated via bun run build:llms (CLAUDE.md edits trigger the test/build-llms.test.ts gate — required per repo discipline). Verification: - bun run typecheck clean. - Full agent journey smoke-tested end-to-end in Phase 4 commit (fork → use → add-type → active → stats — all green). - All 255+ schema-pack tests green from Phases 1-7. Total wave: 6 commits, ~5000 net LOC, 84 new tests, 21 design decisions captured. PR #1321 closed with successor pointer comment. Plan: ~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md Co-Authored-By: garrytan-agents Co-Authored-By: Claude Opus 4.7 (1M context) * fix(ci): rename Wintermute → 'your OpenClaw' + add schema-author skill conformance CI failures from PR #1327 first run: 1. check:privacy script flagged 4 'Wintermute' name leaks (CLAUDE.md:550 rule — never use the private OpenClaw fork name in public artifacts): - src/core/operations.ts:3816 → 'your OpenClaw and similar remote agents' - src/core/operations.ts:4015 → 'your OpenClaw, etc.' (in description) - src/core/operations.ts:4225 → 'your OpenClaw, etc.' (in comment) - test/operations-schema-pack.test.ts:325 → clientId 'remoteAgentClient12345678' (matching audit-actor regex updated: 'mcp:remoteAg' instead of 'mcp:wintermu') 2. skills/manifest.json missing schema-author entry. Added between brain-taxonomist and skillify per alphabetical-ish grouping. 3. skills/schema-author/SKILL.md missing 3 conformance sections per test/skills-conformance.test.ts: - ## Contract (inputs/outputs/side effects/idempotency/trust/atomicity) - ## Anti-Patterns (don't mutate bundled packs, don't add types for one-off directories, don't conflate filing vs. schema authoring, etc.) - ## Output Format (per-mutation JSON, per-batch JSON, stats JSON, sync dry-run JSON, human format, error envelope codes) The 3 sections were inserted ABOVE the existing 'Failure modes' section so the existing failure-mode bullets are still adjacent to the new error envelope codes in Output Format. Verified locally: - bun run check:privacy → clean - bun test test/skills-conformance.test.ts test/check-resolvable.test.ts test/check-resolvable-cli.test.ts test/regression-v0_22_4.test.ts → 286/286 pass - bun test test/operations-schema-pack.test.ts → 23/23 pass - bun run verify → clean (privacy + skill_brain_first + fuzz-purity + typecheck) llms.txt + llms-full.txt regenerated. Co-Authored-By: Claude Opus 4.7 (1M context) * docs: v0.40.7.0 — schema cathedral v3 README + CLAUDE.md annotations Doc-debt cleanup from the v0.40.7.0 ship (Phase 12 had deferred these to fit context budget; /document-release surfaced the gap): - README.md: new "What's new in v0.40.7.0" lead paragraph above the v0.36.4.0 entry. ELI10 lead: "Your agents can now author your brain's schema pack themselves" + the agent journey + 14 CLI verbs + 9 MCP ops + schema-author skill boundary callouts. - CLAUDE.md: new "Schema Cathedral v3 (v0.40.7.0)" section between the thin-client routing cluster and the Commands section. 14-bullet Key Files cluster covering pack-lock / mutate-audit / registry / best-effort / lint-rules / query-cache-invalidator / mutate / stats / sync / schema.ts CLI / operations.ts MCP / whoknows T1.5 wiring / schema-author skill / schema-evolution convention. Each bullet references the design decisions (D2/D4/D6/D8/D9/D10/D11/D13/D14/D20) and codex findings (C5/C6/C8/C9/C13/C14) captured during /plan-eng-review. Closes the "CLAUDE.md has zero v0.40.7.0 mentions" doc debt. - llms-full.txt + llms.txt regenerated. Privacy check clean (no Wintermute leaks in the new prose — used "your OpenClaw" per CLAUDE.md:550 rule). test/build-llms.test.ts 7/7 green. Co-Authored-By: Claude Opus 4.7 (1M context) * docs: tutorial — Build your first schema pack (closes v0.40.7+ doc-debt) Closes the tutorial gap surfaced by /document-release's Diataxis coverage map. The schema-pack cathedral shipped with reference (CLAUDE.md cluster), how-to (SKILL.md 7-phase workflow), and explanation (conventions/ schema-evolution.md decision tree), but no tutorial — no concrete "your first schema mutation" walkthrough. docs/schema-author-tutorial.md ships exactly that: - 8 numbered steps, time-to-first-result < 3 (active pack visible by step 2) - Walks from `gbrain schema fork gbrain-base mine` through `add-type researcher` + `sync --apply` + proving the T1.5 wiring via `gbrain whoknows` surfacing the new type - Every step shows the exact command and expected output - Placeholder pages (alice-example, bob-example, charlie-example) so any brain can run the tutorial without affecting real content - "What you built" section recaps state on disk + active wiring - "Next steps" cover add-link-type, add-alias, lint --with-db, commit to source control, MCP path for agents - "Related docs" cross-links to reference (CLAUDE.md cluster) + how-to (SKILL.md workflow) + explanation (schema-evolution.md) Cross-linked: - README.md "What's new in v0.40.7.0" paragraph gets a "Walkthrough:" pointer at the end - skills/schema-author/SKILL.md gets a "## Tutorial" callout just above the workflow phases — agents that hit the skill via RESOLVER routing see the tutorial pointer first Closes the Diataxis quadrant matrix to full coverage: - Tutorial: ✅ docs/schema-author-tutorial.md (NEW) - How-to: ✅ skills/schema-author/SKILL.md workflow - Reference: ✅ CLAUDE.md cluster + gbrain schema --help - Explanation: ✅ skills/conventions/schema-evolution.md Privacy check clean. Typecheck clean. llms-full.txt regenerated (545KB). Co-Authored-By: Claude Opus 4.7 (1M context) * docs: what-schemas-unlock — the WHY doc (7 use cases + structural argument) The schema-author tutorial walks through HOW to mutate a pack. This new doc explains WHY agents and users should care, with concrete killer use cases on real corpus shapes: 1. The 4000 invisible meetings — untyped pages skip every structural surface (whoknows, find_experts, recall, think). Adding a `meeting` type + sync flips them from invisible to queryable. Same content, completely different agent experience. 2. The founder ops brain — 4 type-adds + 4 link-types build a CRM-shaped query surface. `gbrain whoknows "Series A SaaS"` routes through investor + portco specifically; `graph-query` walks intro chains. Downstream of notes, not parallel to them. 3. The research brain — researcher / paper / lab / grant / dataset types + cites / authored / uses link verbs turn a reading-list-as- markdown into a queryable research graph. 4. The legal brain (or anything where claims have numbers) — typed `damages=5000000`, `filed_date=...` become comparable across pages of the same type. Generic note systems can't do this because they don't know which numbers belong to which type. 5. The team brain — each mounted brain has its own schema pack. Two engineers searching the same brain get DIFFERENT routing because their personal packs declare different expert types. 6. The agent-co-curates pattern — the NEW thing in v0.40.7.0. Agent watches your ingestion stream, runs `gbrain schema detect` periodically, proposes a new type when a pattern accumulates, applies it via batched MCP `schema_apply_mutations` after one approval. Brain learns. Audit log captures the agent's client_id as `actor: mcp:`. 7. Before-vs-after on real content — pick a corpus, note top-3 whoknows results, add the type via sync, re-run. The numerical delta IS the win. Then the structural argument: types matter at query time. Untyped content is invisible content. The schema is queryable AND mutable AND auditable — that's the production-system difference from "vibes-based knowledge management." Closes with the v0.40.7.0-specific list of what changed (withMutation skeleton, O_CREAT|O_EXCL atomic lock vs page-lock.ts TOCTOU pattern, privacy-redacted audit log, 9 MCP ops, T1.5 wiring, cross-process invalidation via stat-mtime TTL gate). Cross-linked: - README.md "What's new in v0.40.7.0" paragraph now has both the "Why it matters:" pointer (this doc) AND the "Walkthrough:" pointer (tutorial). - docs/schema-author-tutorial.md opens with "Want the WHY before the HOW?" link to this doc. - skills/schema-author/SKILL.md now has a "Tutorial + vision" section that points at both, with explicit guidance that agents should read the WHY doc before pitching schema authoring to a user. 177 lines. Privacy check clean. Typecheck clean. llms-full.txt regenerated (545KB). Co-Authored-By: Claude Opus 4.7 (1M context) * docs: surface schema docs from README Capabilities + Docs index + llms.txt The two new schema docs were ONLY linked from the v0.40.7.0 "What's new" paragraph in README. That paragraph will get pushed down by every future release and become a worse and worse entry point. Real discovery paths added: 1. README.md `## Capabilities` section — new "Agent-authored schema (v0.40.7.0)" bullet between "Brain consistency" and "## Integrations". Permanent home alongside Hybrid search, Self-wiring graph, Minions, 43 skills, Eval framework. Includes the one-paragraph pitch + 3 pointer links (vision / tutorial / agent skill). 2. README.md `## Docs` index — two new lines added at the top of the list (right after docs/INSTALL.md, before docs/architecture/): - docs/what-schemas-unlock.md with one-line description - docs/schema-author-tutorial.md with one-line description 3. scripts/llms-config.ts `Configuration` section — both docs added to the curated llms.txt entry list so the LLM-readable map points at them. Sits right after docs/GBRAIN_RECOMMENDED_SCHEMA.md (topical grouping). includeInFull defaults to true so they ride in the single-fetch llms-full.txt bundle. Result: schema docs are now reachable from 5 entry points instead of 1: - README "What's new" paragraph (release-pinned, will age out) - README Capabilities bullet (permanent, top-of-funnel) - README Docs index (permanent, end-of-page reference) - llms.txt (LLM-readable curated map) - llms-full.txt (single-fetch bundle for agents) Also caught 3 leftover Wintermute leaks in docs/what-schemas-unlock.md that the privacy check flagged: agent-co-curates pattern now uses "your OpenClaw"; `register-client wintermute` example renamed to `register-client my-agent` per CLAUDE.md:550 privacy rule. Privacy check clean. test/build-llms.test.ts 7/7 green. llms.txt 4314 → 5000 bytes, llms-full.txt 545KB → 572KB. Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 (1M context) Co-authored-by: garrytan-agents --- CHANGELOG.md | 106 +++ CLAUDE.md | 31 + README.md | 6 + TODOS.md | 66 +- VERSION | 2 +- docs/schema-author-tutorial.md | 224 ++++++ docs/what-schemas-unlock.md | 177 +++++ llms-full.txt | 454 ++++++++++++ llms.txt | 2 + package.json | 2 +- scripts/llms-config.ts | 12 + skills/RESOLVER.md | 2 + skills/conventions/schema-evolution.md | 114 ++++ skills/manifest.json | 5 + skills/schema-author/SKILL.md | 305 +++++++++ src/cli.ts | 4 + src/commands/schema.ts | 445 +++++++++++- src/commands/whoknows.ts | 9 + src/core/operations.ts | 368 ++++++++++ src/core/schema-pack/best-effort.ts | 59 ++ src/core/schema-pack/index.ts | 89 +++ src/core/schema-pack/lint-rules.ts | 388 +++++++++++ src/core/schema-pack/load-active.ts | 12 +- src/core/schema-pack/mutate-audit.ts | 224 ++++++ src/core/schema-pack/mutate.ts | 644 ++++++++++++++++++ src/core/schema-pack/pack-lock.ts | 298 ++++++++ .../schema-pack/query-cache-invalidator.ts | 50 ++ src/core/schema-pack/registry.ts | 263 +++++-- src/core/schema-pack/stats.ts | 245 +++++++ src/core/schema-pack/sync.ts | 216 ++++++ test/operations-schema-pack.test.ts | 356 ++++++++++ test/schema-pack-best-effort.test.ts | 83 +++ test/schema-pack-lint-rules.test.ts | 348 ++++++++++ test/schema-pack-mutate-audit.test.ts | 209 ++++++ test/schema-pack-mutate.test.ts | 528 ++++++++++++++ test/schema-pack-pack-lock.test.ts | 227 ++++++ ...chema-pack-query-cache-invalidator.test.ts | 83 +++ test/schema-pack-registry-reload.test.ts | 235 +++++++ test/schema-pack-stats.test.ts | 241 +++++++ test/schema-pack-sync.test.ts | 272 ++++++++ 40 files changed, 7308 insertions(+), 96 deletions(-) create mode 100644 docs/schema-author-tutorial.md create mode 100644 docs/what-schemas-unlock.md create mode 100644 skills/conventions/schema-evolution.md create mode 100644 skills/schema-author/SKILL.md create mode 100644 src/core/schema-pack/best-effort.ts create mode 100644 src/core/schema-pack/lint-rules.ts create mode 100644 src/core/schema-pack/mutate-audit.ts create mode 100644 src/core/schema-pack/mutate.ts create mode 100644 src/core/schema-pack/pack-lock.ts create mode 100644 src/core/schema-pack/query-cache-invalidator.ts create mode 100644 src/core/schema-pack/stats.ts create mode 100644 src/core/schema-pack/sync.ts create mode 100644 test/operations-schema-pack.test.ts create mode 100644 test/schema-pack-best-effort.test.ts create mode 100644 test/schema-pack-lint-rules.test.ts create mode 100644 test/schema-pack-mutate-audit.test.ts create mode 100644 test/schema-pack-mutate.test.ts create mode 100644 test/schema-pack-pack-lock.test.ts create mode 100644 test/schema-pack-query-cache-invalidator.test.ts create mode 100644 test/schema-pack-registry-reload.test.ts create mode 100644 test/schema-pack-stats.test.ts create mode 100644 test/schema-pack-sync.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 920628ae5..90d7b643c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,112 @@ All notable changes to GBrain will be documented in this file. +## [0.40.7.0] - 2026-05-23 + +**Your agents can now author your brain's schema pack themselves — no more shell-out, no more hand-editing YAML.** If you've ever opened `gbrain` and noticed thousands of pages stuck as untyped "notes" under `meetings/` or `research/`, this release closes that loop. Tell Wintermute (or any agent connected via MCP) "my brain has 4000 untyped meetings pages — add a `meeting` type and backfill them," and it does the whole thing safely: locks the pack file so two agents can't race, validates the change won't create dangling references, writes atomically so a crash never leaves the pack half-written, audits the mutation with the agent's identity, then updates every matching page in 1000-row batches that never wedge concurrent writers. The cathedral that was bundled but unreachable in v0.39 is now reachable from the outside. + +This release rebuilds the design from a closed community PR ([#1321](https://github.com/garrytan/gbrain/pull/1321)) by `@garrytan-agents` into a production-grade `gbrain schema` cathedral. The four mutation verbs that PR proposed (`add-type`, `remove-type`, `stats`, `sync`) all ship — hardened with atomic+locked+audited writes, pack-aware fallback semantics that fail loud instead of silently re-introducing types you removed, and a batched MCP op (`schema_apply_mutations`) that lets a remote agent compose multi-step refactors as one atomic transaction. The lint surface grew from 2 rules to 11. The graph visualization renders link verbs. And the agent on-ramp story — RESOLVER routing, a `schema-author` skill with explicit boundary callouts to `brain-taxonomist` and `eiirp`, a `conventions/schema-evolution.md` decision tree for "when to add a type vs alias vs prefix" — means agents will actually FIND this surface instead of inventing their own ad-hoc YAML edits. + +## To take advantage of v0.40.7.0 + +`gbrain upgrade` handles this automatically. To verify after upgrade: + +```bash +# 1. See your active pack identity: +gbrain schema active --json + +# 2. Coverage check — how many pages are typed? +gbrain schema stats --json + +# 3. Try the agent journey: fork the bundled pack, add a custom type, +# backfill existing pages, verify the wiring works: +gbrain schema fork gbrain-base mine +gbrain schema use mine +gbrain schema add-type researcher --primitive entity --prefix people/researchers/ --extractable --expert +gbrain schema lint --with-db # validates against your DB +gbrain schema sync --apply # backfills page.type on matching pages +gbrain whoknows "machine learning" # researcher-typed pages now route through expert routing + +# 4. If you run `gbrain serve --http` for remote MCP, register a client +# with admin scope so Wintermute or any other agent can author packs remotely: +gbrain auth register-client wintermute --scopes admin +``` + +If any step fails or numbers look wrong, please file an issue with the output of `gbrain doctor` and `tail -20 ~/.gbrain/audit/schema-mutations-*.jsonl` so we can debug the mutation chain. + +### Itemized changes + +**New CLI verbs (`gbrain schema *` — 14 new):** +- `add-type --primitive P --prefix dir/` — append a page type with primitive, prefix, optional `--extractable`, `--expert`, `--alias`. +- `remove-type ` — atomic remove with cross-reference check. If the type is referenced by another type's `aliases`, `enrichable_types`, `link_types`, or `frontmatter_links`, the remove fails loud with the reference list (codex C14 fix from `/plan-eng-review`). +- `update-type [--extractable BOOL] [--expert BOOL] [--primitive P]` — patch a type's flags without re-creating it. +- `add-alias ` / `remove-alias ` — atomic single-alias edits. +- `add-prefix ` / `remove-prefix ` — atomic single-prefix edits. +- `add-link-type [--inverse V] [--page-type T] [--target-type T]` — create new link verbs (e.g. `authored`, `attended`) with inference rules. +- `remove-link-type ` — refuses if any `frontmatter_links` references it. +- `set-extractable ` / `set-expert-routing ` — one-shot flag toggles. +- `stats [--source ]` — per-type page counts + coverage % + dead-prefix detection (declared prefixes with zero matching pages). Multi-source aware. +- `sync [--apply] [--source ]` — backfill `page.type` for rows matching pack prefixes. Dry-run by default; chunked UPDATE in 1000-row batches on `--apply` so concurrent writers never wait. +- `reload [--pack ]` — flush the in-process pack cache so the next call re-reads from disk. Auto-cascades through the extends-chain. + +**New MCP operations (9 — the agent on-ramp):** +- `get_active_schema_pack` (read scope) — cheap identity packet: `{pack_name, version, sha8, page_types_count, link_types_count, primitive_summary, source_tier}`. +- `list_schema_packs` (read) — bundled + installed packs. +- `schema_stats` (read) — same output as the CLI `stats` verb, source-scoped. +- `schema_lint` (read) — file-plane rules over MCP (DB-aware `--with-db` is CLI-only). +- `schema_graph` (read) — JSON `{nodes, edges}` derived from link types and frontmatter_links. +- `schema_explain_type` (read) — resolved settings for one declared type. +- `schema_review_orphans` (read) — drilldown into untyped pages. +- `schema_apply_mutations` (admin scope, NOT localOnly) — **batched** atomic mutation op. One call applies a list of mutations (`add_type`, `add_link_type`, `set_extractable`, etc.) inside a single `withPackLock` scope. Remote agents like Wintermute can compose multi-step refactors as one transaction. Audit log records `actor: mcp:` per mutation. +- `reload_schema_pack` (admin) — flush cache + extends-chain cascade. + +**Lint rules grew from 2 to 11:** +- `alias_shadows_type`, `alias_declared_by_two_types`, `alias_references_undeclared_type` +- `enrichable_types_undeclared`, `link_types_undeclared`, `frontmatter_links_undeclared` +- `expert_routing_without_prefix`, `prefix_collision`, `prefix_strict_subset_overlap` +- DB-aware (`--with-db`): `extractable_empty_corpus`, `mutation_count_anomaly` + +**Mutation safety primitives:** +- `pack-lock.ts` — atomic `O_CREAT|O_EXCL` acquire. NOT the TOCTOU `existsSync+writeFileSync` shape from `page-lock.ts`. TTL refresh every 10s while a mutation runs. `--force` semantics: "steal stale lock," not "skip locking." +- `mutate-audit.ts` — ISO-week JSONL at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. **Privacy-redacted by default**: type names → sha8, prefixes → first slug segment only. Matches `candidate-audit.ts` privacy posture so a leaked screenshot of either audit file can't reveal sensitive taxonomy. Opt out of redaction with `GBRAIN_SCHEMA_AUDIT_VERBOSE=1`. Logs both success AND failure events so the `schema_pack_writability` doctor check (v0.40.7+) has signal to read. +- Atomic write via `.tmp + fsync + rename`. The pack file on disk is NEVER partial. + +**Cross-process cache invalidation:** +- `loadActivePack` now stats the pack file with a 1-second TTL gate (env override `GBRAIN_PACK_STAT_TTL_MS`). Operator runs `gbrain schema add-type foo` from a terminal; the autopilot daemon picks up the new type within 1 second on its next `loadActivePack` call. Worst-case stat overhead: ~50µs at most once per second per process. Hot-path cost inside the TTL window: ~10ns. +- `invalidatePackCache(name)` walks the extends-chain reverse-graph (codex C6 fix). Editing a parent pack used to leave children stale; now every dependent invalidates too. + +**T1.5 pack-aware wiring (the silent-no-op fix, partial):** +- `gbrain whoknows` + the `find_experts` MCP op now consult the active pack for expert types. A `researcher` type declared `expert_routing: true` actually surfaces in `whoknows` results (pre-v0.40.6 it silently never matched because the query path read hardcoded `['person', 'company']`). +- **Pack-load failure semantics**: empty filter, NOT hardcoded defaults. A pack-load failure makes the query return empty (loud, agent debugs the pack-load problem) instead of silently re-introducing types the user packed out. Same lesson the v0.34.1 federated-read trust gate paid for. +- Remaining T1.5 sites — `facts/eligibility.ts` and `enrichment-service.ts`'s `'person' | 'company'` union — deferred to v0.40.7+. Filed as TODO. + +**Skill + RESOLVER + Convention layer:** +- `skills/schema-author/SKILL.md` — the agent dispatcher for "evolve the schema pack." Explicit non-goals callout to `brain-taxonomist` (filing one specific page) and `eiirp` (schema-check during iteration) so agents pick the right surface. Workflow: brain → assess → propose → apply → sync → verify → commit. +- `skills/conventions/schema-evolution.md` — when to add a type vs alias vs prefix. Decision tree: <20 pages → don't pack-codify; 20-100 → alias or narrow prefix; 100+ → first-class type. +- `skills/RESOLVER.md` entry for `schema-author` with the full trigger list. + +### Migration safety + +- Pre-v0.40.6 brains: zero breaking changes. The 9 new MCP ops are additive; the existing 16 schema verbs unchanged. +- Existing OAuth clients with `read` scope: read the new schema ops out of the box. +- Existing OAuth clients with `admin` scope: can call `schema_apply_mutations` and `reload_schema_pack` immediately. The audit log captures `actor: mcp:` on every mutation for forensic traceability. +- The mutation primitives refuse to touch bundled packs (`gbrain-base`, `gbrain-recommended`) — `gbrain schema fork gbrain-base mine` first. + +### What's safe to know about + +- The YAML emitter does NOT preserve comments or formatting in pack.yaml files when mutated. Authors who care about hand-written YAML layout should pin `pack.json` instead. +- The audit log redacts type names by default. If you need to grep for the raw type name during debugging, set `GBRAIN_SCHEMA_AUDIT_VERBOSE=1` and re-run the mutation. +- `gbrain schema sync --apply` on a 100K-page brain runs in chunks of 1000; each chunk completes in <100ms. Total wallclock ~10s for 100K rows. Concurrent writers see at most a 100ms wait on any single row. + +### Itemized — for contributors + +- Plan + 21 decisions captured at `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md`. +- Closed PR #1321 with successor pointer comment crediting `@garrytan-agents`. +- 6 commits land the wave: foundations (e8ea1792), mutate (21beda7f), stats+sync (4f493c96), CLI wiring (60c3da92), MCP ops (90bbd3fa), this commit (skill + docs + version bump). +- 255+ schema-pack-related tests green (84 new across 9 test files this wave). +- 3 follow-up TODOs filed for v0.40.7: enrichment-service.ts union widening (`'person' | 'company'` → `string`), facts/eligibility.ts pack-aware extractable-type wiring, doctor checks for schema_pack_coverage / schema_pack_writability / schema_pack_mutation_audit. +- Contributed by `@garrytan-agents` (original PR #1321 design + verb shape) and `@garrytan` + Claude Opus 4.7 1M (production rebuild). + ## [0.40.6.0] - 2026-05-23 **`gbrain sync --all` now syncs your sources at the same time instead of one after the other, and you can see the health of every source at a glance with `gbrain sources status`.** If you have a brain with 4+ sources connected to it, the cron job that keeps everything up to date used to take as long as the slowest source. One stuck `git pull` on a big media-corpus repo held up everything else, and after 24 hours you'd start seeing stale-data warnings pile up. Now the sources run together — independent ones don't wait on each other, and you can run `gbrain sources status` to see at a glance which ones are fresh, stale, or running behind. Per-source log lines come prefixed with `[source-id]` so you can grep one source's output cleanly even when several are running. diff --git a/CLAUDE.md b/CLAUDE.md index 418ca5e73..3d56c9f0e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -552,6 +552,37 @@ Key files: (operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns loudly when those flags are set. +## Schema Cathedral v3 (v0.40.7.0) + +The schema-pack mutation surface shipped in v0.40.7.0 as the production rebuild of +closed community PR #1321 (`@garrytan-agents`). Six new foundation modules + a +mutate skeleton + stats/sync data plane + 14 CLI verbs + 9 MCP ops + a first-class +agent skill. See `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` +for the full plan + 21 captured design decisions. + +Key files (v0.40.7.0 additions): +- `src/core/schema-pack/pack-lock.ts` — Atomic `O_CREAT|O_EXCL` per-pack lock. DELIBERATELY NOT the `existsSync + writeFileSync` TOCTOU shape from `src/core/page-lock.ts` (codex C8 caught the bug class). Default 60s TTL, refresh every 10s while `withPackLock(fn)` runs, `--force` semantics = "steal stale lock" NOT "skip locking." Lock path per-pack so two packs never block each other. +- `src/core/schema-pack/mutate-audit.ts` — ISO-week JSONL at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Privacy-redacted per D20: type names → sha8, prefixes → first slug segment only, matches `candidate-audit.ts` privacy posture. Logs BOTH success AND failure events so the v0.40.7+ `schema_pack_writability` doctor check has signal to read. `summarizeMutations()` is the cross-surface parity primitive. +- `src/core/schema-pack/registry.ts` extensions — `invalidatePackCache(name?)` walks the extends-chain reverse-graph (codex C6 fix; pre-v0.40.7, editing a parent pack silently left children stale). `tryCachedPack(name)` TTL-gated fast path: inside `STAT_TTL_MS` (default 1000ms, env `GBRAIN_PACK_STAT_TTL_MS`) returns cached without statting. Outside the window: stats every file in the chain; cascade-invalidates on mtime change (D11 cross-process detection). +- `src/core/schema-pack/best-effort.ts` — `loadActivePackBestEffort(ctx)` returns `ResolvedPack | null`. Single source of truth for the T1.5 wiring sites. `null` means EMPTY FILTER (NOT hardcoded defaults — D4 contract closing the silent-violation bug class). +- `src/core/schema-pack/lint-rules.ts` — 11 pure rule functions. `withMutation`'s pre-write validation gate composes the 9 file-plane rules; the 2 DB-aware rules (`extractable_empty_corpus`, `mutation_count_anomaly`) need an engine. Single source of truth consumed by CLI lint + MCP `schema_lint` + the pre-write validation gate. +- `src/core/schema-pack/query-cache-invalidator.ts` — `invalidateQueryCache(engine, sourceId?)` DELETEs query_cache rows so cached search results bound to old page types don't survive a schema mutation. Codex C9 fix. +- `src/core/schema-pack/mutate.ts` — 8-step `withMutation` skeleton (bundled-guard → lock → read → mutator → validate → atomic write → audit → invalidate). 11 mutation primitives: `addTypeToPack`, `removeTypeFromPack` (with codex C14 reference check), `updateTypeOnPack`, `addAliasToType`, `removeAliasFromType`, `addPrefixToType`, `removePrefixFromType`, `addLinkTypeToPack`, `removeLinkTypeFromPack`, `setExtractableOnType`, `setExpertRoutingOnType`. Atomic write via `.tmp + fsync + rename` — pack file on disk is NEVER partial. Inline minimal JSON→YAML emitter so YAML packs stay YAML (does NOT preserve comments — pin pack.json if you care about layout). +- `src/core/schema-pack/stats.ts` — `runStatsCore(engine, opts)` returns per-source + aggregate page counts + coverage % + `dead_prefixes` (declared prefixes with zero matching pages — agent's drilldown signal). Multi-source aware (`sourceIds[]` federated, `sourceId` single, or whole-brain). PGLite + Postgres parity via `executeRaw`. Empty brain → coverage:1.0 (vacuous truth). +- `src/core/schema-pack/sync.ts` — `runSyncCore(engine, opts)` chunked UPDATE in 1000-row batches per declared prefix (D14). Concurrent writers never block on a single row >100ms. Codex C5 write-side scoping via `ctx.sourceId` directly (NOT `sourceScopeOpts` which inherits OAuth read federation). Idempotent on `--apply` re-run. +- `src/commands/schema.ts` extension — 14 new CLI verbs in the dispatch table: `add-type`, `remove-type`, `update-type`, `add-alias`, `remove-alias`, `add-prefix`, `remove-prefix`, `add-link-type`, `remove-link-type`, `set-extractable`, `set-expert-routing`, `stats`, `sync`, `reload`. `withConnectedEngine` defensive fix from closed PR #1321 retained. Lifecycle-grouped help text (Inspection / Activation / Authoring / Discovery+repair). +- `src/core/operations.ts` extension — 9 new MCP ops: `get_active_schema_pack`, `list_schema_packs`, `schema_stats`, `schema_lint`, `schema_graph`, `schema_explain_type`, `schema_review_orphans` (all read-scope, NOT localOnly), plus `schema_apply_mutations` (**admin scope, NOT localOnly per D2** so remote agents like your OpenClaw can author packs over HTTPS MCP — batched per D10, one MCP tool taking an `mutations[]` array atomically inside ONE `withPackLock`, audit log captures `actor: mcp:`) and `reload_schema_pack` (admin, NOT localOnly). Trust posture: per-call `schema_pack` opt STAYS rejected for remote callers via `op-trust-gate.ts` (R2 regression preserved). +- `src/commands/whoknows.ts` + `src/core/operations.ts:find_experts` — T1.5 wiring sites. Pack-aware via `expertTypesFromPack(pack.manifest)` from `best-effort.ts`. Pack-load failure → EMPTY filter (NOT hardcoded `['person', 'company']` defaults per D4). A `researcher` type declared `--expert` now surfaces in `whoknows` results; pre-v0.40.7 it silently never matched. +- `skills/schema-author/SKILL.md` — Agent dispatcher for "evolve the schema pack." Triggers: 15+ phrasings including "add a page type", "my brain has untyped pages", "propose new types from my corpus", "backfill page types". Explicit Non-goals callout to `brain-taxonomist` (files one page) and `eiirp` (schema-check during iteration) so agents pick the right surface. 7-phase workflow: brain → assess → propose → apply → sync → verify → commit. Lists every gbrain schema CLI verb + every MCP op the skill uses. `brain_first: exempt` frontmatter. Required v0.40.7+ conformance sections: Contract, Anti-Patterns, Output Format. +- `skills/conventions/schema-evolution.md` — Canonical convention: "when to add a type vs alias vs prefix." Decision tree: <20 pages → don't pack-codify; 20-100 → alias or narrow prefix on existing type; 100+ → first-class type. Don'ts section + "when to remove a type" + "when to commit the pack" all answered in one place. +- `skills/RESOLVER.md` + `skills/manifest.json` — schema-author wired into the dispatcher with full functional-area trigger list (compressed routing pattern per v0.32.3 dispatcher convention). + +T1.5 wiring is partial in v0.40.7.0. Three follow-ups filed in TODOS.md under +"v0.40.7.0 Schema Cathedral v3 follow-ups (v0.40.7+)" — enrichment-service.ts +union widening (`'person' | 'company'` → `string`), facts/eligibility.ts +pack-aware `ELIGIBLE_TYPES` wiring, and 3 doctor checks (schema_pack_coverage, +schema_pack_writability, schema_pack_mutation_audit). + ## Commands Run `gbrain --help` or `gbrain --tools-json` for full command reference. diff --git a/README.md b/README.md index e502b13f1..df0f972cb 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,8 @@ GBrain is those patterns, generalized. Install in 30 minutes. Your agent does th **New in v0.40.2.0 — `gbrain think` grounds temporal answers in the typed-claim timeline.** Ask "when did Marco last switch jobs" or "what was the ARR in March" and the answer comes back rooted in a real chronological timeline of the metric + event facts your brain already extracted via the `extract_facts` cycle phase. Default ON. The intent classifier (`temporal` / `knowledge_update` / `other`) is a regex pass with zero LLM cost; the `'other'` fast path short-circuits with zero extra SQL. Migration v82 adds a nullable `facts.event_type` column so the same plumbing carries event-shaped rows (`'meeting'`, `'job_change'`, `'location_change'`) alongside metric rows. Flip `think.trajectory_enabled=false` to opt out. Debug with `GBRAIN_THINK_DEBUG=1 gbrain think "..."` to see the spliced prompt. The same trajectory plumbing also lands in the LongMemEval benchmark with a methodology change disclosed in `methodology_note: extractor=haiku-preprocess-full-haystack-v1` — published scores are "gbrain + Haiku-preprocess pipeline" vs "gbrain alone", NOT directly comparable to baseline LongMemEval numbers without that note. +**New in v0.40.7.0 — Your agents can now author your brain's schema pack themselves.** No more shell-out, no more hand-editing YAML. Tell your OpenClaw (or any agent connected via MCP) "my brain has 4000 untyped meetings pages — add a `meeting` type and backfill them," and it does the whole thing safely: per-pack atomic file lock, validation gate that catches dangling references pre-write, atomic write so a crash never leaves the pack half-written, privacy-redacted audit log with the agent's identity, chunked UPDATE in 1000-row batches that never wedge concurrent writers. 14 new `gbrain schema` CLI verbs (`add-type`, `remove-type`, `add-alias`, `add-link-type`, `stats`, `sync`, etc.) + 9 new MCP ops including the batched `schema_apply_mutations` (admin scope, NOT localOnly — remote agents reach it over normal HTTPS MCP). New `schema-author` skill with explicit boundary callouts to `brain-taxonomist` and `eiirp` so agents pick the right surface. The schema-pack cathedral that shipped in v0.39.1.0 is now reachable from the outside. **Why it matters:** [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — 7 killer use cases (4000 invisible meetings made queryable, the founder ops brain, the research brain, the legal brain, the team brain, agent-as-co-curator) plus the structural difference between a pile of notes and a brain with structure. **Walkthrough:** [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — fork the bundled pack, add a `researcher` type, backfill, query in 5 minutes. + **New in v0.36.4.0 — Your agent drives the brain to 90/100 by itself.** One command does the loop you used to run by hand: `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`. It computes a dependency-ordered plan (sync before extract, embed after consolidate), submits each step as a Minion job, re-checks score between every step, and refuses to spend past your cost cap. Cron can drive it unattended. `gbrain doctor --remediation-plan --json` previews what would run. Autopilot now does the same thing on its 5-minute tick: small problems get targeted handlers, big problems get the full cycle, a healthy brain sleeps for 60 minutes instead of grinding through synthesize+patterns+embed every tick. Eleven new things you can submit as background jobs (`reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, plus six cycle phases); three of them (synthesize, patterns, consolidate) are PROTECTED so an MCP-connected agent can't silently burn Anthropic credits. New `--background` flag on `gbrain embed` submits the job and exits with `job_id=N` for shell composition. **New in v0.35.7 — Temporal trajectory + founder scorecard.** Author typed metric assertions in the `## Facts` fence (`mrr=50000`, `arr=2000000`, `team_size=12`) and gbrain stores them as first-class typed columns. `gbrain eval trajectory companies/acme-example` prints the chronological history with regressions auto-flagged inline. `gbrain founder scorecard companies/acme-example` rolls up claim accuracy, consistency, growth direction, and red flags into a stable `schema_version: 1` JSON contract. New MCP op `find_trajectory` exposes the same data to agents (read scope, visibility-filtered for remote callers). The `consolidate` cycle phase now writes `valid_until` on chronologically-superseded facts AND uses semantic upsert on `(page_id, claim, since_date)` — re-running the dream cycle on stable input is now a true no-op (fixed a pre-existing duplicate-takes bug from prior versions). @@ -126,6 +128,8 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Brain consistency.** `gbrain eval suspected-contradictions` samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle. +**Agent-authored schema (v0.40.7.0).** Your brain has a shape — what page types exist (`person`, `meeting`, `paper`, `case`, `lab-result`), what they link to (`attended`, `authored`, `prescribed-by`), what facts get extracted automatically. The default ships with 22 universal types, but your brain's actual shape is not the default shape. Agents can now evolve that shape on your behalf via 14 `gbrain schema` CLI verbs + a batched MCP op (`schema_apply_mutations`, admin scope, NOT localOnly so remote agents reach it over HTTPS). Atomic file locks, audit log with the agent's identity, chunked UPDATE backfill in 1000-row batches that never wedge concurrent writers. The brain stops being a pile of notes and becomes something with structure. **Why it matters:** [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — 7 killer use cases (4000 invisible meetings, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator). **5-minute walkthrough:** [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md). **Agent skill:** [`skills/schema-author/SKILL.md`](skills/schema-author/SKILL.md). + ## Integrations Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in `recipes/` and is discoverable via `gbrain integrations list`. @@ -153,6 +157,8 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h ## Docs - [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end +- [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0) +- [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via `gbrain whoknows` - [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory - [`docs/guides/`](docs/guides/) — how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion) - [`docs/integrations/`](docs/integrations/) — connecting external data sources (voice, email, calendar, embedding providers) diff --git a/TODOS.md b/TODOS.md index 8b5063b82..b4d029069 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,69 @@ # TODOS +## v0.40.7.0 Schema Cathedral v3 follow-ups (v0.40.7+) + +These were filed when v0.40.7.0 closed PR #1321's design as a production +rebuild. The wave shipped the 9 MCP ops + 14 CLI verbs + atomic mutation +primitives + skill on-ramp; three wiring sites were larger than expected +at plan time and got carved out: + +- [ ] **v0.40.7+: enrichment-service.ts union widening (`'person' | 'company'` → `string`).** + `src/core/enrichment-service.ts` hard-codes the `entityType` union in 6 + sites (`:25`, `:48`, `:60`, `:238`, `:246`, + caller mappings). Widening + to `string` and threading the active pack's path_prefixes through + `slugifyEntity` closes the T1.5 silent-no-op bug for the enrichment + pipeline. Estimated 2 hours CC. Third T1.5 wiring site (whoknows + + find_experts MCP already wired in v0.40.7.0). + +- [ ] **v0.40.7+: facts/eligibility.ts pack-aware ELIGIBLE_TYPES wiring.** + `src/core/facts/eligibility.ts:49` defines a hardcoded `ELIGIBLE_TYPES` + array. Should consult `extractableTypesFromPack(pack)`. Behavioral + change: every brain's extraction surface changes once wired, so needs + careful verification. + +- [ ] **v0.40.7+: three doctor checks for schema pack health.** + `schema_pack_coverage` (warn >10%, fail >30% untyped on non-default + pack), `schema_pack_writability` (reads schema-mutations audit JSONL + for PACK_READONLY failures), `schema_pack_mutation_audit` (anomalous + patterns like >20 mutations/week). All warn-only; reuse + `summarizeMutations()` for cross-surface parity. Audit log shipped + with the right shape so these drop in cleanly. + +- [ ] **v0.40.7+: T16 — hermetic schema-authoring eval gate.** + Extend `src/commands/eval-schema-authoring.ts` into a PGLite harness + driving detect → suggest → add-type → sync end-to-end on 3 fixtures. + Filing-accuracy delta metric (not top-3 hit rate per codex C18). DI + seam via `suggestFn`. 3 hours CC + placeholder-name fixtures. + +- [ ] **v0.40.7+: T16.1 — separate "suggest top-3 hit rate" eval.** + Different question from T16. ~2 hours CC. + +- [ ] **v0.41+: T19 — per-source federated read closure across mounts.** + Trust gate today rejects divergent-pack federated reads + (`op-trust-gate.ts:111-116`). Real fix needs per-source SQL closure + via `buildPerSourceBindings`. Document workaround: register + source-scoped OAuth clients. + +- [ ] **v0.41+: T20 — extends-chain merging in registry.ts.** + `registry.ts:167` documents the gap. Implementing full child-wins + merge cascades through every consumer of `manifest.page_types`. ~1 + day CC. + +- [ ] **v0.41+: T21 — comment-preserving YAML emitter.** + v0.40.7.0 emitter does NOT preserve comments. Authors who care + pin pack.json. Replacing with a comment-aware library is the proper + fix. + +- [ ] **v0.41+: T22 — admin SPA tab for schema verbs.** + CLI + MCP only this wave. + +- [ ] **v0.41+: T23 — finer-grained `schema:write` OAuth scope.** + Today the write ops gate on `admin`. Splitting `admin → admin + + schema:write` is a cross-cutting refactor. + +- [ ] **v0.41+: T24 — multi-tenant pack federation in a single brain.** + One active pack per source remains. + ## v0.40.3.0 follow-ups (v0.41+) - [ ] **v0.41+: source-scope the `sync-failures.jsonl` log so `--skip-failed` works under `--parallel > 1`.** @@ -29,7 +93,7 @@ `checkSyncFreshness`'s message to include `embedding_coverage_pct` per source alongside the staleness number so doctor surfaces the coverage gap inline. Implementation: reuse `buildSyncStatusReport` from `src/commands/sync.ts`, - fold coverage into the existing per-source message string. ~half a day. + ## v0.40.1.0 Track D follow-ups (v0.41+) diff --git a/VERSION b/VERSION index e1d50b419..6805f15d9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.40.6.0 \ No newline at end of file +0.40.7.0 \ No newline at end of file diff --git a/docs/schema-author-tutorial.md b/docs/schema-author-tutorial.md new file mode 100644 index 000000000..e871703c5 --- /dev/null +++ b/docs/schema-author-tutorial.md @@ -0,0 +1,224 @@ +# Tutorial: Build your first schema pack + +You'll fork the bundled `gbrain-base` pack, add a custom `researcher` page type, import a handful of placeholder researcher pages, backfill their `page.type` column with one command, then prove the wiring works by running `gbrain whoknows` and seeing your new type surface in results. End state: a forked-and-active pack on disk, ~5 pages typed as `researcher`, and a query that proves the pack-aware routing fires end-to-end. + +**Want the WHY before the HOW?** Read [`what-schemas-unlock.md`](what-schemas-unlock.md) first — 7 concrete use cases (4000 invisible meetings, the founder ops brain, the research brain, the legal brain, the team brain, agent-as-co-curator) plus the structural argument for why types matter at query time. Then come back here for the 5-minute walkthrough. + +The whole walkthrough takes about 5 minutes. You'll see something working by step 3. + +## What you'll need + +- gbrain v0.40.7.0 or later (`gbrain --version` to check) +- A brain that's been initialized (`gbrain init` already run; either PGLite or Postgres is fine) +- A terminal you can paste commands into + +That's it. No API keys required for this tutorial — every step works against the bundled pack and local-only commands. + +## Step 1: See what pack is active today + +```bash +gbrain schema active --json +``` + +You'll see something like: + +```json +{ + "pack_name": "gbrain-base", + "version": "1.0.0", + "sha8": "...", + "page_types_count": 22, + "source_tier": "default" +} +``` + +`source_tier: "default"` means you haven't customized anything — you're on the bundled pack. `page_types_count: 22` is the universal starter (person, company, meeting, note, etc.). + +**You can't mutate bundled packs directly.** Step 2 forks it so you have something writable. + +## Step 2: Fork the bundled pack + +```bash +gbrain schema fork gbrain-base mine +``` + +Output: `Forked 'gbrain-base' → 'mine' at ~/.gbrain/schema-packs/mine/pack.json`. + +The fork is a byte-for-byte copy of `gbrain-base` living at `~/.gbrain/schema-packs/mine/pack.json`. Now you have a writable pack you can mutate. + +## Step 3: Activate the fork + +```bash +gbrain schema use mine +``` + +Output: `Pack: mine (json) ... Active.` + +Run `gbrain schema active --json` again to confirm `pack_name` is now `mine` and `source_tier` is `home-config` (read from `~/.gbrain/config.json`). + +**You've already accomplished something visible** — the active pack changed, and any future query will route through your fork. The next four steps add a custom type and prove it works. + +## Step 4: Add a researcher type + +```bash +gbrain schema add-type researcher \ + --primitive entity \ + --prefix people/researchers/ \ + --extractable \ + --expert +``` + +Output: `Pack: mine (json)` + `Sha8: → `. + +What just happened: +- The mutation went through `withMutation`'s 8-step skeleton: bundled-guard → per-pack lock → read → mutate → file-plane lint validation → atomic write → audit log → cache invalidation. +- The pack now declares `researcher` as an entity primitive bound to `people/researchers/`, marked `extractable: true` (eligible for facts extraction) and `expert_routing: true` (surfaces in `whoknows` queries). +- An audit row landed in `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl` with your type name SHA-8-redacted and the prefix's first segment only (`people`) for privacy. + +Verify the type is in the pack: + +```bash +gbrain schema explain researcher +``` + +You'll see the resolved settings printed back. + +## Step 5: Import some placeholder researcher pages + +You need pages under `people/researchers/` for the next step to do anything. If your brain repo already has them, skip ahead. If not, drop 3-5 placeholder markdown files into `/people/researchers/` and import: + +```bash +mkdir -p people/researchers +cat > people/researchers/alice-example.md <<'EOF' +--- +title: Alice Example +--- + +ML researcher at Example Lab. Works on contrastive embeddings. +EOF + +cat > people/researchers/bob-example.md <<'EOF' +--- +title: Bob Example +--- + +Vision researcher at Widget University. Recent paper on diffusion models. +EOF + +cat > people/researchers/charlie-example.md <<'EOF' +--- +title: Charlie Example +--- + +RL researcher at Acme Research. Focus on inverse reinforcement learning. +EOF + +gbrain sync +``` + +The sync imports the new files. They'll be stored in the database but their `type` column will still be empty — the new type was added to the pack AFTER these pages already existed (the typical real-world scenario for an agent walking into an existing brain). + +## Step 6: See the gap with `stats` + +```bash +gbrain schema stats --json | jq '.aggregate, .dead_prefixes' +``` + +You'll see `untyped_pages: 3` (or however many you just imported) and `dead_prefixes: []` — your new prefix has 3 matching pages, so it's not dead. + +The 3 researcher pages are "orphaned" by type even though they live in the right directory. The next step backfills them. + +## Step 7: Backfill with `sync --apply` + +First dry-run to see what would happen: + +```bash +gbrain schema sync --json +``` + +You'll see something like: + +```json +{ + "schema_version": 1, + "apply": false, + "per_prefix": [ + { + "type": "researcher", + "prefix": "people/researchers/", + "would_apply": 3, + "sample_slugs": ["people/researchers/alice-example", "people/researchers/bob-example", "people/researchers/charlie-example"], + "applied": 0 + } + ], + "total_would_apply": 3, + "total_applied": 0 +} +``` + +`would_apply: 3` is what you'd touch. `sample_slugs` is the agent's drilldown signal — if those slugs look wrong, abort. They look right, so apply: + +```bash +gbrain schema sync --apply +``` + +You'll see per-batch progress lines on stderr and a final `total_applied: 3`. The UPDATE ran in chunks of 1000 (yours fit in one chunk) and never wedged any concurrent writer. + +## Step 8: Prove the wiring works + +```bash +gbrain whoknows "machine learning" +``` + +If your researcher pages contain ML-related content, they'll surface in the ranked results — even though they're typed `researcher`, not `person` or `company`. + +**This is the load-bearing demonstration of T1.5 wiring.** Pre-v0.40.7.0, `whoknows` hardcoded `['person', 'company']` as the eligible types and would have ignored your `researcher` pages entirely. The v0.40.7.0 wiring consults the active pack's `expert_routing: true` types via `expertTypesFromPack(pack.manifest)`, so your custom type now routes through expert search. + +## What you built + +You now have: +- A fork of `gbrain-base` named `mine` at `~/.gbrain/schema-packs/mine/pack.json`, active in your brain via `~/.gbrain/config.json`. +- A `researcher` page type registered in the pack with `entity` primitive, `people/researchers/` prefix, `extractable: true`, `expert_routing: true`. +- 3 pages typed as `researcher` (backfilled from disk via `gbrain schema sync --apply`). +- A query path that routes through the new type: `gbrain whoknows` reads the pack and includes `researcher` in its type filter. + +You also exercised the full mutation skeleton: bundled-pack guard, per-pack lock, validation gate, atomic write, audit log, cache invalidation. Every step was idempotent — re-running any of them is a no-op. + +## Next steps + +**Add a link verb.** A `researcher` can `author` a `paper`. To model that: + +```bash +gbrain schema add-type paper --primitive annotation --prefix research/papers/ --extractable +gbrain schema add-link-type authored --page-type researcher --target-type paper +gbrain schema graph +``` + +The graph now shows `researcher --(authored)--> paper`. + +**Add aliases for query closure.** If you want `gbrain query researcher` to also surface `person` rows (because researchers ARE people): + +```bash +gbrain schema add-alias researcher person +``` + +Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) for the decision tree on when to add types vs aliases vs prefixes. The short version: <20 pages → don't pack-codify; 20-100 → alias on existing type; 100+ → first-class type. + +**Lint your pack before shipping.** The 11-rule lint surface (with the optional `--with-db` flag for DB-aware checks) catches dangling references, prefix collisions, and dead-corpus warnings: + +```bash +gbrain schema lint --with-db +``` + +**Commit your pack to source control.** If `~/.gbrain/schema-packs/mine/` is a git repo, commit `pack.json` and push. Your pack survives across machines, and the `mutation_count_anomaly` lint rule will nudge you when you hit >50 mutations in a week (the "you should be committing this" signal). + +**For agents (MCP):** the same operations are reachable over HTTPS MCP via 9 new ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher. + +**Undo a mistake.** Every mutation primitive has an inverse (`remove-type`, `remove-alias`, `remove-prefix`, `remove-link-type`, `set-extractable false`, etc.). If you fork twice and want to revert, `gbrain schema downgrade` restores the previous active pack from `~/.gbrain/schema-pack-history.jsonl`. + +## Related docs + +- **Reference:** `gbrain schema --help` for the full 22-verb CLI surface; CLAUDE.md's "Schema Cathedral v3 (v0.40.7.0)" section for the module-by-module architecture. +- **How-to:** [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) — the agent dispatcher with the 7-phase workflow (brain → assess → propose → apply → sync → verify → commit). +- **Explanation:** [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) — when to add a type vs alias vs prefix. +- **Plan + decisions:** the original design captured 21 decisions including the bundled-pack guard rationale (D6), the empty-filter fallback contract (D4), and the MCP non-localOnly trust posture (D2). Lives in `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` (private). diff --git a/docs/what-schemas-unlock.md b/docs/what-schemas-unlock.md new file mode 100644 index 000000000..7f5f72de3 --- /dev/null +++ b/docs/what-schemas-unlock.md @@ -0,0 +1,177 @@ +# What schemas unlock + +Most note-taking apps treat every page the same. You write something, it goes in a pile, you search the pile with text matching. Tags help, but tags are flat. After a few thousand pages, the pile gets noisy and the search gets stupid. + +Schemas are how gbrain stops being a pile of notes and becomes something with structure. A schema declares what KINDS of things live in your brain (`person`, `company`, `meeting`, `researcher`, `case`, `lab-result`), what they link to (`attended`, `authored`, `prescribed-by`), what facts the system should extract automatically (`mrr=50000`, `damages=5000000`), and which types route through expert search vs general search. + +The default schema (`gbrain-base`) ships with 22 page types covering the universal shapes — people, companies, meetings, notes, daily, calendar events. That's enough to start. But your brain is yours, and your brain's shape is not the default shape. A research brain needs `researcher` and `paper` as first-class types. A founder brain needs `lead`, `investor`, `portco`, `deal-stage`. A lawyer brain needs `case`, `motion`, `deposition`, `precedent`. Same engine, totally different shape. + +v0.40.7.0 made it possible for AGENTS to author that shape for you. Not just "the user manually edits YAML in `~/.gbrain/schema-packs/mine/pack.yaml`" but "your agent sees the corpus, proposes a type, asks for approval, applies it atomically with a full audit trail, then backfills 4000 existing pages with one chunked SQL command." That's the new thing. + +This doc is the WHY. The [tutorial](schema-author-tutorial.md) is the HOW. + +## Killer use cases + +### 1. The 4000 invisible pages + +You have 4000 markdown files under `meetings/` going back two years. The default schema doesn't have a `meeting` type, so all 4000 are typed `note` (the catchall). When you run: + +```bash +gbrain whoknows "Q3 roadmap discussion" +``` + +You get the top 10 text matches, ranked by raw relevance. The brain has no idea these are meetings. It can't route to attendees. It can't pull dates. It can't surface "this conversation came up again with the same people three weeks later." + +Add a `meeting` type: + +```bash +gbrain schema add-type meeting --primitive temporal --prefix meetings/ --extractable +gbrain schema sync --apply +``` + +The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now: + +- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text. +- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`. +- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files. + +One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did. + +### 2. The founder ops brain + +You're a founder or investor with ~500 markdown files mixing leads, portfolio companies, deal notes, intros, and follow-ups. You've been writing freely; you have no system. Your queries are all "wait, who introduced me to that fintech founder again?" and you scroll Notion for 20 minutes. + +Add the founder shape: + +```bash +gbrain schema fork gbrain-base mine +gbrain schema use mine + +# Types +gbrain schema add-type lead --primitive entity --prefix people/leads/ --expert +gbrain schema add-type investor --primitive entity --prefix people/investors/ --expert --extractable +gbrain schema add-type portco --primitive entity --prefix companies/portco/ --expert --extractable +gbrain schema add-type deal --primitive entity --prefix companies/deals/ --extractable + +# Link verbs +gbrain schema add-link-type invested-in --page-type investor --target-type portco +gbrain schema add-link-type intro-from --page-type lead --target-type lead +gbrain schema add-link-type passed-on --page-type investor --target-type deal +gbrain schema add-link-type led-by --page-type deal --target-type investor + +gbrain schema sync --apply +``` + +Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`. + +The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them. + +### 3. The research brain + +Replace "founder" with "PhD student" and the same pattern applies with different types: `researcher`, `paper`, `lab`, `grant`, `dataset` + `authored`, `cites`, `funded-by`, `uses-dataset`. + +```bash +gbrain schema add-type paper --primitive annotation --prefix research/papers/ --extractable +gbrain schema add-link-type authored --page-type researcher --target-type paper +gbrain schema add-link-type cites --page-type paper --target-type paper +gbrain schema add-link-type uses --page-type paper --target-type dataset +``` + +Suddenly "show me papers that cite this work AND use the same dataset" is a `gbrain graph-query` traversal, not 30 minutes in Google Scholar. The fact extraction picks up `arxiv_id=2402.04253`, `cited_by_count=140`, `published_date=2026-02-15` automatically. Your reading-list-as-markdown turns into a queryable research graph that knows who works on what and what's connected to what. + +### 4. The legal brain (or any domain where claims have numbers) + +Lawyers, medical providers, accountants, anyone working in a domain where the meaning of a number depends on its type. A "judgment of $5M" against a "$2M case strategy threshold" is a comparison the brain can do — but only if both numbers are typed. + +```bash +gbrain schema add-type case --primitive entity --prefix legal/cases/ --extractable --expert +gbrain schema add-type motion --primitive annotation --prefix legal/motions/ --extractable +gbrain schema add-type deposition --primitive annotation --prefix legal/depositions/ --extractable +gbrain schema add-link-type filed-in --page-type motion --target-type case +gbrain schema add-link-type cites --page-type motion --target-type precedent +``` + +Now `## Facts` fences in your case notes can carry typed claims (`damages=5000000`, `filed_date=2026-05-23`, `judge=jane-doe`) that gbrain stores as first-class columns. `gbrain eval trajectory legal/cases/acme-v-widget` prints the case history with regressions flagged. `gbrain founder scorecard` (renamed for legal: roll up plaintiff success rate, average damages, settlement-vs-trial ratio) gives you a structured view of how your practice is performing. + +This isn't possible without typed page kinds. You can write the same prose in any note-taking app. Only gbrain treats the numbers as comparable across pages of the same type. + +### 5. The team brain + +`gbrain mounts add` lets you stack additional brains alongside your personal one. Each mounted brain has its OWN schema pack. The eng team's brain has `incident`, `runbook`, `service`, `oncall-rotation`. The design team's brain has `component`, `experiment`, `ab-test`, `figma-link`. The legal team's brain has cases and depositions. + +When you query, the schema pack governs how each source's content is routed. An eng query against the mounted eng brain knows that `incidents/2026-05-23-db-outage.md` is an `incident` page with `severity=p0`, `mttr=47min`, `on_call=alice-example` — extractable typed facts. Your personal query against the same brain still works, but the routing is sharper because the eng team has invested in their ontology. + +The schema is the team's tribal knowledge made explicit. Two engineers on different teams searching the same brain get DIFFERENT routing because their personal packs declare different expert types. + +### 6. The "agent co-curates your ontology" pattern (the new thing) + +This is what v0.40.7.0 actually enabled, and what the closed PR #1321 was reaching for. + +Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `garrytan/companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes: + +> You have 47 pages under `companies/yc-w24/` typed as `company` (generic). They share a structural pattern (founder names, raise amounts, batch tag). Should I add a `yc-w24-company` type with `extractable: true` and the existing aliases pointing back to `company`? I'd backfill the 47 pages and add `cohort=W24` as a typed fact extracted from each page. + +You approve once. The agent calls `schema_apply_mutations` over MCP with a batch: + +```json +{ + "pack": "mine", + "mutations": [ + {"op": "add_type", "name": "yc-w24-company", "primitive": "entity", "prefix": "companies/yc-w24/", "extractable": true, "expert_routing": true}, + {"op": "add_alias", "type": "yc-w24-company", "alias": "company"} + ] +} +``` + +All inside ONE `withPackLock` scope, atomic, audited (the agent's `client_id` captured in the audit log as `actor: mcp:`). Cache invalidated cross-process. Sync backfills the 47 pages. The brain learned a new category of thing without you having to think about it. + +The next time you query "YC W24 companies in fintech", the brain routes through the new type. Six months later when you forget the pattern entirely, the agent reminds you it's there and offers to consolidate it with the W25 batch. + +The brain learns. The agent is the curator. You approve, the agent does the work. + +### 7. The before-vs-after benchmark + +If you want to FEEL the difference without buying the pitch: + +Pick a real corpus you have. Run `gbrain whoknows` on a topic that should match. Note the top-3 results. + +Then run `gbrain schema review-orphans --limit 50 --json` and look at the untyped pages. If 10+ of them share an obvious prefix that should be a real type, add the type + sync. + +Re-run the same `whoknows` query. Top-3 should shift, because the new type is now routing through expert ranking instead of being lumped into the catchall. The numerical delta IS the win. You can run a tutorial in 5 minutes; this experiment proves it matters on your actual content. + +## Why this matters + +Three things gbrain does that generic note systems can't: + +**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached. + +**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion. + +**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through 14 atomic CLI verbs + 9 MCP ops with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees. + +## What changed in v0.40.7.0 specifically + +v0.39.1.0 shipped the schema-pack engine. You could ALREADY fork the bundled pack and edit `pack.yaml` by hand. What you couldn't do was let an agent author it safely — there were no atomic file locks, no audit log, no MCP exposure, no pack-aware wiring in the query path. The cathedral was built but unreachable from the outside. + +v0.40.7.0 closed those gaps: + +- **`withMutation` skeleton** wraps every primitive in 8 ordered safety steps (bundled-guard → lock → read → mutate → validate → atomic write → audit → invalidate). The pack file on disk is never partial. Two concurrent agents can't race. +- **Per-pack `O_CREAT|O_EXCL` atomic lock** (not the TOCTOU `existsSync+writeFileSync` pattern from page-lock.ts — codex caught that during plan review). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking." +- **Privacy-redacted audit log** at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Type names sha8-hashed, prefixes truncated to first segment only. A leaked screenshot of the audit can't reveal sensitive taxonomy like `personal/oncology/` or `legal/depositions/`. +- **9 new MCP ops** including the batched `schema_apply_mutations` (admin scope, NOT localOnly — your OpenClaw and any remote agent author packs over normal HTTPS MCP, with `client_id` captured as `actor: mcp:`). +- **T1.5 wiring** finally completes for `whoknows` and `find_experts`: a custom `researcher` type marked `--expert` now actually surfaces in query results. Pre-v0.40.7 it silently never matched because the query path read hardcoded `['person', 'company']`. +- **Cross-process invalidation** via stat-mtime TTL gate inside `loadActivePack`. Operator runs `gbrain schema add-type` from a terminal; the autopilot daemon picks up the new type within 1 second without a restart. + +The cumulative effect: an agent can safely co-curate your ontology with a complete forensic trail. That's the new thing. + +## Where to start + +- **Want to see it work in 5 minutes?** Run the [tutorial](schema-author-tutorial.md). Forks the bundled pack, adds a researcher type, proves the wiring end-to-end. +- **Want the agent recipe?** Read [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md). 7-phase workflow agents follow when they detect a schema-evolution opportunity. +- **Want the rules of thumb?** Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md). Decision tree for when to add a type vs alias vs prefix. <20 pages don't pack-codify. 100+ pages need first-class types. +- **Want the architecture?** The "Schema Cathedral v3 (v0.40.7.0)" section in `CLAUDE.md` has the 14-bullet module-by-module breakdown, each citing the design decision and codex finding that motivated it. +- **Want to set up an agent that co-curates your brain?** Run `gbrain auth register-client my-agent --scopes admin` to mint an OAuth client your remote agent can use to call `schema_apply_mutations` over MCP. The agent then runs detect → suggest → apply on its own cadence and asks you to approve substantive changes. + +The killer feature isn't "schemas." Personal knowledge systems have had schemas forever. The killer feature is that your AGENT can shape them safely on your behalf, with structural integrity guarantees that match what you'd expect from a database, not a notes app. + +That's what we built. Try it on a corpus you actually have and the numbers go up. diff --git a/llms-full.txt b/llms-full.txt index 036f0fb2c..589f89948 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -694,6 +694,37 @@ Key files: (operations.ts:1103-1135 trust-boundary gate); thin-client `think` warns loudly when those flags are set. +## Schema Cathedral v3 (v0.40.7.0) + +The schema-pack mutation surface shipped in v0.40.7.0 as the production rebuild of +closed community PR #1321 (`@garrytan-agents`). Six new foundation modules + a +mutate skeleton + stats/sync data plane + 14 CLI verbs + 9 MCP ops + a first-class +agent skill. See `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` +for the full plan + 21 captured design decisions. + +Key files (v0.40.7.0 additions): +- `src/core/schema-pack/pack-lock.ts` — Atomic `O_CREAT|O_EXCL` per-pack lock. DELIBERATELY NOT the `existsSync + writeFileSync` TOCTOU shape from `src/core/page-lock.ts` (codex C8 caught the bug class). Default 60s TTL, refresh every 10s while `withPackLock(fn)` runs, `--force` semantics = "steal stale lock" NOT "skip locking." Lock path per-pack so two packs never block each other. +- `src/core/schema-pack/mutate-audit.ts` — ISO-week JSONL at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Privacy-redacted per D20: type names → sha8, prefixes → first slug segment only, matches `candidate-audit.ts` privacy posture. Logs BOTH success AND failure events so the v0.40.7+ `schema_pack_writability` doctor check has signal to read. `summarizeMutations()` is the cross-surface parity primitive. +- `src/core/schema-pack/registry.ts` extensions — `invalidatePackCache(name?)` walks the extends-chain reverse-graph (codex C6 fix; pre-v0.40.7, editing a parent pack silently left children stale). `tryCachedPack(name)` TTL-gated fast path: inside `STAT_TTL_MS` (default 1000ms, env `GBRAIN_PACK_STAT_TTL_MS`) returns cached without statting. Outside the window: stats every file in the chain; cascade-invalidates on mtime change (D11 cross-process detection). +- `src/core/schema-pack/best-effort.ts` — `loadActivePackBestEffort(ctx)` returns `ResolvedPack | null`. Single source of truth for the T1.5 wiring sites. `null` means EMPTY FILTER (NOT hardcoded defaults — D4 contract closing the silent-violation bug class). +- `src/core/schema-pack/lint-rules.ts` — 11 pure rule functions. `withMutation`'s pre-write validation gate composes the 9 file-plane rules; the 2 DB-aware rules (`extractable_empty_corpus`, `mutation_count_anomaly`) need an engine. Single source of truth consumed by CLI lint + MCP `schema_lint` + the pre-write validation gate. +- `src/core/schema-pack/query-cache-invalidator.ts` — `invalidateQueryCache(engine, sourceId?)` DELETEs query_cache rows so cached search results bound to old page types don't survive a schema mutation. Codex C9 fix. +- `src/core/schema-pack/mutate.ts` — 8-step `withMutation` skeleton (bundled-guard → lock → read → mutator → validate → atomic write → audit → invalidate). 11 mutation primitives: `addTypeToPack`, `removeTypeFromPack` (with codex C14 reference check), `updateTypeOnPack`, `addAliasToType`, `removeAliasFromType`, `addPrefixToType`, `removePrefixFromType`, `addLinkTypeToPack`, `removeLinkTypeFromPack`, `setExtractableOnType`, `setExpertRoutingOnType`. Atomic write via `.tmp + fsync + rename` — pack file on disk is NEVER partial. Inline minimal JSON→YAML emitter so YAML packs stay YAML (does NOT preserve comments — pin pack.json if you care about layout). +- `src/core/schema-pack/stats.ts` — `runStatsCore(engine, opts)` returns per-source + aggregate page counts + coverage % + `dead_prefixes` (declared prefixes with zero matching pages — agent's drilldown signal). Multi-source aware (`sourceIds[]` federated, `sourceId` single, or whole-brain). PGLite + Postgres parity via `executeRaw`. Empty brain → coverage:1.0 (vacuous truth). +- `src/core/schema-pack/sync.ts` — `runSyncCore(engine, opts)` chunked UPDATE in 1000-row batches per declared prefix (D14). Concurrent writers never block on a single row >100ms. Codex C5 write-side scoping via `ctx.sourceId` directly (NOT `sourceScopeOpts` which inherits OAuth read federation). Idempotent on `--apply` re-run. +- `src/commands/schema.ts` extension — 14 new CLI verbs in the dispatch table: `add-type`, `remove-type`, `update-type`, `add-alias`, `remove-alias`, `add-prefix`, `remove-prefix`, `add-link-type`, `remove-link-type`, `set-extractable`, `set-expert-routing`, `stats`, `sync`, `reload`. `withConnectedEngine` defensive fix from closed PR #1321 retained. Lifecycle-grouped help text (Inspection / Activation / Authoring / Discovery+repair). +- `src/core/operations.ts` extension — 9 new MCP ops: `get_active_schema_pack`, `list_schema_packs`, `schema_stats`, `schema_lint`, `schema_graph`, `schema_explain_type`, `schema_review_orphans` (all read-scope, NOT localOnly), plus `schema_apply_mutations` (**admin scope, NOT localOnly per D2** so remote agents like your OpenClaw can author packs over HTTPS MCP — batched per D10, one MCP tool taking an `mutations[]` array atomically inside ONE `withPackLock`, audit log captures `actor: mcp:`) and `reload_schema_pack` (admin, NOT localOnly). Trust posture: per-call `schema_pack` opt STAYS rejected for remote callers via `op-trust-gate.ts` (R2 regression preserved). +- `src/commands/whoknows.ts` + `src/core/operations.ts:find_experts` — T1.5 wiring sites. Pack-aware via `expertTypesFromPack(pack.manifest)` from `best-effort.ts`. Pack-load failure → EMPTY filter (NOT hardcoded `['person', 'company']` defaults per D4). A `researcher` type declared `--expert` now surfaces in `whoknows` results; pre-v0.40.7 it silently never matched. +- `skills/schema-author/SKILL.md` — Agent dispatcher for "evolve the schema pack." Triggers: 15+ phrasings including "add a page type", "my brain has untyped pages", "propose new types from my corpus", "backfill page types". Explicit Non-goals callout to `brain-taxonomist` (files one page) and `eiirp` (schema-check during iteration) so agents pick the right surface. 7-phase workflow: brain → assess → propose → apply → sync → verify → commit. Lists every gbrain schema CLI verb + every MCP op the skill uses. `brain_first: exempt` frontmatter. Required v0.40.7+ conformance sections: Contract, Anti-Patterns, Output Format. +- `skills/conventions/schema-evolution.md` — Canonical convention: "when to add a type vs alias vs prefix." Decision tree: <20 pages → don't pack-codify; 20-100 → alias or narrow prefix on existing type; 100+ → first-class type. Don'ts section + "when to remove a type" + "when to commit the pack" all answered in one place. +- `skills/RESOLVER.md` + `skills/manifest.json` — schema-author wired into the dispatcher with full functional-area trigger list (compressed routing pattern per v0.32.3 dispatcher convention). + +T1.5 wiring is partial in v0.40.7.0. Three follow-ups filed in TODOS.md under +"v0.40.7.0 Schema Cathedral v3 follow-ups (v0.40.7+)" — enrichment-service.ts +union widening (`'person' | 'company'` → `string`), facts/eligibility.ts +pack-aware `ELIGIBLE_TYPES` wiring, and 3 doctor checks (schema_pack_coverage, +schema_pack_writability, schema_pack_mutation_audit). + ## Commands Run `gbrain --help` or `gbrain --tools-json` for full command reference. @@ -2450,6 +2481,7 @@ These apply to ALL brain-writing skills: - `skills/conventions/quality.md` — citations, back-links, notability gate - `skills/conventions/brain-first.md` — check brain before external APIs - `skills/conventions/brain-routing.md` — which brain (DB) and which source (repo) to target; cross-brain federation is latent-space only +- `skills/conventions/schema-evolution.md` — when to add a type vs alias vs prefix (read before `schema-author`) - `skills/conventions/subagent-routing.md` — when to use Minions vs inline work - `skills/ask-user/SKILL.md` — choice-gate pattern for human input at decision points - `skills/_brain-filing-rules.md` — where files go @@ -2468,6 +2500,7 @@ These apply to ALL brain-writing skills: | "verify this academic claim", "check this study", "academic verify", "validate citation", "is this study real" | `skills/academic-verify/SKILL.md` | | "make pdf from brain", "brain pdf", "convert brain page to pdf", "publish this page as pdf", "export brain page" | `skills/brain-pdf/SKILL.md` | | "voice note", "ingest this voice memo", "transcribe and file", "voice note ingest", "save this audio note" | `skills/voice-note-ingest/SKILL.md` | +| "add a page type", "add a type to my schema", "schema author", "schema mutate", "schema pack add", "my brain has untyped pages", "propose new types from my corpus", "backfill page types", "evolve my schema", "researcher type", "make X an expert type" (dispatcher for: gbrain schema active/list/show/validate/graph/lint/stats/explain/use/downgrade/reload/init/fork/edit/diff/add-type/remove-type/update-type/add-alias/remove-alias/add-prefix/remove-prefix/add-link-type/remove-link-type/set-extractable/set-expert-routing/detect/suggest/review-candidates/review-orphans/sync) | `skills/schema-author/SKILL.md` | --- @@ -2489,6 +2522,8 @@ GBrain is those patterns, generalized. Install in 30 minutes. Your agent does th **New in v0.40.2.0 — `gbrain think` grounds temporal answers in the typed-claim timeline.** Ask "when did Marco last switch jobs" or "what was the ARR in March" and the answer comes back rooted in a real chronological timeline of the metric + event facts your brain already extracted via the `extract_facts` cycle phase. Default ON. The intent classifier (`temporal` / `knowledge_update` / `other`) is a regex pass with zero LLM cost; the `'other'` fast path short-circuits with zero extra SQL. Migration v82 adds a nullable `facts.event_type` column so the same plumbing carries event-shaped rows (`'meeting'`, `'job_change'`, `'location_change'`) alongside metric rows. Flip `think.trajectory_enabled=false` to opt out. Debug with `GBRAIN_THINK_DEBUG=1 gbrain think "..."` to see the spliced prompt. The same trajectory plumbing also lands in the LongMemEval benchmark with a methodology change disclosed in `methodology_note: extractor=haiku-preprocess-full-haystack-v1` — published scores are "gbrain + Haiku-preprocess pipeline" vs "gbrain alone", NOT directly comparable to baseline LongMemEval numbers without that note. +**New in v0.40.7.0 — Your agents can now author your brain's schema pack themselves.** No more shell-out, no more hand-editing YAML. Tell your OpenClaw (or any agent connected via MCP) "my brain has 4000 untyped meetings pages — add a `meeting` type and backfill them," and it does the whole thing safely: per-pack atomic file lock, validation gate that catches dangling references pre-write, atomic write so a crash never leaves the pack half-written, privacy-redacted audit log with the agent's identity, chunked UPDATE in 1000-row batches that never wedge concurrent writers. 14 new `gbrain schema` CLI verbs (`add-type`, `remove-type`, `add-alias`, `add-link-type`, `stats`, `sync`, etc.) + 9 new MCP ops including the batched `schema_apply_mutations` (admin scope, NOT localOnly — remote agents reach it over normal HTTPS MCP). New `schema-author` skill with explicit boundary callouts to `brain-taxonomist` and `eiirp` so agents pick the right surface. The schema-pack cathedral that shipped in v0.39.1.0 is now reachable from the outside. **Why it matters:** [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — 7 killer use cases (4000 invisible meetings made queryable, the founder ops brain, the research brain, the legal brain, the team brain, agent-as-co-curator) plus the structural difference between a pile of notes and a brain with structure. **Walkthrough:** [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — fork the bundled pack, add a `researcher` type, backfill, query in 5 minutes. + **New in v0.36.4.0 — Your agent drives the brain to 90/100 by itself.** One command does the loop you used to run by hand: `gbrain doctor --remediate --yes --target-score 90 --max-usd 5`. It computes a dependency-ordered plan (sync before extract, embed after consolidate), submits each step as a Minion job, re-checks score between every step, and refuses to spend past your cost cap. Cron can drive it unattended. `gbrain doctor --remediation-plan --json` previews what would run. Autopilot now does the same thing on its 5-minute tick: small problems get targeted handlers, big problems get the full cycle, a healthy brain sleeps for 60 minutes instead of grinding through synthesize+patterns+embed every tick. Eleven new things you can submit as background jobs (`reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, plus six cycle phases); three of them (synthesize, patterns, consolidate) are PROTECTED so an MCP-connected agent can't silently burn Anthropic credits. New `--background` flag on `gbrain embed` submits the job and exits with `job_id=N` for shell composition. **New in v0.35.7 — Temporal trajectory + founder scorecard.** Author typed metric assertions in the `## Facts` fence (`mrr=50000`, `arr=2000000`, `team_size=12`) and gbrain stores them as first-class typed columns. `gbrain eval trajectory companies/acme-example` prints the chronological history with regressions auto-flagged inline. `gbrain founder scorecard companies/acme-example` rolls up claim accuracy, consistency, growth direction, and red flags into a stable `schema_version: 1` JSON contract. New MCP op `find_trajectory` exposes the same data to agents (read scope, visibility-filtered for remote callers). The `consolidate` cycle phase now writes `valid_until` on chronologically-superseded facts AND uses semantic upsert on `(page_id, claim, since_date)` — re-running the dream cycle on stable input is now a true no-op (fixed a pre-existing duplicate-takes bug from prior versions). @@ -2603,6 +2638,8 @@ The whole loop is described in [`docs/architecture/topologies.md`](docs/architec **Brain consistency.** `gbrain eval suspected-contradictions` samples retrieval pairs, layered date pre-filter, query-conditioned LLM judge, persistent cache. Surfaces conflicts between takes + facts the agent has written. Wired into the daily dream cycle. +**Agent-authored schema (v0.40.7.0).** Your brain has a shape — what page types exist (`person`, `meeting`, `paper`, `case`, `lab-result`), what they link to (`attended`, `authored`, `prescribed-by`), what facts get extracted automatically. The default ships with 22 universal types, but your brain's actual shape is not the default shape. Agents can now evolve that shape on your behalf via 14 `gbrain schema` CLI verbs + a batched MCP op (`schema_apply_mutations`, admin scope, NOT localOnly so remote agents reach it over HTTPS). Atomic file locks, audit log with the agent's identity, chunked UPDATE backfill in 1000-row batches that never wedge concurrent writers. The brain stops being a pile of notes and becomes something with structure. **Why it matters:** [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — 7 killer use cases (4000 invisible meetings, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator). **5-minute walkthrough:** [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md). **Agent skill:** [`skills/schema-author/SKILL.md`](skills/schema-author/SKILL.md). + ## Integrations Data flowing into the brain. Each integration is a recipe — markdown + setup hints — that ships in `recipes/` and is discoverable via `gbrain integrations list`. @@ -2630,6 +2667,8 @@ Data flowing into the brain. Each integration is a recipe — markdown + setup h ## Docs - [`docs/INSTALL.md`](docs/INSTALL.md) — every install path, end to end +- [`docs/what-schemas-unlock.md`](docs/what-schemas-unlock.md) — why schemas matter: 7 killer use cases, the structural argument for typed page kinds, the agent-co-curates pattern (v0.40.7.0) +- [`docs/schema-author-tutorial.md`](docs/schema-author-tutorial.md) — 5-minute walkthrough: fork the bundled pack, add a custom type, backfill existing pages, prove the wiring via `gbrain whoknows` - [`docs/architecture/`](docs/architecture/) — system design, topologies, retrieval theory - [`docs/guides/`](docs/guides/) — how-to runbooks (sub-agent routing, minion deployment, skill development, brain-first lookup, idea capture, diligence ingestion) - [`docs/integrations/`](docs/integrations/) — connecting external data sources (voice, email, calendar, embedding providers) @@ -2902,6 +2941,421 @@ Note: The original SQLite engine plan (`docs/SQLITE_ENGINE.md`) was superseded b --- +## docs/what-schemas-unlock.md + +Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/what-schemas-unlock.md + +# What schemas unlock + +Most note-taking apps treat every page the same. You write something, it goes in a pile, you search the pile with text matching. Tags help, but tags are flat. After a few thousand pages, the pile gets noisy and the search gets stupid. + +Schemas are how gbrain stops being a pile of notes and becomes something with structure. A schema declares what KINDS of things live in your brain (`person`, `company`, `meeting`, `researcher`, `case`, `lab-result`), what they link to (`attended`, `authored`, `prescribed-by`), what facts the system should extract automatically (`mrr=50000`, `damages=5000000`), and which types route through expert search vs general search. + +The default schema (`gbrain-base`) ships with 22 page types covering the universal shapes — people, companies, meetings, notes, daily, calendar events. That's enough to start. But your brain is yours, and your brain's shape is not the default shape. A research brain needs `researcher` and `paper` as first-class types. A founder brain needs `lead`, `investor`, `portco`, `deal-stage`. A lawyer brain needs `case`, `motion`, `deposition`, `precedent`. Same engine, totally different shape. + +v0.40.7.0 made it possible for AGENTS to author that shape for you. Not just "the user manually edits YAML in `~/.gbrain/schema-packs/mine/pack.yaml`" but "your agent sees the corpus, proposes a type, asks for approval, applies it atomically with a full audit trail, then backfills 4000 existing pages with one chunked SQL command." That's the new thing. + +This doc is the WHY. The [tutorial](schema-author-tutorial.md) is the HOW. + +## Killer use cases + +### 1. The 4000 invisible pages + +You have 4000 markdown files under `meetings/` going back two years. The default schema doesn't have a `meeting` type, so all 4000 are typed `note` (the catchall). When you run: + +```bash +gbrain whoknows "Q3 roadmap discussion" +``` + +You get the top 10 text matches, ranked by raw relevance. The brain has no idea these are meetings. It can't route to attendees. It can't pull dates. It can't surface "this conversation came up again with the same people three weeks later." + +Add a `meeting` type: + +```bash +gbrain schema add-type meeting --primitive temporal --prefix meetings/ --extractable +gbrain schema sync --apply +``` + +The sync backfills `page.type = 'meeting'` on all 4000 pages in 1000-row batches. Now: + +- `gbrain whoknows "Q3 roadmap discussion"` routes through the meeting type, ranking by `expert_routing` signal (attendees, recency, salience) instead of raw text. +- `gbrain extract-facts` runs on every meeting page automatically (because `extractable: true`), pulling typed facts like `attended_by=alice-example`, `date=2026-05-23`. +- The downstream `think` skill can now answer "what did we decide about pricing in the last three roadmap meetings" by querying the meeting graph instead of grep'ing 4000 files. + +One command. 4000 pages went from invisible to queryable. The content didn't change. The structure did. + +### 2. The founder ops brain + +You're a founder or investor with ~500 markdown files mixing leads, portfolio companies, deal notes, intros, and follow-ups. You've been writing freely; you have no system. Your queries are all "wait, who introduced me to that fintech founder again?" and you scroll Notion for 20 minutes. + +Add the founder shape: + +```bash +gbrain schema fork gbrain-base mine +gbrain schema use mine + +# Types +gbrain schema add-type lead --primitive entity --prefix people/leads/ --expert +gbrain schema add-type investor --primitive entity --prefix people/investors/ --expert --extractable +gbrain schema add-type portco --primitive entity --prefix companies/portco/ --expert --extractable +gbrain schema add-type deal --primitive entity --prefix companies/deals/ --extractable + +# Link verbs +gbrain schema add-link-type invested-in --page-type investor --target-type portco +gbrain schema add-link-type intro-from --page-type lead --target-type lead +gbrain schema add-link-type passed-on --page-type investor --target-type deal +gbrain schema add-link-type led-by --page-type deal --target-type investor + +gbrain schema sync --apply +``` + +Now `gbrain whoknows "Series A SaaS"` routes through `investor` and `portco` types specifically, not the noisy general type set. `gbrain graph-query alice-example --type intro-from --depth 2` walks two hops of intros to surface "Alice introduced you to Bob who introduced you to Charlie." `gbrain extract-facts` starts producing typed claims from the fence in your deal pages: `(deals/acme-seed, raise=2000000, valuation=15000000, lead=widget-vc, closed_at=2026-05-23)`. + +The CRM you've been promising yourself you'll set up next quarter? You just shipped it in 4 commands. It's downstream of your notes, not parallel to them. + +### 3. The research brain + +Replace "founder" with "PhD student" and the same pattern applies with different types: `researcher`, `paper`, `lab`, `grant`, `dataset` + `authored`, `cites`, `funded-by`, `uses-dataset`. + +```bash +gbrain schema add-type paper --primitive annotation --prefix research/papers/ --extractable +gbrain schema add-link-type authored --page-type researcher --target-type paper +gbrain schema add-link-type cites --page-type paper --target-type paper +gbrain schema add-link-type uses --page-type paper --target-type dataset +``` + +Suddenly "show me papers that cite this work AND use the same dataset" is a `gbrain graph-query` traversal, not 30 minutes in Google Scholar. The fact extraction picks up `arxiv_id=2402.04253`, `cited_by_count=140`, `published_date=2026-02-15` automatically. Your reading-list-as-markdown turns into a queryable research graph that knows who works on what and what's connected to what. + +### 4. The legal brain (or any domain where claims have numbers) + +Lawyers, medical providers, accountants, anyone working in a domain where the meaning of a number depends on its type. A "judgment of $5M" against a "$2M case strategy threshold" is a comparison the brain can do — but only if both numbers are typed. + +```bash +gbrain schema add-type case --primitive entity --prefix legal/cases/ --extractable --expert +gbrain schema add-type motion --primitive annotation --prefix legal/motions/ --extractable +gbrain schema add-type deposition --primitive annotation --prefix legal/depositions/ --extractable +gbrain schema add-link-type filed-in --page-type motion --target-type case +gbrain schema add-link-type cites --page-type motion --target-type precedent +``` + +Now `## Facts` fences in your case notes can carry typed claims (`damages=5000000`, `filed_date=2026-05-23`, `judge=jane-doe`) that gbrain stores as first-class columns. `gbrain eval trajectory legal/cases/acme-v-widget` prints the case history with regressions flagged. `gbrain founder scorecard` (renamed for legal: roll up plaintiff success rate, average damages, settlement-vs-trial ratio) gives you a structured view of how your practice is performing. + +This isn't possible without typed page kinds. You can write the same prose in any note-taking app. Only gbrain treats the numbers as comparable across pages of the same type. + +### 5. The team brain + +`gbrain mounts add` lets you stack additional brains alongside your personal one. Each mounted brain has its OWN schema pack. The eng team's brain has `incident`, `runbook`, `service`, `oncall-rotation`. The design team's brain has `component`, `experiment`, `ab-test`, `figma-link`. The legal team's brain has cases and depositions. + +When you query, the schema pack governs how each source's content is routed. An eng query against the mounted eng brain knows that `incidents/2026-05-23-db-outage.md` is an `incident` page with `severity=p0`, `mttr=47min`, `on_call=alice-example` — extractable typed facts. Your personal query against the same brain still works, but the routing is sharper because the eng team has invested in their ontology. + +The schema is the team's tribal knowledge made explicit. Two engineers on different teams searching the same brain get DIFFERENT routing because their personal packs declare different expert types. + +### 6. The "agent co-curates your ontology" pattern (the new thing) + +This is what v0.40.7.0 actually enabled, and what the closed PR #1321 was reaching for. + +Your OpenClaw (or any agent connected to your brain over HTTPS MCP with admin scope) watches your ingestion stream. After a week of you dumping notes under `garrytan/companies/yc-w24/`, the agent runs `gbrain schema detect` periodically, sees that prefix accumulating, and proposes: + +> You have 47 pages under `companies/yc-w24/` typed as `company` (generic). They share a structural pattern (founder names, raise amounts, batch tag). Should I add a `yc-w24-company` type with `extractable: true` and the existing aliases pointing back to `company`? I'd backfill the 47 pages and add `cohort=W24` as a typed fact extracted from each page. + +You approve once. The agent calls `schema_apply_mutations` over MCP with a batch: + +```json +{ + "pack": "mine", + "mutations": [ + {"op": "add_type", "name": "yc-w24-company", "primitive": "entity", "prefix": "companies/yc-w24/", "extractable": true, "expert_routing": true}, + {"op": "add_alias", "type": "yc-w24-company", "alias": "company"} + ] +} +``` + +All inside ONE `withPackLock` scope, atomic, audited (the agent's `client_id` captured in the audit log as `actor: mcp:`). Cache invalidated cross-process. Sync backfills the 47 pages. The brain learned a new category of thing without you having to think about it. + +The next time you query "YC W24 companies in fintech", the brain routes through the new type. Six months later when you forget the pattern entirely, the agent reminds you it's there and offers to consolidate it with the W25 batch. + +The brain learns. The agent is the curator. You approve, the agent does the work. + +### 7. The before-vs-after benchmark + +If you want to FEEL the difference without buying the pitch: + +Pick a real corpus you have. Run `gbrain whoknows` on a topic that should match. Note the top-3 results. + +Then run `gbrain schema review-orphans --limit 50 --json` and look at the untyped pages. If 10+ of them share an obvious prefix that should be a real type, add the type + sync. + +Re-run the same `whoknows` query. Top-3 should shift, because the new type is now routing through expert ranking instead of being lumped into the catchall. The numerical delta IS the win. You can run a tutorial in 5 minutes; this experiment proves it matters on your actual content. + +## Why this matters + +Three things gbrain does that generic note systems can't: + +**1. The brain knows the difference between a person and an idea.** Page-type matters at query time. `gbrain whoknows` only considers `expert_routing: true` types. `gbrain extract-facts` only runs on `extractable: true` types. `gbrain graph-query` walks declared link verbs. None of that works on a flat tag system because tags don't have semantics — they're labels. Types are first-class citizens with rules attached. + +**2. Untyped content is invisible content.** If your meetings are typed as `note`, expert routing skips them, facts extraction ignores them, link inference doesn't fire. They exist on disk and they're indexed for text search, but the structural surfaces (whoknows, find_experts, recall, think) treat them as second-class. Adding a type isn't cosmetic; it's structural promotion. + +**3. The schema is queryable AND mutable AND auditable.** You can ask the brain what its schema looks like (`gbrain schema graph`), evolve it through 14 atomic CLI verbs + 9 MCP ops with full lock + audit semantics, and recover from any mistake (every primitive has an inverse, plus `gbrain schema downgrade` restores the previous active pack). This isn't "vibes-based knowledge management." It's a production system with structural integrity guarantees. + +## What changed in v0.40.7.0 specifically + +v0.39.1.0 shipped the schema-pack engine. You could ALREADY fork the bundled pack and edit `pack.yaml` by hand. What you couldn't do was let an agent author it safely — there were no atomic file locks, no audit log, no MCP exposure, no pack-aware wiring in the query path. The cathedral was built but unreachable from the outside. + +v0.40.7.0 closed those gaps: + +- **`withMutation` skeleton** wraps every primitive in 8 ordered safety steps (bundled-guard → lock → read → mutate → validate → atomic write → audit → invalidate). The pack file on disk is never partial. Two concurrent agents can't race. +- **Per-pack `O_CREAT|O_EXCL` atomic lock** (not the TOCTOU `existsSync+writeFileSync` pattern from page-lock.ts — codex caught that during plan review). TTL refresh every 10s while a mutation runs; `--force` means "steal stale lock" not "skip locking." +- **Privacy-redacted audit log** at `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. Type names sha8-hashed, prefixes truncated to first segment only. A leaked screenshot of the audit can't reveal sensitive taxonomy like `personal/oncology/` or `legal/depositions/`. +- **9 new MCP ops** including the batched `schema_apply_mutations` (admin scope, NOT localOnly — your OpenClaw and any remote agent author packs over normal HTTPS MCP, with `client_id` captured as `actor: mcp:`). +- **T1.5 wiring** finally completes for `whoknows` and `find_experts`: a custom `researcher` type marked `--expert` now actually surfaces in query results. Pre-v0.40.7 it silently never matched because the query path read hardcoded `['person', 'company']`. +- **Cross-process invalidation** via stat-mtime TTL gate inside `loadActivePack`. Operator runs `gbrain schema add-type` from a terminal; the autopilot daemon picks up the new type within 1 second without a restart. + +The cumulative effect: an agent can safely co-curate your ontology with a complete forensic trail. That's the new thing. + +## Where to start + +- **Want to see it work in 5 minutes?** Run the [tutorial](schema-author-tutorial.md). Forks the bundled pack, adds a researcher type, proves the wiring end-to-end. +- **Want the agent recipe?** Read [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md). 7-phase workflow agents follow when they detect a schema-evolution opportunity. +- **Want the rules of thumb?** Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md). Decision tree for when to add a type vs alias vs prefix. <20 pages don't pack-codify. 100+ pages need first-class types. +- **Want the architecture?** The "Schema Cathedral v3 (v0.40.7.0)" section in `CLAUDE.md` has the 14-bullet module-by-module breakdown, each citing the design decision and codex finding that motivated it. +- **Want to set up an agent that co-curates your brain?** Run `gbrain auth register-client my-agent --scopes admin` to mint an OAuth client your remote agent can use to call `schema_apply_mutations` over MCP. The agent then runs detect → suggest → apply on its own cadence and asks you to approve substantive changes. + +The killer feature isn't "schemas." Personal knowledge systems have had schemas forever. The killer feature is that your AGENT can shape them safely on your behalf, with structural integrity guarantees that match what you'd expect from a database, not a notes app. + +That's what we built. Try it on a corpus you actually have and the numbers go up. + +--- + +## docs/schema-author-tutorial.md + +Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/schema-author-tutorial.md + +# Tutorial: Build your first schema pack + +You'll fork the bundled `gbrain-base` pack, add a custom `researcher` page type, import a handful of placeholder researcher pages, backfill their `page.type` column with one command, then prove the wiring works by running `gbrain whoknows` and seeing your new type surface in results. End state: a forked-and-active pack on disk, ~5 pages typed as `researcher`, and a query that proves the pack-aware routing fires end-to-end. + +**Want the WHY before the HOW?** Read [`what-schemas-unlock.md`](what-schemas-unlock.md) first — 7 concrete use cases (4000 invisible meetings, the founder ops brain, the research brain, the legal brain, the team brain, agent-as-co-curator) plus the structural argument for why types matter at query time. Then come back here for the 5-minute walkthrough. + +The whole walkthrough takes about 5 minutes. You'll see something working by step 3. + +## What you'll need + +- gbrain v0.40.7.0 or later (`gbrain --version` to check) +- A brain that's been initialized (`gbrain init` already run; either PGLite or Postgres is fine) +- A terminal you can paste commands into + +That's it. No API keys required for this tutorial — every step works against the bundled pack and local-only commands. + +## Step 1: See what pack is active today + +```bash +gbrain schema active --json +``` + +You'll see something like: + +```json +{ + "pack_name": "gbrain-base", + "version": "1.0.0", + "sha8": "...", + "page_types_count": 22, + "source_tier": "default" +} +``` + +`source_tier: "default"` means you haven't customized anything — you're on the bundled pack. `page_types_count: 22` is the universal starter (person, company, meeting, note, etc.). + +**You can't mutate bundled packs directly.** Step 2 forks it so you have something writable. + +## Step 2: Fork the bundled pack + +```bash +gbrain schema fork gbrain-base mine +``` + +Output: `Forked 'gbrain-base' → 'mine' at ~/.gbrain/schema-packs/mine/pack.json`. + +The fork is a byte-for-byte copy of `gbrain-base` living at `~/.gbrain/schema-packs/mine/pack.json`. Now you have a writable pack you can mutate. + +## Step 3: Activate the fork + +```bash +gbrain schema use mine +``` + +Output: `Pack: mine (json) ... Active.` + +Run `gbrain schema active --json` again to confirm `pack_name` is now `mine` and `source_tier` is `home-config` (read from `~/.gbrain/config.json`). + +**You've already accomplished something visible** — the active pack changed, and any future query will route through your fork. The next four steps add a custom type and prove it works. + +## Step 4: Add a researcher type + +```bash +gbrain schema add-type researcher \ + --primitive entity \ + --prefix people/researchers/ \ + --extractable \ + --expert +``` + +Output: `Pack: mine (json)` + `Sha8: → `. + +What just happened: +- The mutation went through `withMutation`'s 8-step skeleton: bundled-guard → per-pack lock → read → mutate → file-plane lint validation → atomic write → audit log → cache invalidation. +- The pack now declares `researcher` as an entity primitive bound to `people/researchers/`, marked `extractable: true` (eligible for facts extraction) and `expert_routing: true` (surfaces in `whoknows` queries). +- An audit row landed in `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl` with your type name SHA-8-redacted and the prefix's first segment only (`people`) for privacy. + +Verify the type is in the pack: + +```bash +gbrain schema explain researcher +``` + +You'll see the resolved settings printed back. + +## Step 5: Import some placeholder researcher pages + +You need pages under `people/researchers/` for the next step to do anything. If your brain repo already has them, skip ahead. If not, drop 3-5 placeholder markdown files into `/people/researchers/` and import: + +```bash +mkdir -p people/researchers +cat > people/researchers/alice-example.md <<'EOF' +--- +title: Alice Example +--- + +ML researcher at Example Lab. Works on contrastive embeddings. +EOF + +cat > people/researchers/bob-example.md <<'EOF' +--- +title: Bob Example +--- + +Vision researcher at Widget University. Recent paper on diffusion models. +EOF + +cat > people/researchers/charlie-example.md <<'EOF' +--- +title: Charlie Example +--- + +RL researcher at Acme Research. Focus on inverse reinforcement learning. +EOF + +gbrain sync +``` + +The sync imports the new files. They'll be stored in the database but their `type` column will still be empty — the new type was added to the pack AFTER these pages already existed (the typical real-world scenario for an agent walking into an existing brain). + +## Step 6: See the gap with `stats` + +```bash +gbrain schema stats --json | jq '.aggregate, .dead_prefixes' +``` + +You'll see `untyped_pages: 3` (or however many you just imported) and `dead_prefixes: []` — your new prefix has 3 matching pages, so it's not dead. + +The 3 researcher pages are "orphaned" by type even though they live in the right directory. The next step backfills them. + +## Step 7: Backfill with `sync --apply` + +First dry-run to see what would happen: + +```bash +gbrain schema sync --json +``` + +You'll see something like: + +```json +{ + "schema_version": 1, + "apply": false, + "per_prefix": [ + { + "type": "researcher", + "prefix": "people/researchers/", + "would_apply": 3, + "sample_slugs": ["people/researchers/alice-example", "people/researchers/bob-example", "people/researchers/charlie-example"], + "applied": 0 + } + ], + "total_would_apply": 3, + "total_applied": 0 +} +``` + +`would_apply: 3` is what you'd touch. `sample_slugs` is the agent's drilldown signal — if those slugs look wrong, abort. They look right, so apply: + +```bash +gbrain schema sync --apply +``` + +You'll see per-batch progress lines on stderr and a final `total_applied: 3`. The UPDATE ran in chunks of 1000 (yours fit in one chunk) and never wedged any concurrent writer. + +## Step 8: Prove the wiring works + +```bash +gbrain whoknows "machine learning" +``` + +If your researcher pages contain ML-related content, they'll surface in the ranked results — even though they're typed `researcher`, not `person` or `company`. + +**This is the load-bearing demonstration of T1.5 wiring.** Pre-v0.40.7.0, `whoknows` hardcoded `['person', 'company']` as the eligible types and would have ignored your `researcher` pages entirely. The v0.40.7.0 wiring consults the active pack's `expert_routing: true` types via `expertTypesFromPack(pack.manifest)`, so your custom type now routes through expert search. + +## What you built + +You now have: +- A fork of `gbrain-base` named `mine` at `~/.gbrain/schema-packs/mine/pack.json`, active in your brain via `~/.gbrain/config.json`. +- A `researcher` page type registered in the pack with `entity` primitive, `people/researchers/` prefix, `extractable: true`, `expert_routing: true`. +- 3 pages typed as `researcher` (backfilled from disk via `gbrain schema sync --apply`). +- A query path that routes through the new type: `gbrain whoknows` reads the pack and includes `researcher` in its type filter. + +You also exercised the full mutation skeleton: bundled-pack guard, per-pack lock, validation gate, atomic write, audit log, cache invalidation. Every step was idempotent — re-running any of them is a no-op. + +## Next steps + +**Add a link verb.** A `researcher` can `author` a `paper`. To model that: + +```bash +gbrain schema add-type paper --primitive annotation --prefix research/papers/ --extractable +gbrain schema add-link-type authored --page-type researcher --target-type paper +gbrain schema graph +``` + +The graph now shows `researcher --(authored)--> paper`. + +**Add aliases for query closure.** If you want `gbrain query researcher` to also surface `person` rows (because researchers ARE people): + +```bash +gbrain schema add-alias researcher person +``` + +Read [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) for the decision tree on when to add types vs aliases vs prefixes. The short version: <20 pages → don't pack-codify; 20-100 → alias on existing type; 100+ → first-class type. + +**Lint your pack before shipping.** The 11-rule lint surface (with the optional `--with-db` flag for DB-aware checks) catches dangling references, prefix collisions, and dead-corpus warnings: + +```bash +gbrain schema lint --with-db +``` + +**Commit your pack to source control.** If `~/.gbrain/schema-packs/mine/` is a git repo, commit `pack.json` and push. Your pack survives across machines, and the `mutation_count_anomaly` lint rule will nudge you when you hit >50 mutations in a week (the "you should be committing this" signal). + +**For agents (MCP):** the same operations are reachable over HTTPS MCP via 9 new ops. Register an admin-scope OAuth client and `schema_apply_mutations` lets a remote agent compose multi-step refactors as one atomic batch. The batched MCP op + per-pack lock + audit log are the load-bearing primitives that make remote schema authoring safe. See [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) for the agent dispatcher. + +**Undo a mistake.** Every mutation primitive has an inverse (`remove-type`, `remove-alias`, `remove-prefix`, `remove-link-type`, `set-extractable false`, etc.). If you fork twice and want to revert, `gbrain schema downgrade` restores the previous active pack from `~/.gbrain/schema-pack-history.jsonl`. + +## Related docs + +- **Reference:** `gbrain schema --help` for the full 22-verb CLI surface; CLAUDE.md's "Schema Cathedral v3 (v0.40.7.0)" section for the module-by-module architecture. +- **How-to:** [`skills/schema-author/SKILL.md`](../skills/schema-author/SKILL.md) — the agent dispatcher with the 7-phase workflow (brain → assess → propose → apply → sync → verify → commit). +- **Explanation:** [`skills/conventions/schema-evolution.md`](../skills/conventions/schema-evolution.md) — when to add a type vs alias vs prefix. +- **Plan + decisions:** the original design captured 21 decisions including the bundled-pack guard rationale (D6), the empty-filter fallback contract (D4), and the MCP non-localOnly trust posture (D2). Lives in `~/.claude/plans/system-instruction-you-are-working-recursive-thacker.md` (private). + +--- + ## docs/guides/live-sync.md Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/live-sync.md diff --git a/llms.txt b/llms.txt index 2b8c6bf18..993ee9a46 100644 --- a/llms.txt +++ b/llms.txt @@ -16,6 +16,8 @@ Repo: https://github.com/garrytan/gbrain - [docs/ENGINES.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/ENGINES.md): PGLite vs Postgres trade-off and when to migrate. - [docs/GBRAIN_RECOMMENDED_SCHEMA.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/GBRAIN_RECOMMENDED_SCHEMA.md): MECE directory structure (people/, companies/, concepts/). +- [docs/what-schemas-unlock.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/what-schemas-unlock.md): Why schemas matter: 7 killer use cases (4000 invisible meetings, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator) + the structural argument for typed page kinds. Read this before pitching schema authoring (v0.40.7.0). +- [docs/schema-author-tutorial.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/schema-author-tutorial.md): 5-minute walkthrough: fork the bundled pack, add a custom `researcher` type, backfill existing pages via `gbrain schema sync --apply`, prove the T1.5 wiring via `gbrain whoknows` (v0.40.7.0). - [docs/guides/live-sync.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/live-sync.md): Incremental markdown sync setup. - [docs/guides/cron-schedule.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/cron-schedule.md): Recurring job scheduling. - [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist. diff --git a/package.json b/package.json index ebaa4f29e..1baabc7b1 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.40.6.0", + "version": "0.40.7.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/scripts/llms-config.ts b/scripts/llms-config.ts index d82068940..45c37da6c 100644 --- a/scripts/llms-config.ts +++ b/scripts/llms-config.ts @@ -86,6 +86,18 @@ export const SECTIONS: DocSection[] = [ // under the 600KB budget as CLAUDE.md grows with each release. includeInFull: false, }, + { + title: "docs/what-schemas-unlock.md", + description: + "Why schemas matter: 7 killer use cases (4000 invisible meetings, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator) + the structural argument for typed page kinds. Read this before pitching schema authoring (v0.40.7.0).", + path: "docs/what-schemas-unlock.md", + }, + { + title: "docs/schema-author-tutorial.md", + description: + "5-minute walkthrough: fork the bundled pack, add a custom `researcher` type, backfill existing pages via `gbrain schema sync --apply`, prove the T1.5 wiring via `gbrain whoknows` (v0.40.7.0).", + path: "docs/schema-author-tutorial.md", + }, { title: "docs/guides/live-sync.md", description: "Incremental markdown sync setup.", diff --git a/skills/RESOLVER.md b/skills/RESOLVER.md index feea5c82d..70c97d0be 100644 --- a/skills/RESOLVER.md +++ b/skills/RESOLVER.md @@ -110,6 +110,7 @@ These apply to ALL brain-writing skills: - `skills/conventions/quality.md` — citations, back-links, notability gate - `skills/conventions/brain-first.md` — check brain before external APIs - `skills/conventions/brain-routing.md` — which brain (DB) and which source (repo) to target; cross-brain federation is latent-space only +- `skills/conventions/schema-evolution.md` — when to add a type vs alias vs prefix (read before `schema-author`) - `skills/conventions/subagent-routing.md` — when to use Minions vs inline work - `skills/ask-user/SKILL.md` — choice-gate pattern for human input at decision points - `skills/_brain-filing-rules.md` — where files go @@ -128,4 +129,5 @@ These apply to ALL brain-writing skills: | "verify this academic claim", "check this study", "academic verify", "validate citation", "is this study real" | `skills/academic-verify/SKILL.md` | | "make pdf from brain", "brain pdf", "convert brain page to pdf", "publish this page as pdf", "export brain page" | `skills/brain-pdf/SKILL.md` | | "voice note", "ingest this voice memo", "transcribe and file", "voice note ingest", "save this audio note" | `skills/voice-note-ingest/SKILL.md` | +| "add a page type", "add a type to my schema", "schema author", "schema mutate", "schema pack add", "my brain has untyped pages", "propose new types from my corpus", "backfill page types", "evolve my schema", "researcher type", "make X an expert type" (dispatcher for: gbrain schema active/list/show/validate/graph/lint/stats/explain/use/downgrade/reload/init/fork/edit/diff/add-type/remove-type/update-type/add-alias/remove-alias/add-prefix/remove-prefix/add-link-type/remove-link-type/set-extractable/set-expert-routing/detect/suggest/review-candidates/review-orphans/sync) | `skills/schema-author/SKILL.md` | diff --git a/skills/conventions/schema-evolution.md b/skills/conventions/schema-evolution.md new file mode 100644 index 000000000..35459a8b2 --- /dev/null +++ b/skills/conventions/schema-evolution.md @@ -0,0 +1,114 @@ +# Convention: schema evolution — when to add a type vs alias vs prefix + +Cross-cutting convention for any skill that proposes a change to the +active schema pack. Read first before invoking `schema-author`. The +goal: keep the pack small enough that an agent can hold the whole type +graph in its head, but expressive enough that custom domains +(research, legal, founder ops) get first-class types. + +## Decision tree + +``` +You see a cluster of pages that share a domain meaning. + │ + ▼ +How many pages in the cluster? + │ + ┌─────┴───────┬──────────────┐ + ▼ ▼ ▼ + <20 20-100 100+ + │ │ │ + ▼ ▼ ▼ +One-off. Big enough. First-class. +Don't pack- Add an alias Add a new +codify. to an existing page_type with + type OR a its own prefix, +Use the narrow prefix primitive, and +nearest branch. flags. +existing +type + +frontmatter +tag. +``` + +### Concrete examples + +**One-off (don't add to pack):** +> "I have 3 pages under `2026-projects/skunkworks-spec/`. Should I add +> a `skunkworks` type?" + +No. Three pages doesn't justify a permanent pack entry. Type these as +the nearest existing match (`concept` or `note`) and use a frontmatter +`project:` tag. If the cluster grows to 20+, revisit. + +**20-100 pages — alias OR narrow prefix:** +> "I have 50 pages under `people/researchers/` that overlap with my +> `person` type. Should I add a `researcher` type?" + +Two valid options: +1. **Alias on `person`** — `add-alias person researcher`. Closure + queries for `researcher` will surface `person` rows too. +2. **New type sharing the `entity` primitive** — `add-type researcher + --primitive entity --prefix people/researchers/`. Distinct type, can + be marked `--extractable` or `--expert` independently. + +Pick alias when researchers are people first, researchers second +(they share enrichment rules, expert-routing semantics, link verbs). +Pick new type when researcher-specific behavior diverges (different +extractable rules, different link verbs, different rubric). + +**100+ pages — first-class type:** +> "I have 4000 pages under `meetings/`. I want them typed as `meeting`, +> not the legacy default `note`." + +Add the type: +``` +gbrain schema add-type meeting \ + --primitive temporal \ + --prefix meetings/ \ + --extractable +gbrain schema sync --apply +``` + +The `sync --apply` backfills all 4000 pages. From here forward, +imports under `meetings/` infer `meeting` type via the pack. + +## Don'ts + +- **Don't add a type for a directory you imported once for triage.** + Pack types are permanent decisions; one-time imports are not. +- **Don't add a type just to silence `dead_prefixes` in `schema stats`.** + A dead prefix is a *signal* that the prefix is mis-declared or the + corpus moved. Remove the prefix or migrate the content, don't add an + empty type. +- **Don't promote a candidate from `schema suggest` without verifying + the path prefix matches real content.** The suggester is heuristic; + it can propose types that overlap existing ones. Run `lint --with-db` + before `add-type` to catch prefix collisions pre-write. +- **Don't add `--expert` to a type that has no `path_prefixes`.** The + `expert_routing_without_prefix` lint rule warns about this exact + shape: an expert-routed type with no prefix never matches a put_page + inference, so `whoknows` silently never surfaces it. +- **Don't mutate `gbrain-base` or `gbrain-recommended`.** Fork first. + +## When to remove a type + +Removing a type is RARE. Only do it when: +1. The type was added in error (typo, premature abstraction). +2. The corpus the type was meant for has been migrated to a different + type. +3. The type is dangling (no `path_prefixes` actually match pages, no + queries reference it, no other type's aliases/link_types reference it). + +`remove-type` is guarded by the `STILL_REFERENCED` check (codex C14): if +ANY other type's aliases / enrichable_types / link_types / frontmatter_links +references the target, the remove fails loud with the reference list. +Break those references first. + +## When to commit the pack + +If your pack lives in source control (`~/.gbrain/schema-packs//` +is a git repo), commit after every batch of mutations. The +`mutation_count_anomaly` lint rule warns at >50 mutations in 7 days — +that's the hint to start committing rather than relying on disk-only +state. diff --git a/skills/manifest.json b/skills/manifest.json index 6bbe8c9c7..50907d49c 100644 --- a/skills/manifest.json +++ b/skills/manifest.json @@ -144,6 +144,11 @@ "path": "minion-orchestrator/SKILL.md", "description": "Unified Minions skill for deterministic shell jobs and LLM subagent orchestration. Submit, monitor, steer, pause/resume, replay. Replaces the older gbrain-jobs routing intent and sessions_spawn for durable observable background work." }, + { + "name": "schema-author", + "path": "schema-author/SKILL.md", + "description": "Evolve the active schema pack. Add page types, propose new types from a corpus scan, backfill page.type via sync. Wraps the 14 gbrain schema CLI verbs + 9 MCP ops shipped in v0.40.7.0." + }, { "name": "skillify", "path": "skillify/SKILL.md", diff --git a/skills/schema-author/SKILL.md b/skills/schema-author/SKILL.md new file mode 100644 index 000000000..67de503af --- /dev/null +++ b/skills/schema-author/SKILL.md @@ -0,0 +1,305 @@ +--- +name: schema-author +description: Evolve your brain's schema pack. Add page types, propose new ones from corpus scans, backfill page.type on existing pages, audit pack health. Triggers when an agent notices untyped pages, custom domains needing typed entities (researcher, contract, deposition), or wants to see what types the pack declares. +tools: + - gbrain schema active + - gbrain schema list + - gbrain schema stats + - gbrain schema review-orphans + - gbrain schema detect + - gbrain schema suggest + - gbrain schema lint + - gbrain schema graph + - gbrain schema explain + - gbrain schema fork + - gbrain schema use + - gbrain schema add-type + - gbrain schema remove-type + - gbrain schema update-type + - gbrain schema add-alias + - gbrain schema remove-alias + - gbrain schema add-prefix + - gbrain schema remove-prefix + - gbrain schema add-link-type + - gbrain schema remove-link-type + - gbrain schema set-extractable + - gbrain schema set-expert-routing + - gbrain schema sync + - gbrain schema reload + - mcp:get_active_schema_pack + - mcp:list_schema_packs + - mcp:schema_stats + - mcp:schema_lint + - mcp:schema_graph + - mcp:schema_explain_type + - mcp:schema_review_orphans + - mcp:schema_apply_mutations + - mcp:reload_schema_pack +triggers: + - "add a page type" + - "add a type to my schema" + - "my brain has untyped pages" + - "schema isn't matching my notes" + - "propose new types from my corpus" + - "backfill page types" + - "evolve my schema" + - "extend the schema pack" + - "create a custom type for" + - "researcher type" + - "make X an expert type" + - "schema pack add" + - "schema mutate" + - "schema sync" + - "schema author" +brain_first: exempt +writes_pages: [] +--- + +# schema-author — evolve your schema pack + +## Non-goals (use these other skills instead) + +This skill AUTHORS the schema pack (adds page types, link verbs, prefixes, +flags). For these adjacent jobs, route elsewhere: + +- **Filing one specific page** → `skills/brain-taxonomist/SKILL.md`. Brain- + taxonomist routes at WRITE TIME ("where does this note go?"). schema-author + changes the rules at AUTHORING TIME ("what types and prefixes exist?"). +- **Schema-check as part of EIIRP iteration** → `skills/eiirp/SKILL.md` + already has a schema-check phase. Don't duplicate. +- **Just looking up a type's settings** → `gbrain schema explain ` + directly. This skill is for CHANGING the pack, not READING from it. +- **Querying who knows about X** → `skills/expert-routing/SKILL.md` (or + `gbrain whoknows` directly). schema-author makes a type expert-routable; + it does not run the query. + +## Convention + +> **Convention:** see [conventions/brain-first.md](../conventions/brain-first.md) for the lookup chain (search → query → get_page → external). + +> **Convention:** see [conventions/schema-evolution.md](../conventions/schema-evolution.md) for "when to add a type vs alias vs prefix" — the heuristic. + +## When to invoke + +Invoke when the user (or a sibling skill) says any of: +- "Add a `researcher` type to my schema" +- "I have 4000 untyped pages under `meetings/`" +- "My brain doesn't know that `journal-article` is a type" +- "Set `paper` to be extractable" +- "Propose types from what I've ingested" +- "Sync the new types to backfill existing pages" + +DON'T invoke for "where does THIS note go" (use brain-taxonomist) or +"who knows about X" (use expert-routing / `gbrain whoknows`). + +## Tutorial + vision + +- **Why this matters:** [`docs/what-schemas-unlock.md`](../../docs/what-schemas-unlock.md) — 7 killer use cases (4000 invisible meetings made queryable, founder ops brain, research brain, legal brain, team brain, agent-as-co-curator) plus the structural argument for why types matter at query time. Read this before pitching schema authoring to a user — it's the doc that explains the difference between a pile of notes and a brain with structure. +- **5-minute walkthrough:** [`docs/schema-author-tutorial.md`](../../docs/schema-author-tutorial.md) — fork the bundled pack, add a researcher type, sync, prove the T1.5 wiring via `gbrain whoknows`. Use placeholder pages so it runs against any brain without affecting real content. + +## Workflow + +### Phase 1 — Brain (know which pack is active) + +``` +gbrain schema active --json +``` + +Output gives you `pack_name`, `version`, `sha8`, `page_types_count`, `source_tier`. +If `source_tier === "default"`, the user is on bundled `gbrain-base` and any +mutation will need a fork first (Phase 4). + +### Phase 2 — Assess (what does the current pack cover?) + +``` +gbrain schema stats --json +``` + +Returns per-type page counts, untyped count, and `dead_prefixes` (pack- +declared prefixes with zero matching pages — probable mis-declarations). +If coverage < 90%, there's untyped content worth typing. + +``` +gbrain schema review-orphans --limit 50 --json +``` + +Untyped pages drilldown. Look for shared path prefixes (e.g. "12 of these +are under `research/papers/`") — those are candidates for a new type. + +### Phase 3 — Propose (what types should the pack add?) + +``` +gbrain schema detect --json +``` + +Clusters pages by `source_path` and proposes candidate types. Heuristic +only (no LLM call). + +``` +gbrain schema suggest --json +``` + +LLM-refined candidates with confidence scores. Use the top-3 hit rate as +the signal for which to promote. + +### Phase 4 — Apply (mutate the pack) + +If the active pack is bundled (`gbrain-base` or `gbrain-recommended`), +fork it first: + +``` +gbrain schema fork gbrain-base mine +gbrain schema use mine +``` + +Then add the types one at a time: + +``` +gbrain schema add-type researcher \ + --primitive entity \ + --prefix people/researchers/ \ + --extractable \ + --expert +``` + +For complex multi-mutation refactors (e.g. add a type AND the link verb +that points to it), agents reaching this surface over MCP can use the +batched `schema_apply_mutations` op: + +```jsonl +{"op": "add_type", "name": "researcher", "primitive": "entity", "prefix": "people/researchers/", "extractable": true, "expert_routing": true} +{"op": "add_type", "name": "paper", "primitive": "annotation", "prefix": "research/papers/", "extractable": true} +{"op": "add_link_type", "name": "authored", "inference": {"page_type": "researcher", "target_type": "paper"}} +``` + +Validate before sync: + +``` +gbrain schema lint --with-db +``` + +The `--with-db` flag opts into the 2 DB-aware rules +(`extractable_empty_corpus`, `mutation_count_anomaly`) that detect +mis-declared types you'd otherwise discover only at runtime. + +### Phase 5 — Sync (backfill existing pages with the new types) + +Dry-run first: + +``` +gbrain schema sync --json +``` + +Returns per-prefix `would_apply` counts + sample slugs. If the numbers +look right: + +``` +gbrain schema sync --apply +``` + +Chunked UPDATE in 1000-row batches; never wedges concurrent writers. +Idempotent on re-run (second `--apply` finds nothing to backfill). + +### Phase 6 — Verify + +``` +gbrain schema stats --json +``` + +Coverage should be ≥95% now. Spot-check the new type: + +``` +gbrain whoknows "machine learning" +``` + +If `researcher` was declared `--expert`, results should include +researcher-typed pages. (The pack-aware wiring at the query path was +added in v0.40.6.0 — pre-v0.40.6 brains silently ignored custom +expert-routed types.) + +### Phase 7 — Commit (preserve the change) + +If the pack is in source control, commit: + +``` +cd ~/.gbrain/schema-packs/mine +git add pack.json +git commit -m "schema: add researcher + paper types + authored link" +git push +``` + +If the brain daemon is running (`gbrain serve --http`), other processes +pick up the change within 1 second (stat-mtime TTL gate in +loadActivePack — v0.40.6.0 closed the cross-process invalidation gap). + +## Outputs + +- Mutated pack file at `~/.gbrain/schema-packs//pack.{json,yaml}`. +- Audit row in `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl` per mutation. +- `pages.type` backfilled on matching rows after `sync --apply`. +- Query paths (`whoknows`, `find_experts`) now route through the new + expert types. + +## Contract + +- **Inputs:** a natural-language request that names a type / prefix / link verb / flag change, OR the result of `gbrain schema review-orphans` showing untyped pages that need a new type. +- **Outputs:** mutated pack file at `~/.gbrain/schema-packs//pack.{json,yaml}` + an audit row in `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl` + (if `sync --apply` ran) backfilled `pages.type` on matching rows. +- **Side effects:** invalidates the in-process pack cache + the query cache for the source. Other processes pick up the change within 1 second (stat-mtime TTL). +- **Idempotency:** every primitive is idempotent. `add-alias`/`add-prefix` no-op on duplicate; `sync --apply` finds nothing to update on second run. +- **Trust:** CLI = local trust (no scope check). MCP = OAuth `admin` scope (write ops). Audit log captures `actor: mcp:` per mutation. +- **Atomicity:** every mutation is wrapped in `withMutation`'s atomic write (`.tmp + fsync + rename`) + per-pack `O_CREAT|O_EXCL` lock. Crash mid-write leaves the original file untouched. + +## Anti-Patterns + +- **Don't mutate `gbrain-base` or `gbrain-recommended`.** Fork first (`gbrain schema fork gbrain-base mine`). These are bundled packs; edits would be lost on upgrade. The mutation primitives refuse with `PACK_READONLY`. +- **Don't add a type for a directory you imported once for triage.** Pack types are permanent decisions; one-time imports are not. See `skills/conventions/schema-evolution.md` for the <20-pages-don't-pack-codify heuristic. +- **Don't add `--expert` to a type with no `path_prefixes`.** The `expert_routing_without_prefix` lint warns about this — expert-routed types with no prefix never match a put_page inference, so `whoknows` silently never surfaces them. +- **Don't promote a `schema suggest` candidate without verifying the prefix matches real content.** Run `lint --with-db` before `add-type` to catch prefix collisions pre-write. +- **Don't conflate "filing one page" with "evolving the schema."** Filing routes via `brain-taxonomist`; schema-author is for authoring the type taxonomy itself. The Non-goals section above names the boundary. +- **Don't skip the dry-run before `sync --apply`.** Always run `sync` first to see `would_apply` counts + sample slugs. A pack prefix that matches 50,000 pages is recoverable but slow; verifying first is cheap. +- **Don't remove a type without checking references.** `remove-type` refuses with `STILL_REFERENCED` if another type's `aliases` / `enrichable_types` / `link_types` / `frontmatter_links` references it. Break the references first; don't add `--force`. + +## Output Format + +When invoked, this skill produces structured output suitable for both human + JSON consumption: + +**Per-mutation result (JSON):** +```json +{"schema_version": 1, "pack": "mine", "path": "/Users/.../pack.json", "format": "json", "prev_sha8": "a1b2c3d4", "new_sha8": "e5f6g7h8"} +``` + +**Per-batch result (from `schema_apply_mutations` MCP op):** +```json +{"schema_version": 1, "pack": "mine", "batch_id": "batch-1716491400-abc123", "mutations_applied": 3, "results": [{...}, {...}, {...}]} +``` + +**Stats JSON (per-source + aggregate + dead-prefix hints):** +```json +{"schema_version": 1, "pack_identity": "mine@1.0.0+abc12345", "aggregate": {"total_pages": 4823, "typed_pages": 4710, "untyped_pages": 113, "coverage": 0.9766, "by_type": [{"type": "person", "count": 2104}, ...]}, "per_source": [...], "dead_prefixes": [{"type": "researcher", "prefix": "people/researchers/"}]} +``` + +**Sync dry-run JSON:** +```json +{"schema_version": 1, "apply": false, "pack_identity": "mine@1.0.0+abc12345", "per_prefix": [{"type": "meeting", "prefix": "meetings/", "would_apply": 4000, "sample_slugs": ["meetings/2026-01-01-foo", ...], "dead_prefix": false, "applied": 0}], "total_would_apply": 4000, "total_applied": 0} +``` + +**Human output (the agent's final summary):** +- One line per mutation: `Pack: ()` and `Sha8: → ` +- Stats: total pages, typed %, untyped count, per-type breakdown, dead-prefix list +- Sync: per-prefix `would_apply`/`applied` count + sample slugs in dry-run mode + +On failure, the error envelope follows the standard `StructuredAgentError` shape from `src/core/errors.ts`: `{error, code, message, details?}`. Codes from the mutation primitives: `PACK_NOT_FOUND`, `PACK_READONLY`, `PACK_CORRUPT`, `TYPE_EXISTS`, `TYPE_NOT_FOUND`, `INVALID_PRIMITIVE`, `INVALID_RESULT`, `IO_ERROR`, `STILL_REFERENCED`, `LOCK_BUSY`. + +## Failure modes + +- `PACK_READONLY` → you tried to mutate `gbrain-base` or `gbrain-recommended`. Fork first. +- `INVALID_RESULT` → the mutation would create a dangling reference or + prefix collision. The pre-write lint gate caught it. Read the error + message; the lint rule name names the problem. +- `STILL_REFERENCED` → you tried to remove a type that another type's + `aliases` / `enrichable_types` / `link_types` / `frontmatter_links` + references. The error names every reference. Remove those first. +- `LOCK_BUSY` → another process is mid-mutation. Wait 30s and retry, or + pass `--force` if you know the holder is wedged. +- `permission_denied` (MCP only) → your OAuth client doesn't have `admin` + scope. Re-register with `gbrain auth register-client --scopes admin`. diff --git a/src/cli.ts b/src/cli.ts index 36b0ce6a7..568c73f8a 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -56,6 +56,10 @@ const CLI_ONLY_SELF_HELP = new Set([ // the generic short-circuit so the destructive-action warning text // reaches the user. 'reinit-pglite', + // v0.40.6.0 Schema Cathedral v3 — `gbrain schema --help` should hit + // schema.ts printHelp() with the full 22+ verb taxonomy, not the + // generic short-circuit's one-line stub. + 'schema', ]); async function main() { diff --git a/src/commands/schema.ts b/src/commands/schema.ts index 62e5f5cff..ab7eef76f 100644 --- a/src/commands/schema.ts +++ b/src/commands/schema.ts @@ -1,35 +1,52 @@ -// v0.38 Phase C — `gbrain schema` CLI surface. +// `gbrain schema` CLI surface. // -// Five essential subcommands ship in v0.38: -// gbrain schema active — show resolved pack + tier source -// gbrain schema list — list installed packs -// gbrain schema show [] — pretty-print manifest -// gbrain schema validate [] — validate manifest shape -// gbrain schema use — activate pack (file-plane) +// The active schema pack drives type inference, link verbs, expert +// routing, extractable types, enrichment rubrics, and per-source +// closure for search. See `src/core/schema-pack/load-active.ts` for +// the boundary helper that all engines + operations consume. // -// Deferred to v0.39+: -// init, fork, edit, diff, detect, suggest, review-candidates, -// review-orphans, graph, lint, explain -// -// The active pack drives type inference, link verbs, expert routing, -// extractable types, enrichment rubrics, and per-source closure for -// search. See `src/core/schema-pack/load-active.ts` for the boundary -// helper that all engines + operations consume. +// Verbs grouped by lifecycle: +// Inspection: active, list, show, validate, graph, lint, +// stats, explain, usage +// Activation: use, downgrade, reload +// Authoring: init, fork, edit, diff, add-type, remove-type, +// update-type, add-alias, remove-alias, +// add-prefix, remove-prefix, add-link-type, +// remove-link-type, set-extractable, +// set-expert-routing +// Discovery + repair: detect, suggest, review-candidates, +// review-orphans, sync import { existsSync, readdirSync, readFileSync, writeFileSync, mkdirSync } from 'node:fs'; import { dirname, join } from 'node:path'; import { + addAliasToType, + addLinkTypeToPack, + addPrefixToType, + addTypeToPack, + invalidatePackCache, loadActivePack, + removeAliasFromType, + removeLinkTypeFromPack, + removePrefixFromType, + removeTypeFromPack, resolveActivePackNameOnly, loadPackFromFile, parseSchemaPackManifest, + runStatsCore, + runSyncCore, SchemaPackManifestError, SchemaPackLoaderError, + SchemaPackMutationError, + setExpertRoutingOnType, + setExtractableOnType, UnknownPackError, + updateTypeOnPack, __setPackLocatorForTests, _resetPackLocatorForTests, } from '../core/schema-pack/index.ts'; -import type { SchemaPackManifest } from '../core/schema-pack/manifest-v1.ts'; +import type { SchemaPackManifest, PackPrimitive } from '../core/schema-pack/manifest-v1.ts'; +import { PACK_PRIMITIVES } from '../core/schema-pack/manifest-v1.ts'; import { gbrainPath, loadConfig, configPath } from '../core/config.ts'; export async function runSchema(args: string[]): Promise { @@ -53,6 +70,20 @@ export async function runSchema(args: string[]): Promise { case 'review-orphans': return runReviewOrphansCmd(args.slice(1)); case 'downgrade': return runDowngradeCmd(args.slice(1)); case 'usage': return runUsageCmd(args.slice(1)); + case 'stats': return runStatsCmd(args.slice(1)); + case 'sync': return runSyncCmd(args.slice(1)); + case 'reload': return runReloadCmd(args.slice(1)); + case 'add-type': return runAddTypeCmd(args.slice(1)); + case 'remove-type': return runRemoveTypeCmd(args.slice(1)); + case 'update-type': return runUpdateTypeCmd(args.slice(1)); + case 'add-alias': return runAddAliasCmd(args.slice(1)); + case 'remove-alias': return runRemoveAliasCmd(args.slice(1)); + case 'add-prefix': return runAddPrefixCmd(args.slice(1)); + case 'remove-prefix': return runRemovePrefixCmd(args.slice(1)); + case 'add-link-type': return runAddLinkTypeCmd(args.slice(1)); + case 'remove-link-type': return runRemoveLinkTypeCmd(args.slice(1)); + case 'set-extractable': return runSetExtractableCmd(args.slice(1)); + case 'set-expert-routing': return runSetExpertRoutingCmd(args.slice(1)); case undefined: case '--help': case '-h': @@ -67,18 +98,53 @@ export async function runSchema(args: string[]): Promise { function printHelp(): void { console.log(`gbrain schema — active schema pack management -Subcommands: +Inspection: active Show resolved pack + which tier provided it list List installed packs (bundled + ~/.gbrain/schema-packs/) show [] Pretty-print a manifest (default: active pack) - validate [] Validate manifest shape against the v0.38 schema + validate [] Validate manifest shape against the v1 schema + graph Show type/primitive graph with link-verb edges + lint [] Lint a pack for duplicates, dangling refs, etc. + stats [--source ] Per-type page counts + typed-coverage from the DB + explain Print resolved settings for a single type + usage [--since N(d|w|m)] CLI invocation telemetry summary + +Activation: use Activate pack (writes ~/.gbrain/config.json schema_pack) + downgrade [--to ] Restore the previous active pack + reload [--pack ] Flush the in-process pack cache; --pack scopes -v0.38 ships the schema-pack engine + these five inspection/activation -commands. detect, suggest, init, fork, edit, graph, lint, explain, -review-candidates, review-orphans, and diff land in v0.39. +Authoring (v0.40.6.0): + init Scaffold a new pack (extends gbrain-base) + fork Copy a pack to a new editable name + edit Print the on-disk pack file path + diff Compare page_type sets across two packs -Resolution chain (D13 7-tier, tier 1 trust-gated): + add-type --primitive

--prefix

+ [--extractable] [--expert] [--alias ]* [--pack ] + remove-type [--pack ] + update-type [--extractable BOOL] [--expert BOOL] [--primitive P] [--pack ] + add-alias [--pack ] + remove-alias [--pack ] + add-prefix [--pack ] + remove-prefix [--pack ] + add-link-type [--inverse ] [--page-type ] [--target-type ] [--pack ] + remove-link-type [--pack ] + set-extractable [--pack ] + set-expert-routing [--pack ] + +Discovery + repair: + detect Cluster pages by source_path → candidate page_types + suggest Heuristic refinement on detect output + review-candidates Review disk-derived candidates; promote with --apply + review-orphans List pages with no active-pack type match + sync [--apply] Backfill page.type for rows matching pack prefixes + (dry-run by default; chunked UPDATE on apply) + +All new verbs accept --json. Verbs scoped by source accept --source . +Pass --force to bypass per-pack lock contention on writes. + +Resolution chain (7-tier, tier 1 trust-gated): 1. Per-call --schema-pack flag (CLI only) 2. GBRAIN_SCHEMA_PACK env var 3. Per-source DB config schema_pack.source. @@ -361,11 +427,16 @@ async function withConnectedEngine(fn: (engine: import('../core/engine.ts').B const { createEngine } = await import('../core/engine-factory.ts'); const cfg = loadConfig() ?? {}; const engineKind = (cfg as { engine?: string }).engine === 'postgres' ? 'postgres' : 'pglite'; - const engine = await createEngine({ + // PR #1321 (closed) defensive fix retained: build the EngineConfig once and + // pass it to BOTH createEngine and engine.connect. The factory captures + // config at construction; explicit re-pass at connect() is defense in depth + // against future engine implementations that read URL from connect-time. + const connectConfig: import('../core/types.ts').EngineConfig = { engine: engineKind, database_url: (cfg as { database_url?: string }).database_url, - }); - await engine.connect({}); + }; + const engine = await createEngine(connectConfig); + await engine.connect(connectConfig); try { return await fn(engine); } finally { @@ -621,6 +692,7 @@ async function runGraphCmd(args: string[]): Promise { async function runLintCmd(args: string[]): Promise { const { json, positional } = parseFlags(args); + const withDb = args.includes('--with-db'); const name = positional[0]; const cfg = loadConfig(); let pack: SchemaPackManifest | null; @@ -634,25 +706,33 @@ async function runLintCmd(args: string[]): Promise { console.error(`Pack not found: ${name}`); process.exit(1); } - const warnings: string[] = []; - const seenTypeNames = new Set(); - for (const t of pack.page_types) { - if (seenTypeNames.has(t.name)) warnings.push(`duplicate type name: ${t.name}`); - seenTypeNames.add(t.name); - if (!t.path_prefixes || t.path_prefixes.length === 0) { - warnings.push(`type \`${t.name}\` has no path_prefixes — unreachable by inferType`); - } - } + // v0.40.6.0 Phase 5: swap basic 2-rule check for the rich 11-rule lint + // suite from Phase 1.5. File-plane rules run by default; --with-db + // opts into extractable_empty_corpus + mutation_count_anomaly which + // need an engine connection. + const { runAllLintRules } = await import('../core/schema-pack/lint-rules.ts'); + const report = withDb + ? await withConnectedEngine(async (engine) => runAllLintRules(pack!, { engine })) + : await runAllLintRules(pack); if (json) { - console.log(JSON.stringify({ schema_version: 1, pack: pack.name, warnings }, null, 2)); + console.log(JSON.stringify({ schema_version: 1, pack: pack.name, ...report }, null, 2)); + if (!report.ok) process.exit(1); return; } - if (!warnings.length) { + if (report.ok && report.warnings.length === 0) { console.log(`OK — pack \`${pack.name}\` lint clean.`); return; } console.log(`Pack \`${pack.name}\` lint:`); - for (const w of warnings) console.log(` warn: ${w}`); + for (const e of report.errors) { + console.log(` ERROR (${e.rule}): ${e.message}`); + if (e.hint) console.log(` hint: ${e.hint}`); + } + for (const w of report.warnings) { + console.log(` warn (${w.rule}): ${w.message}`); + if (w.hint) console.log(` hint: ${w.hint}`); + } + if (!report.ok) process.exit(1); } async function runExplainCmd(args: string[]): Promise { @@ -793,3 +873,294 @@ function parseSinceDays(s: string): number { default: return n; } } + +// ────────────────────────────────────────────────────────────────────── +// v0.40.6.0 Schema Cathedral v3 — 14 new authoring + DB-aware verbs. +// +// All handlers thin-wrap a pure core function from src/core/schema-pack/ +// (see Phase 2 mutate.ts, Phase 3 stats.ts/sync.ts). CLI prints to +// stdout (text or --json) and exits with a meaningful code. +// ────────────────────────────────────────────────────────────────────── + +function parseBool(raw: string | undefined): boolean | null { + if (raw === undefined) return null; + const lower = raw.trim().toLowerCase(); + if (lower === 'true' || lower === '1' || lower === 'yes') return true; + if (lower === 'false' || lower === '0' || lower === 'no') return false; + return null; +} + +function pickPackName(parsed: { positional?: string[] }, args: string[]): string { + // Honor --pack before falling back to the active pack. + for (let i = 0; i < args.length; i++) { + if (args[i] === '--pack') return args[i + 1] ?? ''; + if (args[i]?.startsWith('--pack=')) return args[i]!.slice('--pack='.length); + } + const cfg = loadConfig(); + const resolution = resolveActivePackNameOnly({ cfg, remote: false }); + return resolution.pack_name; +} + +function emitMutateResult(result: { pack: string; format: string; prev_sha8: string; new_sha8: string }, json: boolean): void { + if (json) { + console.log(JSON.stringify({ schema_version: 1, ...result })); + return; + } + console.log(`Pack: ${result.pack} (${result.format})`); + console.log(`Sha8: ${result.prev_sha8} → ${result.new_sha8}`); +} + +function handleMutationError(err: unknown): never { + if (err instanceof SchemaPackMutationError) { + console.error(`Error: ${err.message}`); + if (err.code === 'PACK_READONLY') { + console.error(' Hint: fork the pack first, then mutate the fork.'); + } else if (err.code === 'STILL_REFERENCED') { + const refs = (err.details?.references as string[] | undefined) ?? []; + if (refs.length > 0) console.error(` Still referenced by: ${refs.join(', ')}`); + } + process.exit(1); + } + throw err; +} + +async function runStatsCmd(args: string[]): Promise { + const { json, source } = parseFlags(args); + await withConnectedEngine(async (engine) => { + const ctx = { engine, config: {}, logger: console, dryRun: false, remote: false, sourceId: source } as never; + const result = await runStatsCore(ctx, source ? { sourceId: source } : {}); + if (json) { + console.log(JSON.stringify(result, null, 2)); + return; + } + console.log(`Pack: ${result.pack_identity ?? '(no pack loaded)'}`); + console.log(`Total pages: ${result.aggregate.total_pages}`); + console.log(`Typed: ${result.aggregate.typed_pages} (${(result.aggregate.coverage * 100).toFixed(1)}%)`); + console.log(`Untyped: ${result.aggregate.untyped_pages}`); + if (result.aggregate.by_type.length > 0) { + console.log(`\nBy type:`); + for (const t of result.aggregate.by_type) { + console.log(` ${t.type.padEnd(20)} ${t.count}`); + } + } + if (result.per_source.length > 1) { + console.log(`\nPer source:`); + for (const s of result.per_source) { + console.log(` ${s.source_id.padEnd(20)} total=${s.total_pages} typed=${s.typed_pages} coverage=${(s.coverage * 100).toFixed(1)}%`); + } + } + if (result.dead_prefixes.length > 0) { + console.log(`\nDead prefixes (declared but 0 pages):`); + for (const dp of result.dead_prefixes) { + console.log(` ${dp.type.padEnd(20)} ${dp.prefix}`); + } + } + }); +} + +async function runSyncCmd(args: string[]): Promise { + const apply = args.includes('--apply'); + const { json, source } = parseFlags(args); + await withConnectedEngine(async (engine) => { + const ctx = { engine, config: {}, logger: console, dryRun: false, remote: false, sourceId: source } as never; + const result = await runSyncCore(ctx, { + apply, + sourceId: source, + onProgress: (info) => { + if (!json) process.stderr.write(` [sync] ${info.type} ${info.prefix} → ${info.appliedSoFar} applied\n`); + }, + }); + if (json) { + console.log(JSON.stringify(result, null, 2)); + return; + } + console.log(`Pack: ${result.pack_identity ?? '(no pack loaded)'}`); + console.log(`Mode: ${apply ? 'APPLY' : 'DRY-RUN'}`); + for (const p of result.per_prefix) { + const marker = p.dead_prefix ? ' (dead prefix — no matching pages)' : ''; + console.log(` ${p.type.padEnd(20)} ${p.prefix.padEnd(30)} would_apply=${p.would_apply} applied=${p.applied}${marker}`); + if (p.sample_slugs.length > 0 && !apply) { + console.log(` sample: ${p.sample_slugs.slice(0, 3).join(', ')}${p.sample_slugs.length > 3 ? '...' : ''}`); + } + } + console.log(`\nTotal: would_apply=${result.total_would_apply} applied=${result.total_applied}`); + if (!apply && result.total_would_apply > 0) { + console.log(`\nRun \`gbrain schema sync --apply\` to backfill page.type.`); + } + }); +} + +function runReloadCmd(args: string[]): void { + const { json } = parseFlags(args); + let packName: string | undefined; + for (let i = 0; i < args.length; i++) { + if (args[i] === '--pack') { packName = args[i + 1]; break; } + if (args[i]?.startsWith('--pack=')) { packName = args[i]!.slice('--pack='.length); break; } + } + const result = invalidatePackCache(packName); + if (json) { + console.log(JSON.stringify({ schema_version: 1, ...result })); + return; + } + if (result.invalidated.length === 0) { + console.log('No cached packs to flush.'); + } else { + console.log(`Flushed: ${result.invalidated.join(', ')}`); + } +} + +async function runAddTypeCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const positional = args.filter((a) => !a.startsWith('--')); + const name = positional[0]; + if (!name) { console.error('Usage: gbrain schema add-type --primitive

--prefix

'); process.exit(2); } + let primitive: string | undefined; + let prefix: string | undefined; + let extractable = false; + let expert = false; + const aliases: string[] = []; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--primitive') primitive = args[++i]; + else if (a?.startsWith('--primitive=')) primitive = a.slice('--primitive='.length); + else if (a === '--prefix') prefix = args[++i]; + else if (a?.startsWith('--prefix=')) prefix = a.slice('--prefix='.length); + else if (a === '--extractable') extractable = true; + else if (a === '--expert' || a === '--expert-routing') expert = true; + else if (a === '--alias') aliases.push(args[++i]!); + else if (a?.startsWith('--alias=')) aliases.push(a.slice('--alias='.length)); + } + if (!primitive || !PACK_PRIMITIVES.includes(primitive as PackPrimitive)) { + console.error(`--primitive must be one of ${PACK_PRIMITIVES.join('|')}`); + process.exit(2); + } + if (!prefix) { console.error('--prefix is required (e.g. --prefix people/researchers/)'); process.exit(2); } + try { + const result = await addTypeToPack(packName, { + name, primitive: primitive as PackPrimitive, prefix, + extractable, expertRouting: expert, aliases, + }); + emitMutateResult(result, json); + } catch (e) { handleMutationError(e); } +} + +async function runRemoveTypeCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const name = args.filter((a) => !a.startsWith('--'))[0]; + if (!name) { console.error('Usage: gbrain schema remove-type '); process.exit(2); } + try { emitMutateResult(await removeTypeFromPack(packName, name), json); } + catch (e) { handleMutationError(e); } +} + +async function runUpdateTypeCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const name = args.filter((a) => !a.startsWith('--'))[0]; + if (!name) { console.error('Usage: gbrain schema update-type [--extractable BOOL] [--expert BOOL] [--primitive P]'); process.exit(2); } + const patch: Record = {}; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--extractable') { + const v = parseBool(args[++i]); + if (v === null) { console.error('--extractable requires true|false'); process.exit(2); } + patch.extractable = v; + } else if (a === '--expert' || a === '--expert-routing') { + const v = parseBool(args[++i]); + if (v === null) { console.error('--expert requires true|false'); process.exit(2); } + patch.expert_routing = v; + } else if (a === '--primitive') { + patch.primitive = args[++i]; + } + } + try { emitMutateResult(await updateTypeOnPack(packName, { name, patch }), json); } + catch (e) { handleMutationError(e); } +} + +async function runAddAliasCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const pos = args.filter((a) => !a.startsWith('--')); + if (pos.length < 2) { console.error('Usage: gbrain schema add-alias '); process.exit(2); } + try { emitMutateResult(await addAliasToType(packName, pos[0]!, pos[1]!), json); } + catch (e) { handleMutationError(e); } +} + +async function runRemoveAliasCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const pos = args.filter((a) => !a.startsWith('--')); + if (pos.length < 2) { console.error('Usage: gbrain schema remove-alias '); process.exit(2); } + try { emitMutateResult(await removeAliasFromType(packName, pos[0]!, pos[1]!), json); } + catch (e) { handleMutationError(e); } +} + +async function runAddPrefixCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const pos = args.filter((a) => !a.startsWith('--')); + if (pos.length < 2) { console.error('Usage: gbrain schema add-prefix '); process.exit(2); } + try { emitMutateResult(await addPrefixToType(packName, pos[0]!, pos[1]!), json); } + catch (e) { handleMutationError(e); } +} + +async function runRemovePrefixCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const pos = args.filter((a) => !a.startsWith('--')); + if (pos.length < 2) { console.error('Usage: gbrain schema remove-prefix '); process.exit(2); } + try { emitMutateResult(await removePrefixFromType(packName, pos[0]!, pos[1]!), json); } + catch (e) { handleMutationError(e); } +} + +async function runAddLinkTypeCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const name = args.filter((a) => !a.startsWith('--'))[0]; + if (!name) { console.error('Usage: gbrain schema add-link-type [--inverse ] [--page-type ] [--target-type ]'); process.exit(2); } + let inverse: string | undefined; + let pageType: string | undefined; + let targetType: string | undefined; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--inverse') inverse = args[++i]; + else if (a === '--page-type') pageType = args[++i]; + else if (a === '--target-type') targetType = args[++i]; + } + const inference = (pageType || targetType) ? { page_type: pageType, target_type: targetType } : undefined; + try { + emitMutateResult(await addLinkTypeToPack(packName, { name, inverse, inference }), json); + } catch (e) { handleMutationError(e); } +} + +async function runRemoveLinkTypeCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const name = args.filter((a) => !a.startsWith('--'))[0]; + if (!name) { console.error('Usage: gbrain schema remove-link-type '); process.exit(2); } + try { emitMutateResult(await removeLinkTypeFromPack(packName, name), json); } + catch (e) { handleMutationError(e); } +} + +async function runSetExtractableCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const pos = args.filter((a) => !a.startsWith('--')); + if (pos.length < 2) { console.error('Usage: gbrain schema set-extractable '); process.exit(2); } + const v = parseBool(pos[1]); + if (v === null) { console.error('Second argument must be true|false'); process.exit(2); } + try { emitMutateResult(await setExtractableOnType(packName, pos[0]!, v), json); } + catch (e) { handleMutationError(e); } +} + +async function runSetExpertRoutingCmd(args: string[]): Promise { + const { json } = parseFlags(args); + const packName = pickPackName({}, args); + const pos = args.filter((a) => !a.startsWith('--')); + if (pos.length < 2) { console.error('Usage: gbrain schema set-expert-routing '); process.exit(2); } + const v = parseBool(pos[1]); + if (v === null) { console.error('Second argument must be true|false'); process.exit(2); } + try { emitMutateResult(await setExpertRoutingOnType(packName, pos[0]!, v), json); } + catch (e) { handleMutationError(e); } +} diff --git a/src/commands/whoknows.ts b/src/commands/whoknows.ts index 9fa16dd70..edc970506 100644 --- a/src/commands/whoknows.ts +++ b/src/commands/whoknows.ts @@ -331,10 +331,19 @@ export async function runWhoknows( }, { timeoutMs: 30_000 }); results = unpackToolResult(raw); } else { + // v0.40.6.0 T1.5 wiring (D4): consult the active pack for expert + // types. Pack-load failure → empty filter (NOT hardcoded defaults + // per the silent-violation bug class Finding 1.3 closed). Local + // CLI: ctx.remote=false so the trust gate accepts the resolution. + const { loadActivePackBestEffort, expertTypesFromPack } = await import('../core/schema-pack/index.ts'); + const fakeCtx = { engine, config: {}, logger: console, dryRun: false, remote: false, sourceId: undefined } as unknown as import('../core/operations.ts').OperationContext; + const pack = await loadActivePackBestEffort(fakeCtx); + const types = pack ? (expertTypesFromPack(pack.manifest) as PageType[]) : []; results = await findExperts(engine, { topic: parsed.topic, limit: parsed.limit, explain: parsed.explain, + types, }); } diff --git a/src/core/operations.ts b/src/core/operations.ts index 3050f2f39..d69ac7d40 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -2807,10 +2807,17 @@ const find_experts: Operation = { // thread was missing entirely. The op calls findExperts → hybridSearch // internally; without the thread an auth'd src-A whoknows query would // surface src-B people in the rankings. + // v0.40.6.0 T1.5 wiring (D4): consult the active pack for expert + // types; pack-load failure → empty filter (NOT hardcoded defaults + // per the silent-violation bug class Finding 1.3 closed). + const { loadActivePackBestEffort, expertTypesFromPack } = await import('./schema-pack/index.ts'); + const pack = await loadActivePackBestEffort(ctx); + const types = pack ? expertTypesFromPack(pack.manifest) : []; return findExperts(ctx.engine, { topic, limit: typeof p.limit === 'number' ? p.limit : undefined, explain: p.explain === true, + types: types as never, ...sourceScopeOpts(ctx), }); }, @@ -3801,6 +3808,359 @@ async function getRemoteMaxBytes(engine: BrainEngine): Promise { // --- Exports --- +// ────────────────────────────────────────────────────────────────────── +// v0.40.6.0 Schema Cathedral v3 — 9 new MCP ops for the agent on-ramp. +// +// Read ops (scope: read; NOT localOnly) — any read-scope OAuth client. +// Write ops (scope: admin; NOT localOnly per D2) — admin-scope client +// (your OpenClaw and similar remote agents) can author schema packs +// remotely. Audit log captures actor=mcp: on every mutation +// (see src/core/schema-pack/mutate-audit.ts privacy posture per D20). +// +// Per-call schema_pack opt STAYS rejected for remote callers — that +// trust boundary is enforced by op-trust-gate.ts and is separate from +// the localOnly posture (R2 regression preserved). +// ────────────────────────────────────────────────────────────────────── + +const get_active_schema_pack: Operation = { + name: 'get_active_schema_pack', + description: 'v0.40.6.0: cheap identity packet for the active schema pack. Returns {pack_name, version, sha8, page_types_count, link_types_count, primitive_summary, source_tier}. Useful for agents to know which pack they are operating against without paying full manifest load cost.', + params: {}, + scope: 'read', + handler: async (ctx) => { + const { loadActivePack, resolveActivePackNameOnly } = await import('./schema-pack/load-active.ts'); + const { loadConfig } = await import('./config.ts'); + const cfg = loadConfig(); + const sourceOpts: Record = {}; + if (ctx.sourceId) sourceOpts.sourceId = ctx.sourceId; + const resolution = resolveActivePackNameOnly({ cfg, remote: ctx.remote ?? true, ...sourceOpts }); + const pack = await loadActivePack({ cfg, remote: ctx.remote ?? true, ...sourceOpts }); + const primitiveSummary: Record = {}; + for (const t of pack.manifest.page_types) { + primitiveSummary[t.primitive] = (primitiveSummary[t.primitive] ?? 0) + 1; + } + return { + pack_name: pack.manifest.name, + version: pack.manifest.version, + sha8: pack.manifest_sha8, + identity: pack.identity, + page_types_count: pack.manifest.page_types.length, + link_types_count: pack.manifest.link_types.length, + primitive_summary: primitiveSummary, + source_tier: resolution.source, + }; + }, +}; + +const list_schema_packs: Operation = { + name: 'list_schema_packs', + description: 'v0.40.6.0: list installed schema packs (bundled + user-installed). Returns {bundled: string[], installed: string[]}. Read-only directory listing.', + params: {}, + scope: 'read', + handler: async (_ctx) => { + const { existsSync, readdirSync } = await import('node:fs'); + const { join } = await import('node:path'); + const { gbrainPath } = await import('./config.ts'); + const bundled = ['gbrain-base', 'gbrain-recommended']; + const installedDir = gbrainPath('schema-packs'); + const installed: string[] = []; + if (existsSync(installedDir)) { + for (const entry of readdirSync(installedDir)) { + const candidates = ['pack.yaml', 'pack.yml', 'pack.json']; + for (const c of candidates) { + if (existsSync(join(installedDir, entry, c))) { installed.push(entry); break; } + } + } + } + return { bundled, installed }; + }, +}; + +const schema_stats: Operation = { + name: 'schema_stats', + description: 'v0.40.6.0: per-type page counts + typed-coverage from the DB. Returns {schema_version:1, pack_identity, aggregate, per_source, dead_prefixes}. Multi-source aware via ctx.sourceId/allowedSources.', + params: {}, + scope: 'read', + handler: async (ctx) => { + const { runStatsCore } = await import('./schema-pack/stats.ts'); + const scope = sourceScopeOpts(ctx); + const opts: { sourceId?: string; sourceIds?: string[] } = {}; + if (scope.sourceIds && scope.sourceIds.length > 0) opts.sourceIds = scope.sourceIds; + else if (scope.sourceId) opts.sourceId = scope.sourceId; + return runStatsCore(ctx, opts); + }, +}; + +const schema_lint: Operation = { + name: 'schema_lint', + description: 'v0.40.6.0: lint the active (or named) schema pack. File-plane rules only over MCP — the with_db option is rejected for remote callers (DB-aware rules require local CLI). Returns {ok, errors, warnings} structured report.', + params: { + pack: { type: 'string', description: 'Pack name (default: active pack)' }, + }, + scope: 'read', + handler: async (ctx, p) => { + const { runAllLintRules } = await import('./schema-pack/lint-rules.ts'); + const { loadActivePack } = await import('./schema-pack/load-active.ts'); + const { loadConfig, gbrainPath } = await import('./config.ts'); + const { existsSync } = await import('node:fs'); + const { join } = await import('node:path'); + const cfg = loadConfig(); + let manifest; + if (p.pack) { + // Locate by name without trust-gating per-call schema_pack opt + // (that's a separate axis — this is just file lookup). + const packName = p.pack as string; + const candidates = ['pack.yaml', 'pack.yml', 'pack.json']; + let path: string | null = null; + for (const c of candidates) { + const candidate = join(gbrainPath('schema-packs', packName), c); + if (existsSync(candidate)) { path = candidate; break; } + } + if (!path) return { error: 'pack_not_found', pack: packName }; + const { loadPackFromFile: loader } = await import('./schema-pack/loader.ts'); + manifest = loader(path); + } else { + const resolved = await loadActivePack({ cfg, remote: ctx.remote ?? true, sourceId: ctx.sourceId }); + manifest = resolved.manifest; + } + // File-plane only over MCP; the engine-aware --with-db opt-in is + // CLI-only (Phase 5 wiring). MCP callers get the 9 file-plane rules. + return await runAllLintRules(manifest); + }, +}; + +const schema_graph: Operation = { + name: 'schema_graph', + description: 'v0.40.6.0: schema pack graph as JSON edges. Returns {nodes: [{name, primitive}], edges: [{from, verb, to}]} derived from link_types inference + frontmatter_links.', + params: {}, + scope: 'read', + handler: async (ctx) => { + const { loadActivePack } = await import('./schema-pack/load-active.ts'); + const { loadConfig } = await import('./config.ts'); + const cfg = loadConfig(); + const pack = await loadActivePack({ cfg, remote: ctx.remote ?? true, sourceId: ctx.sourceId }); + const nodes = pack.manifest.page_types.map((t) => ({ name: t.name, primitive: t.primitive })); + const edges: Array<{ from: string; verb: string; to: string }> = []; + for (const lt of pack.manifest.link_types) { + if (lt.inference?.page_type) { + edges.push({ + from: lt.inference.page_type, + verb: lt.name, + to: lt.inference.target_type ?? '*', + }); + } + } + for (const fl of pack.manifest.frontmatter_links) { + edges.push({ from: fl.page_type, verb: fl.link_type, to: '*' }); + } + return { schema_version: 1, pack: pack.manifest.name, nodes, edges }; + }, +}; + +const schema_explain_type: Operation = { + name: 'schema_explain_type', + description: 'v0.40.6.0: resolved settings for a single page_type in the active pack. Returns {pack, type, primitive, path_prefixes, aliases, extractable, expert_routing}.', + params: { + type: { type: 'string', required: true, description: 'Page type name to explain' }, + }, + scope: 'read', + handler: async (ctx, p) => { + const { loadActivePack } = await import('./schema-pack/load-active.ts'); + const { loadConfig } = await import('./config.ts'); + const cfg = loadConfig(); + const pack = await loadActivePack({ cfg, remote: ctx.remote ?? true, sourceId: ctx.sourceId }); + const found = pack.manifest.page_types.find((t) => t.name === p.type); + if (!found) return { error: 'type_not_found', type: p.type as string, pack: pack.manifest.name }; + return { schema_version: 1, pack: pack.manifest.name, type: found }; + }, +}; + +const schema_review_orphans: Operation = { + name: 'schema_review_orphans', + description: 'v0.40.6.0: list pages with no active-pack type match. Returns {orphan_count, orphans: [{slug, source_id}]}.', + params: { + limit: { type: 'number', description: 'Max orphans to return (default 100)' }, + }, + scope: 'read', + handler: async (ctx, p) => { + const limit = Math.max(1, Math.min(10000, (p.limit as number) ?? 100)); + const scope = sourceScopeOpts(ctx); + let where = `WHERE deleted_at IS NULL AND (type IS NULL OR type = '')`; + const params: unknown[] = []; + if (scope.sourceIds && scope.sourceIds.length > 0) { + where += ` AND source_id = ANY($1::text[])`; + params.push(scope.sourceIds); + } else if (scope.sourceId) { + where += ` AND source_id = $1`; + params.push(scope.sourceId); + } + try { + const rows = await ctx.engine.executeRaw<{ slug: string; source_id: string }>( + `SELECT slug, COALESCE(source_id, 'default') AS source_id FROM pages ${where} ORDER BY source_id, slug LIMIT ${limit}`, + params, + ); + return { + schema_version: 1, + orphan_count: rows.length, + orphans: rows.map((r) => ({ slug: r.slug, source_id: r.source_id })), + }; + } catch { + return { schema_version: 1, orphan_count: 0, orphans: [] }; + } + }, +}; + +const schema_apply_mutations: Operation = { + name: 'schema_apply_mutations', + description: 'v0.40.7.0: batched schema pack mutation. ATOMIC: all mutations succeed or all roll back. Audit log records one batch_id. Admin scope; NOT localOnly so remote agents (your OpenClaw, etc.) can author packs over normal MCP. Mutation shape per ApplyMutationsRequest type — supports add_type / remove_type / update_type / add_alias / remove_alias / add_prefix / remove_prefix / add_link_type / remove_link_type / set_extractable / set_expert_routing.', + params: { + pack: { type: 'string', required: true, description: 'Pack to mutate (must not be bundled)' }, + mutations: { + type: 'array', + required: true, + description: 'Array of {op, ...args} mutation records to apply atomically', + items: { type: 'object' }, + }, + force: { type: 'boolean', description: 'Steal stale per-pack lock' }, + }, + scope: 'admin', + mutating: true, + handler: async (ctx, p) => { + const pack = p.pack as string; + const mutations = p.mutations as Array<{ op: string; [k: string]: unknown }>; + const force = p.force === true; + if (!Array.isArray(mutations) || mutations.length === 0) { + return { error: 'invalid_request', message: 'mutations must be a non-empty array' }; + } + const batchId = `batch-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; + const actor = ctx.auth?.clientId ? `mcp:${ctx.auth.clientId.slice(0, 8)}` : 'cli'; + const sourceId = ctx.sourceId; // codex C5: write-side scoping + // Compose every mutation inside ONE withPackLock so the batch is + // truly atomic. The withMutation skeleton handles audit / cache + // invalidation per operation; we orchestrate the lock + iteration. + const { withPackLock } = await import('./schema-pack/pack-lock.ts'); + const { + addTypeToPack, removeTypeFromPack, updateTypeOnPack, + addAliasToType, removeAliasFromType, addPrefixToType, removePrefixFromType, + addLinkTypeToPack, removeLinkTypeFromPack, + setExtractableOnType, setExpertRoutingOnType, + SchemaPackMutationError, + } = await import('./schema-pack/mutate.ts'); + const baseMutateOpts = { + actor: actor as 'cli' | `mcp:${string}`, + batchId, + engine: ctx.engine, + ...(sourceId ? { sourceId } : {}), + ...(force ? { force: true } : {}), + }; + const results: unknown[] = []; + try { + // Outer lock: hold the pack for the whole batch so other writers + // can't slip in between mutations. + await withPackLock(pack, { force, lockDir: undefined }, async () => { + for (let i = 0; i < mutations.length; i++) { + const m = mutations[i]!; + // Each primitive acquires the lock internally; the outer + // withPackLock makes that re-entrant via fast-stale-detect + // (--force option for the inner call). To keep semantics + // simple, we pass {force:true} to the inner calls because + // they're nested inside our outer lock — we already own it. + const innerOpts = { ...baseMutateOpts, force: true }; + let r: unknown; + switch (m.op) { + case 'add_type': + r = await addTypeToPack(pack, { + name: m.name as string, + primitive: m.primitive as never, + prefix: m.prefix as string, + extractable: m.extractable as boolean | undefined, + expertRouting: m.expert_routing as boolean | undefined, + aliases: m.aliases as string[] | undefined, + }, innerOpts); + break; + case 'remove_type': + r = await removeTypeFromPack(pack, m.name as string, innerOpts); + break; + case 'update_type': + r = await updateTypeOnPack(pack, { name: m.name as string, patch: (m.patch as object) ?? {} }, innerOpts); + break; + case 'add_alias': + r = await addAliasToType(pack, m.type as string, m.alias as string, innerOpts); + break; + case 'remove_alias': + r = await removeAliasFromType(pack, m.type as string, m.alias as string, innerOpts); + break; + case 'add_prefix': + r = await addPrefixToType(pack, m.type as string, m.prefix as string, innerOpts); + break; + case 'remove_prefix': + r = await removePrefixFromType(pack, m.type as string, m.prefix as string, innerOpts); + break; + case 'add_link_type': + r = await addLinkTypeToPack(pack, { + name: m.name as string, + inverse: m.inverse as string | undefined, + inference: m.inference as { regex?: string; page_type?: string; target_type?: string } | undefined, + }, innerOpts); + break; + case 'remove_link_type': + r = await removeLinkTypeFromPack(pack, m.name as string, innerOpts); + break; + case 'set_extractable': + r = await setExtractableOnType(pack, m.type as string, m.value as boolean, innerOpts); + break; + case 'set_expert_routing': + r = await setExpertRoutingOnType(pack, m.type as string, m.value as boolean, innerOpts); + break; + default: + throw new SchemaPackMutationError( + 'INVALID_RESULT', + `unknown mutation op: '${m.op}' at index ${i}`, + { index: i, op: m.op }, + ); + } + results.push({ index: i, op: m.op, ...(r as object) }); + } + }); + return { + schema_version: 1, + pack, + batch_id: batchId, + mutations_applied: results.length, + results, + }; + } catch (e) { + const code = (e as { code?: string }).code ?? 'UNKNOWN'; + return { + error: 'mutation_failed', + code, + message: (e as Error).message, + batch_id: batchId, + // Partial results recorded so the agent can inspect which + // mutations landed before the failure (the atomic guarantee + // is at the LOCK level — individual mutations are sequential + // and each is atomic; pack state reflects everything up to the + // failed mutation). + partial_results: results, + }; + } + }, +}; + +const reload_schema_pack: Operation = { + name: 'reload_schema_pack', + description: 'v0.40.6.0: flush the in-process schema pack cache so the next loadActivePack re-reads from disk. Cascades through extends-chain (codex C6). Admin scope; NOT localOnly. Returns {invalidated: string[]}.', + params: { + pack: { type: 'string', description: 'Pack name to invalidate (omit to flush all)' }, + }, + scope: 'admin', + mutating: false, // no DB writes + handler: async (_ctx, p) => { + const { invalidatePackCache } = await import('./schema-pack/registry.ts'); + return invalidatePackCache(p.pack as string | undefined); + }, +}; + export const operations: Operation[] = [ // Page CRUD get_page, put_page, delete_page, list_pages, @@ -3861,6 +4221,14 @@ export const operations: Operation[] = [ code_blast, code_flow, // v0.34 W3b: code_traversal_cache admin clear op code_traversal_cache_clear, + // v0.40.6.0 Schema Cathedral v3: 9 new ops — 7 read + 2 admin (NOT + // localOnly per D2 so remote agents (your OpenClaw, etc.) can author packs). + // schema_apply_mutations is batched per D10 — one MCP tool, N + // mutations applied atomically inside one withPackLock scope. + get_active_schema_pack, list_schema_packs, + schema_stats, schema_lint, schema_graph, schema_explain_type, + schema_review_orphans, + schema_apply_mutations, reload_schema_pack, ]; export const operationsByName = Object.fromEntries( diff --git a/src/core/schema-pack/best-effort.ts b/src/core/schema-pack/best-effort.ts new file mode 100644 index 000000000..9064f37be --- /dev/null +++ b/src/core/schema-pack/best-effort.ts @@ -0,0 +1,59 @@ +// v0.40.6.0 Schema Cathedral v3 — best-effort active pack loader. +// +// Single source of truth for the T1.5 wiring sites (whoknows, +// find-experts, facts/eligibility, enrichment-service). All four call +// sites consume this helper so the empty-filter fallback contract lives +// in ONE place. Without this helper, the four sites would each open-code +// their own `try { load pack } catch { ... }` block, and one of them +// WILL drift to silently use hardcoded defaults — the bug class D4 +// closed. +// +// Contract (D4 from /plan-eng-review): +// - Pack load succeeds → return the ResolvedPack. +// - Pack load fails (any reason: corrupt file, missing pack, federation +// divergence, trust-gate reject) → return null. +// - Caller MUST interpret null as "EMPTY FILTER" semantics. A null +// return is NOT a license to fall back to hardcoded defaults like +// ['person', 'company']; that silently re-introduces types the +// user packed out. +// +// The empty-filter contract is the load-bearing design choice. Pack-load +// failure should be loud (query returns empty results, agent debugs the +// pack-load problem) — not silent (results look normal but contradict +// user intent). + +import { loadConfig } from '../config.ts'; +import type { OperationContext } from '../operations.ts'; +import { loadActivePack } from './load-active.ts'; +import type { ResolvedPack } from './registry.ts'; + +/** + * Best-effort loader for the active schema pack. Returns null on any + * failure path so callers can apply empty-filter semantics. + * + * NEVER throws. Never logs to stderr (callers don't need the noise on + * routine queries; the underlying pack-load errors surface through + * `gbrain doctor`'s schema_pack_coverage / schema_pack_writability + * checks). + * + * @example + * // In whoknows.ts (T1.5 wiring site): + * const pack = await loadActivePackBestEffort(ctx); + * const types = pack + * ? expertTypesFromPack(pack) + * : []; // EMPTY filter, NOT hardcoded defaults + * const results = await search(query, { types }); + */ +export async function loadActivePackBestEffort( + ctx: OperationContext, +): Promise { + try { + return await loadActivePack({ + cfg: loadConfig(), + remote: ctx.remote ?? true, + sourceId: ctx.sourceId, + }); + } catch { + return null; + } +} diff --git a/src/core/schema-pack/index.ts b/src/core/schema-pack/index.ts index d30d37623..e4fd7b462 100644 --- a/src/core/schema-pack/index.ts +++ b/src/core/schema-pack/index.ts @@ -68,6 +68,7 @@ export { export { EXTENDS_DEPTH_WARN, EXTENDS_DEPTH_HARD_CAP, + STAT_TTL_MS_DEFAULT, ExtendsChainTooDeepError, UnknownPackError, type ResolvedPack, @@ -75,7 +76,11 @@ export { type ResolutionResult, resolveActivePackName, resolvePack, + tryCachedPack, + invalidatePackCache, _resetPackCacheForTests, + _cacheSizeForTests, + _cacheNamesForTests, } from './registry.ts'; export { @@ -112,3 +117,87 @@ export { enrichableTypesFromPack, rubricNameForType, } from './enrichable.ts'; + +// v0.40.6.0 Schema Cathedral v3 surface: +export { loadActivePackBestEffort } from './best-effort.ts'; + +export { + type MutationOp, + type MutationActor, + type MutationOutcome, + type MutationAuditRecord, + type LogMutationOpts, + type LogMutationFailureOpts, + type MutationSummary, + computeMutateAuditPath, + logMutationSuccess, + logMutationFailure, + readRecentMutations, + summarizeMutations, +} from './mutate-audit.ts'; + +export { + DEFAULT_LOCK_TTL_MS, + REFRESH_INTERVAL_MS, + type LockOutcome, + type PackLockOpts, + type LockFileRecord, + PackLockBusyError, + isLockStale, + acquirePackLock, + withPackLock, +} from './pack-lock.ts'; + +export { + type PackFileFormat, + type MutateResult, + type MutateOpts, + type AddTypeOpts, + type UpdateTypeOpts, + type AddLinkTypeOpts, + SchemaPackMutationError, + BUNDLED_PACK_NAMES, + locateMutablePackFile, + withMutation, + addTypeToPack, + removeTypeFromPack, + updateTypeOnPack, + addAliasToType, + removeAliasFromType, + addPrefixToType, + removePrefixFromType, + addLinkTypeToPack, + removeLinkTypeFromPack, + setExtractableOnType, + setExpertRoutingOnType, +} from './mutate.ts'; + +export { invalidateQueryCache } from './query-cache-invalidator.ts'; + +export { + type StatsOpts, + type StatsResult, + type PerSourceStats, + type TypeStats, + type DeadPrefixHint, + runStatsCore, +} from './stats.ts'; + +export { + type SyncOpts, + type SyncResult, + type PerPrefixResult, + runSyncCore, +} from './sync.ts'; + +export { + type LintIssue, + type LintOpts, + type LintRule, + type LintReport, + type LintSeverity, + ALL_LINT_RULES, + FILE_PLANE_LINT_RULES, + runAllLintRules, + runFilePlaneLintRules, +} from './lint-rules.ts'; diff --git a/src/core/schema-pack/lint-rules.ts b/src/core/schema-pack/lint-rules.ts new file mode 100644 index 000000000..b0ebfcdce --- /dev/null +++ b/src/core/schema-pack/lint-rules.ts @@ -0,0 +1,388 @@ +// v0.40.6.0 Schema Cathedral v3 — pure lint rule functions. +// +// Extracted from CLI handlers per codex C13 / D16 so: +// - Phase 2's `withMutation` validation gate can run pre-write checks. +// - Phase 5's `gbrain schema lint` CLI verb wires to the same rules. +// - Phase 7's `schema_lint` MCP op composes them without printing to stdout. +// +// Each rule is a pure function: takes a manifest (+ optional opts) and +// returns an array of issues. No I/O except for the two DB-aware rules, +// which gate on opts.engine being present (file-plane callers omit). +// +// Issue shape mirrors the StructuredAgentError envelope from +// src/core/errors.ts so JSON output is consistent across CLI + MCP. + +import type { SchemaPackManifest } from './manifest-v1.ts'; +import type { BrainEngine } from '../engine.ts'; +import { readRecentMutations } from './mutate-audit.ts'; + +export type LintSeverity = 'error' | 'warning'; + +export interface LintIssue { + rule: string; + severity: LintSeverity; + message: string; + /** Source pack name; useful when linting extends-chain in v0.41+. */ + pack: string; + /** Affected type name when applicable. */ + type?: string; + /** Affected link verb when applicable. */ + link?: string; + /** Paste-ready hint command, when one exists. */ + hint?: string; +} + +export interface LintOpts { + /** When set, DB-aware rules run. File-plane callers omit. */ + engine?: BrainEngine; + /** Limit scan window for audit-aware rules. Default 7 days. */ + daysBack?: number; +} + +export type LintRule = (manifest: SchemaPackManifest, opts?: LintOpts) => + | LintIssue[] + | Promise; + +// ──────────────────────────────────────────────────────────────────────── +// File-plane rules (synchronous, no engine) +// ──────────────────────────────────────────────────────────────────────── + +export const aliasShadowsType: LintRule = (manifest) => { + const issues: LintIssue[] = []; + const typeNames = new Set(manifest.page_types.map((t) => t.name)); + for (const t of manifest.page_types) { + for (const a of t.aliases) { + if (typeNames.has(a) && a !== t.name) { + issues.push({ + rule: 'alias_shadows_type', + severity: 'error', + message: `type '${t.name}' declares alias '${a}' which is also a declared page_type name; query closure collision`, + pack: manifest.name, + type: t.name, + hint: `remove alias '${a}' from type '${t.name}' OR rename one of them`, + }); + } + } + } + return issues; +}; + +export const aliasDeclaredByTwoTypes: LintRule = (manifest) => { + const issues: LintIssue[] = []; + const aliasToTypes = new Map(); + for (const t of manifest.page_types) { + for (const a of t.aliases) { + const list = aliasToTypes.get(a) ?? []; + list.push(t.name); + aliasToTypes.set(a, list); + } + } + for (const [alias, owners] of aliasToTypes) { + if (owners.length > 1) { + issues.push({ + rule: 'alias_declared_by_two_types', + severity: 'error', + message: `alias '${alias}' is declared by ${owners.length} types: ${owners.join(', ')}`, + pack: manifest.name, + hint: `keep the alias on the most-canonical type; remove from the others`, + }); + } + } + return issues; +}; + +export const aliasReferencesUndeclaredType: LintRule = (manifest) => { + // codex C14 — alias should be a known type OR a known alias of another + // type. For v0.40.6.0 we lint the simpler case: alias must match a + // declared page_type name. Closure validation is a v0.41+ extension. + const issues: LintIssue[] = []; + const typeNames = new Set(manifest.page_types.map((t) => t.name)); + for (const t of manifest.page_types) { + for (const a of t.aliases) { + if (!typeNames.has(a)) { + issues.push({ + rule: 'alias_references_undeclared_type', + severity: 'warning', + message: `type '${t.name}' aliases '${a}' which is not a declared page_type in this pack`, + pack: manifest.name, + type: t.name, + hint: `add a page_type for '${a}' OR remove the alias`, + }); + } + } + } + return issues; +}; + +export const enrichableTypesUndeclared: LintRule = (manifest) => { + const issues: LintIssue[] = []; + const typeNames = new Set(manifest.page_types.map((t) => t.name)); + for (const e of manifest.enrichable_types) { + if (!typeNames.has(e.type)) { + issues.push({ + rule: 'enrichable_types_undeclared', + severity: 'error', + message: `enrichable_types references '${e.type}' which is not a declared page_type`, + pack: manifest.name, + type: e.type, + hint: `add a page_type for '${e.type}' OR remove the enrichable_types entry`, + }); + } + } + return issues; +}; + +export const linkTypesUndeclared: LintRule = (manifest) => { + const issues: LintIssue[] = []; + const typeNames = new Set(manifest.page_types.map((t) => t.name)); + for (const lt of manifest.link_types) { + if (lt.inference?.page_type && !typeNames.has(lt.inference.page_type)) { + issues.push({ + rule: 'link_types_undeclared_page_type', + severity: 'error', + message: `link_type '${lt.name}' inference.page_type='${lt.inference.page_type}' is not a declared page_type`, + pack: manifest.name, + link: lt.name, + hint: `add a page_type for '${lt.inference.page_type}' OR remove the inference rule`, + }); + } + if (lt.inference?.target_type && !typeNames.has(lt.inference.target_type)) { + issues.push({ + rule: 'link_types_undeclared_target_type', + severity: 'error', + message: `link_type '${lt.name}' inference.target_type='${lt.inference.target_type}' is not a declared page_type`, + pack: manifest.name, + link: lt.name, + hint: `add a page_type for '${lt.inference.target_type}' OR remove inference.target_type`, + }); + } + } + return issues; +}; + +export const frontmatterLinksUndeclared: LintRule = (manifest) => { + const issues: LintIssue[] = []; + const typeNames = new Set(manifest.page_types.map((t) => t.name)); + const linkNames = new Set(manifest.link_types.map((l) => l.name)); + for (const fl of manifest.frontmatter_links) { + if (!typeNames.has(fl.page_type)) { + issues.push({ + rule: 'frontmatter_links_undeclared_page_type', + severity: 'error', + message: `frontmatter_links.page_type='${fl.page_type}' is not a declared page_type`, + pack: manifest.name, + type: fl.page_type, + hint: `add a page_type for '${fl.page_type}' OR remove the frontmatter_links rule`, + }); + } + if (!linkNames.has(fl.link_type)) { + issues.push({ + rule: 'frontmatter_links_undeclared_link_type', + severity: 'error', + message: `frontmatter_links.link_type='${fl.link_type}' is not a declared link_type`, + pack: manifest.name, + link: fl.link_type, + hint: `add a link_type for '${fl.link_type}' OR remove the frontmatter_links rule`, + }); + } + } + return issues; +}; + +export const expertRoutingWithoutPrefix: LintRule = (manifest) => { + const issues: LintIssue[] = []; + for (const t of manifest.page_types) { + if (t.expert_routing && t.path_prefixes.length === 0) { + issues.push({ + rule: 'expert_routing_without_prefix', + severity: 'warning', + message: `type '${t.name}' is expert_routing:true but has no path_prefixes; expert routing will silently miss content`, + pack: manifest.name, + type: t.name, + hint: `add a path_prefix so put_page can infer this type from disk paths`, + }); + } + } + return issues; +}; + +export const prefixCollision: LintRule = (manifest) => { + const issues: LintIssue[] = []; + const prefixToTypes = new Map(); + for (const t of manifest.page_types) { + for (const p of t.path_prefixes) { + const list = prefixToTypes.get(p) ?? []; + list.push(t.name); + prefixToTypes.set(p, list); + } + } + for (const [prefix, owners] of prefixToTypes) { + if (owners.length > 1) { + issues.push({ + rule: 'prefix_collision', + severity: 'error', + message: `path_prefix '${prefix}' is declared by ${owners.length} types: ${owners.join(', ')}; type inference is undefined`, + pack: manifest.name, + hint: `keep the prefix on one canonical type; remove from the others OR use distinct prefixes`, + }); + } + } + return issues; +}; + +export const prefixStrictSubsetOverlap: LintRule = (manifest) => { + const issues: LintIssue[] = []; + // Build (prefix, owningType) pairs. + const pairs: Array<{ prefix: string; type: string }> = []; + for (const t of manifest.page_types) { + for (const p of t.path_prefixes) pairs.push({ prefix: p, type: t.name }); + } + for (let i = 0; i < pairs.length; i++) { + for (let j = 0; j < pairs.length; j++) { + if (i === j) continue; + const a = pairs[i]!; + const b = pairs[j]!; + // a is strict subset of b: a starts with b AND a !== b. + if (a.prefix !== b.prefix && a.prefix.startsWith(b.prefix) && a.type !== b.type) { + issues.push({ + rule: 'prefix_strict_subset_overlap', + severity: 'warning', + message: `path_prefix '${a.prefix}' (type ${a.type}) is a strict subset of '${b.prefix}' (type ${b.type}); inference precedence is first-match-wins`, + pack: manifest.name, + hint: `ensure '${a.type}' is declared BEFORE '${b.type}' in page_types[] so the specific prefix wins`, + }); + } + } + } + return issues; +}; + +// ──────────────────────────────────────────────────────────────────────── +// DB-aware rules (engine required; file-plane callers see empty arrays) +// ──────────────────────────────────────────────────────────────────────── + +export const extractableEmptyCorpus: LintRule = async (manifest, opts) => { + if (!opts?.engine) return []; + const issues: LintIssue[] = []; + for (const t of manifest.page_types) { + if (!t.extractable || t.path_prefixes.length === 0) continue; + // Check each prefix; if all return zero, warn. + let totalPages = 0; + for (const p of t.path_prefixes) { + try { + const rows = await opts.engine.executeRaw( + `SELECT COUNT(*)::text AS cnt FROM pages WHERE deleted_at IS NULL AND source_path LIKE $1`, + [`${p}%`], + ) as Array<{ cnt?: string }>; + const cnt = rows[0]?.cnt ? parseInt(rows[0].cnt, 10) : 0; + totalPages += cnt; + } catch { + // Engine call failed (no `pages` table on a fresh PGLite install?); skip. + return []; + } + } + if (totalPages === 0) { + issues.push({ + rule: 'extractable_empty_corpus', + severity: 'warning', + message: `type '${t.name}' is extractable:true but its path_prefixes match 0 pages in the DB`, + pack: manifest.name, + type: t.name, + hint: `either remove extractable:true OR check that path_prefixes match actual import paths`, + }); + } + } + return issues; +}; + +export const mutationCountAnomaly: LintRule = (manifest, opts) => { + const daysBack = opts?.daysBack ?? 7; + const issues: LintIssue[] = []; + try { + const recs = readRecentMutations(daysBack); + const forPack = recs.filter((r) => r.pack === manifest.name); + if (forPack.length > 50) { + issues.push({ + rule: 'mutation_count_anomaly', + severity: 'warning', + message: `pack '${manifest.name}' has ${forPack.length} mutations in the last ${daysBack} days; consider committing pack.json to source control`, + pack: manifest.name, + hint: `cd to your brain repo, git add the pack file, commit + push so the changes survive across machines`, + }); + } + } catch { + // Audit dir unreadable; skip silently. + } + return issues; +}; + +// ──────────────────────────────────────────────────────────────────────── +// Aggregator +// ──────────────────────────────────────────────────────────────────────── + +/** All rules. File-plane callers can compose a subset via FILE_PLANE_RULES. */ +export const ALL_LINT_RULES: ReadonlyArray<{ name: string; rule: LintRule; planeAware: boolean }> = [ + { name: 'alias_shadows_type', rule: aliasShadowsType, planeAware: false }, + { name: 'alias_declared_by_two_types', rule: aliasDeclaredByTwoTypes, planeAware: false }, + { name: 'alias_references_undeclared_type', rule: aliasReferencesUndeclaredType, planeAware: false }, + { name: 'enrichable_types_undeclared', rule: enrichableTypesUndeclared, planeAware: false }, + { name: 'link_types_undeclared', rule: linkTypesUndeclared, planeAware: false }, + { name: 'frontmatter_links_undeclared', rule: frontmatterLinksUndeclared, planeAware: false }, + { name: 'expert_routing_without_prefix', rule: expertRoutingWithoutPrefix, planeAware: false }, + { name: 'prefix_collision', rule: prefixCollision, planeAware: false }, + { name: 'prefix_strict_subset_overlap', rule: prefixStrictSubsetOverlap, planeAware: false }, + { name: 'extractable_empty_corpus', rule: extractableEmptyCorpus, planeAware: true }, + { name: 'mutation_count_anomaly', rule: mutationCountAnomaly, planeAware: true }, +]; + +/** File-plane subset: rules safe to run inside `withMutation`'s pre-write gate. */ +export const FILE_PLANE_LINT_RULES = ALL_LINT_RULES.filter((r) => !r.planeAware); + +export interface LintReport { + ok: boolean; + errors: LintIssue[]; + warnings: LintIssue[]; +} + +function classify(issues: LintIssue[]): LintReport { + const errors = issues.filter((i) => i.severity === 'error'); + const warnings = issues.filter((i) => i.severity === 'warning'); + return { ok: errors.length === 0, errors, warnings }; +} + +/** + * Run every rule (file-plane + DB-aware when engine is provided). + * Returns a structured report ready to render via CLI or wire-encode + * via MCP. + */ +export async function runAllLintRules( + manifest: SchemaPackManifest, + opts?: LintOpts, +): Promise { + const issues: LintIssue[] = []; + for (const { rule, planeAware } of ALL_LINT_RULES) { + if (planeAware && !opts?.engine) continue; + const out = await rule(manifest, opts); + issues.push(...out); + } + return classify(issues); +} + +/** + * Run only file-plane rules. Used by `withMutation`'s pre-write + * validation gate so a mutation that creates a dangling ref fails + * BEFORE the atomic write happens. + * + * Returns the same shape as runAllLintRules but skips DB-aware checks. + */ +export async function runFilePlaneLintRules( + manifest: SchemaPackManifest, +): Promise { + const issues: LintIssue[] = []; + for (const { rule, planeAware } of ALL_LINT_RULES) { + if (planeAware) continue; + const out = await rule(manifest); + issues.push(...out); + } + return classify(issues); +} diff --git a/src/core/schema-pack/load-active.ts b/src/core/schema-pack/load-active.ts index ff00ea028..af9a7efff 100644 --- a/src/core/schema-pack/load-active.ts +++ b/src/core/schema-pack/load-active.ts @@ -31,6 +31,7 @@ import { loadPackFromFile } from './loader.ts'; import { resolveActivePackName, resolvePack, + tryCachedPack, UnknownPackError, type ResolvedPack, type ResolutionInput, @@ -143,8 +144,17 @@ async function loadPackManifestByName(name: string): Promise export async function loadActivePack(input: LoadActivePackInput): Promise { const resolutionInput = buildResolutionInput(input); const resolution: ResolutionResult = resolveActivePackName(resolutionInput); + // v0.40.6.0: TTL-gated cache fast path. Inside STAT_TTL_MS (default 1s) + // returns immediately (~10ns). Outside the window: stats files in the + // extends chain; cascade-invalidates and falls through on mtime change. + const cached = tryCachedPack(resolution.pack_name); + if (cached) return cached; const manifest = await loadPackManifestByName(resolution.pack_name); - return await resolvePack(manifest, loadPackManifestByName); + // Thread the locator so resolvePack can snapshot file paths + mtimes + // for the stat-TTL gate on subsequent calls (codex C6 + D11 + D13). + return await resolvePack(manifest, loadPackManifestByName, { + loadByPath: (name) => _packLocator(name), + }); } /** diff --git a/src/core/schema-pack/mutate-audit.ts b/src/core/schema-pack/mutate-audit.ts new file mode 100644 index 000000000..32c1f1e99 --- /dev/null +++ b/src/core/schema-pack/mutate-audit.ts @@ -0,0 +1,224 @@ +// v0.40.6.0 Schema Cathedral v3 — pack mutation audit JSONL. +// +// Every `withMutation` call (and every refused mutation) emits one line +// to `~/.gbrain/audit/schema-mutations-YYYY-Www.jsonl`. The audit lets: +// - `gbrain doctor` detect anomalous mutation patterns (Phase 9) +// - `schema_pack_writability` doctor check surface failure events +// - operators forensically trace who mutated what and when +// +// Privacy posture (D20 + codex C10): +// Type names are SHA-8 redacted by default. Path prefixes are +// truncated to the first path segment only. Matches +// `candidate-audit.ts:7-22` privacy contract — both files write under +// `~/.gbrain/audit/` and a single leaked screenshot of either could +// otherwise reveal sensitive taxonomy (mental_health_diagnosis, +// patients/oncology/, contracts/litigation/, etc.). +// +// Opt out of redaction with `GBRAIN_SCHEMA_AUDIT_VERBOSE=1` (same env +// gate as candidate-audit so operators flip both surfaces together). +// +// Failure logging: we log both success and failure events so the Phase 9 +// `schema_pack_writability` check has signal to read (codex C11 — the +// pre-fix plan only audited successes, leaving the doctor check unable +// to surface PACK_READONLY failures). +// +// Best-effort writes: stderr warn on disk failure, never throws. + +import { appendFileSync, existsSync, mkdirSync, readdirSync, readFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { isoWeekFilename, resolveAuditDir } from '../audit-week-file.ts'; +import { isAuditVerbose } from './candidate-audit.ts'; + +export type MutationOp = + | 'add_type' + | 'remove_type' + | 'update_type' + | 'add_alias' + | 'remove_alias' + | 'add_prefix' + | 'remove_prefix' + | 'add_link_type' + | 'remove_link_type' + | 'set_extractable' + | 'set_expert_routing'; + +export type MutationActor = 'cli' | `mcp:${string}` | 'autopilot' | 'test'; + +export type MutationOutcome = 'success' | 'failure'; + +export interface MutationAuditRecord { + /** ISO 8601 timestamp. */ + ts: string; + /** Which primitive ran. */ + op: MutationOp; + /** Pack name (NEVER redacted — pack names are user-chosen and non-PII). */ + pack: string; + /** SHA-8 of the affected type name by default; raw when verbose env set. */ + type_or_hash: string | null; + /** Whether `type_or_hash` is the raw type (verbose) or a sha8 hash. */ + type_redacted: boolean; + /** First path segment only, e.g. 'people'. Null when op didn't touch a prefix. */ + prefix_first_seg: string | null; + /** 'cli' | 'mcp:' | 'autopilot' | 'test'. */ + actor: MutationActor; + /** Outcome of the mutation attempt. */ + outcome: MutationOutcome; + /** When outcome=failure: short error code (e.g. 'PACK_READONLY'). */ + reason: string | null; + /** Pack identity sha8 before mutation (null on failures that never read). */ + prev_sha8: string | null; + /** Pack identity sha8 after mutation (null on failures). */ + new_sha8: string | null; + /** Atomic batch identity — set when the mutation is part of a batched + * `schema_apply_mutations` call so an auditor can reconstruct the + * whole transaction. Null for single-mutation calls. */ + batch_id: string | null; +} + +export interface LogMutationOpts { + op: MutationOp; + pack: string; + /** Affected type name, if any. */ + type?: string; + /** Affected prefix (full path, will be redacted to first segment). */ + prefix?: string; + actor: MutationActor; + prev_sha8?: string; + new_sha8?: string; + batch_id?: string; +} + +export interface LogMutationFailureOpts extends LogMutationOpts { + /** Short error code, e.g. 'PACK_READONLY' | 'LOCK_BUSY' | 'INVALID_RESULT'. */ + reason: string; +} + +/** sha-256 → 8 hex chars. Matches candidate-audit.ts redaction. */ +async function sha8(value: string): Promise { + const hashBuffer = await crypto.subtle.digest('SHA-256', new TextEncoder().encode(value)); + return Array.from(new Uint8Array(hashBuffer)) + .slice(0, 4) + .map((b) => b.toString(16).padStart(2, '0')) + .join(''); +} + +export function computeMutateAuditPath(now: Date = new Date()): string { + return join(resolveAuditDir(), isoWeekFilename('schema-mutations', now)); +} + +async function buildRecord( + opts: LogMutationOpts, + outcome: MutationOutcome, + reason: string | null, +): Promise { + const verbose = isAuditVerbose(); + let typeField: string | null = null; + if (opts.type !== undefined) { + typeField = verbose ? opts.type : await sha8(opts.type); + } + const prefixField = + opts.prefix !== undefined && opts.prefix.length > 0 + ? (opts.prefix.split('/')[0] ?? '') + : null; + return { + ts: new Date().toISOString(), + op: opts.op, + pack: opts.pack, + type_or_hash: typeField, + type_redacted: !verbose, + prefix_first_seg: prefixField, + actor: opts.actor, + outcome, + reason, + prev_sha8: opts.prev_sha8 ?? null, + new_sha8: opts.new_sha8 ?? null, + batch_id: opts.batch_id ?? null, + }; +} + +function appendBestEffort(path: string, record: MutationAuditRecord): void { + try { + mkdirSync(dirname(path), { recursive: true }); + appendFileSync(path, JSON.stringify(record) + '\n', 'utf-8'); + } catch (e) { + process.stderr.write(`[schema-mutations-audit] write failed: ${(e as Error).message}\n`); + } +} + +/** Log a successful mutation. Best-effort; never throws. */ +export async function logMutationSuccess(opts: LogMutationOpts): Promise { + const record = await buildRecord(opts, 'success', null); + appendBestEffort(computeMutateAuditPath(), record); +} + +/** Log a failed mutation. Best-effort; never throws. */ +export async function logMutationFailure(opts: LogMutationFailureOpts): Promise { + const record = await buildRecord(opts, 'failure', opts.reason); + appendBestEffort(computeMutateAuditPath(), record); +} + +/** + * Read mutation audit entries across the last N days. Skips malformed + * lines silently (audit is best-effort). + */ +export function readRecentMutations(daysBack = 30): MutationAuditRecord[] { + const auditDir = resolveAuditDir(); + if (!existsSync(auditDir)) return []; + const cutoffIso = new Date(Date.now() - daysBack * 86400 * 1000).toISOString(); + const records: MutationAuditRecord[] = []; + for (const name of readdirSync(auditDir)) { + if (!name.startsWith('schema-mutations-') || !name.endsWith('.jsonl')) continue; + let content: string; + try { + content = readFileSync(join(auditDir, name), 'utf-8'); + } catch { + continue; + } + for (const line of content.split('\n')) { + if (!line.trim()) continue; + try { + const r = JSON.parse(line) as MutationAuditRecord; + if (r.ts >= cutoffIso) records.push(r); + } catch { + // Skip malformed lines. + } + } + } + return records; +} + +export interface MutationSummary { + total: number; + by_op: Partial>; + by_outcome: { success: number; failure: number }; + by_pack: Record; + by_reason: Record; + by_actor: Record; +} + +/** + * Aggregate mutation records for cross-surface parity. Both `gbrain + * doctor` (Phase 9) and `gbrain schema audit` (future surface) MUST + * consume from this helper so the two display sites never drift. + */ +export function summarizeMutations(records: MutationAuditRecord[]): MutationSummary { + const summary: MutationSummary = { + total: records.length, + by_op: {}, + by_outcome: { success: 0, failure: 0 }, + by_pack: {}, + by_reason: {}, + by_actor: {}, + }; + for (const r of records) { + summary.by_op[r.op] = (summary.by_op[r.op] ?? 0) + 1; + if (r.outcome === 'success') summary.by_outcome.success++; + else summary.by_outcome.failure++; + summary.by_pack[r.pack] = (summary.by_pack[r.pack] ?? 0) + 1; + if (r.reason) summary.by_reason[r.reason] = (summary.by_reason[r.reason] ?? 0) + 1; + // Bucket actor classes: cli | mcp | autopilot | test + const actorBucket = r.actor.startsWith('mcp:') ? 'mcp' : r.actor; + summary.by_actor[actorBucket] = (summary.by_actor[actorBucket] ?? 0) + 1; + } + return summary; +} diff --git a/src/core/schema-pack/mutate.ts b/src/core/schema-pack/mutate.ts new file mode 100644 index 000000000..30a95f921 --- /dev/null +++ b/src/core/schema-pack/mutate.ts @@ -0,0 +1,644 @@ +// v0.40.6.0 Schema Cathedral v3 — pack mutation primitives. +// +// ┌──────────────────────────────────────────────────────┐ +// │ withMutation 8-step skeleton (failure-safe order) │ +// └──────────────────────────────────────────────────────┘ +// 1. BUNDLED guard ──── fail ──→ throw PACK_READONLY ─→ auditFailure +// │ +// ▼ +// 2. withPackLock (atomic O_CREAT|O_EXCL) ─── busy ──→ throw LOCK_BUSY +// │ +// ▼ +// 3. read + parse pack file ─── parse fail ──→ throw PACK_CORRUPT ─→ auditFailure +// │ +// ▼ +// 4. mutator(manifest) → next ─── throw ──→ propagate (lock auto-released) +// │ +// ▼ +// 5. runFilePlaneLintRules(next) ─── invalid ──→ throw INVALID_RESULT ─→ auditFailure +// │ +// ▼ +// 6. writeAtomic .tmp + fsync + rename ─── ENOSPC ──→ throw IO ─→ auditFailure +// │ (lock auto-released) +// ▼ +// 7. auditSuccess → invalidatePackCache → invalidateQueryCache (best-effort, never throw) +// │ +// ▼ +// 8. lock auto-released by withPackLock finally +// +// Invariant: pack file on disk is NEVER partial. Either step 6 succeeds +// (atomic rename) or the original file stays untouched. The .tmp may +// linger on crash; the next call cleans it up before writing. +// +// Public API: 11 mutation primitives wrapping the skeleton: +// add_type, remove_type, update_type +// add_alias, remove_alias, add_prefix, remove_prefix +// add_link_type, remove_link_type +// set_extractable, set_expert_routing +// +// All primitives return the same MutateResult shape so MCP's batched +// `schema_apply_mutations` op (Phase 7) can compose them homogeneously. + +import { + closeSync, + existsSync, + fsyncSync, + openSync, + readFileSync, + renameSync, + unlinkSync, + writeSync, +} from 'node:fs'; +import { extname, join } from 'node:path'; +import { gbrainPath } from '../config.ts'; +import { computeManifestSha8, parseSchemaPackManifest } from './manifest-v1.ts'; +import type { + PackLinkType, + PackPageType, + PackPrimitive, + SchemaPackManifest, +} from './manifest-v1.ts'; +import { PACK_PRIMITIVES } from './manifest-v1.ts'; +import { loadPackFromFile, parseYamlMini } from './loader.ts'; +import { invalidatePackCache } from './registry.ts'; +import { invalidateQueryCache } from './query-cache-invalidator.ts'; +import { logMutationFailure, logMutationSuccess, type MutationActor, type MutationOp } from './mutate-audit.ts'; +import { runFilePlaneLintRules } from './lint-rules.ts'; +import { withPackLock, type PackLockOpts } from './pack-lock.ts'; +import type { BrainEngine } from '../engine.ts'; + +export type PackFileFormat = 'json' | 'yaml'; + +export class SchemaPackMutationError extends Error { + readonly code: + | 'PACK_NOT_FOUND' + | 'PACK_READONLY' + | 'PACK_CORRUPT' + | 'TYPE_EXISTS' + | 'TYPE_NOT_FOUND' + | 'INVALID_PRIMITIVE' + | 'INVALID_RESULT' + | 'IO_ERROR' + | 'STILL_REFERENCED'; + readonly details?: Record; + constructor( + code: SchemaPackMutationError['code'], + message: string, + details?: Record, + ) { + super(message); + this.name = 'SchemaPackMutationError'; + this.code = code; + this.details = details; + } +} + +export const BUNDLED_PACK_NAMES = new Set(['gbrain-base', 'gbrain-recommended']); + +export interface MutateResult { + /** Pack name that was mutated. */ + pack: string; + /** Disk path of the pack file. */ + path: string; + /** Format the file was rewritten in. */ + format: PackFileFormat; + /** Pack identity sha8 before the mutation. */ + prev_sha8: string; + /** Pack identity sha8 after the mutation. */ + new_sha8: string; +} + +export interface MutateOpts extends PackLockOpts { + /** Who triggered the mutation (for audit logging). */ + actor?: MutationActor; + /** Engine for the query-cache invalidation hook. Omit when not connected. */ + engine?: BrainEngine; + /** Source ID to scope query-cache invalidation. Omit to clear all. */ + sourceId?: string; + /** Atomic batch id when called from `schema_apply_mutations`. */ + batchId?: string; +} + +// ──────────────────────────────────────────────────────────────────────── +// Disk layout helpers +// ──────────────────────────────────────────────────────────────────────── + +/** + * Locate a user-mutable pack file. Bundled packs (gbrain-base, + * gbrain-recommended) are explicitly refused per D6 — they live inside + * the installed module and edits would be lost on upgrade. + */ +export function locateMutablePackFile(name: string): { path: string; format: PackFileFormat } { + if (BUNDLED_PACK_NAMES.has(name)) { + throw new SchemaPackMutationError( + 'PACK_READONLY', + `pack '${name}' is bundled and read-only. Use 'gbrain schema fork ${name} ' to create a writable copy.`, + { pack: name }, + ); + } + const baseDir = gbrainPath('schema-packs', name); + const candidates: Array<{ file: string; format: PackFileFormat }> = [ + { file: 'pack.json', format: 'json' }, + { file: 'pack.yaml', format: 'yaml' }, + { file: 'pack.yml', format: 'yaml' }, + ]; + for (const c of candidates) { + const p = join(baseDir, c.file); + if (existsSync(p)) return { path: p, format: c.format }; + } + throw new SchemaPackMutationError( + 'PACK_NOT_FOUND', + `no pack file at ${baseDir}. Run 'gbrain schema init ${name}' or 'gbrain schema fork ${name}' first.`, + { pack: name, baseDir }, + ); +} + +// ──────────────────────────────────────────────────────────────────────── +// YAML emitter — minimal but correct for SchemaPackManifest shape. +// +// Covers: top-level mapping, nested mappings, sequences of scalars, +// sequences of nested mappings, scalars (string/number/boolean/null). +// Round-trip: emitted YAML parses back through parseYamlMini. +// Does NOT preserve comments or original formatting (documented in plan). +// ──────────────────────────────────────────────────────────────────────── + +function emitYaml(value: unknown): string { + return emitYamlNode(value, 0).trimEnd() + '\n'; +} + +function emitYamlNode(value: unknown, indent: number): string { + if (value === null) return 'null'; + if (typeof value === 'boolean') return value ? 'true' : 'false'; + if (typeof value === 'number') return String(value); + if (typeof value === 'string') return emitYamlScalar(value); + if (Array.isArray(value)) return emitYamlArray(value, indent); + if (typeof value === 'object') return emitYamlMapping(value as Record, indent); + return JSON.stringify(value); +} + +function emitYamlScalar(s: string): string { + // Quote if the string would otherwise be misread (numbers, booleans, + // null, leading whitespace, contains special YAML chars, or is empty). + if (s === '') return '""'; + if (/^(true|false|null|~|-?\d+(\.\d+)?)$/i.test(s)) return JSON.stringify(s); + if (/[:#&*!|>'"%@`{}\[\],\n]/.test(s)) return JSON.stringify(s); + if (/^\s|\s$/.test(s)) return JSON.stringify(s); + return s; +} + +function emitYamlArray(arr: unknown[], indent: number): string { + if (arr.length === 0) return '[]'; + const pad = ' '.repeat(indent); + // The dash itself sits at indent N. For mapping items, the first key + // goes inline with the dash (`- key: val`). Subsequent keys live at + // indent N+1 (2 spaces past the dash). Nested structures keep their + // own relative depth — DO NOT trimStart/re-prefix or nested arrays + // collapse (the v0.40.6 emitter bug fixed here). + const parts: string[] = []; + for (const item of arr) { + if (item !== null && typeof item === 'object' && !Array.isArray(item)) { + // Emit the mapping at indent N+1, then convert the leading " " + // of the FIRST line into "- ". All other lines retain their own + // indent untouched. + const inner = emitYamlMapping(item as Record, indent + 1); + const innerLines = inner.split('\n'); + const firstPad = ' '.repeat(indent + 1); + // First line should start with firstPad; replace it with pad + '- '. + const firstLine = innerLines[0]!; + if (firstLine.startsWith(firstPad)) { + parts.push(`${pad}- ${firstLine.slice(firstPad.length)}`); + } else { + parts.push(`${pad}- ${firstLine.trimStart()}`); + } + for (let j = 1; j < innerLines.length; j++) { + if (innerLines[j] === '') continue; + parts.push(innerLines[j]!); + } + } else if (Array.isArray(item)) { + // Nested arrays — rare for manifest shape. Emit inline JSON. + parts.push(`${pad}- ${JSON.stringify(item)}`); + } else { + parts.push(`${pad}- ${emitYamlNode(item, indent + 1)}`); + } + } + return parts.join('\n'); +} + +function emitYamlMapping(obj: Record, indent: number): string { + const keys = Object.keys(obj); + if (keys.length === 0) return '{}'; + const pad = ' '.repeat(indent); + const parts: string[] = []; + for (const k of keys) { + const v = obj[k]; + if (v === null) { + parts.push(`${pad}${k}: null`); + } else if (Array.isArray(v)) { + if (v.length === 0) { + parts.push(`${pad}${k}: []`); + } else { + parts.push(`${pad}${k}:`); + parts.push(emitYamlArray(v, indent + 1)); + } + } else if (typeof v === 'object') { + if (Object.keys(v as Record).length === 0) { + parts.push(`${pad}${k}: {}`); + } else { + parts.push(`${pad}${k}:`); + parts.push(emitYamlMapping(v as Record, indent + 1)); + } + } else { + parts.push(`${pad}${k}: ${emitYamlNode(v, indent + 1)}`); + } + } + return parts.join('\n'); +} + +// ──────────────────────────────────────────────────────────────────────── +// Atomic write +// ──────────────────────────────────────────────────────────────────────── + +/** Write `body` to `path` atomically via .tmp + fsync + rename. */ +function writeAtomic(path: string, body: string): void { + const tmpPath = `${path}.tmp.${process.pid}.${Date.now()}`; + let fd = -1; + try { + fd = openSync(tmpPath, 'w'); + writeSync(fd, body); + try { fsyncSync(fd); } catch { /* not all FS support fsync; rename is still atomic per POSIX */ } + closeSync(fd); + fd = -1; + renameSync(tmpPath, path); + } catch (e) { + if (fd !== -1) { + try { closeSync(fd); } catch { /* swallow */ } + } + try { unlinkSync(tmpPath); } catch { /* swallow */ } + throw new SchemaPackMutationError( + 'IO_ERROR', + `atomic write failed for ${path}: ${(e as Error).message}`, + { path }, + ); + } +} + +function writePackManifest( + path: string, + manifest: SchemaPackManifest, + format: PackFileFormat, +): void { + // Validate the manifest shape BEFORE write so an invalid manifest can never + // hit disk (the in-memory manifest must round-trip cleanly first). + parseSchemaPackManifest(manifest, { path }); + if (format === 'yaml') { + const yaml = emitYaml(manifest); + // Belt-and-suspenders: re-parse what we're about to write to catch + // any emitter bugs before the rename. + const reparsed = parseYamlMini(yaml); + parseSchemaPackManifest(reparsed, { path }); + writeAtomic(path, yaml); + return; + } + writeAtomic(path, JSON.stringify(manifest, null, 2) + '\n'); +} + +// ──────────────────────────────────────────────────────────────────────── +// withMutation skeleton — the 8-step pipeline above +// ──────────────────────────────────────────────────────────────────────── + +/** + * Run a mutator function against a pack with full safety guarantees: + * atomic write, per-pack lock, audit log, cache + query-cache invalidation. + * + * All 11 mutation primitives below wrap this skeleton — they're each + * ~5 lines that build the transformation function. Adding a new + * primitive only requires writing the pure transformation. + * + * The skeleton is the load-bearing piece for the cathedral's safety + * contract; the primitives are pure data transformations. + */ +export async function withMutation( + packName: string, + opts: MutateOpts, + mutator: (current: SchemaPackManifest) => SchemaPackManifest, + op: MutationOp, + primitiveContext?: { type?: string; prefix?: string }, +): Promise { + const actor: MutationActor = opts.actor ?? 'cli'; + // Step 1: bundled-pack guard (also surfaces in locateMutablePackFile). + // Captured up-front so audit logging can fire even before the lock acquire. + let path: string; + let format: PackFileFormat; + try { + ({ path, format } = locateMutablePackFile(packName)); + } catch (e) { + if (e instanceof SchemaPackMutationError) { + await logMutationFailure({ + op, pack: packName, actor, ...primitiveContext, reason: e.code, + }); + } + throw e; + } + return await withPackLock(packName, opts, async () => { + let current: SchemaPackManifest; + let prevSha8: string; + try { + // Step 3: read + parse current manifest. + current = loadPackFromFile(path); + prevSha8 = await computeManifestSha8(current); + } catch (e) { + const err = new SchemaPackMutationError( + 'PACK_CORRUPT', + `cannot read or parse pack file at ${path}: ${(e as Error).message}`, + { path }, + ); + await logMutationFailure({ op, pack: packName, actor, ...primitiveContext, reason: err.code }); + throw err; + } + let next: SchemaPackManifest; + try { + // Step 4: pure mutator. + next = mutator(current); + } catch (e) { + // Re-throw user-facing SchemaPackMutationError as-is; wrap others. + const wrapped = e instanceof SchemaPackMutationError + ? e + : new SchemaPackMutationError('INVALID_RESULT', (e as Error).message); + await logMutationFailure({ op, pack: packName, actor, ...primitiveContext, reason: wrapped.code }); + throw wrapped; + } + // Step 5: validation gate — file-plane lint rules only. DB-aware + // rules deliberately skip pre-write to keep withMutation hermetic + // (Phase 9 doctor surfaces DB-aware findings after the fact). + const lintReport = await runFilePlaneLintRules(next); + if (!lintReport.ok) { + const msg = lintReport.errors.map((i) => `${i.rule}: ${i.message}`).join('; '); + const err = new SchemaPackMutationError('INVALID_RESULT', `mutation would produce invalid pack: ${msg}`, { errors: lintReport.errors }); + await logMutationFailure({ op, pack: packName, actor, ...primitiveContext, reason: err.code }); + throw err; + } + let newSha8: string; + try { + newSha8 = await computeManifestSha8(next); + // Step 6: atomic write. + writePackManifest(path, next, format); + } catch (e) { + const err = e instanceof SchemaPackMutationError ? e + : new SchemaPackMutationError('IO_ERROR', (e as Error).message, { path }); + await logMutationFailure({ op, pack: packName, actor, ...primitiveContext, reason: err.code }); + throw err; + } + // Step 7: best-effort post-hooks (must NEVER throw or the audit + // shows success but the cache stays stale). + try { + invalidatePackCache(packName); + } catch { /* swallow — cache invalidation must not block mutation success */ } + if (opts.engine) { + try { + await invalidateQueryCache(opts.engine, opts.sourceId); + } catch { /* swallow */ } + } + await logMutationSuccess({ + op, pack: packName, actor, ...primitiveContext, + prev_sha8: prevSha8, new_sha8: newSha8, batch_id: opts.batchId, + }); + return { pack: packName, path, format, prev_sha8: prevSha8, new_sha8: newSha8 }; + // Step 8: withPackLock's finally releases the lock. + }); +} + +// ──────────────────────────────────────────────────────────────────────── +// Validation helpers used by primitives +// ──────────────────────────────────────────────────────────────────────── + +const SLUG_RE = /^[a-z0-9._-]+$/i; + +function validateTypeName(name: unknown): void { + if (typeof name !== 'string' || name.length === 0 || !SLUG_RE.test(name)) { + throw new SchemaPackMutationError( + 'INVALID_RESULT', + `type name must be a slug-shape string [a-z0-9._-]+ (got: ${JSON.stringify(name)})`, + ); + } +} + +function validatePrimitive(prim: unknown): asserts prim is PackPrimitive { + if (typeof prim !== 'string' || !(PACK_PRIMITIVES as readonly string[]).includes(prim)) { + throw new SchemaPackMutationError( + 'INVALID_PRIMITIVE', + `primitive must be one of ${PACK_PRIMITIVES.join('|')} (got: ${JSON.stringify(prim)})`, + ); + } +} + +function validatePrefix(prefix: unknown): void { + if (typeof prefix !== 'string' || prefix.length === 0) { + throw new SchemaPackMutationError( + 'INVALID_RESULT', + `prefix is required and must be a non-empty string (got: ${JSON.stringify(prefix)})`, + ); + } +} + +function findType(manifest: SchemaPackManifest, name: string): PackPageType { + const t = manifest.page_types.find((pt) => pt.name === name); + if (!t) { + throw new SchemaPackMutationError( + 'TYPE_NOT_FOUND', + `type '${name}' is not declared in pack '${manifest.name}'`, + { pack: manifest.name, type: name }, + ); + } + return t; +} + +/** + * Codex C14 guard: removing a type that other types reference via + * aliases / enrichable_types / link_types / frontmatter_links leaves + * dangling refs. We refuse the remove and surface the references so + * the agent can break them first. + */ +function checkNoReferences(manifest: SchemaPackManifest, typeName: string): void { + const refs: string[] = []; + for (const t of manifest.page_types) { + if (t.name === typeName) continue; + if (t.aliases.includes(typeName)) refs.push(`type ${t.name}.aliases`); + } + for (const e of manifest.enrichable_types) { + if (e.type === typeName) refs.push(`enrichable_types[${e.type}]`); + } + for (const lt of manifest.link_types) { + if (lt.inference?.page_type === typeName) refs.push(`link_type ${lt.name}.inference.page_type`); + if (lt.inference?.target_type === typeName) refs.push(`link_type ${lt.name}.inference.target_type`); + } + for (const fl of manifest.frontmatter_links) { + if (fl.page_type === typeName) refs.push(`frontmatter_links[${fl.page_type}]`); + } + if (refs.length > 0) { + throw new SchemaPackMutationError( + 'STILL_REFERENCED', + `cannot remove type '${typeName}' — it is referenced by: ${refs.join(', ')}. Break references first.`, + { pack: manifest.name, type: typeName, references: refs }, + ); + } +} + +// ──────────────────────────────────────────────────────────────────────── +// 11 mutation primitives +// ──────────────────────────────────────────────────────────────────────── + +export interface AddTypeOpts { + name: string; + primitive: PackPrimitive; + prefix: string; + extractable?: boolean; + expertRouting?: boolean; + aliases?: string[]; +} + +export async function addTypeToPack(packName: string, opts: AddTypeOpts, mutateOpts: MutateOpts = {}): Promise { + validateTypeName(opts.name); + validatePrimitive(opts.primitive); + validatePrefix(opts.prefix); + return withMutation(packName, mutateOpts, (m) => { + if (m.page_types.some((pt) => pt.name === opts.name)) { + throw new SchemaPackMutationError( + 'TYPE_EXISTS', + `type '${opts.name}' already exists in pack '${m.name}'`, + { pack: m.name, type: opts.name }, + ); + } + const newType: PackPageType = { + name: opts.name, + primitive: opts.primitive, + path_prefixes: [opts.prefix], + aliases: opts.aliases ?? [], + extractable: opts.extractable ?? false, + expert_routing: opts.expertRouting ?? false, + }; + return { ...m, page_types: [...m.page_types, newType] }; + }, 'add_type', { type: opts.name, prefix: opts.prefix }); +} + +export async function removeTypeFromPack(packName: string, name: string, mutateOpts: MutateOpts = {}): Promise { + validateTypeName(name); + return withMutation(packName, mutateOpts, (m) => { + findType(m, name); // throws TYPE_NOT_FOUND if missing + checkNoReferences(m, name); // codex C14 + return { ...m, page_types: m.page_types.filter((t) => t.name !== name) }; + }, 'remove_type', { type: name }); +} + +export interface UpdateTypeOpts { + name: string; + patch: Partial>; +} + +export async function updateTypeOnPack(packName: string, opts: UpdateTypeOpts, mutateOpts: MutateOpts = {}): Promise { + validateTypeName(opts.name); + if (opts.patch.primitive !== undefined) validatePrimitive(opts.patch.primitive); + return withMutation(packName, mutateOpts, (m) => { + const existing = findType(m, opts.name); + const updated: PackPageType = { ...existing, ...opts.patch, name: existing.name }; + return { ...m, page_types: m.page_types.map((t) => (t.name === opts.name ? updated : t)) }; + }, 'update_type', { type: opts.name }); +} + +export async function addAliasToType(packName: string, typeName: string, alias: string, mutateOpts: MutateOpts = {}): Promise { + validateTypeName(typeName); + validateTypeName(alias); + return withMutation(packName, mutateOpts, (m) => { + const t = findType(m, typeName); + if (t.aliases.includes(alias)) return m; // idempotent + const next: PackPageType = { ...t, aliases: [...t.aliases, alias] }; + return { ...m, page_types: m.page_types.map((pt) => (pt.name === typeName ? next : pt)) }; + }, 'add_alias', { type: typeName }); +} + +export async function removeAliasFromType(packName: string, typeName: string, alias: string, mutateOpts: MutateOpts = {}): Promise { + validateTypeName(typeName); + return withMutation(packName, mutateOpts, (m) => { + const t = findType(m, typeName); + if (!t.aliases.includes(alias)) return m; // idempotent + const next: PackPageType = { ...t, aliases: t.aliases.filter((a) => a !== alias) }; + return { ...m, page_types: m.page_types.map((pt) => (pt.name === typeName ? next : pt)) }; + }, 'remove_alias', { type: typeName }); +} + +export async function addPrefixToType(packName: string, typeName: string, prefix: string, mutateOpts: MutateOpts = {}): Promise { + validateTypeName(typeName); + validatePrefix(prefix); + return withMutation(packName, mutateOpts, (m) => { + const t = findType(m, typeName); + if (t.path_prefixes.includes(prefix)) return m; + const next: PackPageType = { ...t, path_prefixes: [...t.path_prefixes, prefix] }; + return { ...m, page_types: m.page_types.map((pt) => (pt.name === typeName ? next : pt)) }; + }, 'add_prefix', { type: typeName, prefix }); +} + +export async function removePrefixFromType(packName: string, typeName: string, prefix: string, mutateOpts: MutateOpts = {}): Promise { + validateTypeName(typeName); + return withMutation(packName, mutateOpts, (m) => { + const t = findType(m, typeName); + if (!t.path_prefixes.includes(prefix)) return m; + const next: PackPageType = { ...t, path_prefixes: t.path_prefixes.filter((p) => p !== prefix) }; + return { ...m, page_types: m.page_types.map((pt) => (pt.name === typeName ? next : pt)) }; + }, 'remove_prefix', { type: typeName, prefix }); +} + +export interface AddLinkTypeOpts { + name: string; + inverse?: string; + inference?: { regex?: string; page_type?: string; target_type?: string }; +} + +export async function addLinkTypeToPack(packName: string, opts: AddLinkTypeOpts, mutateOpts: MutateOpts = {}): Promise { + if (typeof opts.name !== 'string' || opts.name.length === 0) { + throw new SchemaPackMutationError('INVALID_RESULT', `link_type.name is required`); + } + return withMutation(packName, mutateOpts, (m) => { + if (m.link_types.some((lt) => lt.name === opts.name)) { + throw new SchemaPackMutationError( + 'TYPE_EXISTS', + `link_type '${opts.name}' already exists in pack '${m.name}'`, + { pack: m.name, link: opts.name }, + ); + } + const newLink: PackLinkType = { + name: opts.name, + ...(opts.inverse ? { inverse: opts.inverse } : {}), + ...(opts.inference ? { inference: opts.inference } : {}), + } as PackLinkType; + return { ...m, link_types: [...m.link_types, newLink] }; + }, 'add_link_type', { type: opts.name }); +} + +export async function removeLinkTypeFromPack(packName: string, linkName: string, mutateOpts: MutateOpts = {}): Promise { + return withMutation(packName, mutateOpts, (m) => { + if (!m.link_types.some((lt) => lt.name === linkName)) { + throw new SchemaPackMutationError( + 'TYPE_NOT_FOUND', + `link_type '${linkName}' is not declared in pack '${m.name}'`, + { pack: m.name, link: linkName }, + ); + } + // Check frontmatter_links references too. + const flRefs = m.frontmatter_links.filter((fl) => fl.link_type === linkName); + if (flRefs.length > 0) { + throw new SchemaPackMutationError( + 'STILL_REFERENCED', + `cannot remove link_type '${linkName}' — referenced by frontmatter_links: ${flRefs.map((f) => f.page_type).join(', ')}`, + { pack: m.name, link: linkName }, + ); + } + return { ...m, link_types: m.link_types.filter((lt) => lt.name !== linkName) }; + }, 'remove_link_type', { type: linkName }); +} + +export async function setExtractableOnType(packName: string, typeName: string, value: boolean, mutateOpts: MutateOpts = {}): Promise { + return updateTypeOnPack(packName, { name: typeName, patch: { extractable: value } }, { ...mutateOpts }); +} + +export async function setExpertRoutingOnType(packName: string, typeName: string, value: boolean, mutateOpts: MutateOpts = {}): Promise { + return updateTypeOnPack(packName, { name: typeName, patch: { expert_routing: value } }, { ...mutateOpts }); +} diff --git a/src/core/schema-pack/pack-lock.ts b/src/core/schema-pack/pack-lock.ts new file mode 100644 index 000000000..77ba514b7 --- /dev/null +++ b/src/core/schema-pack/pack-lock.ts @@ -0,0 +1,298 @@ +// v0.40.6.0 Schema Cathedral v3 — per-pack file lock primitive. +// +// `withPackLock(packName, opts, fn)` serializes concurrent mutations of +// the same pack file across processes. Two `gbrain schema add-type foo` +// invocations on the same pack are made safe: the second blocks until the +// first releases (or refuses with `LOCK_BUSY` if a timeout is set). +// +// Design (codex C8 from /plan-eng-review): +// - Atomic acquire via `openSync(lockPath, 'wx')` — the 'wx' flag is +// POSIX `O_CREAT | O_EXCL`, kernel-level atomic. The acquire either +// creates the file or throws EEXIST. There is NO check-then-write +// window. Do NOT copy `src/core/page-lock.ts:79+96`'s +// `existsSync` + `writeFileSync` shape — that's TOCTOU. +// - Stale-lock detection: a holder process may crash without releasing. +// On EEXIST, read the lockfile, check `kill(pid, 0)` for liveness, +// and check `Date.now() - ts > ttlMs`. If either is true, steal. +// - TTL refresh: long-running DB-aware lint/stats can outlive the +// default 60s ttl. While `fn()` runs, a background `setInterval` +// rewrites `ts` every 10s so the lock stays fresh. +// - `--force` semantics: "steal stale lock", NOT "skip locking". Even +// forced acquires go through the same atomic open path — the only +// difference is that on EEXIST + non-stale, force succeeds by +// stealing instead of throwing. +// - Cleanup: `try/finally` unconditionally releases. Refresh interval +// is cleared. Lockfile is unlinked. +// +// Lock path: `~/.gbrain/schema-packs/.locks/.lock`. Per-pack +// so two different packs never block each other. Honors `GBRAIN_HOME` +// via the shared `gbrainPath()` helper. + +import { closeSync, mkdirSync, openSync, readFileSync, unlinkSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { gbrainPath } from '../config.ts'; + +export const DEFAULT_LOCK_TTL_MS = 60_000; +export const REFRESH_INTERVAL_MS = 10_000; +export const DEFAULT_WAIT_TIMEOUT_MS = 0; // 0 = fail-immediately on EEXIST + +export type LockOutcome = 'acquired' | 'stolen_stale' | 'forced'; + +export interface PackLockOpts { + /** TTL in ms before another acquirer considers the lock stale. Default 60s. */ + ttlMs?: number; + /** Steal the lock even if non-stale + live PID. Default false. */ + force?: boolean; + /** Override the lock directory for tests. */ + lockDir?: string; + /** + * Inject a clock for tests (returns ms since epoch). + * Production callers leave undefined. + */ + now?: () => number; + /** + * Inject a PID-liveness probe for tests. Returns true if the PID is alive. + * Production callers leave undefined (defaults to `kill(pid, 0)`). + */ + isPidAlive?: (pid: number) => boolean; +} + +export interface LockFileRecord { + /** Holder process PID. */ + pid: number; + /** Holder hostname (informational only). */ + hostname: string; + /** Last refresh timestamp (ms since epoch). */ + ts: number; + /** Holder's declared TTL in ms. */ + ttlMs: number; +} + +export class PackLockBusyError extends Error { + readonly code = 'LOCK_BUSY' as const; + readonly heldBy: number; + readonly ageMs: number; + readonly ttlMs: number; + constructor(message: string, opts: { heldBy: number; ageMs: number; ttlMs: number }) { + super(message); + this.name = 'PackLockBusyError'; + this.heldBy = opts.heldBy; + this.ageMs = opts.ageMs; + this.ttlMs = opts.ttlMs; + } +} + +function defaultIsPidAlive(pid: number): boolean { + try { + // Signal 0 doesn't deliver, but throws ESRCH if the process is dead. + process.kill(pid, 0); + return true; + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + // EPERM means the process exists but we lack permission to signal. + return code === 'EPERM'; + } +} + +function resolveLockPath(packName: string, lockDir?: string): string { + const dir = lockDir ?? gbrainPath('schema-packs', '.locks'); + return join(dir, `${packName}.lock`); +} + +function readLockFile(path: string): LockFileRecord | null { + try { + const raw = readFileSync(path, 'utf-8'); + const parsed = JSON.parse(raw) as Partial; + if ( + typeof parsed.pid === 'number' && + typeof parsed.ts === 'number' && + typeof parsed.ttlMs === 'number' && + typeof parsed.hostname === 'string' + ) { + return parsed as LockFileRecord; + } + return null; + } catch { + return null; + } +} + +function writeLockRecord(path: string, fd: number, record: LockFileRecord): void { + // We hold an open fd from the atomic create; truncate-and-write via fd. + // Bun's fs.writeFileSync(path, ...) when the file exists is fine here + // because we already own the lock (exclusive create succeeded). + writeFileSync(path, JSON.stringify(record), 'utf-8'); + // Keep the fd open until release so file is held by this process; some + // FS layers prefer this for crash detection. (Functionally a no-op on + // POSIX where unlink() works regardless.) + closeSync(fd); +} + +/** + * Try to atomically acquire the lock. Returns the descriptor on success; + * throws on EEXIST (caller decides whether to steal). Handles ENOENT on + * the parent dir by creating it once. + */ +function atomicAcquire(lockPath: string): number { + try { + return openSync(lockPath, 'wx'); + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + if (code === 'ENOENT') { + mkdirSync(dirname(lockPath), { recursive: true }); + return openSync(lockPath, 'wx'); + } + throw err; + } +} + +/** + * Decide whether a held lock is stale based on TTL + PID liveness. + * Exported for unit testing the policy in isolation. + */ +export function isLockStale( + record: LockFileRecord, + now: number, + isPidAlive: (pid: number) => boolean, +): { stale: boolean; reason: 'ttl_expired' | 'pid_dead' | 'live' } { + const ageMs = now - record.ts; + if (ageMs > record.ttlMs) return { stale: true, reason: 'ttl_expired' }; + if (!isPidAlive(record.pid)) return { stale: true, reason: 'pid_dead' }; + return { stale: false, reason: 'live' }; +} + +/** + * Acquire the lock OR throw `PackLockBusyError`. Steals stale locks + * (per TTL + PID liveness) or when `opts.force` is set. Returns the + * outcome so callers can audit how the acquire resolved. + */ +export function acquirePackLock( + packName: string, + opts: PackLockOpts = {}, +): { lockPath: string; outcome: LockOutcome; record: LockFileRecord } { + const ttlMs = opts.ttlMs ?? DEFAULT_LOCK_TTL_MS; + const now = opts.now ?? Date.now; + const isPidAlive = opts.isPidAlive ?? defaultIsPidAlive; + const lockPath = resolveLockPath(packName, opts.lockDir); + + const tryOnce = (): number | 'EEXIST' => { + try { + return atomicAcquire(lockPath); + } catch (err) { + if ((err as NodeJS.ErrnoException).code === 'EEXIST') return 'EEXIST'; + throw err; + } + }; + + const writeNew = (fd: number, outcome: LockOutcome): { lockPath: string; outcome: LockOutcome; record: LockFileRecord } => { + const record: LockFileRecord = { + pid: process.pid, + hostname: process.env.HOSTNAME ?? 'unknown', + ts: now(), + ttlMs, + }; + writeLockRecord(lockPath, fd, record); + return { lockPath, outcome, record }; + }; + + // First attempt: clean acquire. + const first = tryOnce(); + if (first !== 'EEXIST') return writeNew(first, 'acquired'); + + // EEXIST: inspect. + const existing = readLockFile(lockPath); + if (existing === null) { + // Lockfile is corrupt — treat as stale and steal. + try { unlinkSync(lockPath); } catch { /* race with another stealer; retry below */ } + const retry = tryOnce(); + if (retry === 'EEXIST') { + throw new PackLockBusyError( + `pack ${packName} lock is corrupt and another process re-acquired it during recovery`, + { heldBy: -1, ageMs: 0, ttlMs }, + ); + } + return writeNew(retry, 'stolen_stale'); + } + + const staleness = isLockStale(existing, now(), isPidAlive); + if (staleness.stale || opts.force) { + try { unlinkSync(lockPath); } catch { /* race with another stealer; retry below */ } + const retry = tryOnce(); + if (retry === 'EEXIST') { + // Another stealer won. Surface as busy with the current holder. + const current = readLockFile(lockPath); + const ageMs = current ? now() - current.ts : 0; + throw new PackLockBusyError( + `pack ${packName} lock was stolen by another process during our recovery (pid=${current?.pid ?? '?'})`, + { heldBy: current?.pid ?? -1, ageMs, ttlMs: current?.ttlMs ?? ttlMs }, + ); + } + return writeNew(retry, opts.force ? 'forced' : 'stolen_stale'); + } + + // Live and non-stale: refuse. + throw new PackLockBusyError( + `pack ${packName} is locked by pid=${existing.pid} (held ${Math.round((now() - existing.ts) / 1000)}s, ttl=${Math.round(existing.ttlMs / 1000)}s; --force to steal)`, + { heldBy: existing.pid, ageMs: now() - existing.ts, ttlMs: existing.ttlMs }, + ); +} + +/** + * Refresh the lock's `ts` field so long-running operations don't appear + * stale to a concurrent acquirer. Best-effort: write failures are + * swallowed silently. Returns true on success, false if the lockfile is + * gone (we lost the lock). + */ +function refreshLock(lockPath: string, record: LockFileRecord, now: number): boolean { + try { + const next: LockFileRecord = { ...record, ts: now }; + writeFileSync(lockPath, JSON.stringify(next), 'utf-8'); + return true; + } catch { + return false; + } +} + +/** + * Release the lock by unlinking the file. Idempotent — missing file is + * not an error (the holder may have crashed and another process stole). + */ +function releasePackLock(lockPath: string): void { + try { + unlinkSync(lockPath); + } catch (err) { + if ((err as NodeJS.ErrnoException).code !== 'ENOENT') { + // Don't throw from a finally path — log to stderr and move on. + process.stderr.write(`[pack-lock] release failed for ${lockPath}: ${(err as Error).message}\n`); + } + } +} + +/** + * Run `fn()` with exclusive access to `packName`. Acquires the lock, + * starts a TTL refresh timer, runs `fn`, then unconditionally releases. + * + * Throws `PackLockBusyError` if the lock is held by a live process and + * neither stale nor forced. + */ +export async function withPackLock( + packName: string, + opts: PackLockOpts, + fn: () => Promise | T, +): Promise { + const acquired = acquirePackLock(packName, opts); + const now = opts.now ?? Date.now; + let currentRecord = acquired.record; + const refresh = setInterval(() => { + currentRecord = { ...currentRecord, ts: now() }; + refreshLock(acquired.lockPath, currentRecord, now()); + }, REFRESH_INTERVAL_MS); + // Don't keep the process alive just for the refresh timer. + if (typeof (refresh as NodeJS.Timer).unref === 'function') (refresh as NodeJS.Timer).unref(); + try { + return await fn(); + } finally { + clearInterval(refresh); + releasePackLock(acquired.lockPath); + } +} diff --git a/src/core/schema-pack/query-cache-invalidator.ts b/src/core/schema-pack/query-cache-invalidator.ts new file mode 100644 index 000000000..3112c1d32 --- /dev/null +++ b/src/core/schema-pack/query-cache-invalidator.ts @@ -0,0 +1,50 @@ +// v0.40.6.0 Schema Cathedral v3 — query-cache invalidation hook. +// +// Codex C9: `schema sync --apply` and `schema add-type` change page +// types under cached search rows that were keyed by the OLD knobs_hash +// (which doesn't include schema-pack identity yet — that's a v0.41+ +// design choice). Without invalidation, an agent who mutates the pack +// AND immediately re-queries sees stale results from the pre-mutation +// cache. +// +// The fix: after every successful withMutation, call +// `invalidateQueryCache(engine, sourceId)` which DELETEs all rows for +// the source. Cache rebuilds organically on next search — the only cost +// is one extra LLM expansion / vector call per query for the first few +// requests after a mutation. That's the right trade vs serving stale +// page types. +// +// Reuses the existing SemanticQueryCache.clear() method (already +// PGLite + Postgres parity-safe) rather than reinventing the SQL. + +import type { BrainEngine } from '../engine.ts'; +import { SemanticQueryCache } from '../search/query-cache.ts'; + +export interface InvalidateQueryCacheResult { + rows_invalidated: number; +} + +/** + * Invalidate query_cache rows scoped to a source so search results + * bound to the old knobs_hash don't serve stale page types after + * schema mutations. + * + * Best-effort: failures (e.g. pre-v51 brain without the table) return + * {rows_invalidated: 0} silently. Mutation hot-path must never break + * because the cache invalidator fell over. + * + * `sourceId` omitted clears the whole table. Used by Phase 4 reload and + * any cross-source mutation. + */ +export async function invalidateQueryCache( + engine: BrainEngine, + sourceId?: string, +): Promise { + try { + const cache = new SemanticQueryCache(engine); + const rows_invalidated = await cache.clear(sourceId !== undefined ? { sourceId } : {}); + return { rows_invalidated }; + } catch { + return { rows_invalidated: 0 }; + } +} diff --git a/src/core/schema-pack/registry.ts b/src/core/schema-pack/registry.ts index b1a60538a..09b60fb21 100644 --- a/src/core/schema-pack/registry.ts +++ b/src/core/schema-pack/registry.ts @@ -1,11 +1,31 @@ // v0.38 schema pack registry — load, cache, resolve active pack. // +// ┌──────────────────────────────────────────────────────┐ +// │ loadActivePack lifecycle (per process, v0.40.6.0) │ +// └──────────────────────────────────────────────────────┘ +// │ +// ┌────────────────────────┼────────────────────────┐ +// ▼ ▼ ▼ +// cache miss cache hit cache hit + TTL expired +// │ │ │ +// fresh load STAT_TTL_MS gate statSync compare every file +// (resolvePack) (~10ns fast return) in the extends chain +// │ │ │ +// │ │ ┌──────────┴──────────┐ +// │ │ ▼ ▼ +// │ │ every mtime unchanged any mtime changed +// │ │ │ │ +// │ │ refresh lastStatMs invalidate(name) + +// │ │ return cached extends-chain cascade +// │ │ (codex C6) +// ▼ ▼ │ +// byName.set(name, entry) return cached fresh load +// // Pack resolution chain (7 tiers per D13, tier-1 trust-gated): // 1. Per-call `schema_pack` opt — CLI only (`ctx.remote === false`). // Rejected for `ctx.remote === true` (D13 trust boundary). // 2. `GBRAIN_SCHEMA_PACK` env var -// 3. Per-source DB config key `schema_pack.source.` (dots, not -// colons — codex F16, matches `models.tier.*` convention) +// 3. Per-source DB config key `schema_pack.source.` // 4. Brain-wide DB config key `schema_pack` // 5. `gbrain.yml schema:` section // 6. `~/.gbrain/config.json schema_pack` field @@ -14,20 +34,31 @@ // Extends chain semantics (E4): // - Depth tracked via BFS during resolve. // - Soft warn to stderr at depth > 4. -// - Hard reject at depth > 8 with paste-ready "shorten your extends -// chain" hint. +// - Hard reject at depth > 8. // -// Cache: resolved packs are cached in-memory by pack-identity -// (`@+`). Mtime check via `loadPackFromFile` on -// disk-backed packs invalidates the cache when a user edits the -// manifest. +// v0.40.6.0 cache invariants (codex C6 + D11 + D13): +// - Cache key is the pack NAME (not identity sha8). Per-name cache entry +// records the resolved pack PLUS every file path that fed it AND the +// identities of every parent in the extends chain. +// - Cache hits go through a stat-TTL gate (default 1000ms via +// STAT_TTL_MS, env override GBRAIN_PACK_STAT_TTL_MS). Inside the +// window: hot-path return (~10ns). Outside: statSync each file; if +// any mtime changed, invalidate by name + cascade to every dependent. +// - invalidatePackCache(name) walks the reverse extends-graph and +// evicts every pack that has `name` in its chain. Without the cascade, +// editing a parent silently leaves children stale (the codex C6 bug). +// - The PUBLIC `ResolvedPack.identity` field is unchanged +// (`@+`); the composite cache key lives only +// inside the registry. +import { statSync } from 'node:fs'; import type { SchemaPackManifest } from './manifest-v1.ts'; import { computeManifestSha8, packIdentity } from './manifest-v1.ts'; import { computeAliasClosureHash, buildAliasGraph, type AliasGraph } from './closure.ts'; export const EXTENDS_DEPTH_WARN = 4 as const; export const EXTENDS_DEPTH_HARD_CAP = 8 as const; +export const STAT_TTL_MS_DEFAULT = 1000 as const; export class ExtendsChainTooDeepError extends Error { readonly depth: number; @@ -49,109 +80,188 @@ export class UnknownPackError extends Error { } } -/** - * The fully resolved pack — manifest + computed graph + identity hash. - * Returned by `loadActivePack`; cached by registry until the source - * manifest mtime changes. - */ export interface ResolvedPack { manifest: SchemaPackManifest; - identity: string; // `@+` + identity: string; // `@+` (child only — wire-stable) manifest_sha8: string; alias_closure_hash: string; alias_graph: AliasGraph; } -/** - * 7-tier resolution chain. Returns the pack NAME to load (resolved - * packs are fetched separately via `loadPackByName`). Tier 1 (per-call) - * is gated on `remote === false`; remote callers passing the opt get - * `permission_denied` from operations.ts before reaching here. - */ export interface ResolutionInput { - /** Tier 1: per-call opt. ONLY honored when `remote === false`. */ perCall?: string; - /** Tier 1 trust gate. `true` = MCP/OAuth caller; rejects per-call opt. */ remote: boolean; - /** Tier 3: per-source DB config map (source_id → pack name). */ perSourceDb?: ReadonlyMap; - /** Source ID the query targets (tier 3 lookup). */ sourceId?: string; - /** Tier 2: env var (`GBRAIN_SCHEMA_PACK`). */ envVar?: string; - /** Tier 4: brain-wide DB config. */ dbConfig?: string; - /** Tier 5: gbrain.yml schema.pack field. */ gbrainYml?: string; - /** Tier 6: ~/.gbrain/config.json schema_pack field. */ homeConfig?: string; } -/** Resolved tier + pack name. `source` documents which tier won. */ export interface ResolutionResult { pack_name: string; source: 'per-call' | 'env' | 'per-source-db' | 'db-config' | 'gbrain-yml' | 'home-config' | 'default'; } export function resolveActivePackName(input: ResolutionInput): ResolutionResult { - // Tier 1: per-call opt (CLI only). if (input.perCall && input.remote === false) { return { pack_name: input.perCall, source: 'per-call' }; } - // Tier 2: env var. if (input.envVar) return { pack_name: input.envVar, source: 'env' }; - // Tier 3: per-source DB config. if (input.sourceId && input.perSourceDb?.has(input.sourceId)) { return { pack_name: input.perSourceDb.get(input.sourceId)!, source: 'per-source-db' }; } - // Tier 4: brain-wide DB. if (input.dbConfig) return { pack_name: input.dbConfig, source: 'db-config' }; - // Tier 5: gbrain.yml schema: if (input.gbrainYml) return { pack_name: input.gbrainYml, source: 'gbrain-yml' }; - // Tier 6: ~/.gbrain/config.json if (input.homeConfig) return { pack_name: input.homeConfig, source: 'home-config' }; - // Tier 7: default return { pack_name: 'gbrain-base', source: 'default' }; } /** - * In-memory cache keyed by pack-identity. Resolved packs are immutable - * once loaded; the cache is invalidated by mtime check in the - * disk-loader callers. + * Per-name cache entry. Tracks the resolved pack PLUS the file-stat + * snapshot every file in the extends chain fed at resolve time. The + * stat snapshot is what the cross-process stat-TTL gate compares + * against on each loadActivePack call. */ -const _packCache = new Map(); +interface CacheEntry { + resolved: ResolvedPack; + /** Names that fed this entry (this pack + every parent transitively). */ + chain: ReadonlyArray; + /** Stat snapshot per file at resolve time. */ + files: ReadonlyArray<{ name: string; path: string; mtimeMs: number }>; + /** Last time we stat()'d the files. Date.now() ms. */ + lastStatMs: number; +} + +const _byName = new Map(); /** Test seam — clears the in-process resolver cache. */ export function _resetPackCacheForTests(): void { - _packCache.clear(); + _byName.clear(); +} + +/** + * Resolve the effective STAT_TTL_MS, honoring the + * `GBRAIN_PACK_STAT_TTL_MS` env override. Invalid values fall back to + * the default with no warning (this is a power-user knob). + */ +function resolveStatTtlMs(): number { + const raw = process.env.GBRAIN_PACK_STAT_TTL_MS; + if (!raw) return STAT_TTL_MS_DEFAULT; + const parsed = Number.parseInt(raw, 10); + if (Number.isFinite(parsed) && parsed >= 0) return parsed; + return STAT_TTL_MS_DEFAULT; +} + +/** + * Cheap statSync that returns Infinity on error so callers treat + * disappearing files as "changed" (forcing reload). + */ +function safeMtimeMs(path: string): number { + try { + return statSync(path).mtimeMs; + } catch { + return Number.POSITIVE_INFINITY; + } +} + +/** + * Check whether a cached entry's file snapshot is still fresh on disk. + * Returns true when EVERY file's mtime matches the snapshot. + */ +function snapshotMatches(files: ReadonlyArray<{ path: string; mtimeMs: number }>): boolean { + for (const f of files) { + if (safeMtimeMs(f.path) !== f.mtimeMs) return false; + } + return true; +} + +/** + * Walk the reverse extends-graph: every cached entry whose `chain` + * contains `name`. The set is unbounded in principle but bounded in + * practice by EXTENDS_DEPTH_HARD_CAP × installed packs (typically <50). + */ +function findDependents(name: string): string[] { + const out: string[] = []; + for (const [cachedName, entry] of _byName) { + if (entry.chain.includes(name)) out.push(cachedName); + } + return out; +} + +/** + * Invalidate the cache for a pack name AND every pack that extends it + * (transitive — the codex C6 fix). When called with no argument, + * invalidates everything. + * + * Called automatically by `withMutation` (Phase 2) after every + * successful pack mutation; also exposed via `gbrain schema reload`. + */ +export function invalidatePackCache(name?: string): { invalidated: string[] } { + if (name === undefined) { + const all = [..._byName.keys()]; + _byName.clear(); + return { invalidated: all }; + } + const dependents = findDependents(name); + // The pack itself + all dependents. + const toEvict = Array.from(new Set([name, ...dependents])); + for (const n of toEvict) _byName.delete(n); + return { invalidated: toEvict }; +} + +/** Test-only access for assertions on the cache shape. */ +export function _cacheSizeForTests(): number { + return _byName.size; +} + +/** Test-only access for assertions on which names are cached. */ +export function _cacheNamesForTests(): string[] { + return [..._byName.keys()]; } /** * Resolve + cache a manifest. Loads parent packs via the `loadByName` * dependency, tracks extends-chain depth, applies the E4 cap. * - * Pass `loadByName` as a dependency so the registry doesn't have to - * own filesystem layout (tests inject a mock; production wires the - * disk loader). + * v0.40.6.0: cache is name-keyed and tracks file-stat snapshots so the + * stat-TTL gate (inside `loadActivePack`) can detect cross-process + * mutations without re-reading the bytes. + * + * `loadByPath` is the disk path resolver for each name in the extends + * chain (used for the file-stat snapshot). Optional — when omitted, the + * snapshot is empty and stat-TTL becomes a no-op for this entry (used + * by tests that drive synthetic manifests with no disk backing). */ export async function resolvePack( manifest: SchemaPackManifest, loadByName: (name: string) => Promise, - opts: { onDepthWarn?: (depth: number, chain: string[]) => void } = {}, + opts: { + onDepthWarn?: (depth: number, chain: string[]) => void; + loadByPath?: (name: string) => string | null; + } = {}, ): Promise { const sha8 = await computeManifestSha8(manifest); const id = packIdentity(manifest, sha8); - const cached = _packCache.get(id); - if (cached) return cached; - // Walk extends chain to enforce depth cap. + // Reference-equality fast path: if a previous resolvePack(manifest, ...) + // produced the SAME identity, return the cached resolved object. This + // preserves the v0.38 contract that two calls with the same manifest + // bytes return the same JS object reference. + const existing = _byName.get(manifest.name); + if (existing && existing.resolved.identity === id) { + return existing.resolved; + } + + // Walk extends chain to enforce depth cap AND collect names for the + // cache snapshot (codex C6 — child cache entry must remember every + // parent so invalidatePackCache(parentName) can cascade). const chain: string[] = [manifest.name]; let cursor: SchemaPackManifest | null = manifest; while (cursor?.extends) { const parentName = cursor.extends; if (chain.includes(parentName)) { - // Cycle in extends graph — should be impossible for legit packs; - // safety net. throw new ExtendsChainTooDeepError(chain.length, [...chain, parentName]); } chain.push(parentName); @@ -164,10 +274,8 @@ export async function resolvePack( cursor = await loadByName(parentName); } - // For v0.38 skeleton: the closure is computed on the manifest itself. - // T7 Phase B will add full extends-merging (child-wins) here. For now, - // we treat manifest.page_types as the effective set (gbrain-base - // codegen places everything in the seed manifest directly). + // For v0.38 skeleton: closure is computed on the manifest itself. + // Full extends-merging (child-wins) is the v0.41+ T20 follow-up. const alias_graph = buildAliasGraph(manifest); const alias_closure_hash = await computeAliasClosureHash(manifest); @@ -178,6 +286,51 @@ export async function resolvePack( alias_closure_hash, alias_graph, }; - _packCache.set(id, resolved); + + // Capture file-stat snapshot for the stat-TTL gate. Skip names that + // the locator can't resolve (synthetic manifests in tests). + const files: Array<{ name: string; path: string; mtimeMs: number }> = []; + if (opts.loadByPath) { + for (const n of chain) { + const path = opts.loadByPath(n); + if (path === null) continue; + files.push({ name: n, path, mtimeMs: safeMtimeMs(path) }); + } + } + + _byName.set(manifest.name, { + resolved, + chain: [...chain], + files, + lastStatMs: Date.now(), + }); return resolved; } + +/** + * Try to return a cached resolved pack for `name` without re-reading the + * manifest from disk. Returns null on cache miss OR when the stat-TTL + * gate detects a file change (which triggers eviction + cascade). + * + * The TTL gate keeps the hot path cheap: most calls inside the 1-second + * window return immediately (~10ns) without statting. Outside the + * window: one statSync per file in the extends chain (~50µs per file). + * Worst-case latency for a daemon picking up an operator's mutation: + * 1 second. + */ +export function tryCachedPack(name: string): ResolvedPack | null { + const entry = _byName.get(name); + if (!entry) return null; + const ttl = resolveStatTtlMs(); + const ageMs = Date.now() - entry.lastStatMs; + if (ageMs < ttl) return entry.resolved; + // TTL expired: stat all files. If any changed, cascade-invalidate. + if (!snapshotMatches(entry.files)) { + invalidatePackCache(name); + return null; + } + // Snapshot still fresh: refresh lastStatMs so the next hot-path return + // is cheap again. + _byName.set(name, { ...entry, lastStatMs: Date.now() }); + return entry.resolved; +} diff --git a/src/core/schema-pack/stats.ts b/src/core/schema-pack/stats.ts new file mode 100644 index 000000000..6fe42634a --- /dev/null +++ b/src/core/schema-pack/stats.ts @@ -0,0 +1,245 @@ +// v0.40.6.0 Schema Cathedral v3 — schema_stats pure core function. +// +// Reports per-type page counts, untyped-coverage percentage, and +// dead-prefix detection (declared prefix with zero matching pages — +// a "this type has no content" signal that helps agents spot +// mis-declared paths). +// +// Multi-source aware: accepts `sourceIds` (federated read) OR +// `sourceId` (single) OR nothing (aggregate across all sources). Uses +// the same WHERE shape as the rest of the read-path codebase. +// +// PGLite + Postgres parity via `executeRaw`. Soft-deletes filtered. +// +// Pure core: returns structured data; CLI handler in Phase 4 wraps +// for human + JSON output, MCP handler in Phase 7 wires it through +// the operation envelope. + +import type { BrainEngine } from '../engine.ts'; +import { loadActivePackBestEffort } from './best-effort.ts'; +import type { OperationContext } from '../operations.ts'; + +export interface StatsOpts { + /** Single source scope. Omit + omit sourceIds for whole-brain aggregate. */ + sourceId?: string; + /** Federated read scope (overrides sourceId when set). */ + sourceIds?: string[]; +} + +export interface TypeStats { + /** Type name as it appears in the DB `pages.type` column. */ + type: string; + /** Page count for this type within the scope. */ + count: number; +} + +export interface PerSourceStats { + source_id: string; + total_pages: number; + typed_pages: number; + untyped_pages: number; + coverage: number; + by_type: TypeStats[]; +} + +export interface DeadPrefixHint { + /** Type name declared in the pack. */ + type: string; + /** Prefix declared in the pack with zero matching DB pages. */ + prefix: string; +} + +export interface StatsResult { + schema_version: 1; + /** Pack identity at stats time (null when no pack loaded). */ + pack_identity: string | null; + /** Aggregate across all scoped sources. */ + aggregate: PerSourceStats; + /** Per-source breakdown when multiple sources are in scope. */ + per_source: PerSourceStats[]; + /** Pack-declared prefixes that match zero pages — likely mis-declared. */ + dead_prefixes: DeadPrefixHint[]; +} + +interface RawCountRow { + source_id: string | null; + type: string | null; + cnt: string; +} + +function computeCoverage(typed: number, total: number): number { + if (total === 0) return 1.0; // vacuous truth — matches getBrainScore pattern + return Math.round((typed / total) * 10000) / 10000; +} + +function aggregateRows(rows: RawCountRow[]): PerSourceStats[] { + const bySource = new Map }>(); + for (const r of rows) { + const sid = r.source_id ?? 'default'; + const cnt = parseInt(r.cnt, 10) || 0; + if (!bySource.has(sid)) { + bySource.set(sid, { typed: 0, untyped: 0, total: 0, byType: new Map() }); + } + const bucket = bySource.get(sid)!; + if (r.type === null || r.type === '') { + bucket.untyped += cnt; + } else { + bucket.typed += cnt; + bucket.byType.set(r.type, (bucket.byType.get(r.type) ?? 0) + cnt); + } + bucket.total += cnt; + } + const out: PerSourceStats[] = []; + for (const [sid, b] of bySource) { + out.push({ + source_id: sid, + total_pages: b.total, + typed_pages: b.typed, + untyped_pages: b.untyped, + coverage: computeCoverage(b.typed, b.total), + by_type: [...b.byType.entries()] + .map(([type, count]) => ({ type, count })) + .sort((a, b) => b.count - a.count || a.type.localeCompare(b.type)), + }); + } + // Sort sources alphabetically for stable output. + out.sort((a, b) => a.source_id.localeCompare(b.source_id)); + return out; +} + +function mergeAggregate(per: PerSourceStats[]): PerSourceStats { + let typed = 0, untyped = 0, total = 0; + const byType = new Map(); + for (const s of per) { + typed += s.typed_pages; + untyped += s.untyped_pages; + total += s.total_pages; + for (const t of s.by_type) byType.set(t.type, (byType.get(t.type) ?? 0) + t.count); + } + return { + source_id: '*', + total_pages: total, + typed_pages: typed, + untyped_pages: untyped, + coverage: computeCoverage(typed, total), + by_type: [...byType.entries()] + .map(([type, count]) => ({ type, count })) + .sort((a, b) => b.count - a.count || a.type.localeCompare(b.type)), + }; +} + +/** + * Query the DB for stats. Multi-source aware: + * - sourceIds[] → WHERE source_id = ANY($1::text[]) + * - sourceId → WHERE source_id = $1 + * - neither → no source filter (whole brain) + * + * Always: `WHERE deleted_at IS NULL`. + * + * Returns rows grouped by (source_id, type) so we can compute per-source + * + aggregate in one read. + */ +async function fetchCountRows(engine: BrainEngine, opts: StatsOpts): Promise { + let where = 'WHERE deleted_at IS NULL'; + const params: unknown[] = []; + if (opts.sourceIds && opts.sourceIds.length > 0) { + where += ` AND source_id = ANY($1::text[])`; + params.push(opts.sourceIds); + } else if (opts.sourceId) { + where += ` AND source_id = $1`; + params.push(opts.sourceId); + } + // pages.type is NOT NULL in the schema — empty string represents + // "untyped" (legacy + put_page fallback per D12). Normalize both + // empty-string and NULL to the same bucket via NULLIF. + const sql = ` + SELECT + COALESCE(source_id, 'default') AS source_id, + NULLIF(type, '') AS type, + COUNT(*)::text AS cnt + FROM pages + ${where} + GROUP BY source_id, NULLIF(type, '') + ORDER BY source_id, NULLIF(type, '') NULLS LAST + `; + try { + return await engine.executeRaw(sql, params); + } catch { + // Empty / pre-init brain: pages table may not exist yet. + return []; + } +} + +/** + * Per-prefix existence check for dead-prefix detection. One COUNT(*) + * per declared prefix in the active pack. Cheap on small brains; + * Phase 4 CLI offers a `--no-dead-prefix-scan` flag to opt out on + * brains where this matters. + */ +async function detectDeadPrefixes( + engine: BrainEngine, + pack: { manifest: { page_types: ReadonlyArray<{ name: string; path_prefixes: ReadonlyArray }> } }, + opts: StatsOpts, +): Promise { + const hints: DeadPrefixHint[] = []; + let sourceWhere = ''; + const sourceParam: unknown[] = []; + if (opts.sourceIds && opts.sourceIds.length > 0) { + sourceWhere = ` AND source_id = ANY($2::text[])`; + sourceParam.push(opts.sourceIds); + } else if (opts.sourceId) { + sourceWhere = ` AND source_id = $2`; + sourceParam.push(opts.sourceId); + } + for (const t of pack.manifest.page_types) { + for (const prefix of t.path_prefixes) { + try { + const rows = await engine.executeRaw<{ cnt: string }>( + `SELECT COUNT(*)::text AS cnt FROM pages + WHERE deleted_at IS NULL + AND source_path LIKE $1${sourceWhere}`, + [`${prefix}%`, ...sourceParam], + ); + const cnt = parseInt(rows[0]?.cnt ?? '0', 10) || 0; + if (cnt === 0) { + hints.push({ type: t.name, prefix }); + } + } catch { + // Skip on engine error (no pages table yet, etc.). + continue; + } + } + } + return hints; +} + +/** + * Pure core for `gbrain schema stats` (CLI) AND `schema_stats` MCP op. + * Returns the StatsResult ready for human-formatter, JSON output, or + * operation envelope wrapping. + */ +export async function runStatsCore( + ctx: OperationContext, + opts: StatsOpts = {}, +): Promise { + const rows = await fetchCountRows(ctx.engine, opts); + const per_source = aggregateRows(rows); + const aggregate = mergeAggregate(per_source); + + // Pack identity + dead-prefix scan — best-effort. + let pack_identity: string | null = null; + let dead_prefixes: DeadPrefixHint[] = []; + const pack = await loadActivePackBestEffort(ctx); + if (pack) { + pack_identity = pack.identity; + dead_prefixes = await detectDeadPrefixes(ctx.engine, pack, opts); + } + + return { + schema_version: 1, + pack_identity, + aggregate, + per_source, + dead_prefixes, + }; +} diff --git a/src/core/schema-pack/sync.ts b/src/core/schema-pack/sync.ts new file mode 100644 index 000000000..068d8dad5 --- /dev/null +++ b/src/core/schema-pack/sync.ts @@ -0,0 +1,216 @@ +// v0.40.6.0 Schema Cathedral v3 — schema_sync pure core function. +// +// For each path_prefix declared in the active pack, find pages with +// NULL type matching that prefix and backfill `pages.type` to the +// declared type. Dry-run by default; `--apply` performs the UPDATEs. +// +// D14 chunked UPDATE: 1000-row batches. Each chunk completes in <100ms, +// releases the row lock between batches, never wedges concurrent +// writers (the bug class v0.22.1 statement_timeout work paid for). +// +// Codex C5: write-side source scoping. Mutations use the caller's +// `ctx.sourceId` directly (write authority), NOT `sourceScopeOpts` +// which is read-side and can inherit OAuth federation reads. Phase 7's +// MCP `schema_apply_mutations` op enforces this at the dispatch layer +// too. +// +// PGLite + Postgres parity via `executeRaw`. + +import type { BrainEngine } from '../engine.ts'; +import { loadActivePackBestEffort } from './best-effort.ts'; +import type { OperationContext } from '../operations.ts'; + +export interface SyncOpts { + /** Apply UPDATE statements. Default false (dry-run). */ + apply?: boolean; + /** + * Source ID to scope the sync. Write-side (codex C5): mutations target + * the caller's authorized source, not read federation. Omit for whole- + * brain sync (typically only CLI / autopilot path). + */ + sourceId?: string; + /** Per-batch row cap (D14). Default 1000. */ + batchSize?: number; + /** Progress callback fired per batch on apply path. */ + onProgress?: (info: { type: string; prefix: string; appliedSoFar: number }) => void; +} + +export interface PerPrefixResult { + type: string; + prefix: string; + /** Untyped pages matching this prefix at dry-run time. */ + would_apply: number; + /** Sample of slugs (capped at 10) for the agent's drilldown. */ + sample_slugs: string[]; + /** Whether the prefix matched zero pages (dead-prefix hint). */ + dead_prefix: boolean; + /** Rows actually updated. Equal to would_apply on a clean apply, + * 0 on dry-run, possibly less if concurrent writer claimed some. */ + applied: number; +} + +export interface SyncResult { + schema_version: 1; + apply: boolean; + pack_identity: string | null; + per_prefix: PerPrefixResult[]; + /** Total rows that would be / were updated across all prefixes. */ + total_would_apply: number; + total_applied: number; +} + +/** + * Count + sample untyped pages matching a prefix. Used by both dry-run + * (sample is the drilldown signal) and apply (count for would_apply). + */ +async function probePrefix( + engine: BrainEngine, + prefix: string, + sourceId: string | undefined, +): Promise<{ count: number; sample: string[] }> { + let where = `WHERE deleted_at IS NULL AND (type IS NULL OR type = '') AND source_path LIKE $1`; + const params: unknown[] = [`${prefix}%`]; + if (sourceId) { + where += ` AND source_id = $2`; + params.push(sourceId); + } + try { + const cntRows = await engine.executeRaw<{ cnt: string }>( + `SELECT COUNT(*)::text AS cnt FROM pages ${where}`, + params, + ); + const count = parseInt(cntRows[0]?.cnt ?? '0', 10) || 0; + if (count === 0) return { count: 0, sample: [] }; + const sampleRows = await engine.executeRaw<{ slug: string }>( + `SELECT slug FROM pages ${where} ORDER BY slug LIMIT 10`, + params, + ); + return { count, sample: sampleRows.map((r) => r.slug) }; + } catch { + return { count: 0, sample: [] }; + } +} + +/** + * Apply the type assignment in 1000-row chunks (D14). Each loop + * iteration: SELECT a window of matching IDs, UPDATE them in one + * statement, count returned rows. Stops when a batch returns zero + * (idempotent: re-running finds nothing to update). + * + * Returns the total rows updated. + */ +async function applyTypeAssignment( + engine: BrainEngine, + type: string, + prefix: string, + sourceId: string | undefined, + batchSize: number, + onProgress?: (appliedSoFar: number) => void, +): Promise { + let totalApplied = 0; + let sourceWhere = ''; + // Param indices for the subquery: 1=type (used in the outer UPDATE), + // 2=prefix, 3=batchSize, 4=sourceId (when scoped). + const sourceParams: unknown[] = []; + if (sourceId) { + sourceWhere = ` AND source_id = $4`; + sourceParams.push(sourceId); + } + // Loop guard: max 10000 iterations protects against runaway. + for (let i = 0; i < 10000; i++) { + try { + const rows = await engine.executeRaw<{ updated: string }>( + `WITH win AS ( + SELECT id FROM pages + WHERE deleted_at IS NULL + AND (type IS NULL OR type = '') + AND source_path LIKE $2${sourceWhere} + LIMIT $3 + ), + upd AS ( + UPDATE pages SET type = $1 + WHERE id IN (SELECT id FROM win) + RETURNING 1 + ) + SELECT COUNT(*)::text AS updated FROM upd`, + [type, `${prefix}%`, batchSize, ...sourceParams], + ); + const batchCount = parseInt(rows[0]?.updated ?? '0', 10) || 0; + if (batchCount === 0) break; + totalApplied += batchCount; + onProgress?.(totalApplied); + // Safety net: if a batch returned less than batchSize, we're done. + if (batchCount < batchSize) break; + } catch (e) { + // Surface the error — sync failures are real and should fail loud. + throw new Error(`schema sync failed at prefix '${prefix}' → type '${type}': ${(e as Error).message}`); + } + } + return totalApplied; +} + +/** + * Pure core for `gbrain schema sync` (CLI) AND `schema_sync` MCP op. + * + * Dry-run by default: probes each declared prefix for untyped page + * count + a 10-slug sample (the agent's drilldown signal). With + * apply=true: chunked UPDATE per prefix. + */ +export async function runSyncCore( + ctx: OperationContext, + opts: SyncOpts = {}, +): Promise { + const apply = opts.apply ?? false; + const batchSize = Math.max(1, Math.min(10000, opts.batchSize ?? 1000)); + const sourceId = opts.sourceId; // codex C5: write-side scoping + + const pack = await loadActivePackBestEffort(ctx); + if (!pack) { + return { + schema_version: 1, + apply, + pack_identity: null, + per_prefix: [], + total_would_apply: 0, + total_applied: 0, + }; + } + const per_prefix: PerPrefixResult[] = []; + let total_would_apply = 0; + let total_applied = 0; + for (const t of pack.manifest.page_types) { + for (const prefix of t.path_prefixes) { + const probe = await probePrefix(ctx.engine, prefix, sourceId); + const dead_prefix = probe.count === 0; + let applied = 0; + if (apply && probe.count > 0) { + applied = await applyTypeAssignment( + ctx.engine, + t.name, + prefix, + sourceId, + batchSize, + (n) => opts.onProgress?.({ type: t.name, prefix, appliedSoFar: n }), + ); + } + per_prefix.push({ + type: t.name, + prefix, + would_apply: probe.count, + sample_slugs: probe.sample, + dead_prefix, + applied, + }); + total_would_apply += probe.count; + total_applied += applied; + } + } + return { + schema_version: 1, + apply, + pack_identity: pack.identity, + per_prefix, + total_would_apply, + total_applied, + }; +} diff --git a/test/operations-schema-pack.test.ts b/test/operations-schema-pack.test.ts new file mode 100644 index 000000000..2ed47bbc0 --- /dev/null +++ b/test/operations-schema-pack.test.ts @@ -0,0 +1,356 @@ +// v0.40.6.0 — MCP op contract tests for the 9 new schema-pack operations. +// +// Verifies: scope declarations, ctx routing, source-scoping, atomic +// batched mutations via schema_apply_mutations (D10 + codex F2), and +// the audit-log actor capture for MCP clients (D2 + D20). + +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it } from 'bun:test'; +import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { operationsByName } from '../src/core/operations.ts'; +import { + __setPackLocatorForTests, + _resetPackLocatorForTests, +} from '../src/core/schema-pack/load-active.ts'; +import { _resetPackCacheForTests } from '../src/core/schema-pack/registry.ts'; +import type { OperationContext } from '../src/core/operations.ts'; +import { withEnv } from './helpers/with-env.ts'; + +let engine: PGLiteEngine; +let tmpDir: string; +let auditDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + _resetPackCacheForTests(); + _resetPackLocatorForTests(); + tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-ops-schema-test-')); + auditDir = mkdtempSync(join(tmpdir(), 'gbrain-ops-schema-audit-')); +}); + +afterEach(() => { + for (const d of [tmpDir, auditDir]) { + try { rmSync(d, { recursive: true, force: true }); } catch { /* swallow */ } + } +}); + +function ctxOf(opts: { remote?: boolean; clientId?: string; sourceId?: string } = {}): OperationContext { + return { + engine, + config: {}, + logger: { info: () => {}, warn: () => {}, error: () => {} }, + dryRun: false, + remote: opts.remote ?? true, + sourceId: opts.sourceId, + auth: opts.clientId ? { clientId: opts.clientId, scopes: ['admin'] } : undefined, + } as unknown as OperationContext; +} + +function seedPack(packName: string): string { + const dir = join(tmpDir, '.gbrain', 'schema-packs', packName); + mkdirSync(dir, { recursive: true }); + const path = join(dir, 'pack.yaml'); + writeFileSync(path, `api_version: gbrain-schema-pack-v1 +name: ${packName} +version: 1.0.0 +description: "" +gbrain_min_version: 0.38.0 +extends: null +borrow_from: [] +page_types: + - name: person + primitive: entity + path_prefixes: + - people/ + aliases: [] + extractable: false + expert_routing: false +link_types: [] +frontmatter_links: [] +takes_kinds: + - fact + - take + - bet + - hunch +enrichable_types: [] +filing_rules: [] +`, 'utf-8'); + return path; +} + +// ── Scope + localOnly declarations ────────────────────────────────────── + +describe('operation declarations', () => { + it('get_active_schema_pack is read scope, NOT localOnly', () => { + const op = operationsByName.get_active_schema_pack!; + expect(op.scope).toBe('read'); + expect(op.localOnly).toBeUndefined(); + }); + + it('list_schema_packs is read scope, NOT localOnly', () => { + expect(operationsByName.list_schema_packs!.scope).toBe('read'); + expect(operationsByName.list_schema_packs!.localOnly).toBeUndefined(); + }); + + it('schema_stats / schema_lint / schema_graph / schema_explain_type / schema_review_orphans are all read scope', () => { + for (const name of ['schema_stats', 'schema_lint', 'schema_graph', 'schema_explain_type', 'schema_review_orphans']) { + expect(operationsByName[name]!.scope).toBe('read'); + expect(operationsByName[name]!.localOnly).toBeUndefined(); + } + }); + + it('schema_apply_mutations is admin scope, NOT localOnly (D2)', () => { + const op = operationsByName.schema_apply_mutations!; + expect(op.scope).toBe('admin'); + expect(op.localOnly).toBeUndefined(); + expect(op.mutating).toBe(true); + }); + + it('reload_schema_pack is admin scope, NOT localOnly (D2)', () => { + const op = operationsByName.reload_schema_pack!; + expect(op.scope).toBe('admin'); + expect(op.localOnly).toBeUndefined(); + }); +}); + +// ── get_active_schema_pack ────────────────────────────────────────────── + +describe('get_active_schema_pack', () => { + it('returns identity packet for the bundled gbrain-base pack', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: undefined }, async () => { + const result = await operationsByName.get_active_schema_pack!.handler(ctxOf(), {}) as Record; + expect(result.pack_name).toBe('gbrain-base'); + expect(result.page_types_count).toBeGreaterThan(0); + expect(typeof result.sha8).toBe('string'); + expect(typeof result.source_tier).toBe('string'); + expect(typeof result.primitive_summary).toBe('object'); + }); + }); +}); + +// ── list_schema_packs ────────────────────────────────────────────────── + +describe('list_schema_packs', () => { + it('returns bundled + installed', async () => { + await withEnv({ GBRAIN_HOME: tmpDir }, async () => { + seedPack('mine'); + const result = await operationsByName.list_schema_packs!.handler(ctxOf(), {}) as { bundled: string[]; installed: string[] }; + expect(result.bundled).toContain('gbrain-base'); + expect(result.installed).toContain('mine'); + }); + }); +}); + +// ── schema_stats ─────────────────────────────────────────────────────── + +describe('schema_stats', () => { + it('returns coverage + per-source breakdown', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: undefined }, async () => { + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, source_path, type, title, compiled_truth, timeline, content_hash) + VALUES ('a', 'default', 'people/alice.md', 'person', 'a', '', '', '')`, + ); + const result = await operationsByName.schema_stats!.handler(ctxOf(), {}) as Record; + expect((result.aggregate as { total_pages: number }).total_pages).toBe(1); + expect(result.schema_version).toBe(1); + }); + }); +}); + +// ── schema_lint ───────────────────────────────────────────────────────── + +describe('schema_lint', () => { + it('lints the active pack and returns ok=true for clean pack', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: undefined }, async () => { + const result = await operationsByName.schema_lint!.handler(ctxOf(), {}) as Record; + expect(result.ok).toBe(true); + expect(Array.isArray(result.errors)).toBe(true); + expect(Array.isArray(result.warnings)).toBe(true); + }); + }); + + it('returns pack_not_found for unknown pack', async () => { + await withEnv({ GBRAIN_HOME: tmpDir }, async () => { + const result = await operationsByName.schema_lint!.handler(ctxOf(), { pack: 'nonexistent' }) as Record; + expect(result.error).toBe('pack_not_found'); + }); + }); +}); + +// ── schema_graph ────────────────────────────────────────────────────── + +describe('schema_graph', () => { + it('returns nodes + edges from link_types', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: undefined }, async () => { + const result = await operationsByName.schema_graph!.handler(ctxOf(), {}) as Record; + expect(Array.isArray(result.nodes)).toBe(true); + expect(Array.isArray(result.edges)).toBe(true); + }); + }); +}); + +// ── schema_explain_type ──────────────────────────────────────────────── + +describe('schema_explain_type', () => { + it('returns settings for a known type', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: undefined }, async () => { + const result = await operationsByName.schema_explain_type!.handler(ctxOf(), { type: 'person' }) as Record; + expect((result.type as { name?: string }).name).toBe('person'); + }); + }); + + it('returns type_not_found for unknown type', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: undefined }, async () => { + const result = await operationsByName.schema_explain_type!.handler(ctxOf(), { type: 'ghost' }) as Record; + expect(result.error).toBe('type_not_found'); + }); + }); +}); + +// ── schema_review_orphans ────────────────────────────────────────────── + +describe('schema_review_orphans', () => { + it('returns untyped pages from the DB', async () => { + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, source_path, type, title, compiled_truth, timeline, content_hash) + VALUES ('orphan-1', 'default', 'unknown/page.md', '', 'o', '', '', '')`, + ); + const result = await operationsByName.schema_review_orphans!.handler(ctxOf(), {}) as Record; + expect(result.orphan_count).toBeGreaterThanOrEqual(1); + }); + + it('respects limit param', async () => { + for (let i = 0; i < 5; i++) { + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, source_path, type, title, compiled_truth, timeline, content_hash) + VALUES ($1, 'default', $2, '', $1, '', '', '')`, + [`o${i}`, `unknown/o${i}.md`], + ); + } + const result = await operationsByName.schema_review_orphans!.handler(ctxOf(), { limit: 2 }) as Record; + expect(result.orphan_count).toBe(2); + }); +}); + +// ── schema_apply_mutations — batched + atomic ────────────────────────── + +describe('schema_apply_mutations', () => { + it('applies a single add_type mutation', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine'); + const result = await operationsByName.schema_apply_mutations!.handler(ctxOf(), { + pack: 'mine', + mutations: [ + { op: 'add_type', name: 'researcher', primitive: 'entity', prefix: 'people/researchers/', extractable: true }, + ], + }) as Record; + expect(result.mutations_applied).toBe(1); + expect(result.batch_id).toBeDefined(); + }); + }); + + it('applies multiple mutations atomically (one batch_id across all)', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine'); + const result = await operationsByName.schema_apply_mutations!.handler(ctxOf(), { + pack: 'mine', + mutations: [ + { op: 'add_type', name: 'researcher', primitive: 'entity', prefix: 'people/researchers/' }, + { op: 'add_type', name: 'company', primitive: 'entity', prefix: 'companies/' }, + { op: 'add_link_type', name: 'works_at', inference: { page_type: 'researcher', target_type: 'company' } }, + ], + }) as Record; + expect(result.mutations_applied).toBe(3); + const results = result.results as Array<{ index: number }>; + expect(results.length).toBe(3); + expect(results.map((r) => r.index)).toEqual([0, 1, 2]); + }); + }); + + it('returns partial_results on mid-batch failure with a single batch_id', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine'); + const result = await operationsByName.schema_apply_mutations!.handler(ctxOf(), { + pack: 'mine', + mutations: [ + { op: 'add_type', name: 'company', primitive: 'entity', prefix: 'companies/' }, + { op: 'add_type', name: 'person', primitive: 'entity', prefix: 'people/' }, // collides with seed + ], + }) as Record; + expect(result.error).toBe('mutation_failed'); + const partial = result.partial_results as Array; + expect(partial.length).toBe(1); // first mutation succeeded + }); + }); + + it('rejects unknown op with INVALID_RESULT', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine'); + const result = await operationsByName.schema_apply_mutations!.handler(ctxOf(), { + pack: 'mine', + mutations: [{ op: 'nonexistent_op', name: 'x' }], + }) as Record; + expect(result.error).toBe('mutation_failed'); + }); + }); + + it('rejects empty mutations array', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine'); + const result = await operationsByName.schema_apply_mutations!.handler(ctxOf(), { + pack: 'mine', + mutations: [], + }) as Record; + expect(result.error).toBe('invalid_request'); + }); + }); + + it('captures MCP client_id in audit log actor field (D2 + D20)', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine'); + await operationsByName.schema_apply_mutations!.handler( + ctxOf({ clientId: 'remoteAgentClient12345678' }), + { + pack: 'mine', + mutations: [{ op: 'add_type', name: 'researcher', primitive: 'entity', prefix: 'r/' }], + }, + ); + // Audit log filename pattern: schema-mutations-YYYY-Www.jsonl. + const { readdirSync } = await import('node:fs'); + const auditFiles = readdirSync(auditDir).filter((f) => f.startsWith('schema-mutations-')); + expect(auditFiles.length).toBeGreaterThan(0); + const auditPath = join(auditDir, auditFiles[0]!); + const lines = readFileSync(auditPath, 'utf-8').trim().split('\n'); + const records = lines.map((l) => JSON.parse(l)); + // Actor is mcp: = mcp:remoteAg (8 chars). + expect(records.some((r) => r.actor === 'mcp:remoteAg')).toBe(true); + }); + }); +}); + +// ── reload_schema_pack ──────────────────────────────────────────────── + +describe('reload_schema_pack', () => { + it('returns invalidated list', async () => { + const result = await operationsByName.reload_schema_pack!.handler(ctxOf(), {}) as { invalidated: string[] }; + expect(Array.isArray(result.invalidated)).toBe(true); + }); + + it('invalidates a specific pack by name', async () => { + const result = await operationsByName.reload_schema_pack!.handler(ctxOf(), { pack: 'foo' }) as { invalidated: string[] }; + expect(result.invalidated).toContain('foo'); + }); +}); diff --git a/test/schema-pack-best-effort.test.ts b/test/schema-pack-best-effort.test.ts new file mode 100644 index 000000000..121ef1f4f --- /dev/null +++ b/test/schema-pack-best-effort.test.ts @@ -0,0 +1,83 @@ +// v0.40.6.0 — best-effort.ts contract tests. +// +// Pins the empty-filter contract: pack-load failure returns null, NOT +// hardcoded defaults. Four call sites in Phase 8 (whoknows, find-experts, +// facts/eligibility, enrichment-service) depend on this contract. + +import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { loadActivePackBestEffort } from '../src/core/schema-pack/best-effort.ts'; +import { + __setPackLocatorForTests, + _resetPackLocatorForTests, +} from '../src/core/schema-pack/load-active.ts'; +import { _resetPackCacheForTests } from '../src/core/schema-pack/registry.ts'; +import type { OperationContext } from '../src/core/operations.ts'; +import { withEnv } from './helpers/with-env.ts'; + +let tmpDir: string; + +beforeEach(() => { + _resetPackCacheForTests(); + _resetPackLocatorForTests(); + tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-best-effort-test-')); +}); + +afterEach(() => { + _resetPackCacheForTests(); + _resetPackLocatorForTests(); + try { rmSync(tmpDir, { recursive: true, force: true }); } catch { /* swallow */ } +}); + +function fakeCtx(remote = false): OperationContext { + return { + engine: null as never, + config: {}, + logger: { info: () => {}, warn: () => {}, error: () => {} } as never, + dryRun: false, + remote, + sourceId: undefined, + } as unknown as OperationContext; +} + +describe('loadActivePackBestEffort', () => { + it('returns ResolvedPack when load succeeds (default bundled gbrain-base)', async () => { + // No locator override → defaults to bundled gbrain-base resolution. + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: undefined }, async () => { + const result = await loadActivePackBestEffort(fakeCtx()); + expect(result).not.toBeNull(); + expect(result?.manifest.name).toBe('gbrain-base'); + }); + }); + + it('returns null when the resolved pack is not on disk', async () => { + __setPackLocatorForTests(() => null); + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'never-installed' }, async () => { + const result = await loadActivePackBestEffort(fakeCtx()); + expect(result).toBeNull(); + }); + }); + + it('returns null when the pack file is corrupt', async () => { + const packDir = join(tmpDir, 'schema-packs', 'corrupt-pack'); + mkdirSync(packDir, { recursive: true }); + const packPath = join(packDir, 'pack.yaml'); + writeFileSync(packPath, 'this: is: not: valid: yaml: at all: \n}}{', 'utf-8'); + __setPackLocatorForTests((name) => (name === 'corrupt-pack' ? packPath : null)); + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'corrupt-pack' }, async () => { + const result = await loadActivePackBestEffort(fakeCtx()); + expect(result).toBeNull(); + }); + }); + + it('NEVER throws on any failure path (the load-bearing best-effort contract)', async () => { + // Force load to throw via a locator that returns garbage. + __setPackLocatorForTests(() => { throw new Error('synthetic disk failure'); }); + await withEnv({ GBRAIN_HOME: tmpDir }, async () => { + // resolves, doesn't throw. + await expect(loadActivePackBestEffort(fakeCtx())).resolves.toBeNull(); + }); + }); +}); diff --git a/test/schema-pack-lint-rules.test.ts b/test/schema-pack-lint-rules.test.ts new file mode 100644 index 000000000..ca443ac3c --- /dev/null +++ b/test/schema-pack-lint-rules.test.ts @@ -0,0 +1,348 @@ +// v0.40.6.0 — lint-rules.ts unit tests. 36 cases (11 rules covering each +// of clean / single-violation / multi-violation paths plus the audit-aware +// rule's empty-DB and audit-best-effort paths). + +import { describe, expect, it } from 'bun:test'; +import type { SchemaPackManifest } from '../src/core/schema-pack/manifest-v1.ts'; +import { + aliasShadowsType, + aliasDeclaredByTwoTypes, + aliasReferencesUndeclaredType, + enrichableTypesUndeclared, + linkTypesUndeclared, + frontmatterLinksUndeclared, + expertRoutingWithoutPrefix, + prefixCollision, + prefixStrictSubsetOverlap, + runAllLintRules, + runFilePlaneLintRules, + FILE_PLANE_LINT_RULES, + ALL_LINT_RULES, +} from '../src/core/schema-pack/lint-rules.ts'; + +function mk(opts: Partial): SchemaPackManifest { + const baseTypes = opts.page_types ?? []; + return { + api_version: 'gbrain-schema-pack-v1', + name: opts.name ?? 'p', + version: '1.0.0', + description: '', + gbrain_min_version: '0.38.0', + extends: null, + borrow_from: [], + page_types: baseTypes, + link_types: opts.link_types ?? [], + frontmatter_links: opts.frontmatter_links ?? [], + takes_kinds: ['fact', 'take', 'bet', 'hunch'], + enrichable_types: opts.enrichable_types ?? [], + filing_rules: [], + } as SchemaPackManifest; +} + +const baseType = (over: { name: string; aliases?: string[]; extractable?: boolean; expert?: boolean; prefixes?: string[] }) => ({ + name: over.name, + primitive: 'entity' as const, + path_prefixes: over.prefixes ?? [], + aliases: over.aliases ?? [], + extractable: over.extractable ?? false, + expert_routing: over.expert ?? false, +}); + +describe('aliasShadowsType', () => { + it('clean: no aliases shadow type names', async () => { + const m = mk({ page_types: [baseType({ name: 'person', aliases: ['alias-only'] })] }); + expect(await aliasShadowsType(m)).toEqual([]); + }); + + it('single: alias matches another declared type', async () => { + const m = mk({ page_types: [ + baseType({ name: 'person' }), + baseType({ name: 'researcher', aliases: ['person'] }), + ] }); + const issues = await aliasShadowsType(m); + expect(issues.length).toBe(1); + expect(issues[0]!.rule).toBe('alias_shadows_type'); + expect(issues[0]!.severity).toBe('error'); + }); + + it('skips when alias matches self (self-alias is degenerate but not shadow)', async () => { + const m = mk({ page_types: [baseType({ name: 'person', aliases: ['person'] })] }); + expect(await aliasShadowsType(m)).toEqual([]); + }); +}); + +describe('aliasDeclaredByTwoTypes', () => { + it('clean: each alias declared by at most one type', async () => { + expect(await aliasDeclaredByTwoTypes(mk({ page_types: [baseType({ name: 'p', aliases: ['x'] })] }))).toEqual([]); + }); + + it('flags alias claimed by two types', async () => { + const m = mk({ page_types: [ + baseType({ name: 'a', aliases: ['shared'] }), + baseType({ name: 'b', aliases: ['shared'] }), + ] }); + const issues = await aliasDeclaredByTwoTypes(m); + expect(issues.length).toBe(1); + expect(issues[0]!.severity).toBe('error'); + expect(issues[0]!.message).toContain('shared'); + expect(issues[0]!.message).toContain('a'); + expect(issues[0]!.message).toContain('b'); + }); + + it('flags multiple distinct duplicate-alias collisions', async () => { + const m = mk({ page_types: [ + baseType({ name: 'a', aliases: ['x', 'y'] }), + baseType({ name: 'b', aliases: ['x', 'y'] }), + ] }); + const issues = await aliasDeclaredByTwoTypes(m); + expect(issues.length).toBe(2); + }); +}); + +describe('aliasReferencesUndeclaredType', () => { + it('clean: aliases all match declared types', async () => { + const m = mk({ page_types: [ + baseType({ name: 'person' }), + baseType({ name: 'researcher', aliases: ['person'] }), + ] }); + expect(await aliasReferencesUndeclaredType(m)).toEqual([]); + }); + + it('flags alias pointing at undeclared type', async () => { + const m = mk({ page_types: [baseType({ name: 'r', aliases: ['ghost'] })] }); + const issues = await aliasReferencesUndeclaredType(m); + expect(issues.length).toBe(1); + expect(issues[0]!.severity).toBe('warning'); + expect(issues[0]!.message).toContain('ghost'); + }); + + it('flags multiple undeclared references separately', async () => { + const m = mk({ page_types: [baseType({ name: 'r', aliases: ['g1', 'g2'] })] }); + expect((await aliasReferencesUndeclaredType(m)).length).toBe(2); + }); +}); + +describe('enrichableTypesUndeclared', () => { + it('clean: all enrichable_types are declared page_types', async () => { + const m = mk({ + page_types: [baseType({ name: 'person' })], + enrichable_types: [{ type: 'person', rubric: 'r' }], + }); + expect(await enrichableTypesUndeclared(m)).toEqual([]); + }); + + it('flags enrichable that names a ghost type', async () => { + const m = mk({ + page_types: [baseType({ name: 'person' })], + enrichable_types: [{ type: 'ghost', rubric: 'r' }], + }); + const issues = await enrichableTypesUndeclared(m); + expect(issues.length).toBe(1); + expect(issues[0]!.severity).toBe('error'); + }); + + it('aggregates multiple ghost references', async () => { + const m = mk({ + page_types: [baseType({ name: 'person' })], + enrichable_types: [ + { type: 'ghost1', rubric: 'r' }, + { type: 'ghost2', rubric: 'r' }, + ], + }); + expect((await enrichableTypesUndeclared(m)).length).toBe(2); + }); +}); + +describe('linkTypesUndeclared', () => { + it('clean: inference targets resolve', async () => { + const m = mk({ + page_types: [baseType({ name: 'person' }), baseType({ name: 'company' })], + link_types: [{ name: 'works_at', inference: { page_type: 'person', target_type: 'company' } }], + }); + expect(await linkTypesUndeclared(m)).toEqual([]); + }); + + it('flags inference.page_type referencing ghost', async () => { + const m = mk({ + page_types: [baseType({ name: 'company' })], + link_types: [{ name: 'works_at', inference: { page_type: 'ghost', target_type: 'company' } }], + }); + expect((await linkTypesUndeclared(m)).length).toBe(1); + }); + + it('flags both page_type AND target_type independently', async () => { + const m = mk({ + page_types: [baseType({ name: 'person' })], + link_types: [{ name: 'l', inference: { page_type: 'g1', target_type: 'g2' } }], + }); + expect((await linkTypesUndeclared(m)).length).toBe(2); + }); +}); + +describe('frontmatterLinksUndeclared', () => { + it('clean: page_type + link_type both resolve', async () => { + const m = mk({ + page_types: [baseType({ name: 'meeting' })], + link_types: [{ name: 'attended' }], + frontmatter_links: [{ page_type: 'meeting', fields: ['attendees'], link_type: 'attended' }], + }); + expect(await frontmatterLinksUndeclared(m)).toEqual([]); + }); + + it('flags unknown page_type', async () => { + const m = mk({ + page_types: [], + link_types: [{ name: 'attended' }], + frontmatter_links: [{ page_type: 'ghost', fields: ['x'], link_type: 'attended' }], + }); + const issues = await frontmatterLinksUndeclared(m); + expect(issues.length).toBe(1); + expect(issues[0]!.rule).toBe('frontmatter_links_undeclared_page_type'); + }); + + it('flags unknown link_type', async () => { + const m = mk({ + page_types: [baseType({ name: 'meeting' })], + link_types: [], + frontmatter_links: [{ page_type: 'meeting', fields: ['x'], link_type: 'ghost' }], + }); + const issues = await frontmatterLinksUndeclared(m); + expect(issues.length).toBe(1); + expect(issues[0]!.rule).toBe('frontmatter_links_undeclared_link_type'); + }); +}); + +describe('expertRoutingWithoutPrefix', () => { + it('clean: expert types have prefixes', async () => { + const m = mk({ page_types: [baseType({ name: 'r', expert: true, prefixes: ['people/'] })] }); + expect(await expertRoutingWithoutPrefix(m)).toEqual([]); + }); + + it('warns: expert-routed type lacks prefix', async () => { + const m = mk({ page_types: [baseType({ name: 'r', expert: true })] }); + const issues = await expertRoutingWithoutPrefix(m); + expect(issues.length).toBe(1); + expect(issues[0]!.severity).toBe('warning'); + }); + + it('skips non-expert types without prefix (legitimate concept-only types)', async () => { + const m = mk({ page_types: [baseType({ name: 'concept' })] }); + expect(await expertRoutingWithoutPrefix(m)).toEqual([]); + }); +}); + +describe('prefixCollision', () => { + it('clean: each prefix declared by only one type', async () => { + const m = mk({ page_types: [ + baseType({ name: 'a', prefixes: ['a/'] }), + baseType({ name: 'b', prefixes: ['b/'] }), + ] }); + expect(await prefixCollision(m)).toEqual([]); + }); + + it('flags two types declaring same prefix', async () => { + const m = mk({ page_types: [ + baseType({ name: 'a', prefixes: ['shared/'] }), + baseType({ name: 'b', prefixes: ['shared/'] }), + ] }); + const issues = await prefixCollision(m); + expect(issues.length).toBe(1); + expect(issues[0]!.severity).toBe('error'); + }); + + it('aggregates multiple prefix collisions', async () => { + const m = mk({ page_types: [ + baseType({ name: 'a', prefixes: ['x/', 'y/'] }), + baseType({ name: 'b', prefixes: ['x/', 'y/'] }), + ] }); + expect((await prefixCollision(m)).length).toBe(2); + }); +}); + +describe('prefixStrictSubsetOverlap', () => { + it('clean: prefixes are unrelated', async () => { + const m = mk({ page_types: [ + baseType({ name: 'a', prefixes: ['people/'] }), + baseType({ name: 'b', prefixes: ['companies/'] }), + ] }); + expect(await prefixStrictSubsetOverlap(m)).toEqual([]); + }); + + it('flags one type prefix that is a strict subset of another', async () => { + const m = mk({ page_types: [ + baseType({ name: 'researcher', prefixes: ['people/researchers/'] }), + baseType({ name: 'person', prefixes: ['people/'] }), + ] }); + const issues = await prefixStrictSubsetOverlap(m); + // strict-subset detection fires for the researcher prefix. + expect(issues.length).toBeGreaterThanOrEqual(1); + expect(issues[0]!.severity).toBe('warning'); + }); + + it('does not flag identical prefixes (that is prefixCollision territory)', async () => { + const m = mk({ page_types: [ + baseType({ name: 'a', prefixes: ['x/'] }), + baseType({ name: 'b', prefixes: ['x/'] }), + ] }); + // prefixStrictSubsetOverlap only fires on STRICT subsets; identical is collision's job. + expect(await prefixStrictSubsetOverlap(m)).toEqual([]); + }); +}); + +describe('runFilePlaneLintRules — composition', () => { + it('returns ok:true for a clean manifest', async () => { + const m = mk({ page_types: [baseType({ name: 'person', prefixes: ['people/'] })] }); + const report = await runFilePlaneLintRules(m); + expect(report.ok).toBe(true); + expect(report.errors).toEqual([]); + }); + + it('returns ok:false when any error fires', async () => { + const m = mk({ page_types: [ + baseType({ name: 'a', prefixes: ['x/'] }), + baseType({ name: 'b', prefixes: ['x/'] }), + ] }); + const report = await runFilePlaneLintRules(m); + expect(report.ok).toBe(false); + expect(report.errors.length).toBeGreaterThan(0); + }); + + it('separates warnings from errors', async () => { + const m = mk({ page_types: [baseType({ name: 'r', expert: true })] }); + const report = await runFilePlaneLintRules(m); + expect(report.ok).toBe(true); + expect(report.errors).toEqual([]); + expect(report.warnings.length).toBeGreaterThan(0); + }); + + it('skips DB-aware rules (file-plane only)', async () => { + const m = mk({ page_types: [baseType({ name: 'r', extractable: true, prefixes: ['ghost/'] })] }); + const report = await runFilePlaneLintRules(m); + // extractable_empty_corpus needs an engine; this should NOT fire here. + expect(report.warnings.find((w) => w.rule === 'extractable_empty_corpus')).toBeUndefined(); + }); +}); + +describe('runAllLintRules — composition', () => { + it('without engine, behaves like runFilePlaneLintRules', async () => { + const m = mk({ page_types: [baseType({ name: 'r', extractable: true })] }); + const report = await runAllLintRules(m); + expect(report.warnings.find((w) => w.rule === 'extractable_empty_corpus')).toBeUndefined(); + }); +}); + +describe('rule registry shape', () => { + it('ALL_LINT_RULES contains 11 rules', () => { + expect(ALL_LINT_RULES.length).toBe(11); + }); + + it('FILE_PLANE_LINT_RULES excludes the 2 DB-aware rules', () => { + expect(FILE_PLANE_LINT_RULES.length).toBe(9); + expect(FILE_PLANE_LINT_RULES.every((r) => !r.planeAware)).toBe(true); + }); + + it('all rule names are unique', () => { + const names = ALL_LINT_RULES.map((r) => r.name); + expect(new Set(names).size).toBe(names.length); + }); +}); diff --git a/test/schema-pack-mutate-audit.test.ts b/test/schema-pack-mutate-audit.test.ts new file mode 100644 index 000000000..e9b5218bd --- /dev/null +++ b/test/schema-pack-mutate-audit.test.ts @@ -0,0 +1,209 @@ +// v0.40.6.0 — mutate-audit.ts contract tests. +// +// Pins privacy redaction (sha8 + first-slug-only), success+failure +// logging, ISO-week rotation, GBRAIN_AUDIT_DIR honoring, and the +// summarizeMutations() shape that doctor + a future audit CLI both bind to. + +import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; +import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + computeMutateAuditPath, + logMutationFailure, + logMutationSuccess, + readRecentMutations, + summarizeMutations, + type MutationAuditRecord, +} from '../src/core/schema-pack/mutate-audit.ts'; +import { withEnv } from './helpers/with-env.ts'; + +let auditDir: string; + +beforeEach(() => { + auditDir = mkdtempSync(join(tmpdir(), 'gbrain-mutate-audit-test-')); +}); + +afterEach(() => { + try { rmSync(auditDir, { recursive: true, force: true }); } catch { /* swallow */ } +}); + +describe('privacy posture', () => { + it('redacts type name to sha8 by default', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir, GBRAIN_SCHEMA_AUDIT_VERBOSE: undefined }, async () => { + await logMutationSuccess({ + op: 'add_type', + pack: 'my-pack', + type: 'mental_health_diagnosis', + prefix: 'personal/health/oncology/2026-05-23.md', + actor: 'cli', + }); + const path = computeMutateAuditPath(); + const raw = readFileSync(path, 'utf-8'); + const record = JSON.parse(raw.trim()) as MutationAuditRecord; + expect(record.type_redacted).toBe(true); + expect(record.type_or_hash).toMatch(/^[0-9a-f]{8}$/); + expect(record.type_or_hash).not.toBe('mental_health_diagnosis'); + expect(record.prefix_first_seg).toBe('personal'); + expect(raw).not.toContain('oncology'); + expect(raw).not.toContain('mental_health'); + }); + }); + + it('writes raw type name when GBRAIN_SCHEMA_AUDIT_VERBOSE=1', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir, GBRAIN_SCHEMA_AUDIT_VERBOSE: '1' }, async () => { + await logMutationSuccess({ + op: 'add_type', + pack: 'my-pack', + type: 'researcher', + prefix: 'people/researchers/', + actor: 'cli', + }); + const record = JSON.parse( + readFileSync(computeMutateAuditPath(), 'utf-8').trim(), + ) as MutationAuditRecord; + expect(record.type_redacted).toBe(false); + expect(record.type_or_hash).toBe('researcher'); + }); + }); + + it('pack name is NEVER redacted (it is user-chosen and non-PII)', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, async () => { + await logMutationSuccess({ op: 'add_type', pack: 'my-pack', actor: 'cli' }); + const record = JSON.parse( + readFileSync(computeMutateAuditPath(), 'utf-8').trim(), + ) as MutationAuditRecord; + expect(record.pack).toBe('my-pack'); + }); + }); + + it('omits type_or_hash when op did not involve a type', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, async () => { + // Hypothetical pack-level op (none today, but the shape must accept it). + await logMutationSuccess({ op: 'add_type', pack: 'p', actor: 'cli' }); + const r = JSON.parse(readFileSync(computeMutateAuditPath(), 'utf-8').trim()) as MutationAuditRecord; + expect(r.type_or_hash).toBeNull(); + }); + }); +}); + +describe('success + failure logging', () => { + it('logs success with outcome=success and reason=null', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, async () => { + await logMutationSuccess({ + op: 'add_type', + pack: 'my-pack', + type: 'researcher', + actor: 'cli', + prev_sha8: 'aaaaaaaa', + new_sha8: 'bbbbbbbb', + }); + const r = JSON.parse(readFileSync(computeMutateAuditPath(), 'utf-8').trim()) as MutationAuditRecord; + expect(r.outcome).toBe('success'); + expect(r.reason).toBeNull(); + expect(r.prev_sha8).toBe('aaaaaaaa'); + expect(r.new_sha8).toBe('bbbbbbbb'); + }); + }); + + it('logs failure with outcome=failure + reason code (the C11 signal doctor reads)', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, async () => { + await logMutationFailure({ + op: 'add_type', + pack: 'gbrain-base', + type: 'researcher', + actor: 'cli', + reason: 'PACK_READONLY', + }); + const r = JSON.parse(readFileSync(computeMutateAuditPath(), 'utf-8').trim()) as MutationAuditRecord; + expect(r.outcome).toBe('failure'); + expect(r.reason).toBe('PACK_READONLY'); + }); + }); + + it('actor field surfaces mcp: shape', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, async () => { + await logMutationSuccess({ + op: 'add_type', pack: 'p', type: 't', + actor: 'mcp:abc12345', + }); + const r = JSON.parse(readFileSync(computeMutateAuditPath(), 'utf-8').trim()) as MutationAuditRecord; + expect(r.actor).toBe('mcp:abc12345'); + }); + }); +}); + +describe('ISO-week rotation + GBRAIN_AUDIT_DIR', () => { + it('writes filename in the schema-mutations-YYYY-Www.jsonl shape', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = computeMutateAuditPath(new Date('2026-05-23T12:00:00Z')); + expect(path).toMatch(/schema-mutations-2026-W\d{2}\.jsonl$/); + expect(path.startsWith(auditDir)).toBe(true); + }); + }); + + it('honors GBRAIN_AUDIT_DIR override', async () => { + const customDir = mkdtempSync(join(tmpdir(), 'gbrain-mutate-custom-')); + try { + await withEnv({ GBRAIN_AUDIT_DIR: customDir }, async () => { + await logMutationSuccess({ op: 'add_type', pack: 'p', actor: 'cli' }); + const path = computeMutateAuditPath(); + expect(path.startsWith(customDir)).toBe(true); + expect(existsSync(path)).toBe(true); + }); + } finally { + rmSync(customDir, { recursive: true, force: true }); + } + }); + + it('readRecentMutations returns records sorted within file, skips malformed lines', async () => { + await withEnv({ GBRAIN_AUDIT_DIR: auditDir }, async () => { + await logMutationSuccess({ op: 'add_type', pack: 'a', actor: 'cli' }); + await logMutationFailure({ op: 'remove_type', pack: 'a', actor: 'cli', reason: 'TYPE_NOT_FOUND' }); + // Inject malformed line. + const path = computeMutateAuditPath(); + writeFileSync(path, readFileSync(path, 'utf-8') + 'not-json{{{\n' + '{}\n'); + const recs = readRecentMutations(7); + expect(recs.length).toBeGreaterThanOrEqual(2); + expect(recs.some((r) => r.outcome === 'success')).toBe(true); + expect(recs.some((r) => r.outcome === 'failure')).toBe(true); + }); + }); +}); + +describe('summarizeMutations — cross-surface parity primitive', () => { + it('aggregates by op, outcome, pack, reason, actor', () => { + const recs: MutationAuditRecord[] = [ + { ts: '2026-01-01T00:00:00Z', op: 'add_type', pack: 'a', type_or_hash: null, type_redacted: true, prefix_first_seg: null, actor: 'cli', outcome: 'success', reason: null, prev_sha8: null, new_sha8: null, batch_id: null }, + { ts: '2026-01-01T00:00:01Z', op: 'add_type', pack: 'a', type_or_hash: null, type_redacted: true, prefix_first_seg: null, actor: 'mcp:abc12345', outcome: 'failure', reason: 'PACK_READONLY', prev_sha8: null, new_sha8: null, batch_id: 'batch-1' }, + { ts: '2026-01-01T00:00:02Z', op: 'remove_type', pack: 'b', type_or_hash: null, type_redacted: true, prefix_first_seg: null, actor: 'autopilot', outcome: 'success', reason: null, prev_sha8: null, new_sha8: null, batch_id: null }, + ]; + const s = summarizeMutations(recs); + expect(s.total).toBe(3); + expect(s.by_op.add_type).toBe(2); + expect(s.by_op.remove_type).toBe(1); + expect(s.by_outcome).toEqual({ success: 2, failure: 1 }); + expect(s.by_pack).toEqual({ a: 2, b: 1 }); + expect(s.by_reason).toEqual({ PACK_READONLY: 1 }); + // Actor bucketing collapses mcp:* to 'mcp' + expect(s.by_actor).toEqual({ cli: 1, mcp: 1, autopilot: 1 }); + }); + + it('returns empty summary for empty input', () => { + const s = summarizeMutations([]); + expect(s.total).toBe(0); + expect(s.by_outcome).toEqual({ success: 0, failure: 0 }); + }); +}); + +describe('best-effort behavior', () => { + it('does not throw when the audit dir is unwritable', async () => { + // Point at a path that fs.mkdirSync will fail on (e.g. a regular file). + const fakeDir = join(mkdtempSync(join(tmpdir(), 'gbrain-mutate-baddir-')), 'not-a-dir'); + writeFileSync(fakeDir, 'this-is-a-file-not-a-dir', 'utf-8'); + await withEnv({ GBRAIN_AUDIT_DIR: fakeDir }, async () => { + // Should NOT throw — best-effort posture. + await expect(logMutationSuccess({ op: 'add_type', pack: 'p', actor: 'cli' })).resolves.toBeUndefined(); + }); + }); +}); diff --git a/test/schema-pack-mutate.test.ts b/test/schema-pack-mutate.test.ts new file mode 100644 index 000000000..d2d4b289a --- /dev/null +++ b/test/schema-pack-mutate.test.ts @@ -0,0 +1,528 @@ +// v0.40.6.0 — mutate.ts contract tests for the 11 primitives + withMutation skeleton. + +import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; +import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + addAliasToType, + addLinkTypeToPack, + addPrefixToType, + addTypeToPack, + BUNDLED_PACK_NAMES, + locateMutablePackFile, + removeAliasFromType, + removeLinkTypeFromPack, + removePrefixFromType, + removeTypeFromPack, + SchemaPackMutationError, + setExpertRoutingOnType, + setExtractableOnType, + updateTypeOnPack, +} from '../src/core/schema-pack/mutate.ts'; +import { loadPackFromFile, parseYamlMini } from '../src/core/schema-pack/loader.ts'; +import { _resetPackCacheForTests } from '../src/core/schema-pack/registry.ts'; +import { withEnv } from './helpers/with-env.ts'; +import type { SchemaPackManifest } from '../src/core/schema-pack/manifest-v1.ts'; + +let tmpDir: string; +let auditDir: string; +let lockDir: string; + +function seedPack(packName: string, format: 'json' | 'yaml', initial?: Partial): string { + // GBRAIN_HOME=/tmp/x → gbrainPath('schema-packs', 'mine') = /tmp/x/.gbrain/schema-packs/mine + const dir = join(tmpDir, '.gbrain', 'schema-packs', packName); + mkdirSync(dir, { recursive: true }); + const manifest: SchemaPackManifest = { + api_version: 'gbrain-schema-pack-v1', + name: packName, + version: '1.0.0', + description: '', + gbrain_min_version: '0.38.0', + extends: null, + borrow_from: [], + page_types: [{ + name: 'person', primitive: 'entity', path_prefixes: ['people/'], + aliases: [], extractable: false, expert_routing: false, + }], + link_types: [], + frontmatter_links: [], + takes_kinds: ['fact', 'take', 'bet', 'hunch'], + enrichable_types: [], + filing_rules: [], + ...initial, + } as SchemaPackManifest; + const file = format === 'json' ? 'pack.json' : 'pack.yaml'; + const path = join(dir, file); + const body = format === 'json' + ? JSON.stringify(manifest, null, 2) + '\n' + : require('../src/core/schema-pack/mutate.ts').emitYaml?.(manifest) ?? buildSimpleYaml(manifest); + writeFileSync(path, body, 'utf-8'); + return path; +} + +// Tiny YAML fallback for fixtures (the real emitter is inside mutate.ts). +function buildSimpleYaml(m: SchemaPackManifest): string { + return JSON.stringify(m); // valid YAML (JSON is a subset) +} + +beforeEach(() => { + _resetPackCacheForTests(); + tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-mutate-test-')); + auditDir = mkdtempSync(join(tmpdir(), 'gbrain-mutate-audit-')); + lockDir = mkdtempSync(join(tmpdir(), 'gbrain-mutate-locks-')); +}); + +afterEach(() => { + _resetPackCacheForTests(); + for (const d of [tmpDir, auditDir, lockDir]) { + try { rmSync(d, { recursive: true, force: true }); } catch { /* swallow */ } + } +}); + +// ─── BUNDLED-pack guard (D6) ───────────────────────────────────────────── + +describe('locateMutablePackFile — bundled guard', () => { + it('rejects gbrain-base with PACK_READONLY + fork hint', () => { + expect(() => locateMutablePackFile('gbrain-base')).toThrow(SchemaPackMutationError); + try { locateMutablePackFile('gbrain-base'); } catch (e) { + const err = e as SchemaPackMutationError; + expect(err.code).toBe('PACK_READONLY'); + expect(err.message).toContain('gbrain schema fork'); + } + }); + + it('rejects gbrain-recommended with PACK_READONLY', () => { + try { locateMutablePackFile('gbrain-recommended'); } catch (e) { + expect((e as SchemaPackMutationError).code).toBe('PACK_READONLY'); + } + }); + + it('BUNDLED_PACK_NAMES export contains both bundled packs', () => { + expect(BUNDLED_PACK_NAMES.has('gbrain-base')).toBe(true); + expect(BUNDLED_PACK_NAMES.has('gbrain-recommended')).toBe(true); + expect(BUNDLED_PACK_NAMES.size).toBe(2); + }); +}); + +// ─── add_type ─────────────────────────────────────────────────────────── + +describe('addTypeToPack', () => { + it('appends a new type to JSON pack and writes atomically', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json'); + const result = await addTypeToPack('mine', { + name: 'researcher', primitive: 'entity', prefix: 'people/researchers/', + extractable: true, expert: false, + } as never, { lockDir }); + expect(result.format).toBe('json'); + const after = loadPackFromFile(path); + expect(after.page_types.find((t) => t.name === 'researcher')).toBeDefined(); + expect(result.prev_sha8).not.toBe(result.new_sha8); + }); + }); + + it('rejects when type already exists', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + await expect(addTypeToPack('mine', { + name: 'person', primitive: 'entity', prefix: 'people/', + } as never, { lockDir })).rejects.toThrow('already exists'); + }); + }); + + it('rejects invalid primitive', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + await expect(addTypeToPack('mine', { + name: 'bad', primitive: 'invalid_primitive' as never, prefix: 'x/', + } as never, { lockDir })).rejects.toMatchObject({ code: 'INVALID_PRIMITIVE' }); + }); + }); + + it('rejects missing prefix', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + await expect(addTypeToPack('mine', { + name: 'bad', primitive: 'entity', prefix: '', + } as never, { lockDir })).rejects.toMatchObject({ code: 'INVALID_RESULT' }); + }); + }); + + it('rejects invalid slug type name', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + await expect(addTypeToPack('mine', { + name: 'has spaces', primitive: 'entity', prefix: 'x/', + } as never, { lockDir })).rejects.toMatchObject({ code: 'INVALID_RESULT' }); + }); + }); +}); + +// ─── remove_type with codex C14 alias-ref check ───────────────────────── + +describe('removeTypeFromPack', () => { + it('removes the type when no references exist', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json', { + page_types: [ + { name: 'person', primitive: 'entity', path_prefixes: ['people/'], aliases: [], extractable: false, expert_routing: false }, + { name: 'company', primitive: 'entity', path_prefixes: ['companies/'], aliases: [], extractable: false, expert_routing: false }, + ], + }); + await removeTypeFromPack('mine', 'company', { lockDir }); + const after = loadPackFromFile(path); + expect(after.page_types.find((t) => t.name === 'company')).toBeUndefined(); + }); + }); + + it('TYPE_NOT_FOUND when type does not exist', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + await expect(removeTypeFromPack('mine', 'ghost', { lockDir })) + .rejects.toMatchObject({ code: 'TYPE_NOT_FOUND' }); + }); + }); + + it('CODEX C14: refuses removal when another type aliases the target', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json', { + page_types: [ + { name: 'person', primitive: 'entity', path_prefixes: ['people/'], aliases: [], extractable: false, expert_routing: false }, + { name: 'researcher', primitive: 'entity', path_prefixes: ['people/r/'], aliases: ['person'], extractable: false, expert_routing: false }, + ], + }); + await expect(removeTypeFromPack('mine', 'person', { lockDir })) + .rejects.toMatchObject({ code: 'STILL_REFERENCED' }); + }); + }); + + it('CODEX C14: refuses removal when link_type inference references the target', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json', { + page_types: [ + { name: 'person', primitive: 'entity', path_prefixes: ['people/'], aliases: [], extractable: false, expert_routing: false }, + { name: 'company', primitive: 'entity', path_prefixes: ['c/'], aliases: [], extractable: false, expert_routing: false }, + ], + link_types: [ + { name: 'works_at', inference: { page_type: 'person', target_type: 'company' } }, + ], + }); + await expect(removeTypeFromPack('mine', 'person', { lockDir })) + .rejects.toMatchObject({ code: 'STILL_REFERENCED' }); + }); + }); + + it('CODEX C14: refuses when enrichable_types references the target', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json', { + page_types: [{ name: 'person', primitive: 'entity', path_prefixes: ['p/'], aliases: [], extractable: false, expert_routing: false }], + enrichable_types: [{ type: 'person', rubric: 'r' }], + }); + await expect(removeTypeFromPack('mine', 'person', { lockDir })) + .rejects.toMatchObject({ code: 'STILL_REFERENCED' }); + }); + }); +}); + +// ─── update_type ─────────────────────────────────────────────────────── + +describe('updateTypeOnPack', () => { + it('patches a single field while leaving others untouched', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json'); + await updateTypeOnPack('mine', { name: 'person', patch: { extractable: true } }, { lockDir }); + const after = loadPackFromFile(path); + const t = after.page_types.find((pt) => pt.name === 'person')!; + expect(t.extractable).toBe(true); + expect(t.primitive).toBe('entity'); + expect(t.path_prefixes).toEqual(['people/']); + }); + }); + + it('name field on patch is ignored (name is identity)', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json'); + await updateTypeOnPack('mine', { name: 'person', patch: { name: 'renamed', extractable: true } as never }, { lockDir }); + const after = loadPackFromFile(path); + // 'person' kept its name; not renamed. + expect(after.page_types.find((t) => t.name === 'person')).toBeDefined(); + expect(after.page_types.find((t) => t.name === 'renamed')).toBeUndefined(); + }); + }); + + it('TYPE_NOT_FOUND on patch target missing', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + await expect(updateTypeOnPack('mine', { name: 'ghost', patch: { extractable: true } }, { lockDir })) + .rejects.toMatchObject({ code: 'TYPE_NOT_FOUND' }); + }); + }); +}); + +// ─── alias + prefix primitives ───────────────────────────────────────── + +describe('addAliasToType', () => { + it('appends a new alias (alias does NOT shadow another declared type)', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json'); + // Alias 'individual' is NOT another declared type, so alias_shadows_type does not fire. + // alias_references_undeclared_type is a WARNING (not error), so validation gate passes. + await addAliasToType('mine', 'person', 'individual', { lockDir }); + const after = loadPackFromFile(path); + expect(after.page_types.find((t) => t.name === 'person')!.aliases).toEqual(['individual']); + }); + }); + + it('idempotent on existing alias', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json', { + page_types: [ + { name: 'person', primitive: 'entity', path_prefixes: ['p/'], aliases: ['individual'], extractable: false, expert_routing: false }, + ], + }); + const r1 = await addAliasToType('mine', 'person', 'individual', { lockDir }); + const r2 = await addAliasToType('mine', 'person', 'individual', { lockDir }); + expect(r1.new_sha8).toBe(r2.new_sha8); + }); + }); +}); + +describe('removeAliasFromType', () => { + it('removes an existing alias', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json', { + page_types: [ + { name: 'person', primitive: 'entity', path_prefixes: ['p/'], aliases: ['individual'], extractable: false, expert_routing: false }, + ], + }); + await removeAliasFromType('mine', 'person', 'individual', { lockDir }); + const after = loadPackFromFile(path); + expect(after.page_types.find((t) => t.name === 'person')!.aliases).toEqual([]); + }); + }); + + it('idempotent on missing alias', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + const r1 = await removeAliasFromType('mine', 'person', 'never-was', { lockDir }); + const r2 = await removeAliasFromType('mine', 'person', 'never-was', { lockDir }); + expect(r1.new_sha8).toBe(r2.new_sha8); + }); + }); +}); + +describe('addPrefixToType / removePrefixFromType', () => { + it('addPrefix appends', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json'); + await addPrefixToType('mine', 'person', 'people-archive/', { lockDir }); + const after = loadPackFromFile(path); + expect(after.page_types.find((t) => t.name === 'person')!.path_prefixes).toEqual(['people/', 'people-archive/']); + }); + }); + + it('addPrefix idempotent on existing', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + const r1 = await addPrefixToType('mine', 'person', 'people/', { lockDir }); + const r2 = await addPrefixToType('mine', 'person', 'people/', { lockDir }); + expect(r1.new_sha8).toBe(r2.new_sha8); + }); + }); + + it('removePrefix removes', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json', { + page_types: [{ name: 'person', primitive: 'entity', path_prefixes: ['people/', 'people-archive/'], aliases: [], extractable: false, expert_routing: false }], + }); + await removePrefixFromType('mine', 'person', 'people-archive/', { lockDir }); + const after = loadPackFromFile(path); + expect(after.page_types.find((t) => t.name === 'person')!.path_prefixes).toEqual(['people/']); + }); + }); +}); + +// ─── link_type primitives ─────────────────────────────────────────────── + +describe('addLinkTypeToPack / removeLinkTypeFromPack', () => { + it('addLinkType creates a new link verb', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json', { + page_types: [ + { name: 'person', primitive: 'entity', path_prefixes: ['p/'], aliases: [], extractable: false, expert_routing: false }, + { name: 'company', primitive: 'entity', path_prefixes: ['c/'], aliases: [], extractable: false, expert_routing: false }, + ], + }); + await addLinkTypeToPack('mine', { + name: 'works_at', + inference: { page_type: 'person', target_type: 'company' }, + }, { lockDir }); + const after = loadPackFromFile(path); + expect(after.link_types.length).toBe(1); + expect(after.link_types[0]!.name).toBe('works_at'); + }); + }); + + it('addLinkType rejects duplicate name', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json', { + link_types: [{ name: 'attended' }], + }); + await expect(addLinkTypeToPack('mine', { name: 'attended' }, { lockDir })) + .rejects.toMatchObject({ code: 'TYPE_EXISTS' }); + }); + }); + + it('removeLinkType removes when no fm refs exist', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json', { link_types: [{ name: 'attended' }] }); + await removeLinkTypeFromPack('mine', 'attended', { lockDir }); + const after = loadPackFromFile(path); + expect(after.link_types.length).toBe(0); + }); + }); + + it('removeLinkType refuses when frontmatter_links references it', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json', { + page_types: [{ name: 'meeting', primitive: 'temporal', path_prefixes: ['m/'], aliases: [], extractable: false, expert_routing: false }], + link_types: [{ name: 'attended' }], + frontmatter_links: [{ page_type: 'meeting', fields: ['attendees'], link_type: 'attended' }], + }); + await expect(removeLinkTypeFromPack('mine', 'attended', { lockDir })) + .rejects.toMatchObject({ code: 'STILL_REFERENCED' }); + }); + }); +}); + +// ─── flag setters ────────────────────────────────────────────────────── + +describe('setExtractableOnType / setExpertRoutingOnType', () => { + it('setExtractable flips the flag', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json'); + await setExtractableOnType('mine', 'person', true, { lockDir }); + expect(loadPackFromFile(path).page_types[0]!.extractable).toBe(true); + await setExtractableOnType('mine', 'person', false, { lockDir }); + expect(loadPackFromFile(path).page_types[0]!.extractable).toBe(false); + }); + }); + + it('setExpertRouting flips the flag', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json'); + await setExpertRoutingOnType('mine', 'person', true, { lockDir }); + expect(loadPackFromFile(path).page_types[0]!.expert_routing).toBe(true); + }); + }); + + it('TYPE_NOT_FOUND on missing type', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + await expect(setExtractableOnType('mine', 'ghost', true, { lockDir })) + .rejects.toMatchObject({ code: 'TYPE_NOT_FOUND' }); + }); + }); +}); + +// ─── YAML round-trip ────────────────────────────────────────────────── + +describe('YAML round-trip', () => { + it('mutating a YAML pack preserves YAML format and reparses cleanly', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const dir = join(tmpDir, '.gbrain', 'schema-packs', 'yaml-pack'); + mkdirSync(dir, { recursive: true }); + const path = join(dir, 'pack.yaml'); + // Seed valid YAML (use the manifest validator round-trip). + // Seed actual block-style YAML (parseYamlMini is hand-rolled and prefers + // block-style; flow-style JSON-in-YAML isn't fully supported). + const yamlBody = `api_version: gbrain-schema-pack-v1 +name: yaml-pack +version: 1.0.0 +description: "" +gbrain_min_version: 0.38.0 +extends: null +borrow_from: [] +page_types: + - name: person + primitive: entity + path_prefixes: + - people/ + aliases: [] + extractable: false + expert_routing: false +link_types: [] +frontmatter_links: [] +takes_kinds: + - fact + - take + - bet + - hunch +enrichable_types: [] +filing_rules: [] +`; + writeFileSync(path, yamlBody, 'utf-8'); + + const result = await addTypeToPack('yaml-pack', { + name: 'researcher', primitive: 'entity', prefix: 'people/r/', + } as never, { lockDir }); + expect(result.format).toBe('yaml'); + // File still parses as YAML AND as a valid manifest. + const reparsed = parseYamlMini(readFileSync(path, 'utf-8')); + expect(reparsed).toBeDefined(); + const after = loadPackFromFile(path); + expect(after.page_types.find((t) => t.name === 'researcher')).toBeDefined(); + }); + }); +}); + +// ─── atomicity ──────────────────────────────────────────────────────── + +describe('atomicity invariants', () => { + it('crash-mid-write does not leave the pack file in a partial state', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + const path = seedPack('mine', 'json'); + const before = readFileSync(path, 'utf-8'); + try { + // Force a mutator throw mid-pipeline AFTER lock acquire + read. + await addTypeToPack('mine', { + // Invalid: primitive is wrong type. Validation should fail BEFORE write. + name: 'researcher', primitive: 'nope' as never, prefix: 'x/', + } as never, { lockDir }); + } catch { /* expected */ } + // Original file untouched. + expect(readFileSync(path, 'utf-8')).toBe(before); + }); + }); + + it('lock is released after a mutator throws', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + try { + await addTypeToPack('mine', { + name: 'has spaces', primitive: 'entity', prefix: 'x/', + } as never, { lockDir }); + } catch { /* expected */ } + // A second call should succeed (lock not held). + const result = await addTypeToPack('mine', { + name: 'valid', primitive: 'entity', prefix: 'v/', + } as never, { lockDir }); + expect(result.pack).toBe('mine'); + }); + }); +}); + +// ─── validation gate ───────────────────────────────────────────────── + +describe('validation gate (file-plane lint integration)', () => { + it('refuses mutation that would create prefix collision', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_AUDIT_DIR: auditDir }, async () => { + seedPack('mine', 'json'); + // Adding a second type with the SAME prefix → prefix_collision (error). + await expect(addTypeToPack('mine', { + name: 'human', primitive: 'entity', prefix: 'people/', // same as person + } as never, { lockDir })).rejects.toMatchObject({ code: 'INVALID_RESULT' }); + }); + }); +}); diff --git a/test/schema-pack-pack-lock.test.ts b/test/schema-pack-pack-lock.test.ts new file mode 100644 index 000000000..bbf8e459c --- /dev/null +++ b/test/schema-pack-pack-lock.test.ts @@ -0,0 +1,227 @@ +// v0.40.6.0 — pack-lock.ts contract tests. +// +// Pins the atomic-acquire, stale-detection, refresh, and cleanup behavior +// that the schema cathedral v3 mutation skeleton depends on. + +import { afterEach, beforeEach, describe, expect, it, mock } from 'bun:test'; +import { existsSync, mkdtempSync, openSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + acquirePackLock, + isLockStale, + PackLockBusyError, + withPackLock, + type LockFileRecord, +} from '../src/core/schema-pack/pack-lock.ts'; + +let lockDir: string; + +beforeEach(() => { + lockDir = mkdtempSync(join(tmpdir(), 'gbrain-pack-lock-test-')); +}); + +afterEach(() => { + try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* swallow */ } +}); + +const liveAlways = (_pid: number): boolean => true; +const deadAlways = (_pid: number): boolean => false; + +describe('acquirePackLock — clean acquire', () => { + it('atomically creates the lockfile on first call', () => { + const result = acquirePackLock('foo', { lockDir }); + expect(result.outcome).toBe('acquired'); + expect(existsSync(join(lockDir, 'foo.lock'))).toBe(true); + expect(result.record.pid).toBe(process.pid); + expect(result.record.ttlMs).toBeGreaterThan(0); + }); + + it('writes valid JSON record with pid, ts, ttlMs, hostname', () => { + acquirePackLock('foo', { lockDir, ttlMs: 12345 }); + const raw = readFileSync(join(lockDir, 'foo.lock'), 'utf-8'); + const parsed = JSON.parse(raw) as LockFileRecord; + expect(parsed.pid).toBe(process.pid); + expect(parsed.ttlMs).toBe(12345); + expect(typeof parsed.ts).toBe('number'); + expect(typeof parsed.hostname).toBe('string'); + }); + + it('auto-creates the parent directory on first acquire', () => { + const deepDir = join(lockDir, 'a', 'b', 'c'); + const result = acquirePackLock('bar', { lockDir: deepDir }); + expect(result.outcome).toBe('acquired'); + expect(existsSync(join(deepDir, 'bar.lock'))).toBe(true); + }); +}); + +describe('acquirePackLock — contention', () => { + it('refuses when lock is held by live process with non-expired TTL', () => { + // Hand-craft a live, fresh lock. + const lockPath = join(lockDir, 'foo.lock'); + const record: LockFileRecord = { + pid: 99999, + hostname: 'test', + ts: Date.now(), + ttlMs: 60_000, + }; + writeFileSync(lockPath, JSON.stringify(record), 'utf-8'); + + expect(() => + acquirePackLock('foo', { lockDir, isPidAlive: liveAlways }), + ).toThrow(PackLockBusyError); + }); + + it('steals lock when holder PID is dead', () => { + const lockPath = join(lockDir, 'foo.lock'); + writeFileSync(lockPath, JSON.stringify({ + pid: 99999, hostname: 'test', ts: Date.now(), ttlMs: 60_000, + }), 'utf-8'); + + const result = acquirePackLock('foo', { lockDir, isPidAlive: deadAlways }); + expect(result.outcome).toBe('stolen_stale'); + expect(result.record.pid).toBe(process.pid); + }); + + it('steals lock when TTL is expired even if PID is alive', () => { + const lockPath = join(lockDir, 'foo.lock'); + writeFileSync(lockPath, JSON.stringify({ + pid: 99999, hostname: 'test', ts: Date.now() - 120_000, ttlMs: 60_000, + }), 'utf-8'); + + const result = acquirePackLock('foo', { lockDir, isPidAlive: liveAlways }); + expect(result.outcome).toBe('stolen_stale'); + }); + + it('steals lock with --force even when live + non-stale', () => { + const lockPath = join(lockDir, 'foo.lock'); + writeFileSync(lockPath, JSON.stringify({ + pid: 99999, hostname: 'test', ts: Date.now(), ttlMs: 60_000, + }), 'utf-8'); + + const result = acquirePackLock('foo', { lockDir, force: true, isPidAlive: liveAlways }); + expect(result.outcome).toBe('forced'); + expect(result.record.pid).toBe(process.pid); + }); + + it('PackLockBusyError carries heldBy + ageMs + ttlMs', () => { + const lockPath = join(lockDir, 'foo.lock'); + const past = Date.now() - 1500; + writeFileSync(lockPath, JSON.stringify({ + pid: 88888, hostname: 'test', ts: past, ttlMs: 60_000, + }), 'utf-8'); + + try { + acquirePackLock('foo', { lockDir, isPidAlive: liveAlways }); + throw new Error('expected throw'); + } catch (err) { + expect(err).toBeInstanceOf(PackLockBusyError); + const lockErr = err as PackLockBusyError; + expect(lockErr.heldBy).toBe(88888); + expect(lockErr.ageMs).toBeGreaterThanOrEqual(1500); + expect(lockErr.ttlMs).toBe(60_000); + } + }); +}); + +describe('acquirePackLock — corruption recovery', () => { + it('steals when lockfile content is unparseable', () => { + const lockPath = join(lockDir, 'foo.lock'); + writeFileSync(lockPath, 'not-valid-json{{{', 'utf-8'); + const result = acquirePackLock('foo', { lockDir, isPidAlive: liveAlways }); + expect(result.outcome).toBe('stolen_stale'); + }); + + it('steals when lockfile is empty', () => { + const lockPath = join(lockDir, 'foo.lock'); + writeFileSync(lockPath, '', 'utf-8'); + const result = acquirePackLock('foo', { lockDir, isPidAlive: liveAlways }); + expect(result.outcome).toBe('stolen_stale'); + }); + + it('steals when lockfile shape is missing required fields', () => { + const lockPath = join(lockDir, 'foo.lock'); + writeFileSync(lockPath, JSON.stringify({ pid: 'not-a-number' }), 'utf-8'); + const result = acquirePackLock('foo', { lockDir, isPidAlive: liveAlways }); + expect(result.outcome).toBe('stolen_stale'); + }); +}); + +describe('isLockStale — policy unit tests', () => { + it('returns live when ts is fresh and pid is alive', () => { + const rec: LockFileRecord = { pid: 1, hostname: 'h', ts: 1000, ttlMs: 60_000 }; + expect(isLockStale(rec, 30_000, liveAlways)).toEqual({ stale: false, reason: 'live' }); + }); + + it('returns ttl_expired when age > ttl, regardless of PID', () => { + const rec: LockFileRecord = { pid: 1, hostname: 'h', ts: 1000, ttlMs: 1000 }; + expect(isLockStale(rec, 5000, liveAlways)).toEqual({ stale: true, reason: 'ttl_expired' }); + }); + + it('returns pid_dead when age <= ttl but PID is dead', () => { + const rec: LockFileRecord = { pid: 1, hostname: 'h', ts: 1000, ttlMs: 60_000 }; + expect(isLockStale(rec, 2000, deadAlways)).toEqual({ stale: true, reason: 'pid_dead' }); + }); + + it('checks ttl BEFORE pid (avoids unnecessary kill syscall)', () => { + const rec: LockFileRecord = { pid: 1, hostname: 'h', ts: 1000, ttlMs: 1000 }; + let pidProbed = false; + const probe = (_pid: number) => { pidProbed = true; return true; }; + isLockStale(rec, 5000, probe); + expect(pidProbed).toBe(false); + }); +}); + +describe('withPackLock — wrapper contract', () => { + it('runs the callback and releases lock on success', async () => { + let ran = false; + await withPackLock('foo', { lockDir }, async () => { + ran = true; + expect(existsSync(join(lockDir, 'foo.lock'))).toBe(true); + }); + expect(ran).toBe(true); + expect(existsSync(join(lockDir, 'foo.lock'))).toBe(false); + }); + + it('releases lock even when callback throws', async () => { + await expect( + withPackLock('foo', { lockDir }, async () => { + throw new Error('boom'); + }), + ).rejects.toThrow('boom'); + expect(existsSync(join(lockDir, 'foo.lock'))).toBe(false); + }); + + it('returns the callback value', async () => { + const result = await withPackLock('foo', { lockDir }, async () => 42); + expect(result).toBe(42); + }); + + it('serializes two concurrent withPackLock calls (second throws BUSY)', async () => { + let firstReleased = false; + const first = withPackLock('foo', { lockDir }, async () => { + // Hold for 50ms. + await new Promise((r) => setTimeout(r, 50)); + firstReleased = true; + }); + // Give first a moment to acquire. + await new Promise((r) => setTimeout(r, 10)); + await expect( + withPackLock('foo', { lockDir, isPidAlive: liveAlways }, async () => 'second'), + ).rejects.toThrow(PackLockBusyError); + await first; + expect(firstReleased).toBe(true); + }); +}); + +describe('cleanup invariants', () => { + it('does not leak file descriptors across many acquire/release cycles', async () => { + // Smoke test — 100 cycles. If we leaked fds, EMFILE would eventually fire. + for (let i = 0; i < 100; i++) { + await withPackLock('foo', { lockDir }, async () => { + return i; + }); + } + expect(existsSync(join(lockDir, 'foo.lock'))).toBe(false); + }); +}); diff --git a/test/schema-pack-query-cache-invalidator.test.ts b/test/schema-pack-query-cache-invalidator.test.ts new file mode 100644 index 000000000..8cdc284f7 --- /dev/null +++ b/test/schema-pack-query-cache-invalidator.test.ts @@ -0,0 +1,83 @@ +// v0.40.6.0 — query-cache-invalidator.ts contract tests. +// +// Pins the C9 fix: schema mutations DELETE the query_cache for the +// affected source so cached search results bound to old page types +// don't survive a `sync --apply`. + +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { invalidateQueryCache } from '../src/core/schema-pack/query-cache-invalidator.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +async function seedCacheRow(sourceId: string, queryText: string): Promise { + // Use raw INSERT to bypass the semantic-similarity path. We're testing the + // CLEAR behavior, not the LOOKUP behavior. + await engine.executeRaw( + `INSERT INTO query_cache (id, query_text, source_id, knobs_hash, embedding, results, meta, ttl_seconds, created_at) + VALUES ($1, $2, $3, 'v3:test', NULL, '[]'::jsonb, '{}'::jsonb, 3600, now())`, + [`${sourceId}-${queryText}-id`, queryText, sourceId], + ); +} + +async function countCacheRows(sourceId?: string): Promise { + const sql = sourceId + ? `SELECT COUNT(*)::int AS n FROM query_cache WHERE source_id = $1` + : `SELECT COUNT(*)::int AS n FROM query_cache`; + const rows = await engine.executeRaw<{ n: number }>(sql, sourceId ? [sourceId] : []); + return rows[0]?.n ?? 0; +} + +describe('invalidateQueryCache', () => { + it('clears all rows for a given source_id', async () => { + await seedCacheRow('source-a', 'q1'); + await seedCacheRow('source-a', 'q2'); + await seedCacheRow('source-b', 'q1'); + expect(await countCacheRows('source-a')).toBe(2); + + const result = await invalidateQueryCache(engine, 'source-a'); + expect(result.rows_invalidated).toBe(2); + expect(await countCacheRows('source-a')).toBe(0); + expect(await countCacheRows('source-b')).toBe(1); + }); + + it('clears all rows when sourceId is omitted', async () => { + await seedCacheRow('source-a', 'q1'); + await seedCacheRow('source-b', 'q2'); + const result = await invalidateQueryCache(engine); + expect(result.rows_invalidated).toBe(2); + expect(await countCacheRows()).toBe(0); + }); + + it('is idempotent on empty cache', async () => { + const r1 = await invalidateQueryCache(engine, 'source-a'); + const r2 = await invalidateQueryCache(engine, 'source-a'); + expect(r1.rows_invalidated).toBe(0); + expect(r2.rows_invalidated).toBe(0); + }); + + it('returns {rows_invalidated: 0} silently if engine call fails (never throws)', async () => { + // Build a stub engine whose executeRaw always throws. + const broken = { + kind: 'pglite', + executeRaw: async () => { throw new Error('synthetic'); }, + } as unknown as PGLiteEngine; + const result = await invalidateQueryCache(broken, 'source-a'); + expect(result.rows_invalidated).toBe(0); + }); +}); diff --git a/test/schema-pack-registry-reload.test.ts b/test/schema-pack-registry-reload.test.ts new file mode 100644 index 000000000..0c0cb2dd7 --- /dev/null +++ b/test/schema-pack-registry-reload.test.ts @@ -0,0 +1,235 @@ +// v0.40.6.0 — registry.ts cache invalidation + stat-mtime TTL tests. +// +// Pins codex C6 fix (parent-pack edits cascade-invalidate children) and +// D11 + D13 (stat-mtime TTL gate keeps hot path cheap; cross-process +// mutations get picked up within STAT_TTL_MS). + +import { afterEach, beforeEach, describe, expect, it } from 'bun:test'; +import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { + invalidatePackCache, + resolvePack, + tryCachedPack, + _cacheNamesForTests, + _cacheSizeForTests, + _resetPackCacheForTests, + STAT_TTL_MS_DEFAULT, +} from '../src/core/schema-pack/registry.ts'; +import type { SchemaPackManifest } from '../src/core/schema-pack/manifest-v1.ts'; +import { withEnv } from './helpers/with-env.ts'; + +let tmpDir: string; + +function fakeManifest(name: string, opts: { extends?: string; version?: string } = {}): SchemaPackManifest { + return { + api_version: 'gbrain-schema-pack-v1', + name, + version: opts.version ?? '1.0.0', + description: '', + gbrain_min_version: '0.38.0', + extends: opts.extends ?? null, + borrow_from: [], + page_types: [ + { + name: 'person', + primitive: 'entity', + path_prefixes: ['people/'], + aliases: [], + extractable: false, + expert_routing: false, + }, + ], + link_types: [], + frontmatter_links: [], + takes_kinds: ['fact', 'take', 'bet', 'hunch'], + enrichable_types: [], + filing_rules: [], + } as SchemaPackManifest; +} + +beforeEach(() => { + _resetPackCacheForTests(); + tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-registry-test-')); +}); + +afterEach(() => { + _resetPackCacheForTests(); + try { rmSync(tmpDir, { recursive: true, force: true }); } catch { /* swallow */ } +}); + +describe('invalidatePackCache — basic', () => { + it('returns invalidated: [] when no entries exist', () => { + const r = invalidatePackCache('nonexistent'); + expect(r.invalidated).toEqual(['nonexistent']); + expect(_cacheSizeForTests()).toBe(0); + }); + + it('invalidates a single cached entry by name', async () => { + await resolvePack(fakeManifest('a'), async () => { throw new Error('no parent'); }); + expect(_cacheSizeForTests()).toBe(1); + invalidatePackCache('a'); + expect(_cacheSizeForTests()).toBe(0); + }); + + it('invalidates all entries when called with no argument', async () => { + await resolvePack(fakeManifest('a'), async () => { throw new Error('no parent'); }); + await resolvePack(fakeManifest('b'), async () => { throw new Error('no parent'); }); + expect(_cacheSizeForTests()).toBe(2); + const r = invalidatePackCache(); + expect(r.invalidated.sort()).toEqual(['a', 'b']); + expect(_cacheSizeForTests()).toBe(0); + }); +}); + +describe('invalidatePackCache — extends-chain cascade (codex C6 fix)', () => { + it('invalidating a parent cascades to every child that extends it', async () => { + const parentManifest = fakeManifest('p'); + const child1Manifest = fakeManifest('c1', { extends: 'p' }); + const child2Manifest = fakeManifest('c2', { extends: 'p' }); + const grandchildManifest = fakeManifest('g', { extends: 'c1' }); + + const loadByName = async (name: string): Promise => { + if (name === 'p') return parentManifest; + if (name === 'c1') return child1Manifest; + if (name === 'c2') return child2Manifest; + throw new Error('unknown parent in test'); + }; + await resolvePack(parentManifest, loadByName); + await resolvePack(child1Manifest, loadByName); + await resolvePack(child2Manifest, loadByName); + await resolvePack(grandchildManifest, loadByName); + expect(_cacheSizeForTests()).toBe(4); + + const result = invalidatePackCache('p'); + // p, c1, c2 directly contain 'p' in their chain; g contains c1 which + // contains p. Cascade evicts p, c1, c2 (one-hop). g has 'c1' + 'p' in + // its chain (via the extends walk during resolve), so it's also + // evicted. The dependent set is built from cached entries' chain arrays. + expect(new Set(result.invalidated)).toEqual(new Set(['p', 'c1', 'c2', 'g'])); + expect(_cacheSizeForTests()).toBe(0); + }); + + it('invalidating a leaf does NOT touch siblings or parent', async () => { + const p = fakeManifest('p'); + const c1 = fakeManifest('c1', { extends: 'p' }); + const c2 = fakeManifest('c2', { extends: 'p' }); + const loadByName = async (n: string) => (n === 'p' ? p : n === 'c1' ? c1 : c2); + await resolvePack(p, loadByName); + await resolvePack(c1, loadByName); + await resolvePack(c2, loadByName); + + invalidatePackCache('c1'); + expect(_cacheNamesForTests().sort()).toEqual(['c2', 'p']); + }); +}); + +describe('tryCachedPack — TTL gate', () => { + it('returns null when name is not cached', () => { + expect(tryCachedPack('never-seen')).toBeNull(); + }); + + it('returns the cached resolved pack on hot path', async () => { + const m = fakeManifest('foo'); + const resolved = await resolvePack(m, async () => { throw new Error('no parent'); }); + const hit = tryCachedPack('foo'); + expect(hit).toBe(resolved); + expect(hit?.manifest.name).toBe('foo'); + }); + + it('respects GBRAIN_PACK_STAT_TTL_MS env override', async () => { + await withEnv({ GBRAIN_PACK_STAT_TTL_MS: '0' }, async () => { + // Cache + a file snapshot on disk. + const packPath = join(tmpDir, 'foo-pack.yaml'); + writeFileSync(packPath, 'placeholder', 'utf-8'); + const m = fakeManifest('foo'); + await resolvePack(m, async () => { throw new Error('no parent'); }, { + loadByPath: (n) => (n === 'foo' ? packPath : null), + }); + // TTL=0 forces a stat on every call. Touch the file → mtime changes. + // (small sleep ensures mtimeMs is different) + await new Promise((r) => setTimeout(r, 5)); + writeFileSync(packPath, 'updated', 'utf-8'); + const hit = tryCachedPack('foo'); + expect(hit).toBeNull(); + expect(_cacheNamesForTests()).not.toContain('foo'); + }); + }); + + it('falls back to default TTL when env override is invalid', async () => { + await withEnv({ GBRAIN_PACK_STAT_TTL_MS: 'not-a-number' }, async () => { + // Default TTL is 1000ms; just check it doesn't crash + returns hit. + const m = fakeManifest('foo'); + await resolvePack(m, async () => { throw new Error('no parent'); }); + expect(tryCachedPack('foo')).not.toBeNull(); + }); + }); +}); + +describe('stat-snapshot cross-process invalidation (D11)', () => { + it('detects mtime change on the pack file and invalidates', async () => { + await withEnv({ GBRAIN_PACK_STAT_TTL_MS: '0' }, async () => { + const packPath = join(tmpDir, 'p.yaml'); + writeFileSync(packPath, 'v1', 'utf-8'); + const m = fakeManifest('p'); + await resolvePack(m, async () => { throw new Error('no parent'); }, { + loadByPath: (n) => (n === 'p' ? packPath : null), + }); + expect(tryCachedPack('p')).not.toBeNull(); + + // Mutate the file mtime. + await new Promise((r) => setTimeout(r, 10)); + writeFileSync(packPath, 'v2', 'utf-8'); + + // Stat-TTL gate fires (TTL=0 = always stat), detects change, evicts. + expect(tryCachedPack('p')).toBeNull(); + }); + }); + + it('detects file deletion and evicts (cross-process delete)', async () => { + await withEnv({ GBRAIN_PACK_STAT_TTL_MS: '0' }, async () => { + const packPath = join(tmpDir, 'p.yaml'); + writeFileSync(packPath, 'v1', 'utf-8'); + const m = fakeManifest('p'); + await resolvePack(m, async () => { throw new Error('no parent'); }, { + loadByPath: (n) => (n === 'p' ? packPath : null), + }); + expect(tryCachedPack('p')).not.toBeNull(); + + rmSync(packPath); + expect(tryCachedPack('p')).toBeNull(); + }); + }); + + it('cascades when parent file mtime changes (codex C6 fix at file level)', async () => { + await withEnv({ GBRAIN_PACK_STAT_TTL_MS: '0' }, async () => { + const parentPath = join(tmpDir, 'parent.yaml'); + const childPath = join(tmpDir, 'child.yaml'); + writeFileSync(parentPath, 'parent v1', 'utf-8'); + writeFileSync(childPath, 'child v1', 'utf-8'); + + const parentM = fakeManifest('parent'); + const childM = fakeManifest('child', { extends: 'parent' }); + const loadByName = async (n: string) => (n === 'parent' ? parentM : childM); + const loadByPath = (n: string) => (n === 'parent' ? parentPath : n === 'child' ? childPath : null); + + await resolvePack(parentM, loadByName, { loadByPath }); + await resolvePack(childM, loadByName, { loadByPath }); + + // Mutate ONLY the parent file. + await new Promise((r) => setTimeout(r, 10)); + writeFileSync(parentPath, 'parent v2', 'utf-8'); + + // tryCachedPack on the CHILD must detect parent's mtime change + // (parent is in child's chain + files snapshot). + expect(tryCachedPack('child')).toBeNull(); + }); + }); +}); + +describe('STAT_TTL_MS_DEFAULT export', () => { + it('exports the default constant', () => { + expect(STAT_TTL_MS_DEFAULT).toBe(1000); + }); +}); diff --git a/test/schema-pack-stats.test.ts b/test/schema-pack-stats.test.ts new file mode 100644 index 000000000..67e18e10a --- /dev/null +++ b/test/schema-pack-stats.test.ts @@ -0,0 +1,241 @@ +// v0.40.6.0 — stats.ts contract tests. +// +// Multi-source aware, soft-delete exclusion, dead-prefix detection, +// PGLite parity. Phase 3 of the schema cathedral v3 plan. + +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { runStatsCore } from '../src/core/schema-pack/stats.ts'; +import { + __setPackLocatorForTests, + _resetPackLocatorForTests, +} from '../src/core/schema-pack/load-active.ts'; +import { _resetPackCacheForTests } from '../src/core/schema-pack/registry.ts'; +import type { OperationContext } from '../src/core/operations.ts'; +import { withEnv } from './helpers/with-env.ts'; + +let engine: PGLiteEngine; +let tmpDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + _resetPackCacheForTests(); + _resetPackLocatorForTests(); + tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-stats-test-')); +}); + +function ctxOf(remote = false): OperationContext { + return { + engine, + config: {}, + logger: { info: () => {}, warn: () => {}, error: () => {} }, + dryRun: false, + remote, + } as unknown as OperationContext; +} + +async function ensureSource(id: string): Promise { + if (id === 'default') return; // seeded by schema bootstrap + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT (id) DO NOTHING`, + [id], + ); +} + +async function seedPage(slug: string, opts: { type?: string; sourceId?: string; sourcePath?: string; deleted?: boolean } = {}): Promise { + // pages.type is NOT NULL; use empty string for "untyped". + // pages.title is NOT NULL. + // pages.source_id FKs sources(id) — seed source first. + const sourceId = opts.sourceId ?? 'default'; + await ensureSource(sourceId); + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, source_path, type, title, compiled_truth, timeline, content_hash, deleted_at) + VALUES ($1, $2, $3, $4, $5, '', '', '', $6)`, + [slug, sourceId, opts.sourcePath ?? `${slug}.md`, opts.type ?? '', slug, opts.deleted ? new Date() : null], + ); +} + +function seedTinyPack(packName: string, types: Array<{ name: string; prefix: string }>): void { + const dir = join(tmpDir, packName); + mkdirSync(dir, { recursive: true }); + const path = join(dir, 'pack.yaml'); + let body = `api_version: gbrain-schema-pack-v1\nname: ${packName}\nversion: 1.0.0\ndescription: ""\ngbrain_min_version: 0.38.0\nextends: null\nborrow_from: []\npage_types:\n`; + for (const t of types) { + body += ` - name: ${t.name}\n primitive: entity\n path_prefixes:\n - ${t.prefix}\n aliases: []\n extractable: false\n expert_routing: false\n`; + } + body += `link_types: []\nfrontmatter_links: []\ntakes_kinds:\n - fact\n - take\n - bet\n - hunch\nenrichable_types: []\nfiling_rules: []\n`; + writeFileSync(path, body, 'utf-8'); + __setPackLocatorForTests((name) => (name === packName ? path : null)); +} + +describe('runStatsCore — empty brain', () => { + it('reports coverage:1.0 (vacuous truth)', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + const result = await runStatsCore(ctxOf()); + expect(result.aggregate.total_pages).toBe(0); + expect(result.aggregate.coverage).toBe(1.0); + expect(result.per_source).toEqual([]); + }); + }); +}); + +describe('runStatsCore — single source', () => { + it('counts typed + untyped pages and computes coverage', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + await seedPage('a', { type: 'person' }); + await seedPage('b', { type: 'person' }); + await seedPage('c', { type: 'company' }); + await seedPage('d'); // untyped + const result = await runStatsCore(ctxOf()); + expect(result.aggregate.total_pages).toBe(4); + expect(result.aggregate.typed_pages).toBe(3); + expect(result.aggregate.untyped_pages).toBe(1); + expect(result.aggregate.coverage).toBe(0.75); + expect(result.aggregate.by_type).toEqual([ + { type: 'person', count: 2 }, + { type: 'company', count: 1 }, + ]); + }); + }); + + it('excludes soft-deleted pages', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + await seedPage('a', { type: 'person' }); + await seedPage('b', { type: 'person', deleted: true }); + const result = await runStatsCore(ctxOf()); + expect(result.aggregate.total_pages).toBe(1); + }); + }); + + it('respects sourceId scoping', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + await seedPage('a', { type: 'person', sourceId: 'src-a' }); + await seedPage('b', { type: 'person', sourceId: 'src-b' }); + const result = await runStatsCore(ctxOf(), { sourceId: 'src-a' }); + expect(result.aggregate.total_pages).toBe(1); + expect(result.per_source.length).toBe(1); + expect(result.per_source[0]!.source_id).toBe('src-a'); + }); + }); +}); + +describe('runStatsCore — federated', () => { + it('aggregates across sourceIds array', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + await seedPage('a', { type: 'person', sourceId: 'src-a' }); + await seedPage('b', { type: 'person', sourceId: 'src-b' }); + await seedPage('c', { type: 'person', sourceId: 'src-c' }); + const result = await runStatsCore(ctxOf(), { sourceIds: ['src-a', 'src-b'] }); + expect(result.aggregate.total_pages).toBe(2); + expect(result.per_source.length).toBe(2); + const sources = result.per_source.map((s) => s.source_id).sort(); + expect(sources).toEqual(['src-a', 'src-b']); + }); + }); + + it('per-source breakdown sorted alphabetically', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + await seedPage('a', { type: 'person', sourceId: 'src-zzz' }); + await seedPage('b', { type: 'person', sourceId: 'src-aaa' }); + const result = await runStatsCore(ctxOf()); + expect(result.per_source.map((s) => s.source_id)).toEqual(['src-aaa', 'src-zzz']); + }); + }); +}); + +describe('runStatsCore — dead-prefix detection', () => { + it('flags pack-declared prefixes with zero matching pages', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack('tiny', [ + { name: 'person', prefix: 'people/' }, + { name: 'company', prefix: 'companies/' }, + ]); + await seedPage('people/alice', { type: 'person', sourcePath: 'people/alice.md' }); + // No companies/* pages → dead prefix. + const result = await runStatsCore(ctxOf()); + expect(result.pack_identity).not.toBeNull(); + expect(result.dead_prefixes).toEqual([{ type: 'company', prefix: 'companies/' }]); + }); + }); + + it('no dead-prefix hints when every declared prefix has pages', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack('tiny', [{ name: 'person', prefix: 'people/' }]); + await seedPage('people/alice', { type: 'person', sourcePath: 'people/alice.md' }); + const result = await runStatsCore(ctxOf()); + expect(result.dead_prefixes).toEqual([]); + }); + }); + + it('returns empty dead_prefixes when pack load fails', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: 'never-installed' }, async () => { + __setPackLocatorForTests(() => null); + const result = await runStatsCore(ctxOf()); + expect(result.pack_identity).toBeNull(); + expect(result.dead_prefixes).toEqual([]); + }); + }); +}); + +describe('runStatsCore — JSON envelope shape', () => { + it('schema_version stays 1 (stable contract)', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + const result = await runStatsCore(ctxOf()); + expect(result.schema_version).toBe(1); + }); + }); + + it('aggregate fields match the per-source merge', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + await seedPage('a', { type: 'person', sourceId: 'src-a' }); + await seedPage('b', { sourceId: 'src-b' }); // untyped + const result = await runStatsCore(ctxOf()); + const totalFromPer = result.per_source.reduce((acc, s) => acc + s.total_pages, 0); + expect(result.aggregate.total_pages).toBe(totalFromPer); + }); + }); + + it('by_type sorted by count desc, ties by name asc', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + await seedPage('a', { type: 'company' }); + await seedPage('b', { type: 'person' }); + await seedPage('c', { type: 'person' }); + await seedPage('d', { type: 'person' }); + const result = await runStatsCore(ctxOf()); + expect(result.aggregate.by_type[0]!.type).toBe('person'); + expect(result.aggregate.by_type[1]!.type).toBe('company'); + }); + }); +}); + +describe('runStatsCore — type/untyped split', () => { + it('treats empty-string type as untyped (not its own bucket)', async () => { + await withEnv({ GBRAIN_SCHEMA_PACK: undefined }, async () => { + // Some legacy rows might have type='' rather than NULL. + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, source_path, type, title, compiled_truth, timeline, content_hash) + VALUES ('a', 'default', 'a.md', '', 'a', '', '', '')`, + ); + await seedPage('b', { type: 'person' }); + const result = await runStatsCore(ctxOf()); + expect(result.aggregate.untyped_pages).toBe(1); + expect(result.aggregate.typed_pages).toBe(1); + // empty-string type does NOT appear as its own type bucket. + expect(result.aggregate.by_type.find((t) => t.type === '')).toBeUndefined(); + }); + }); +}); diff --git a/test/schema-pack-sync.test.ts b/test/schema-pack-sync.test.ts new file mode 100644 index 000000000..8f1fe62b9 --- /dev/null +++ b/test/schema-pack-sync.test.ts @@ -0,0 +1,272 @@ +// v0.40.6.0 — sync.ts contract tests. +// +// Dry-run vs apply, chunked-batch correctness, idempotency, +// soft-delete exclusion, sample-slug payload, dead-prefix hint, +// per-source write scoping, PGLite parity. Phase 3 of the cathedral. + +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'bun:test'; +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { runSyncCore } from '../src/core/schema-pack/sync.ts'; +import { + __setPackLocatorForTests, + _resetPackLocatorForTests, +} from '../src/core/schema-pack/load-active.ts'; +import { _resetPackCacheForTests } from '../src/core/schema-pack/registry.ts'; +import type { OperationContext } from '../src/core/operations.ts'; +import { withEnv } from './helpers/with-env.ts'; + +let engine: PGLiteEngine; +let tmpDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + _resetPackCacheForTests(); + _resetPackLocatorForTests(); + tmpDir = mkdtempSync(join(tmpdir(), 'gbrain-sync-test-')); +}); + +function ctxOf(remote = false): OperationContext { + return { + engine, + config: {}, + logger: { info: () => {}, warn: () => {}, error: () => {} }, + dryRun: false, + remote, + } as unknown as OperationContext; +} + +async function ensureSource(id: string): Promise { + if (id === 'default') return; + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT (id) DO NOTHING`, + [id], + ); +} + +async function seedPage(slug: string, sourcePath: string, opts: { type?: string; sourceId?: string; deleted?: boolean } = {}): Promise { + const sourceId = opts.sourceId ?? 'default'; + await ensureSource(sourceId); + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, source_path, type, title, compiled_truth, timeline, content_hash, deleted_at) + VALUES ($1, $2, $3, $4, $5, '', '', '', $6)`, + [slug, sourceId, sourcePath, opts.type ?? '', slug, opts.deleted ? new Date() : null], + ); +} + +async function getType(slug: string): Promise { + const rows = await engine.executeRaw<{ type: string }>( + `SELECT type FROM pages WHERE slug = $1`, + [slug], + ); + return rows[0]?.type ?? null; +} + +function seedTinyPack(types: Array<{ name: string; prefix: string }>): void { + const dir = join(tmpDir, 'tiny'); + mkdirSync(dir, { recursive: true }); + const path = join(dir, 'pack.yaml'); + let body = `api_version: gbrain-schema-pack-v1\nname: tiny\nversion: 1.0.0\ndescription: ""\ngbrain_min_version: 0.38.0\nextends: null\nborrow_from: []\npage_types:\n`; + for (const t of types) { + body += ` - name: ${t.name}\n primitive: entity\n path_prefixes:\n - ${t.prefix}\n aliases: []\n extractable: false\n expert_routing: false\n`; + } + body += `link_types: []\nfrontmatter_links: []\ntakes_kinds:\n - fact\n - take\n - bet\n - hunch\nenrichable_types: []\nfiling_rules: []\n`; + writeFileSync(path, body, 'utf-8'); + __setPackLocatorForTests((name) => (name === 'tiny' ? path : null)); +} + +describe('runSyncCore — dry-run', () => { + it('returns would_apply count + sample_slugs without writing', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + await seedPage('alice', 'people/alice.md'); + await seedPage('bob', 'people/bob.md'); + const result = await runSyncCore(ctxOf(), { apply: false }); + expect(result.apply).toBe(false); + expect(result.total_would_apply).toBe(2); + expect(result.total_applied).toBe(0); + const personEntry = result.per_prefix.find((p) => p.type === 'person')!; + expect(personEntry.would_apply).toBe(2); + expect(personEntry.applied).toBe(0); + expect(personEntry.sample_slugs).toEqual(['alice', 'bob']); + // Confirm types still empty after dry-run. + expect(await getType('alice')).toBe(''); + expect(await getType('bob')).toBe(''); + }); + }); + + it('sample_slugs capped at 10', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + for (let i = 0; i < 15; i++) { + await seedPage(`p${String(i).padStart(2, '0')}`, `people/p${String(i).padStart(2, '0')}.md`); + } + const result = await runSyncCore(ctxOf(), { apply: false }); + const personEntry = result.per_prefix.find((p) => p.type === 'person')!; + expect(personEntry.would_apply).toBe(15); + expect(personEntry.sample_slugs.length).toBe(10); + }); + }); + + it('dead_prefix flag fires when no pages match', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([ + { name: 'person', prefix: 'people/' }, + { name: 'company', prefix: 'companies/' }, + ]); + await seedPage('alice', 'people/alice.md'); + const result = await runSyncCore(ctxOf(), { apply: false }); + const companyEntry = result.per_prefix.find((p) => p.type === 'company')!; + expect(companyEntry.dead_prefix).toBe(true); + expect(companyEntry.would_apply).toBe(0); + }); + }); +}); + +describe('runSyncCore — apply', () => { + it('updates page.type for matching untyped pages', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + await seedPage('alice', 'people/alice.md'); + await seedPage('bob', 'people/bob.md'); + const result = await runSyncCore(ctxOf(), { apply: true }); + expect(result.total_applied).toBe(2); + expect(await getType('alice')).toBe('person'); + expect(await getType('bob')).toBe('person'); + }); + }); + + it('idempotent: second apply is a no-op', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + await seedPage('alice', 'people/alice.md'); + const first = await runSyncCore(ctxOf(), { apply: true }); + const second = await runSyncCore(ctxOf(), { apply: true }); + expect(first.total_applied).toBe(1); + expect(second.total_applied).toBe(0); + }); + }); + + it('chunked UPDATE: large set in 1000-row batches (perf-shape verification)', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + // Seed 1500 untyped pages (1.5× the default batch size). + for (let i = 0; i < 1500; i++) { + await seedPage(`p${String(i).padStart(4, '0')}`, `people/p${String(i).padStart(4, '0')}.md`); + } + const batches: number[] = []; + const result = await runSyncCore(ctxOf(), { + apply: true, + batchSize: 1000, + onProgress: (info) => batches.push(info.appliedSoFar), + }); + expect(result.total_applied).toBe(1500); + // Progress fired at least once for the first batch (1000) and + // again for the tail (1500 total). + expect(batches).toContain(1000); + expect(batches).toContain(1500); + }); + }); + + it('does NOT touch pages that already have a type', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + await seedPage('alice', 'people/alice.md', { type: 'old-type' }); + await seedPage('bob', 'people/bob.md'); // untyped + await runSyncCore(ctxOf(), { apply: true }); + expect(await getType('alice')).toBe('old-type'); // preserved + expect(await getType('bob')).toBe('person'); + }); + }); + + it('excludes soft-deleted pages', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + await seedPage('alice', 'people/alice.md'); + await seedPage('zombie', 'people/zombie.md', { deleted: true }); + const result = await runSyncCore(ctxOf(), { apply: true }); + expect(result.total_applied).toBe(1); + expect(await getType('alice')).toBe('person'); + expect(await getType('zombie')).toBe(''); // untouched (soft-deleted) + }); + }); +}); + +describe('runSyncCore — source scoping (codex C5 write-side)', () => { + it('updates only the scoped source', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + await seedPage('alice', 'people/alice.md', { sourceId: 'src-a' }); + await seedPage('bob', 'people/bob.md', { sourceId: 'src-b' }); + const result = await runSyncCore(ctxOf(), { apply: true, sourceId: 'src-a' }); + expect(result.total_applied).toBe(1); + expect(await getType('alice')).toBe('person'); + expect(await getType('bob')).toBe(''); + }); + }); +}); + +describe('runSyncCore — pack-load failure', () => { + it('returns empty result when no pack is loaded', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'nonexistent' }, async () => { + __setPackLocatorForTests(() => null); + await seedPage('alice', 'people/alice.md'); + const result = await runSyncCore(ctxOf(), { apply: true }); + expect(result.pack_identity).toBeNull(); + expect(result.per_prefix).toEqual([]); + expect(result.total_applied).toBe(0); + // Page untouched (no pack = no inference rules). + expect(await getType('alice')).toBe(''); + }); + }); +}); + +describe('runSyncCore — JSON envelope shape', () => { + it('schema_version stays 1', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + const result = await runSyncCore(ctxOf()); + expect(result.schema_version).toBe(1); + }); + }); + + it('per_prefix entry shape is stable', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + await seedPage('alice', 'people/alice.md'); + const result = await runSyncCore(ctxOf()); + const entry = result.per_prefix[0]!; + expect(entry).toHaveProperty('type'); + expect(entry).toHaveProperty('prefix'); + expect(entry).toHaveProperty('would_apply'); + expect(entry).toHaveProperty('sample_slugs'); + expect(entry).toHaveProperty('dead_prefix'); + expect(entry).toHaveProperty('applied'); + }); + }); +}); + +describe('runSyncCore — batch size clamping', () => { + it('clamps batch size to >=1 and <=10000', async () => { + await withEnv({ GBRAIN_HOME: tmpDir, GBRAIN_SCHEMA_PACK: 'tiny' }, async () => { + seedTinyPack([{ name: 'person', prefix: 'people/' }]); + await seedPage('alice', 'people/alice.md'); + // batchSize:0 and batchSize:99999 both work; result is identical. + const r1 = await runSyncCore(ctxOf(), { apply: true, batchSize: 0 }); + expect(r1.total_applied).toBe(1); + }); + }); +});