From 26c54588fe5e802a1526f6019e428e6c8799eb47 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 22 May 2026 08:00:42 -0700 Subject: [PATCH] =?UTF-8?q?v0.38.0.0=20ingestion=20cathedral=20=E2=80=94?= =?UTF-8?q?=20gbrain=20capture=20+=20write-through=20+=20IngestionSource?= =?UTF-8?q?=20contract=20(#1275)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(ingestion): v0.38 substrate — daemon + IngestionSource contract + 2 sources The foundation for the ingestion cathedral (CEO+DX+Eng plan-reviewed). Plan: ~/.claude/plans/system-instruction-you-are-working-ethereal-riddle.md WHAT YOU CAN NOW DO The IngestionSource public contract is locked. Skillpack publishers can build third-party ingestion sources (Granola, Linear, Mail, voice, OCR, etc.) and ship them through the v0.37 skillpack registry. The locked surface lives at the new package subpaths: import { IngestionSource, IngestionEvent } from 'gbrain/ingestion'; import { IngestionTestHarness, expectEvent } from 'gbrain/ingestion/test-harness'; Both subpaths are pinned by test/public-exports.test.ts — breaking either is a major-version change. WHAT THIS COMMIT BUILDS Foundation: - src/core/ingestion/types.ts (IngestionSource, IngestionEvent, IngestionSourceContext, validateIngestionEvent, computeContentHash, INGESTION_SOURCE_API_VERSION, INGESTION_CONTENT_TYPES) - src/core/ingestion/dedup.ts (24h content-hash LRU, 5000-entry cap) - src/core/ingestion/skillpack-load.ts (gbrain.plugin.json discovery for third-party sources, api_version compat with paste-ready upgrade hints, in-process trust model for v1) - src/core/ingestion/daemon.ts (IngestionDaemon: in-process source supervision sibling to v0.34.3.0 ChildWorkerSupervisor pattern, plus validate -> dedup -> rate-limit -> dispatch pipeline + health surface) - src/core/ingestion/test-harness.ts (publisher-facing test utility with fake clock + in-memory event bus + expectEvent matchers + engine proxy that throws on access so publishers know what they're depending on) - src/core/ingestion/index.ts (barrel for gbrain/ingestion subpath) First two built-in sources prove the abstraction: - file-watcher (chokidar over the brain repo; 1s debounce; honors pruneDir from src/core/sync.ts; symlinks rejected; Linux ENOSPC surfaces a paste-ready sysctl hint at runtime) - inbox-folder (~/.gbrain/inbox/ target for iOS Shortcuts / AirDrop / Drafts; auto-archives processed files into .archived/YYYY-MM-DD/; symlink rejection; world-writable dir warning; routes content-type by extension) Public exports surface (count 18 -> 20) pinned in: - package.json exports map - test/public-exports.test.ts EXPECTED_EXPORTS + count gate - scripts/check-exports-count.sh baseline ARCHITECTURE-LOCKED DECISIONS (from /plan-eng-review) E1 webhook source process boundary: webhook source will live INSIDE serve --http (NOT this daemon) when it lands in the next commit. Daemon supervises only daemon-side sources. E2 content-type processor execution: hybrid by size (inline <1MB, Minion handlers >1MB). Processors land in a later commit. E3 publisher TTHW: chokidar v4.0.3 across platforms; ephemeral PGLite persistence and Linux inotify-limit doctor probe land in later commits. E4 migration v80 (provenance columns) + forward-reference bootstrap: lands with put_page write-through in a later commit. DX-locked decisions (from /plan-devex-review): - Source error semantics: throws bubble to daemon; supervisor backoff. - IngestionTestHarness exported as gbrain/ingestion/test-harness. - api_version field on gbrain.plugin.json with loud-fail on mismatch. TESTS 192 cases across 8 test files, 0 failures: - test/ingestion/types.test.ts (28 cases pinning the contract) - test/ingestion/dedup.test.ts (15 cases for LRU + TTL + collision) - test/ingestion/skillpack-load.test.ts (22 cases for manifest validation + api_version compat + collision policy + module load) - test/ingestion/test-harness.test.ts (24 cases for harness lifecycle + clock + healthCheck + every expectEvent matcher) - test/ingestion/daemon.test.ts (19 cases for supervision + dispatch pipeline + health surface + per-source config + logger wrapping) - test/ingestion/sources/file-watcher.test.ts (10 cases including ENOSPC sysctl-hint surfacing) - test/ingestion/sources/inbox-folder.test.ts (24 cases including symlink rejection + world-writable warning + archive-loop-prevention) - test/public-exports.test.ts (2 new cases for the new subpaths) typecheck clean. bun run verify gate passes. NEXT IN WAVE Subsequent commits in this PR ship webhook source (serve --http route), cron-scheduler refactor + OpenClaw credential auto-migrate, content-type processors (PDF + image OCR + audio transcribe + video keyframe), put_page write-through with serializePageToMarkdown DRY extract, migration v80 + bootstrap probes, gbrain capture verb, publisher DX cathedral (init scaffold extension + gbrain ingest test [--watch] + tail + validate), daemon rename autopilot -> ingest with forever-alias, doctor inotify probe on Linux, skillpack contract docs + reference pack. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(ingestion): webhook source — POST /ingest + ingest_capture Minion handler Lands the v0.38 ingestion cathedral's webhook source. Per the /plan-eng-review E1 decision, the webhook source lives INSIDE `serve --http` (NOT the ingestion daemon) so there is no new IPC: the HTTP route submits Minion jobs directly into the existing queue, and the daemon supervises only daemon-side sources. WHAT YOU CAN NOW DO With `gbrain serve --http` running and an OAuth client minted, any HTTP caller (Zapier, IFTTT, n8n, Make, Apple Shortcuts) can POST a captured thought into the brain: curl -X POST https://your-brain.example.com/ingest \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: text/markdown" \ -d "# captured from my Shortcut" The route auths via OAuth (write scope required), validates the content-type, enforces a 1MB payload cap and per-IP rate limit (100 events / 10s), submits an `ingest_capture` Minion job tagged `untrusted_payload: true`, and returns 202 Accepted with the job id. The job materializes the page under `inbox/YYYY-MM-DD-` by default (overridable via X-Gbrain-Slug header) so the user has a predictable triage location. WHAT THIS COMMIT BUILDS - src/core/minions/handlers/ingest-capture.ts (new) — handler that takes an IngestionEvent payload, resolves a slug via fallback chain (job.data.slug -> event.metadata.slug -> inbox/-), validates the event at the handler boundary, REJECTS binary content_types with a paste-ready hint to install a processor skillpack, and routes through importFromContent. Defaults noEmbed: true (embed is a separate Minion job, matching the sync handler's pattern). - src/commands/jobs.ts — registers `ingest_capture` in registerBuiltinHandlers alongside sync/embed/extract. - src/commands/serve-http.ts — POST /ingest route with: - OAuth write-scope gate via requireBearerAuth({requiredScopes:['write']}) - 100 events / 10s rate limiter (sibling to ccRateLimiter) - Content-type allowlist: text/markdown, text/plain, text/html, application/json; binary REJECTED with HTTP 415 - 1 MB payload cap (configurable via GBRAIN_INGEST_MAX_BYTES) - Caller-overridable source identity via X-Gbrain-Source-Id / X-Gbrain-Source-Uri / X-Gbrain-Content-Type / X-Gbrain-Slug headers — useful for downstream tools that want clean provenance - untrusted_payload: true ALWAYS (network input) - Idempotency on (client_id, content_hash) so simultaneous retries collapse to one job - maxWaiting: 50 per client so a runaway integration can't monopolize the queue - Audit row in mcp_request_log + SSE broadcast for the admin feed TESTS test/ingestion/ingest-capture.test.ts (15 cases against PGLite): - defaultSlugForEvent helper (3 cases pinning shape + UTC + determinism) - slug resolution fallback chain (3 cases) - validation + content-type routing (5 cases including binary rejection + untrusted_payload round-trip) - importFromContent integration (3 cases including content_hash dedup via status='skipped' on repeat) 207 total ingestion tests passing. typecheck clean. NEXT IN WAVE cron-scheduler refactor + OpenClaw credential auto-migrate; content-type processors (PDF + image OCR + audio transcribe + video keyframe); put_page write-through + serializePageToMarkdown DRY extract + migration v80 + bootstrap probes; gbrain capture verb; publisher DX cathedral (init scaffold + gbrain ingest test --watch + tail + validate); daemon rename autopilot -> ingest with forever-alias; doctor inotify probe; skillpack contract docs + reference pack + VERSION bump. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(ingestion): put_page write-through + migration v80 + DRY extract WHAT YOU CAN NOW DO The drift class is dead. Every `gbrain put_page` (CLI or MCP, local or remote) now lands its markdown file on disk alongside the DB row whenever `sync.repo_path` is configured. The page is queryable immediately AND visible to git, your editor, and downstream tools. Pre-v0.38, put_page wrote ONLY to the DB and synthesize/extract paths had to reverse-render later. The v0.35.6.0 phantom-redirect pass was the cleanup for what THIS commit prevents in the first place. # local CLI gbrain put inbox/test < my-thought.md # file lands at ${sync.repo_path}/inbox/test.md AND in the DB # MCP remote (Zapier / Cursor / Claude Desktop) curl -X POST /mcp ... '{"method":"tools/call","params":{"name":"put_page",...}}' # server-side write-through fires, agent gets a normal success response # untrusted_payload tagging applied (no auto-link, slug-allowlist gate) Provenance frontmatter stamped on every write so future sync round-trips know where the page came from: ingested_via: put_page # local CLI ingested_via: 'mcp:put_page' # MCP remote ingested_at: 2026-05-21T04:... WHAT THIS COMMIT BUILDS 1. Migration v80 — `pages_provenance_columns` adds four nullable columns to `pages`: `ingested_via`, `ingested_at`, `source_uri`, `source_kind`. ADD COLUMN with no DEFAULT is metadata-only on Postgres 11+ and PGLite 17.5; instant on tables of any size. The four columns get NULL on every historical page (pre-v0.38 pages never had provenance). 2. DRY extract — `serializePageToMarkdown(page, tags, opts)` and `resolvePageFilePath(brainDir, slug, sourceId)` in `src/core/markdown.ts`. The dream-cycle's `renderPageToMarkdown` (synthesize.ts) and the new put_page write-through path were going to have 90% duplicate bodies. They now share one foundation; the dream version is a 4-line wrapper that passes `frontmatterOverrides: {dream_generated: true, ...}`. Future markdown-shape changes happen in one place. 3. put_page write-through (`src/core/operations.ts`) — after importFromContent succeeds, resolves sync.repo_path, computes the v0.32.8 source-aware path layout (default: brainDir/.md; non-default: brainDir/.sources//.md), serializes the freshly-written Page via `serializePageToMarkdown`, writes the file. Returns a `write_through: {written, path}` field in the put_page response so callers can see what happened. Trust gating: - subagent sandbox (viaSubagent without allowedSlugPrefixes) → DB-only - dry-run → DB-only (handler's early-return short-circuits before write-through; documented via the dry_run response field) - no sync.repo_path configured → DB-only, skipped reason returned - sync.repo_path points at a non-existent dir → DB-only, skipped - all other writes → write-through Failure isolation: disk-write failures are LOGGED loud but do NOT roll back the DB write. DB is the durable record; the phantom-redirect pass exists for drift cleanup if it ever shows up. TESTS - test/ingestion/put-page-write-through.test.ts (10 cases against PGLite): happy path (file land, provenance stamp local + remote), trust gating (subagent sandbox, dry-run, trusted-workspace), config edges (no repo_path, missing dir), multi-source filing (.sources//), failure isolation (DB write survives a disk failure). - Migration v80 verified across both engines via the existing test/migrate.test.ts + test/bootstrap.test.ts coverage (~125 cases). 369 total tests passing in the ingestion + markdown + migrate bundle. typecheck clean. NOTES - Bootstrap probes for the v80 provenance columns are NOT yet added to applyForwardReferenceBootstrap on either engine. This is safe for v0.38 because no SCHEMA_SQL CREATE INDEX or FK references the new columns — migration v80 is the only consumer, and it runs AFTER SCHEMA_SQL replay. A future commit may add bootstrap probes + REQUIRED_BOOTSTRAP_COVERAGE entries as defense-in-depth (eng review E4). - The trusted-workspace path (dream cycle's reverseWriteRefs in synthesize.ts) still runs its own write at synthesize phase time. Both paths writing the same file is idempotent (byte-identical serialization), but a future commit may simplify reverseWriteRefs to skip pages whose file already matches. NEXT IN WAVE gbrain capture verb (the single human-facing entrypoint); daemon rename autopilot -> ingest with forever-alias + plist migration; doctor inotify probe (Linux); content-type processor router (PDF + image OCR + audio transcribe stubs); cron-scheduler refactor + OpenClaw credential auto-migrate; skillpack contract docs + reference pack; VERSION bump. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(ingestion): gbrain capture — the single human-facing entrypoint WHAT YOU CAN NOW DO One command, local or thin-client, synchronous receipt with the resulting page slug. The answer to "what is the best way to get data into the brain?" is now: just type `gbrain capture` and the right thing happens. # the basic case gbrain capture "remember to follow up on the X deal" # from a file gbrain capture --file ./notes/today.md --slug daily/2026-05-21 # from a pipe (shell pipelines) echo "from stdin" | gbrain capture --stdin # script-friendly: print just the slug SLUG=$(gbrain capture "a thought" --quiet) # JSON for agents gbrain capture "..." --json Default slug is `inbox/YYYY-MM-DD-` — deterministic for the same content so re-running idempotently lands the same page. Receipt block on stdout shows slug + status + content_hash + on-disk path so you can confirm where the page went without rerunning `gbrain query`. The local-install path routes through the put_page operation with the v0.38 write-through plumbing landed in the prior commit, so the page hits both the DB AND the file tree in one move. Thin-client installs route through `callRemoteTool('put_page', ...)` so the server's write-through handles disk persistence the same way. WHAT THIS COMMIT BUILDS - src/commands/capture.ts (new ~290 LOC): - `defaultSlug(content)` — UTC-stable `inbox/YYYY-MM-DD-` - `parseArgs(args)` — positional + flag parsing with --file / --stdin / --slug / --type / --source / --quiet / --json / --help - `buildContent(rawBody, opts)` — wraps unstructured prose in frontmatter (type + title + captured_via + captured_at) and a leading `# Title` heading; passes through if the body already looks like markdown - `runCapture(engine, args)` — local install routes through the in-process put_page operation; thin-client routes through MCP. `--quiet` prints just the slug; `--json` prints structured output; default prints a 5-line receipt block. - src/cli.ts: - Adds `case 'capture'` dispatch - Adds `'capture'` to the CLI_ONLY set so cli.ts wires it correctly TESTS test/commands/capture.test.ts (21 cases against PGLite): - defaultSlug helper: shape + determinism + UTC math - parseArgs: positional + multi-token join + every flag - buildContent: prose wrapping, --type override, no double-wrap for pre-frontmattered content, title cap at 80 chars, --source provenance stamp - Integration: inline content lands in DB + on disk, default slug shape, --file reads from disk, --json structured output, --help returns without engine roundtrip 271 total tests passing in the bundle. typecheck clean. NOTES - Thin-client routing relies on `callRemoteTool('put_page', ...)` from src/core/mcp-client.ts. Identical UX to the local path because the server's put_page handler runs the same write-through plumbing. - buildContent's "looks like markdown" heuristic is intentionally simple — first-line heading or frontmatter delimiter is the trigger. Users who care about exact formatting pass a pre-formatted --file. NEXT IN WAVE Daemon rename autopilot -> ingest with forever-alias + plist migration; doctor inotify probe (Linux); content-type processor router (PDF + image OCR + audio transcribe stubs); cron-scheduler refactor + OpenClaw credential auto-migrate; skillpack contract docs + reference pack; VERSION bump. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(ingestion): v0.38.0.0 release hygiene + e2e roundtrip + capture skill VERSION 0.37.1.0 → 0.38.0.0 (trio: VERSION, package.json, CHANGELOG header). CHANGELOG entry written in user-facing ELI10-lead voice per CLAUDE.md release-summary rules. README's pre-loop section gains a new "How to get data in (v0.38+)" block leading with `gbrain capture`. skills/capture/SKILL.md (NEW) so agents route "capture this" / "save this thought" / "remember this" / "drop this in the inbox" / "save to brain" to the capture verb. RESOLVER.md updated with the new triggers (sits above idea-ingest/media-ingest/meeting-ingestion in the content-ingestion section as the "simple thought" path). E2E roundtrip test (test/e2e/ingestion-roundtrip.test.ts) covers the gap: inbox-folder source -> daemon -> ingest_capture handler -> DB page, including: - Full pipeline: file drop appears as page in DB + file moves to .archived/ - Dedup catches byte-identical content from a different filename - Multi-source coordination: two distinct inbox dirs, two sources, daemon ingests both events independently The test runs against an in-memory PGLite (no DATABASE_URL needed) so it exercises the substrate-level wiring in the standard test suite. A follow-up commit can add a full-process e2e (gbrain serve --http + real OAuth client + POST /ingest) that requires DATABASE_URL. 399/399 v0.38 wave tests passing (910 assertions). typecheck clean. bun run verify gate green across all 14 shell checks. DEFERRED TO FOLLOW-UP RELEASES (called out in CHANGELOG) - Daemon rename autopilot -> ingest + forever-alias + plist migration - cron-scheduler skill refactor + OpenClaw credential auto-migrate - Content-type processors (PDF / OCR / audio / video) - gbrain doctor inotify probe (Linux) - Publisher DX cathedral: gbrain skillpack init --kind=ingestion-source, gbrain ingest test --watch, ingest tail, ingest validate - Reference pack at examples/skillpack-ingestion-reference/ + 3-stage tutorial in docs/ingestion-source-skillpack.md These are polish items; the substrate is shipped and queryable, and skillpack publishers can build sources against the IngestionTestHarness public export today. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(ingestion): test-gap fills — bootstrap probes + manifest entry + conformance The v0.38.0.0 release-hygiene commit landed cleanly against the v0.38 wave suite but tripped 3 categories of full-suite tests. This commit fixes each. The remaining failure (doctorReportRemote > "healthy" status) was verified pre-existing via `git stash + bun test` and is not caused by v0.38; left alone. Fix 1 — `schema-bootstrap-coverage.test.ts` (s1) The test parses MIGRATIONS for ALTER TABLE ADD COLUMN statements and fails if any column is not covered by `applyForwardReferenceBootstrap` on both engines. Migration v80's four provenance columns triggered the failure. Bootstrap probes added to both engines + 4 entries appended to REQUIRED_BOOTSTRAP_COVERAGE: - src/core/pglite-engine.ts — 4 EXISTS probes + state field + needs flag + ALTER TABLE block when bootstrap fires - src/core/postgres-engine.ts — same pattern - test/schema-bootstrap-coverage.test.ts — 4 coverage entries Fix 2 — `check-resolvable.test.ts` (s3 — orphan_trigger) RESOLVER.md references skills via name; check-resolvable cross-checks against skills/manifest.json. The new `capture` skill was missing the manifest entry; added between `brain-ops` and `idea-ingest` so the manifest order mirrors the resolver order. Fix 3 — `skills-conformance.test.ts` (s8) Every SKILL.md must have `## Contract`, `## Output Format`, and `## Anti-Patterns` sections. skills/capture/SKILL.md was missing all three (initial draft skipped them); now compliant with concrete content per the v0.38 contract. Fix 4 — `build-llms.test.ts` (s6) README + CHANGELOG edits in the release-hygiene commit caused llms-full.txt to drift behind. Regenerated via `bun run build:llms`. Per CLAUDE.md: any user-facing docs edit MUST run build:llms before push. The full bun-test parallel runner now passes everywhere except the pre-existing `doctorReportRemote > healthy status` failure (50/100 score on an empty fresh brain — this is a pre-v0.38 health-score tuning issue and orthogonal to ingestion work). Co-Authored-By: Claude Opus 4.7 (1M context) * chore(version): bump 0.38.0.0 → 0.38.1.0 Renumbers the in-flight ingestion-cathedral release to v0.38.1.0. Trio (VERSION, package.json, CHANGELOG.md) bumped together. bun run typecheck → clean. * chore(version): bump 0.38.1.0 → 0.38.0.0 Master sits at 0.37.11.0; 0.38.0.0 is the natural next slot rather than skipping a release. Trio (VERSION, package.json, CHANGELOG.md) bumped together. Migration v81 + ingestion substrate stay identical — this is a header-only renumber. bun run typecheck → clean. * test(ingestion): fill v0.38 test gaps — markdown helpers + migration v81 + webhook E2E Three gaps surfaced from a v0.38 audit against what shipped vs what was covered. All three filled: 1. **test/markdown-serializer.test.ts** (NEW, 19 cases) — pure-function coverage of `serializePageToMarkdown` + `resolvePageFilePath`, the DRY extract that the dream-cycle reverse-render and put_page write-through both consume. Pre-fix nothing pinned the frontmatter-override merge precedence, the type/title defaults, or the source-aware filing layout (default → `/.md`, non-default → `/.sources//.md`). Future schema-shape changes to either helper now surface immediately. 2. **test/migrate.test.ts — v81 cases** (10 new cases, two describe blocks) — structural assertions on `pages_provenance_columns` (four nullable columns, no NOT NULL, no DEFAULT, no index — the ADD COLUMN stays metadata-only) plus a PGLite round-trip that asserts the columns appear post-`initSchema`, accept direct UPDATEs, and survive the historical-page NULL scenario. The schema-bootstrap-coverage test already pinned the forward-reference probe contract; this fills the migrate.test.ts contract gap. 3. **test/e2e/serve-http-ingest-webhook.test.ts** (NEW, 16 cases) — HTTP contract coverage for POST /ingest. The pre-existing ingestion-roundtrip E2E explicitly notes "e2e (gbrain serve --http + POST /ingest + real OAuth) is a separate" thing — it covers the in-process daemon → handler → DB pipeline, NOT the real HTTP route. This file fills that gap. Spawns real gbrain serve --http against real Postgres, mints OAuth tokens with various scopes, exercises: - Auth gate (missing → 401; read-only → 403) - Body validation (empty → 400 with error: empty_body) - Content-type allowlist (image/png → 415 with skillpack hint; application/pdf → 415; text/plain + application/json + text/html all accepted; unknown text/* falls through to text/plain) - X-Gbrain-Content-Type / Source-Id / Source-Uri / Slug header overrides - Idempotency (same content + same client = identical job_id via queue dedup on content_hash) Also wires three new entries into `scripts/e2e-test-map.ts` so changes to `src/commands/serve-http.ts`, `src/core/ingestion/**`, or the `ingest-capture` Minion handler auto-trigger the relevant E2Es under `bun run ci:local:diff`. Verified locally: - bun test test/markdown-serializer.test.ts → 19/19 green - bun test test/migrate.test.ts -t "v81" → 10/10 green - bun test test/e2e/serve-http-ingest-webhook.test.ts (real Postgres on ephemeral 5435) → 16/16 green - bun test test/select-e2e.test.ts → 24/24 green (selector test still honors the v0.38 entries) - bun run typecheck → clean E2E DB lifecycle handled per CLAUDE.md (spin up pgvector:pg16 on a free port, bootstrap via `gbrain doctor --json`, run, tear down). --------- Co-authored-by: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 161 ++++++ README.md | 32 ++ VERSION | 2 +- bun.lock | 5 + llms-full.txt | 33 ++ package.json | 7 +- scripts/check-exports-count.sh | 2 +- scripts/e2e-test-map.ts | 16 + skills/RESOLVER.md | 1 + skills/capture/SKILL.md | 105 ++++ skills/manifest.json | 5 + src/cli.ts | 8 +- src/commands/capture.ts | 322 +++++++++++ src/commands/jobs.ts | 11 + src/commands/serve-http.ts | 211 +++++++ src/core/cycle/synthesize.ts | 27 +- src/core/ingestion/daemon.ts | 524 ++++++++++++++++++ src/core/ingestion/dedup.ts | 169 ++++++ src/core/ingestion/index.ts | 33 ++ src/core/ingestion/skillpack-load.ts | 345 ++++++++++++ src/core/ingestion/sources/file-watcher.ts | 309 +++++++++++ src/core/ingestion/sources/inbox-folder.ts | 367 ++++++++++++ src/core/ingestion/test-harness.ts | 323 +++++++++++ src/core/ingestion/types.ts | 318 +++++++++++ src/core/markdown.ts | 82 ++- src/core/migrate.ts | 48 ++ src/core/minions/handlers/ingest-capture.ts | 127 +++++ src/core/operations.ts | 74 +++ src/core/pglite-engine.ts | 39 +- src/core/postgres-engine.ts | 40 +- test/commands/capture.test.ts | 230 ++++++++ test/e2e/ingestion-roundtrip.test.ts | 253 +++++++++ test/e2e/serve-http-ingest-webhook.test.ts | 365 ++++++++++++ test/ingestion/daemon.test.ts | 513 +++++++++++++++++ test/ingestion/dedup.test.ts | 160 ++++++ test/ingestion/ingest-capture.test.ts | 192 +++++++ test/ingestion/put-page-write-through.test.ts | 226 ++++++++ test/ingestion/skillpack-load.test.ts | 318 +++++++++++ test/ingestion/sources/file-watcher.test.ts | 294 ++++++++++ test/ingestion/sources/inbox-folder.test.ts | 358 ++++++++++++ test/ingestion/test-harness.test.ts | 292 ++++++++++ test/ingestion/types.test.ts | 195 +++++++ test/markdown-serializer.test.ts | 221 ++++++++ test/migrate.test.ts | 180 ++++++ test/public-exports.test.ts | 4 +- test/schema-bootstrap-coverage.test.ts | 11 + 46 files changed, 7532 insertions(+), 26 deletions(-) create mode 100644 skills/capture/SKILL.md create mode 100644 src/commands/capture.ts create mode 100644 src/core/ingestion/daemon.ts create mode 100644 src/core/ingestion/dedup.ts create mode 100644 src/core/ingestion/index.ts create mode 100644 src/core/ingestion/skillpack-load.ts create mode 100644 src/core/ingestion/sources/file-watcher.ts create mode 100644 src/core/ingestion/sources/inbox-folder.ts create mode 100644 src/core/ingestion/test-harness.ts create mode 100644 src/core/ingestion/types.ts create mode 100644 src/core/minions/handlers/ingest-capture.ts create mode 100644 test/commands/capture.test.ts create mode 100644 test/e2e/ingestion-roundtrip.test.ts create mode 100644 test/e2e/serve-http-ingest-webhook.test.ts create mode 100644 test/ingestion/daemon.test.ts create mode 100644 test/ingestion/dedup.test.ts create mode 100644 test/ingestion/ingest-capture.test.ts create mode 100644 test/ingestion/put-page-write-through.test.ts create mode 100644 test/ingestion/skillpack-load.test.ts create mode 100644 test/ingestion/sources/file-watcher.test.ts create mode 100644 test/ingestion/sources/inbox-folder.test.ts create mode 100644 test/ingestion/test-harness.test.ts create mode 100644 test/ingestion/types.test.ts create mode 100644 test/markdown-serializer.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b8d94f301..ebcb2230e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,144 @@ All notable changes to GBrain will be documented in this file. +## [0.38.0.0] - 2026-05-21 + +**One command to capture anything into your brain. Local OR hosted, doesn't matter.** + +You type `gbrain capture "the thought I want to remember"` and the right +thing happens. The page lands in your brain — both as a queryable row AND +as a markdown file on disk, in one move. You see the slug come back as a +five-line receipt. You can search for it 200ms later. If you piped from +stdin or pointed at a file, same deal. If you're on a hosted brain +(thin-client install), it routes through MCP to the server transparently +— same command, same UX. + +Pre-v0.38 the answer to "how do I get a thought into the brain?" was +three different answers depending on context: `put_page` over MCP (DB +only, file drifts), commit a file then run sync (works but commit-then- +sync is friction), or autopilot wait (poll every 5min, latency-bound). +v0.38 collapses these into one mental model: **the brain is a markdown +file tree wherever the DB lives, and `gbrain capture` is the front door.** + +### How to use it + +``` +gbrain capture "remember to follow up on X" +gbrain capture --file ./notes/today.md +echo "from a pipe" | gbrain capture --stdin +SLUG=$(gbrain capture "..." --quiet) # script-friendly +gbrain capture "..." --json # for agents +``` + +### The plumbing + +| Surface | What it does | +|---|---| +| `gbrain capture` | Single human-facing entrypoint. Local installs route through `put_page` directly; thin-client installs route through `callRemoteTool('put_page', ...)`. Same UX both ways. | +| `put_page` write-through | After a page lands in the DB, the markdown file is written to disk too. Closes the drift class the v0.35.6.0 phantom-redirect pass was cleaning up. Subagent sandbox + dry-run writes stay DB-only. | +| `POST /ingest` on `serve --http` | OAuth-gated webhook so Zapier / IFTTT / Apple Shortcuts can drop into your brain. Tagged `untrusted_payload: true`; rate-limited; 1MB cap; content-type allowlist. | +| `IngestionSource` contract | Versioned public API at `gbrain/ingestion` and `gbrain/ingestion/test-harness` package subpaths. Third-party skillpack publishers can build sources (Granola, Linear, voice, OCR) against the contract. | +| Ingestion daemon | Supervises file-watcher + inbox-folder + future built-in sources. Per-source crash-counter + exponential backoff (the v0.34.3.0 ChildWorkerSupervisor pattern adapted for in-process modules). 24h content-hash dedup window. Per-source rate limit. | +| file-watcher source | chokidar-based watcher over your brain repo. 1s debounce coalesces editor save-storms. Honors `pruneDir` (single source of truth with sync). Linux ENOSPC surfaces a paste-ready sysctl hint. | +| inbox-folder source | Drop a file into `~/.gbrain/inbox/` from iOS Shortcuts / AirDrop / Drafts / Finder. Daemon picks it up, ingests, auto-archives to `.archived/YYYY-MM-DD/`. | +| `IngestionTestHarness` | Publisher-facing test utility with fake clock + in-memory event bus + `expectEvent` matchers. Exported as a versioned public API so skillpack authors can write unit tests without spinning up a daemon. | + +### Provenance + +Every page captured via the v0.38 paths now carries provenance frontmatter: + +```yaml +ingested_via: put_page # local CLI +ingested_via: 'mcp:put_page' # MCP remote +ingested_via: capture-cli # via `gbrain capture` +ingested_via: webhook # via POST /ingest +ingested_at: 2026-05-21T04:15:00Z +``` + +Migration v80 adds the four nullable columns (`ingested_via`, +`ingested_at`, `source_uri`, `source_kind`). Historical pages stay +NULL — pre-v0.38 pages never had provenance and the columns are +additive. + +### IngestionSource contract for skillpack authors + +```ts +import { IngestionSource, IngestionEvent } from 'gbrain/ingestion'; +import { IngestionTestHarness, expectEvent } from 'gbrain/ingestion/test-harness'; + +export default function createSource(config: Record): IngestionSource { + return { + id: 'voice-granola', + kind: 'voice-whisper', + async start(ctx) { + // Poll Granola, emit events: + ctx.emit({ + source_id: 'voice-granola', + source_kind: 'voice-whisper', + source_uri: 'granola://transcript/123', + received_at: new Date().toISOString(), + content_type: 'text/markdown', + content: transcript, + content_hash: computeContentHash(transcript), + }); + }, + async stop() { /* drain */ }, + }; +} +``` + +Declared in `gbrain.plugin.json` with `api_version: 'gbrain-ingestion-source-v1'`. Mismatched API versions fail loudly with a paste-ready upgrade hint, never silently break. Both subpaths are pinned by `test/public-exports.test.ts` so breaking the contract is a major-version change. + +### What this commit ships + +- v0.38 ingestion substrate: ~1500 LOC + ~150 LOC tests across 6 modules (types, dedup, daemon, skillpack-load, test-harness, two built-in sources) +- POST /ingest webhook source on `serve --http` +- `ingest_capture` Minion handler +- `put_page` write-through (server-side, source-aware filing layout) +- `serializePageToMarkdown` + `resolvePageFilePath` DRY extract (synthesize.ts dream-cycle render is now a 4-line wrapper) +- Migration v80 — provenance columns on `pages` +- `gbrain capture` CLI verb +- 271+ new test cases across ingestion + commands + public-exports + +### Itemized changes + +- `src/core/ingestion/types.ts` (NEW) — IngestionSource + IngestionEvent + IngestionSourceContext + validateIngestionEvent + computeContentHash + IngestionEventError + INGESTION_SOURCE_API_VERSION + INGESTION_CONTENT_TYPES +- `src/core/ingestion/dedup.ts` (NEW) — 24h content-hash LRU with 5000-entry cap +- `src/core/ingestion/skillpack-load.ts` (NEW) — `gbrain.plugin.json` discovery with api_version compat + collision policy + in-process trust model for v1 +- `src/core/ingestion/test-harness.ts` (NEW) — publisher-facing test utility exported as `gbrain/ingestion/test-harness` +- `src/core/ingestion/daemon.ts` (NEW) — IngestionDaemon: SourceSupervisor pattern + validate → dedup → rate-limit → dispatch pipeline + health surface +- `src/core/ingestion/sources/file-watcher.ts` (NEW) — chokidar source with 1s debounce, atomic-write handling, ENOSPC sysctl-hint surfacing +- `src/core/ingestion/sources/inbox-folder.ts` (NEW) — Shortcuts/AirDrop target with auto-archive to `.archived/YYYY-MM-DD/` +- `src/core/ingestion/index.ts` (NEW) — barrel for `gbrain/ingestion` +- `src/core/minions/handlers/ingest-capture.ts` (NEW) — `ingest_capture` Minion handler with slug-resolution fallback chain +- `src/commands/serve-http.ts` — POST /ingest webhook route with OAuth write scope + rate limit + content-type allowlist + 1MB payload cap + `untrusted_payload: true` tagging +- `src/commands/jobs.ts` — registers `ingest_capture` in `registerBuiltinHandlers` +- `src/core/operations.ts` — put_page write-through after `importFromContent` with trust gating (subagent sandbox / dry-run stay DB-only) + provenance frontmatter stamp +- `src/core/markdown.ts` — `serializePageToMarkdown(page, tags, opts)` + `resolvePageFilePath(brainDir, slug, sourceId)` DRY extract +- `src/core/cycle/synthesize.ts` — `renderPageToMarkdown` becomes a 4-line wrapper around `serializePageToMarkdown` +- `src/core/migrate.ts` — migration v81 `pages_provenance_columns` adds 4 nullable columns (renumbered from v80 during master merge with v0.37.2.0 takes hotfix) +- `src/commands/capture.ts` (NEW) — `gbrain capture` CLI verb with local + thin-client routing, --file / --stdin / --slug / --type / --source / --quiet / --json +- `src/cli.ts` — registers `capture` dispatch case + `CLI_ONLY` entry +- `package.json` — adds `chokidar` dep + `gbrain/ingestion` + `gbrain/ingestion/test-harness` exports +- `test/public-exports.test.ts` + `scripts/check-exports-count.sh` — pin new public subpaths (count 18 → 20) + +### Deferred to follow-up releases + +These were in the v0.38 plan but did not land in this release: + +- Daemon rename `gbrain autopilot` → `gbrain ingest` with forever-alias + launchd plist migration +- cron-scheduler skill refactor + OpenClaw credential auto-migrate (the existing skill still works untouched; the migration into a daemon-side source ships in a follow-up) +- Content-type processors (PDF text extract, image OCR, audio transcribe, video keyframe). The webhook source currently rejects binary content_types with HTTP 415 and a paste-ready hint; processors land as skillpack-distributed sources. +- `gbrain doctor` inotify-limit probe (Linux) — chokidar still surfaces ENOSPC at runtime with the sysctl hint; the doctor-time static probe is a polish item. +- Publisher DX cathedral: `gbrain skillpack init --kind=ingestion-source ` scaffold extension, `gbrain ingest test [--watch]`, `gbrain ingest tail `, `gbrain ingest validate `. The IngestionTestHarness export is the foundation publishers can use today; the CLI helpers come next. +- Skillpack reference pack at `examples/skillpack-ingestion-reference/` and the 3-stage tutorial in `docs/ingestion-source-skillpack.md`. + +These do not block the v0.38 release: the substrate is shipped and queryable; sources can be built against the contract today using the IngestionTestHarness; the cathedral commits add polish around the publisher experience. + +## To take advantage of v0.38.0.0 + +`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` +warns about a partial migration: ## [0.37.11.0] - 2026-05-21 **Fresh `gbrain init --pglite` works out of the box now.** @@ -358,6 +496,29 @@ Credited contributors per the CHANGELOG attribution convention; closing comments ```bash gbrain apply-migrations --yes ``` +2. **Try the capture verb:** + ```bash + gbrain capture "first thought into v0.38" + gbrain query "first thought" + ``` + The receipt block should show the slug + file path; the query should + return the page within a second. +3. **For webhook ingestion** (only if you run `gbrain serve --http`): + ```bash + curl -X POST https://your-brain/ingest \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: text/markdown" \ + -d "# webhook test" + ``` + You should see HTTP 202 + a `job_id`. Run `gbrain query "webhook test"` + to confirm the page landed. +4. **If any step fails or the numbers look wrong,** please file an issue: + https://github.com/garrytan/gbrain/issues with: + - output of `gbrain doctor` + - contents of `~/.gbrain/upgrade-errors.jsonl` if it exists + - which step broke + + This feedback loop is how the gbrain maintainers find fragile upgrade paths. Thank you. 2. **Verify the source-routing fix on your federated brains:** ```bash gbrain sources current diff --git a/README.md b/README.md index 931ccfa0f..6999b10bc 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,38 @@ gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard Per-client guides (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork) live under [`docs/mcp/`](docs/mcp/). HTTP server supports DCR-style client registration, scope-gated access (`read`/`write`/`admin`), and built-in rate limiting. +## How to get data in (v0.38+) + +One command, local or hosted, synchronous receipt: + +```bash +gbrain capture "the thought I want to remember" +gbrain capture --file ./notes/today.md +echo "from a pipe" | gbrain capture --stdin +SLUG=$(gbrain capture "..." --quiet) +``` + +The page lands in the DB AND on disk in one move (the v0.38 `put_page` +write-through plumbing). Default slug `inbox/YYYY-MM-DD-` so +captures cluster in a predictable triage location. On thin-client installs +the verb routes through MCP to the server — same command, same UX. + +For webhook ingestion (Zapier / IFTTT / Apple Shortcuts): + +```bash +curl -X POST https://your-brain/ingest \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: text/markdown" \ + -d "# a thought from a Shortcut" +``` + +For mobile capture, the inbox folder source picks up anything dropped into +`~/.gbrain/inbox/` from iOS Shortcuts / AirDrop / Drafts / Finder. + +Third-party skillpacks can ship custom ingestion sources (Granola, Linear, +voice, OCR) against the versioned `IngestionSource` contract at +`gbrain/ingestion`. See [`docs/skillpack-anatomy.md`](docs/skillpack-anatomy.md). + ## What it does (the loop) ``` diff --git a/VERSION b/VERSION index 5d4245b47..60c25453a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.37.11.0 \ No newline at end of file +0.38.0.0 \ No newline at end of file diff --git a/bun.lock b/bun.lock index 21eacd6cb..0e2ad1807 100644 --- a/bun.lock +++ b/bun.lock @@ -17,6 +17,7 @@ "@jsquash/png": "^3.1.1", "@modelcontextprotocol/sdk": "1.29.0", "ai": "^6.0.168", + "chokidar": "^4.0.3", "cookie-parser": "^1.4.7", "cors": "^2.8.5", "eventsource-parser": "^3.0.8", @@ -322,6 +323,8 @@ "call-bound": ["call-bound@1.0.4", "", { "dependencies": { "call-bind-apply-helpers": "^1.0.2", "get-intrinsic": "^1.3.0" } }, "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg=="], + "chokidar": ["chokidar@4.0.3", "", { "dependencies": { "readdirp": "^4.0.1" } }, "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA=="], + "combined-stream": ["combined-stream@1.0.8", "", { "dependencies": { "delayed-stream": "~1.0.0" } }, "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg=="], "content-disposition": ["content-disposition@1.0.1", "", {}, "sha512-oIXISMynqSqm241k6kcQ5UwttDILMK4BiurCfGEREw6+X9jkkpEe5T9FZaApyLGGOnFuyMWZpdolTXMtvEJ08Q=="], @@ -506,6 +509,8 @@ "raw-body": ["raw-body@3.0.2", "", { "dependencies": { "bytes": "~3.1.2", "http-errors": "~2.0.1", "iconv-lite": "~0.7.0", "unpipe": "~1.0.0" } }, "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA=="], + "readdirp": ["readdirp@4.1.2", "", {}, "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg=="], + "require-from-string": ["require-from-string@2.0.2", "", {}, "sha512-Xf0nWe6RseziFMu+Ap9biiUbmplq6S9/p+7w7YXP/JBHhrUDDUhwa+vANyubuqfZWTveU//DYVGsDG7RKL/vEw=="], "router": ["router@2.2.0", "", { "dependencies": { "debug": "^4.4.0", "depd": "^2.0.0", "is-promise": "^4.0.0", "parseurl": "^1.3.3", "path-to-regexp": "^8.0.0" } }, "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ=="], diff --git a/llms-full.txt b/llms-full.txt index 2782721c8..e6054071b 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -2331,6 +2331,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef | Trigger | Skill | |---------|-------| +| "capture this", "save this thought", "remember this", "drop this in the inbox", "save to brain" | `skills/capture/SKILL.md` | | User shares a link, article, tweet, or idea | `skills/idea-ingest/SKILL.md` | | "watch this video", "process this YouTube link", "ingest this PDF", "save this podcast", "process this book", "summarize this book", "PDF book", "ingest it into my brain", "what's in this screenshot", "check out this repo" | `skills/media-ingest/SKILL.md` | | Meeting transcript received | `skills/meeting-ingestion/SKILL.md` | @@ -2499,6 +2500,38 @@ gbrain serve --http # HTTP MCP with OAuth 2.1 + admin dashboard Per-client guides (Claude Desktop, Code, Cursor, ChatGPT, Perplexity, Cowork) live under [`docs/mcp/`](docs/mcp/). HTTP server supports DCR-style client registration, scope-gated access (`read`/`write`/`admin`), and built-in rate limiting. +## How to get data in (v0.38+) + +One command, local or hosted, synchronous receipt: + +```bash +gbrain capture "the thought I want to remember" +gbrain capture --file ./notes/today.md +echo "from a pipe" | gbrain capture --stdin +SLUG=$(gbrain capture "..." --quiet) +``` + +The page lands in the DB AND on disk in one move (the v0.38 `put_page` +write-through plumbing). Default slug `inbox/YYYY-MM-DD-` so +captures cluster in a predictable triage location. On thin-client installs +the verb routes through MCP to the server — same command, same UX. + +For webhook ingestion (Zapier / IFTTT / Apple Shortcuts): + +```bash +curl -X POST https://your-brain/ingest \ + -H "Authorization: Bearer $TOKEN" \ + -H "Content-Type: text/markdown" \ + -d "# a thought from a Shortcut" +``` + +For mobile capture, the inbox folder source picks up anything dropped into +`~/.gbrain/inbox/` from iOS Shortcuts / AirDrop / Drafts / Finder. + +Third-party skillpacks can ship custom ingestion sources (Granola, Linear, +voice, OCR) against the versioned `IngestionSource` contract at +`gbrain/ingestion`. See [`docs/skillpack-anatomy.md`](docs/skillpack-anatomy.md). + ## What it does (the loop) ``` diff --git a/package.json b/package.json index 6313bcc3c..8533b9c82 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.37.11.0", + "version": "0.38.0.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", @@ -25,7 +25,9 @@ "./search/hybrid": "./src/core/search/hybrid.ts", "./search/expansion": "./src/core/search/expansion.ts", "./ai/gateway": "./src/core/ai/gateway.ts", - "./extract": "./src/commands/extract.ts" + "./extract": "./src/commands/extract.ts", + "./ingestion": "./src/core/ingestion/index.ts", + "./ingestion/test-harness": "./src/core/ingestion/test-harness.ts" }, "scripts": { "dev": "bun run src/cli.ts", @@ -93,6 +95,7 @@ "@jsquash/png": "^3.1.1", "@modelcontextprotocol/sdk": "1.29.0", "ai": "^6.0.168", + "chokidar": "^4.0.3", "cookie-parser": "^1.4.7", "cors": "^2.8.5", "eventsource-parser": "^3.0.8", diff --git a/scripts/check-exports-count.sh b/scripts/check-exports-count.sh index af6816f30..a02d1a015 100755 --- a/scripts/check-exports-count.sh +++ b/scripts/check-exports-count.sh @@ -19,7 +19,7 @@ set -euo pipefail -EXPECTED_COUNT=18 +EXPECTED_COUNT=20 # Count top-level keys in the exports object. `node -e` parses JSON # reliably without needing jq (which isn't in every CI environment). diff --git a/scripts/e2e-test-map.ts b/scripts/e2e-test-map.ts index 329de18e0..1900a86fd 100644 --- a/scripts/e2e-test-map.ts +++ b/scripts/e2e-test-map.ts @@ -85,4 +85,20 @@ export const E2E_TEST_MAP: Record = { "src/commands/doctor.ts": ["test/e2e/doctor-progress.test.ts"], // Knowledge graph layer feeds graph-quality. "src/core/link-extraction.ts": ["test/e2e/graph-quality.test.ts"], + // v0.38 ingestion substrate. POST /ingest lives inside serve-http.ts + // (per the plan-eng-review E1 decision); the daemon + built-in sources + // + ingest_capture Minion handler all feed the in-process roundtrip + // E2E AND the HTTP contract E2E for the webhook route. + "src/commands/serve-http.ts": [ + "test/e2e/serve-http-ingest-webhook.test.ts", + "test/e2e/serve-http-oauth.test.ts", + ], + "src/core/ingestion/**": [ + "test/e2e/ingestion-roundtrip.test.ts", + "test/e2e/serve-http-ingest-webhook.test.ts", + ], + "src/core/minions/handlers/ingest-capture.ts": [ + "test/e2e/ingestion-roundtrip.test.ts", + "test/e2e/serve-http-ingest-webhook.test.ts", + ], }; diff --git a/skills/RESOLVER.md b/skills/RESOLVER.md index 1c676bfd5..dd9af4f3e 100644 --- a/skills/RESOLVER.md +++ b/skills/RESOLVER.md @@ -29,6 +29,7 @@ This is the dispatcher. Skills are the implementation. **Read the skill file bef | Trigger | Skill | |---------|-------| +| "capture this", "save this thought", "remember this", "drop this in the inbox", "save to brain" | `skills/capture/SKILL.md` | | User shares a link, article, tweet, or idea | `skills/idea-ingest/SKILL.md` | | "watch this video", "process this YouTube link", "ingest this PDF", "save this podcast", "process this book", "summarize this book", "PDF book", "ingest it into my brain", "what's in this screenshot", "check out this repo" | `skills/media-ingest/SKILL.md` | | Meeting transcript received | `skills/meeting-ingestion/SKILL.md` | diff --git a/skills/capture/SKILL.md b/skills/capture/SKILL.md new file mode 100644 index 000000000..868534eb7 --- /dev/null +++ b/skills/capture/SKILL.md @@ -0,0 +1,105 @@ +--- +name: capture +description: Save any thought or content into the brain via one CLI command. The single human-facing entrypoint that replaces "put_page vs commit-then-sync vs autopilot-wait" with one command that just works. +triggers: + - "capture this" + - "save this thought" + - "remember this" + - "ingest this into my brain" + - "drop this in the inbox" + - "save to brain" +writes_pages: + - "inbox/*" +--- + +# capture — the single ingestion entrypoint + +When the user wants to save a thought, an article snippet, a transcript +fragment, or any text into their brain, run `gbrain capture`. Don't reach +for `gbrain put` or commit-then-sync — `capture` is the front door and it +handles both local and thin-client installs the same way. + +## Contract + +- **Input:** the content to save (inline arg, `--file PATH`, or `--stdin`). +- **Output:** a page in the brain DB AND a markdown file on disk under + `/.md`. Receipt printed to stdout. +- **Side effect:** the page becomes immediately queryable via `gbrain query`, + `gbrain search`, or any MCP-bound agent. +- **Idempotency:** same content → same `inbox/YYYY-MM-DD-` slug. The + daemon's 24h content-hash dedup catches re-captures. +- **Trust:** all captures via this skill are local-CLI trust (`remote: false`). + Untrusted webhook ingestion goes through `POST /ingest`, not this verb. + +## When to invoke + +- "Capture this thought" / "save this" / "drop this into my brain" / "remember this" +- The user pastes content and asks to keep it +- After a meeting summary, a research note, or any synthesis that should land as a brain page + +## What it does + +`gbrain capture` resolves to a `put_page` call (local) or a remote MCP call +(thin-client). Either way the page lands in the DB AND on disk in one move +via the v0.38 write-through plumbing. The default slug is +`inbox/YYYY-MM-DD-` so captures cluster in a predictable triage +location. + +## How to use + +```bash +gbrain capture "the thought I want to remember" +gbrain capture --file ./notes/today.md +echo "from a pipe" | gbrain capture --stdin +gbrain capture "..." --slug daily/2026-05-21 +gbrain capture "..." --type idea --source voice-whisper +gbrain capture "..." --quiet # script-friendly: prints just the slug +gbrain capture "..." --json # structured output for agents +``` + +## Defaults + +- **Slug:** `inbox/YYYY-MM-DD-` (stable for same content; the daemon's 24h dedup catches re-captures). +- **Type:** `note` (override with `--type idea` etc.). +- **Frontmatter stamps:** `captured_via: capture-cli`, `captured_at: `. +- **Title:** first non-empty line of the body, capped at 80 chars. + +## Output Format + +Default prints a 5-line receipt: + +``` +captured: + slug: inbox/2026-05-21-abcdef12 + status: created_or_updated + content_hash: f3a7b9c0d1e2f3a4… + file: /Users/you/brain/inbox/2026-05-21-abcdef12.md + captured_at: 2026-05-21T04:15:00.000Z +``` + +`--quiet` prints only the slug (use for `SLUG=$(gbrain capture "..." --quiet)`). +`--json` prints structured output for downstream tools. + +## Anti-Patterns + +- **Don't reach for `gbrain put`.** That's the old per-page primitive that + doesn't know about default slug generation, content-type heuristics, or + the receipt block. `capture` is the human-facing wrapper. +- **Don't try to bulk-import dozens of files by looping over `gbrain capture`.** + That's what `gbrain sync` (or `gbrain import`) is for. Capture is for + single thoughts, single notes, single transcripts. +- **Don't pre-format the content yourself with frontmatter if you don't need to.** + Capture wraps plain prose in sensible frontmatter (type + title + + captured_via + captured_at). The body becomes `# Title\n\n`. + Pass `--file PATH` if you already have a fully-formatted markdown file. +- **Don't pass secrets as inline content.** Inline args land in shell + history. Use `--file` or `--stdin` instead. + +## When NOT to use this skill + +- Bulk ingestion of many files → `skills/media-ingest/SKILL.md` or `gbrain sync` instead +- Article/link with author + publication metadata → `skills/idea-ingest/SKILL.md` (it knows to build the people page) +- Meeting transcripts → `skills/meeting-ingestion/SKILL.md` (attendee enrichment) + +This skill is for the simple "I have a thought, save it" case. Specialized +ingestion paths handle their own slugging + cross-referencing. diff --git a/skills/manifest.json b/skills/manifest.json index 7b39ae415..35e07b2f9 100644 --- a/skills/manifest.json +++ b/skills/manifest.json @@ -59,6 +59,11 @@ "path": "brain-ops/SKILL.md", "description": "Brain-first lookup, read-enrich-write loop, source attribution, ambient enrichment. The core read/write cycle." }, + { + "name": "capture", + "path": "capture/SKILL.md", + "description": "Single human-facing entrypoint for ingestion. Routes through put_page (local) or MCP (thin-client); replaces put-page-vs-commit-vs-autopilot confusion with one verb." + }, { "name": "idea-ingest", "path": "idea-ingest/SKILL.md", diff --git a/src/cli.ts b/src/cli.ts index 6b47fe38d..d361ad147 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -27,7 +27,7 @@ for (const op of operations) { } // CLI-only commands that bypass the operation layer -const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd']); +const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'capture']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -1219,6 +1219,12 @@ async function handleCliOnly(command: string, args: string[]) { await runAnomalies(engine, args); break; } + // v0.38 — Capture: single human-facing entrypoint for ingestion. + case 'capture': { + const { runCapture } = await import('./commands/capture.ts'); + await runCapture(engine, args); + break; + } case 'edges-backfill': { // v0.34 W6 — operator escape hatch for the symbol-resolution backfill. // Resumable via the edges_backfilled_at watermark; per-batch transactions diff --git a/src/commands/capture.ts b/src/commands/capture.ts new file mode 100644 index 000000000..c89a5e155 --- /dev/null +++ b/src/commands/capture.ts @@ -0,0 +1,322 @@ +/** + * gbrain capture — the single human-facing entrypoint for getting content + * into the brain. Replaces the confusion of "do I call put_page, commit + * a file, or wait for autopilot?" with one command that just works. + * + * gbrain capture "thought to remember" + * gbrain capture --file ./notes/2026-05-20.md + * echo "from stdin" | gbrain capture --stdin + * gbrain capture "..." --slug inbox/specific + * gbrain capture "..." --quiet # slug-only output for pipelines + * + * Behavior: + * - Local install: writes to ~/.gbrain/inbox/.md OR routes through + * put_page (which now writes through to disk via the v0.38 plumbing). + * Synchronous result with the slug, status, content_hash, and queue + * job id (when applicable). + * - Thin-client install: routes through callRemoteTool('put_page', ...) + * so the server's daemon handles ingestion. Same UX, transparent to + * the caller. + * + * Default slug: `inbox/YYYY-MM-DD-`. Stable for same + * content (the daemon's 24h content-hash dedup will catch duplicates if + * you re-capture the same thought twice). + * + * Output: + * - Default: 5-line receipt block (slug, ingested_at, source_kind, + * content_hash, queue job id where applicable). + * - --quiet: just the slug on stdout for shell pipelines like + * `JOB=$(gbrain capture "..." --quiet)`. + * - --json: structured response for agents. + */ + +import { readFileSync } from 'node:fs'; +import type { BrainEngine } from '../core/engine.ts'; +import { loadConfig, isThinClient } from '../core/config.ts'; +import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts'; +import { computeContentHash } from '../core/ingestion/types.ts'; +import { operations } from '../core/operations.ts'; +import type { OperationContext } from '../core/operations.ts'; + +interface RunOpts { + content?: string; + filePath?: string; + stdin?: boolean; + slug?: string; + type?: string; + source?: string; + quiet?: boolean; + json?: boolean; +} + +function parseArgs(args: string[]): RunOpts | { help: true; positional: string | undefined } { + const opts: RunOpts = {}; + const positional: string[] = []; + for (let i = 0; i < args.length; i++) { + const a = args[i]; + if (a === '--help' || a === '-h') return { help: true, positional: undefined }; + if (a === '--quiet' || a === '-q') { opts.quiet = true; continue; } + if (a === '--json') { opts.json = true; continue; } + if (a === '--stdin') { opts.stdin = true; continue; } + if (a === '--file') { + const v = args[++i]; + if (v) opts.filePath = v; + continue; + } + if (a === '--slug') { + const v = args[++i]; + if (v) opts.slug = v; + continue; + } + if (a === '--type') { + const v = args[++i]; + if (v) opts.type = v; + continue; + } + if (a === '--source') { + const v = args[++i]; + if (v) opts.source = v; + continue; + } + if (a.startsWith('--')) continue; // unknown flag, ignore + positional.push(a); + } + if (positional.length > 0) { + opts.content = positional.join(' '); + } + return opts; +} + +const HELP = `Usage: gbrain capture [content] [options] + +The single entrypoint for getting content into the brain. One command, +local OR thin-client, synchronous receipt with the resulting page slug. + +Modes (mutually exclusive — first match wins): + gbrain capture "thought" inline content + gbrain capture --file PATH read content from a file + gbrain capture --stdin read content from stdin (piped) + +Options: + --slug SLUG Override the default inbox/YYYY-MM-DD- slug + --type TYPE Override the page type (default: note) + --source ID Multi-source brains: write under a non-default source + --quiet, -q Print just the slug on stdout (for shell pipelines) + --json JSON output for agents + --help, -h Show this help + +Examples: + gbrain capture "remember to follow up on the X deal" + echo "from a pipe" | gbrain capture --stdin + gbrain capture --file ./notes/today.md --slug daily/2026-05-20 + JOB=$(gbrain capture "..." --quiet) +`; + +function defaultSlug(content: string, now: Date = new Date()): string { + const y = now.getUTCFullYear(); + const m = String(now.getUTCMonth() + 1).padStart(2, '0'); + const d = String(now.getUTCDate()).padStart(2, '0'); + const hashPrefix = computeContentHash(content).slice(0, 8); + return `inbox/${y}-${m}-${d}-${hashPrefix}`; +} + +async function readStdin(): Promise { + const chunks: Buffer[] = []; + for await (const chunk of process.stdin) { + chunks.push(typeof chunk === 'string' ? Buffer.from(chunk, 'utf8') : (chunk as Buffer)); + } + return Buffer.concat(chunks).toString('utf8'); +} + +/** + * Build the put_page content (frontmatter + body). The user's --type and + * the auto-stamped capture provenance go in the frontmatter so future + * tools (e.g. the inbox triage UI) can find captures. + */ +function buildContent(rawBody: string, opts: RunOpts): string { + const frontmatterLines: string[] = ['---']; + frontmatterLines.push(`type: ${opts.type ?? 'note'}`); + // Title defaults to the first non-empty line of the body, capped at 80 chars. + const firstLine = rawBody.split('\n').find((l) => l.trim().length > 0) ?? ''; + const title = firstLine.replace(/^#+\s*/, '').slice(0, 80) || 'Capture'; + frontmatterLines.push(`title: ${JSON.stringify(title)}`); + frontmatterLines.push(`captured_via: ${opts.source ?? 'capture-cli'}`); + frontmatterLines.push(`captured_at: ${new Date().toISOString()}`); + frontmatterLines.push('---'); + // Heuristic: if the raw body doesn't already look like markdown (no headings, + // no leading frontmatter), wrap in a heading for readability. Inline thoughts + // are usually unstructured. + const lookskMarkdown = /^#{1,6}\s/.test(rawBody.trimStart()) || rawBody.startsWith('---'); + const body = lookskMarkdown ? rawBody : `# ${title}\n\n${rawBody}`; + return frontmatterLines.join('\n') + '\n\n' + body; +} + +interface CaptureResult { + slug: string; + status?: string; + chunks?: number; + content_hash: string; + written?: boolean; + path?: string; + source_kind: string; + captured_at: string; +} + +function printReceipt(result: CaptureResult, quiet: boolean, json: boolean): void { + if (json) { + console.log(JSON.stringify(result, null, 2)); + return; + } + if (quiet) { + console.log(result.slug); + return; + } + console.log('captured:'); + console.log(` slug: ${result.slug}`); + console.log(` status: ${result.status ?? 'unknown'}`); + console.log(` content_hash: ${result.content_hash.slice(0, 16)}…`); + if (result.path) { + console.log(` file: ${result.path}`); + } + console.log(` captured_at: ${result.captured_at}`); +} + +export async function runCapture(engine: BrainEngine | null, args: string[]): Promise { + const parsed = parseArgs(args); + if ('help' in parsed) { + console.log(HELP); + return; + } + + // Resolve the source content. + let rawBody: string; + if (parsed.stdin) { + rawBody = await readStdin(); + } else if (parsed.filePath) { + try { + rawBody = readFileSync(parsed.filePath, 'utf8'); + } catch (e) { + console.error( + `gbrain capture: failed to read ${parsed.filePath}: ${e instanceof Error ? e.message : String(e)}`, + ); + process.exit(1); + } + } else if (parsed.content) { + rawBody = parsed.content; + } else { + console.error('gbrain capture: provide content positionally, --file PATH, or --stdin'); + console.error('Run `gbrain capture --help` for examples.'); + process.exit(1); + } + + rawBody = rawBody.trim(); + if (rawBody.length === 0) { + console.error('gbrain capture: refusing to capture empty content'); + process.exit(1); + } + + const slug = parsed.slug ?? defaultSlug(rawBody); + const fullContent = buildContent(rawBody, parsed); + const capturedAt = new Date().toISOString(); + const contentHash = computeContentHash(fullContent); + + // Thin-client install: route through put_page over MCP. The server's + // write-through plumbing handles disk persistence. + const cfg = loadConfig(); + if (isThinClient(cfg)) { + let raw: unknown; + try { + raw = await callRemoteTool( + cfg!, + 'put_page', + { slug, content: fullContent }, + { timeoutMs: 30_000 }, + ); + } catch (e) { + console.error( + `gbrain capture: remote put_page failed: ${e instanceof Error ? e.message : String(e)}`, + ); + console.error('Run `gbrain remote doctor` to diagnose the connection.'); + process.exit(1); + } + const remoteResult = unpackToolResult<{ + slug: string; + status?: string; + chunks?: number; + write_through?: { written: boolean; path?: string }; + }>(raw); + const result: CaptureResult = { + slug: remoteResult.slug, + status: remoteResult.status, + chunks: remoteResult.chunks, + content_hash: contentHash, + written: remoteResult.write_through?.written ?? false, + path: remoteResult.write_through?.path, + source_kind: parsed.source ?? 'capture-cli', + captured_at: capturedAt, + }; + printReceipt(result, parsed.quiet ?? false, parsed.json ?? false); + return; + } + + // Local install: route through put_page operation directly so we + // exercise the same write-through path the MCP server uses. + if (!engine) { + console.error('gbrain capture: engine not connected'); + process.exit(1); + } + const putPageOp = operations.find((o) => o.name === 'put_page'); + if (!putPageOp) { + console.error('gbrain capture: put_page operation missing (gbrain build issue)'); + process.exit(1); + } + const ctx: OperationContext = { + engine, + config: cfg ?? { engine: 'pglite' as const }, + logger: { + info: (msg: string) => { process.stderr.write(`[capture] ${msg}\n`); }, + warn: (msg: string) => { process.stderr.write(`[capture] WARN: ${msg}\n`); }, + error: (msg: string) => { process.stderr.write(`[capture] ERROR: ${msg}\n`); }, + }, + dryRun: false, + remote: false, + // OperationContext.sourceId is REQUIRED as of v0.34.1 D4 — match the + // dispatcher's behavior of defaulting to 'default' when the caller + // didn't pass --source. + sourceId: parsed.source ?? 'default', + }; + try { + const result = (await putPageOp.handler(ctx, { slug, content: fullContent })) as { + slug: string; + status?: string; + chunks?: number; + write_through?: { written: boolean; path?: string; skipped?: string }; + }; + printReceipt( + { + slug: result.slug, + status: result.status, + chunks: result.chunks, + content_hash: contentHash, + written: result.write_through?.written ?? false, + path: result.write_through?.path, + source_kind: parsed.source ?? 'capture-cli', + captured_at: capturedAt, + }, + parsed.quiet ?? false, + parsed.json ?? false, + ); + } catch (e) { + console.error( + `gbrain capture: put_page failed: ${e instanceof Error ? e.message : String(e)}`, + ); + process.exit(1); + } +} + +/** Test seam. */ +export const __testing = { + defaultSlug, + buildContent, + parseArgs, +}; diff --git a/src/commands/jobs.ts b/src/commands/jobs.ts index 6793836e3..c17f2a5ec 100644 --- a/src/commands/jobs.ts +++ b/src/commands/jobs.ts @@ -1201,6 +1201,17 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai worker.register('subagent_aggregator', subagentAggregatorHandler); process.stderr.write('[minion worker] subagent handlers enabled\n'); + // ============================================================ + // v0.38 ingestion substrate — ingest_capture handler. Receives + // IngestionEvent payloads from the daemon's dispatcher (file-watcher, + // inbox-folder, cron-scheduler sources) and from serve --http's + // POST /ingest route (webhook source). Routes through importFromContent + // to land as a brain page under inbox/YYYY-MM-DD- (or the + // caller-provided slug). + // ============================================================ + const { makeIngestCaptureHandler } = await import('../core/minions/handlers/ingest-capture.ts'); + worker.register('ingest_capture', makeIngestCaptureHandler(engine)); + // ============================================================ // v0.36+ brain-health-100 wave: 11 new handlers for autonomous // remediation via `gbrain doctor --remediate` and autopilot. diff --git a/src/commands/serve-http.ts b/src/commands/serve-http.ts index b26a5c532..68fb06582 100644 --- a/src/commands/serve-http.ts +++ b/src/commands/serve-http.ts @@ -35,6 +35,13 @@ import { buildError, serializeError } from '../core/errors.ts'; import { VERSION } from '../version.ts'; import * as db from '../core/db.ts'; import { sqlQueryForEngine, executeRawJsonb } from '../core/sql-query.ts'; +import { MinionQueue } from '../core/minions/queue.ts'; +import { + computeContentHash, + validateIngestionEvent, + type IngestionContentType, + type IngestionEvent, +} from '../core/ingestion/types.ts'; /** * /health endpoint timeout. 3s rather than 5s: Fly.io's default @@ -1424,6 +1431,210 @@ export async function runServeHttp(engine: BrainEngine, options: ServeHttpOption } }); + // --------------------------------------------------------------------------- + // v0.38 ingestion substrate — POST /ingest (webhook source) + // + // The webhook ingestion source lives INSIDE serve --http (NOT in the + // ingestion daemon) per the /plan-eng-review E1 decision. This avoids + // cross-process IPC: the daemon supervises only daemon-side sources + // (file-watcher, inbox-folder, cron-scheduler) while serve --http hosts + // the network surface and submits Minion jobs directly. + // + // Auth: existing OAuth `write` scope. Rate limit: 100 events / 10s per + // IP (reuses the IP-keyed pattern from ccRateLimiter; a future tweak + // could key on authInfo.clientId for fairer per-agent fairness). + // Payload cap: 1 MB default. Content-type allowlist: markdown, plain, + // HTML, JSON. Binary content is REJECTED with HTTP 415 in v1 — the + // binary-upload flow ships as a separate route in a later wave when + // content-type processors land. + // + // Events always carry untrusted_payload: true because the input came + // over the network from an OAuth-authenticated but otherwise untrusted + // source (Zapier / IFTTT / Apple Shortcuts). The downstream + // ingest_capture handler logs the flag; a future v2 wave wires it + // through the put_page op to skip auto-link. + // --------------------------------------------------------------------------- + const ingestRateLimiter = rateLimit({ + windowMs: 10_000, // 10 seconds + limit: 100, // 100 events per IP per window + standardHeaders: 'draft-7', + legacyHeaders: false, + message: { error: 'rate_limit_exceeded', message: 'too many /ingest events; backoff and retry' }, + }); + + // Maximum payload bytes for POST /ingest. Configurable via env. Default 1 MB. + const ingestMaxBytes = (() => { + const fromEnv = process.env.GBRAIN_INGEST_MAX_BYTES; + if (!fromEnv) return 1_048_576; + const n = parseInt(fromEnv, 10); + return Number.isFinite(n) && n > 0 ? n : 1_048_576; + })(); + + // Content-type allowlist: text-shaped types only in v1. The handler + // routes binary content_types with HTTP 415; a future wave + skillpack + // processors will accept image/audio/video/pdf via a separate flow. + const INGEST_ALLOWED_CONTENT_TYPES: ReadonlySet = new Set([ + 'text/markdown', + 'text/plain', + 'text/html', + 'application/json', + ]); + + // Single MinionQueue instance shared across POST /ingest invocations + // (the queue is stateless beyond the engine handle; reusing avoids + // per-request construction). + const ingestQueue = new MinionQueue(engine); + + app.post( + '/ingest', + ingestRateLimiter, + requireBearerAuth({ verifier: oauthProvider, requiredScopes: ['write'] }), + express.raw({ type: '*/*', limit: ingestMaxBytes }), + async (req: Request, res: Response) => { + const startTime = Date.now(); + const authInfo = (req as Request & { auth?: AuthInfo }).auth as AuthInfo; + const agentName = authInfo.clientName ?? authInfo.clientId; + + // Express raw() returns a Buffer. Decode as UTF-8; reject non-UTF-8 + // bytes loudly so callers know their payload was garbled. + let body: Buffer; + if (Buffer.isBuffer(req.body)) { + body = req.body; + } else if (typeof req.body === 'string') { + body = Buffer.from(req.body, 'utf8'); + } else { + // express.json or urlencoded fired earlier in the chain and parsed + // for us. Re-serialize so we can hash and forward. + body = Buffer.from(JSON.stringify(req.body), 'utf8'); + } + + if (body.length === 0) { + res.status(400).json({ error: 'empty_body', message: 'POST /ingest requires a non-empty body' }); + return; + } + + // Detect content_type. Caller can override via the X-Gbrain-Content-Type + // header for the JSON case (since the request's Content-Type would say + // application/json but the user might intend the body to be markdown). + const declared = (req.header('x-gbrain-content-type') || req.header('content-type') || '').toLowerCase(); + let contentType: IngestionContentType; + if (declared.startsWith('text/markdown')) { + contentType = 'text/markdown'; + } else if (declared.startsWith('text/html')) { + contentType = 'text/html'; + } else if (declared.startsWith('text/plain')) { + contentType = 'text/plain'; + } else if (declared.startsWith('application/json')) { + contentType = 'application/json'; + } else if (declared.startsWith('text/')) { + // Unknown text/* sub-types pass through as text/plain. + contentType = 'text/plain'; + } else { + // Binary or unknown — rejected in v1. + res.status(415).json({ + error: 'unsupported_content_type', + message: `content_type '${declared}' not supported. Use one of: ${[...INGEST_ALLOWED_CONTENT_TYPES].join(', ')}. ` + + 'Binary content (image/audio/video/pdf) is not yet supported via POST /ingest — install a content-type processor skillpack.', + }); + return; + } + + if (!INGEST_ALLOWED_CONTENT_TYPES.has(contentType)) { + res.status(415).json({ + error: 'unsupported_content_type', + message: `content_type '${contentType}' is in the taxonomy but not currently accepted by POST /ingest`, + }); + return; + } + + const content = body.toString('utf8'); + const contentHash = computeContentHash(content); + const sourceUri = (req.header('x-gbrain-source-uri') || `mcp-webhook:${authInfo.clientId}:${Date.now()}`).slice(0, 1024); + const sourceId = (req.header('x-gbrain-source-id') || `webhook-${authInfo.clientId}`).slice(0, 256); + const callerSlug = req.header('x-gbrain-slug'); + + const event: IngestionEvent = { + source_id: sourceId, + source_kind: 'webhook', + source_uri: sourceUri, + received_at: new Date().toISOString(), + content_type: contentType, + content, + content_hash: contentHash, + untrusted_payload: true, // ALWAYS true for network input + metadata: { + ip: req.ip, + user_agent: req.header('user-agent') ?? '', + client_id: authInfo.clientId, + ...(callerSlug ? { slug: callerSlug } : {}), + }, + }; + + const validationErr = validateIngestionEvent(event); + if (validationErr) { + res.status(400).json({ + error: 'invalid_event', + message: validationErr.message, + field: validationErr.field, + }); + return; + } + + try { + const job = await ingestQueue.add( + 'ingest_capture', + { + event, + ...(callerSlug ? { slug: callerSlug } : {}), + }, + { + // Idempotency: same content from the same client within the + // queue's lifetime is a single job. Different content gets + // different jobs. Daemon-side dedup catches the 24h window; + // the queue-level idempotency catches simultaneous retries. + idempotency_key: `ingest:webhook:${authInfo.clientId}:${contentHash}`, + // Cap waiting jobs from a single client so a runaway integration + // can't fill the queue. + maxWaiting: 50, + }, + ); + + const latency = Date.now() - startTime; + try { + await executeRawJsonb( + engine, + `INSERT INTO mcp_request_log (token_name, agent_name, operation, latency_ms, status, params) + VALUES ($1, $2, $3, $4, $5, $6::jsonb)`, + [authInfo.clientId, agentName, 'webhook_ingest', latency, 'success'], + [{ content_type: contentType, content_hash: contentHash, bytes: body.length, job_id: job.id }], + ); + } catch { /* best effort */ } + broadcastEvent({ + agent: agentName, + operation: 'webhook_ingest', + scopes: authInfo.scopes.join(','), + latency_ms: latency, + status: 'success', + timestamp: new Date().toISOString(), + }); + + res.status(202).json({ + job_id: job.id, + content_hash: contentHash, + source_id: sourceId, + message: 'Accepted. Event queued for ingestion.', + }); + } catch (err) { + const msg = err instanceof Error ? err.message : String(err); + console.error('POST /ingest queue submission error:', msg); + res.status(500).json({ + error: 'queue_submission_failed', + message: msg, + }); + } + }, + ); + // --------------------------------------------------------------------------- // Start server // --------------------------------------------------------------------------- diff --git a/src/core/cycle/synthesize.ts b/src/core/cycle/synthesize.ts index 10ab88cb7..d680bc62d 100644 --- a/src/core/cycle/synthesize.ts +++ b/src/core/cycle/synthesize.ts @@ -35,7 +35,7 @@ import { MinionQueue } from '../minions/queue.ts'; import { waitForCompletion, TimeoutError } from '../minions/wait-for-completion.ts'; import type { MinionJobInput, SubagentHandlerData } from '../minions/types.ts'; import { discoverTranscripts, type DiscoveredTranscript } from './transcript-discovery.ts'; -import { serializeMarkdown } from '../markdown.ts'; +import { serializeMarkdown, serializePageToMarkdown } from '../markdown.ts'; import type { Page, PageType } from '../types.ts'; import { validateSourceId } from '../utils.ts'; @@ -939,21 +939,18 @@ async function reverseWriteRefs( * `serializeMarkdown` does not embed the page slug in the body. */ export function renderPageToMarkdown(page: Page, tags: string[]): string { - const frontmatter: Record = { - ...((page.frontmatter ?? {}) as Record), - dream_generated: true, - dream_cycle_date: today(), - }; - return serializeMarkdown( - frontmatter, - page.compiled_truth ?? '', - page.timeline ?? '', - { - type: (page.type as PageType) ?? 'note', - title: page.title ?? '', - tags, + // v0.38 DRY: the dream-output identity stamp (dream_generated + + // dream_cycle_date) is the ONLY thing that differs from the v0.38 + // put_page write-through renderer. Both call the shared + // serializePageToMarkdown helper in markdown.ts; this wrapper passes + // the dream-specific overrides. Future markdown-shape changes happen + // in one place. + return serializePageToMarkdown(page, tags, { + frontmatterOverrides: { + dream_generated: true, + dream_cycle_date: today(), }, - ); + }); } // ── Summary index page ─────────────────────────────────────────────── diff --git a/src/core/ingestion/daemon.ts b/src/core/ingestion/daemon.ts new file mode 100644 index 000000000..46687204c --- /dev/null +++ b/src/core/ingestion/daemon.ts @@ -0,0 +1,524 @@ +/** + * IngestionDaemon — supervises N pluggable IngestionSource instances, + * dispatches their events into the Minion queue. + * + * Composition: + * - per-source supervision via SourceSupervisor (sibling pattern to the + * v0.34.3.0 ChildWorkerSupervisor, but for in-process modules instead + * of subprocesses) + * - daemon-side validation (validateIngestionEvent at the boundary) + * - 24h content-hash dedup (DedupWindow) + * - per-source rate limit (token bucket, default 100 events / 10s) + * - pluggable event dispatch (constructor-injected). Production wires + * this to MinionQueue.add('ingest_capture', ...). Tests inject a + * recorder so the daemon can be exercised in isolation. + * + * Scope notes: + * - The webhook source does NOT live here. It lives in `serve --http` + * (E1 eng-review decision) and submits ingest_capture Minion jobs + * directly via the existing OAuth-gated route. The daemon supervises + * only daemon-side sources: file-watcher, inbox-folder, cron-scheduler, + * and skillpack-distributed sources. + * + * - Content-type processors (PDF, OCR, audio transcribe, video keyframe) + * are wired through the daemon dispatcher in a later commit. The + * contract here is "daemon emits ingest_capture with the IngestionEvent + * payload"; the processor router decides inline-vs-Minion handler + * based on byte size before the event leaves the daemon. + * + * - Daemon process model: subsumes the v0.37 autopilot daemon. A separate + * commit handles the launchd plist rename and backward-compat alias. + * + * Error model: + * - Source.start() throws → SourceSupervisor catches, increments crash + * counter, applies exponential backoff, restarts up to maxCrashes. + * - Source emits invalid event → daemon logs and drops; source keeps + * running. Bug surfaces in `gbrain doctor ingestion_health`. + * - Dispatcher throws → daemon logs but does NOT crash the source. The + * queue write failed (DB blip, etc.); subsequent emits will retry. + * - Rate-limit exceeded → daemon drops the event silently. Source + * enters `warn` health when sustained backpressure exceeds threshold. + */ + +import type { + IngestionEvent, + IngestionSource, + IngestionSourceContext, + IngestionSourceHealth, +} from './types.ts'; +import { validateIngestionEvent } from './types.ts'; +import { DedupWindow } from './dedup.ts'; +import type { Logger } from '../operations.ts'; +import type { BrainEngine } from '../engine.ts'; + +interface RateLimitConfig { + /** Events allowed per window. */ + capacity: number; + /** Window length in ms. */ + windowMs: number; +} + +interface SupervisionConfig { + /** Max consecutive crashes before the source is marked `fail`. */ + maxCrashes: number; + /** Stable run window: a crash after this duration resets crashCount to 1. */ + stableRunResetMs: number; + /** Initial backoff in ms. Doubles on each crash up to ceiling. */ + initialBackoffMs: number; + /** Backoff ceiling. */ + maxBackoffMs: number; +} + +const DEFAULT_RATE_LIMIT: RateLimitConfig = { + capacity: 100, + windowMs: 10_000, +}; + +const DEFAULT_SUPERVISION: SupervisionConfig = { + maxCrashes: 10, + stableRunResetMs: 5 * 60_000, + initialBackoffMs: 1_000, + maxBackoffMs: 60_000, +}; + +/** Outcome the dispatcher returns. Lets the daemon distinguish queue + * retryable failures from genuine drops (e.g. invalid payload). */ +export type DispatchOutcome = + | { kind: 'queued'; jobId?: number } + | { kind: 'failed'; error: string }; + +export type IngestionDispatcher = ( + event: IngestionEvent, +) => Promise; + +export interface IngestionDaemonOpts { + /** Engine handle exposed to sources via ctx.engine. Sources should only + * read; the daemon routes writes via the dispatcher. */ + engine: BrainEngine; + /** Daemon-wide logger. Source-specific log lines route through a per-source + * wrapper so messages carry the source id. */ + logger: Logger; + /** Where events go after validation + dedup + rate-limit. Production + * wires this to MinionQueue.add('ingest_capture', ...). */ + dispatch: IngestionDispatcher; + /** Per-source rate limit override. */ + rateLimit?: Partial; + /** Per-source supervision override. */ + supervision?: Partial; + /** Test seam: alternative clock. */ + _now?: () => number; +} + +export interface SourceRegistration { + source: IngestionSource; + /** Source-specific config. Merged with the source's declared + * default_config (if from a skillpack) at registration time. */ + config?: Record; +} + +export interface DaemonHealth { + /** Overall daemon status: `ok` if every source is `ok`, `warn` if any + * source is `warn` (but daemon is still functional), `fail` if any + * source has exceeded maxCrashes and is offline. */ + status: 'ok' | 'warn' | 'fail'; + /** Per-source breakdown. */ + sources: Array<{ + id: string; + kind: string; + status: 'ok' | 'warn' | 'fail'; + message?: string; + crashCount: number; + eventCount: number; + rateLimitHits: number; + }>; + /** Dedup stats since daemon start. */ + dedup: { + total: number; + hits: number; + evictions: number; + size: number; + }; +} + +/** Per-source supervision state. Internal to the daemon. */ +interface SourceState { + registration: SourceRegistration; + abortController: AbortController; + crashCount: number; + lastStartTime: number; + /** Rolling event-timestamp buffer for the token-bucket rate limiter. */ + rateLimitTimestamps: number[]; + /** Total events emitted since daemon start (post-dedup, pre-dispatch). */ + eventCount: number; + /** Rate-limit drops since daemon start. */ + rateLimitHits: number; + /** Marked once start() resolves successfully — used by health to + * distinguish "still starting" from "running". */ + started: boolean; + /** Set when the source has exceeded maxCrashes; daemon stops trying. */ + exhausted: boolean; + /** Last error message from start() / supervisor — for the health surface. */ + lastError: string | null; +} + +export class IngestionDaemon { + private readonly opts: IngestionDaemonOpts; + private readonly dedup: DedupWindow; + private readonly sources: Map = new Map(); + private readonly supervision: SupervisionConfig; + private readonly rateLimit: RateLimitConfig; + private _running = false; + private _stopping = false; + + constructor(opts: IngestionDaemonOpts) { + this.opts = opts; + this.dedup = new DedupWindow({ _now: opts._now }); + this.supervision = { ...DEFAULT_SUPERVISION, ...(opts.supervision ?? {}) }; + this.rateLimit = { ...DEFAULT_RATE_LIMIT, ...(opts.rateLimit ?? {}) }; + } + + /** Whether the daemon is currently in its start/run lifecycle. */ + get running(): boolean { + return this._running; + } + + /** + * Register a source. Throws on duplicate id — the daemon's identity + * model requires unique source instance ids. + */ + register(registration: SourceRegistration): void { + if (this._running) { + throw new Error( + `IngestionDaemon.register: cannot register source '${registration.source.id}' ` + + `after daemon has started; call register before start()`, + ); + } + const id = registration.source.id; + if (this.sources.has(id)) { + throw new Error(`IngestionDaemon.register: duplicate source id '${id}'`); + } + this.sources.set(id, { + registration, + abortController: new AbortController(), + crashCount: 0, + lastStartTime: 0, + rateLimitTimestamps: [], + eventCount: 0, + rateLimitHits: 0, + started: false, + exhausted: false, + lastError: null, + }); + } + + /** + * Start every registered source. Each source's lifecycle runs + * independently — a crash in one does not stop others. Resolves when + * all sources have either started successfully OR exhausted maxCrashes. + */ + async start(): Promise { + if (this._running) { + throw new Error('IngestionDaemon.start: already running'); + } + this._running = true; + this._stopping = false; + + // Fire each source's supervisor in parallel. Each returns when the + // source is either running or exhausted. + const startPromises = Array.from(this.sources.keys()).map((id) => + this.superviseSource(id), + ); + + // The supervisor returns once start() resolves (source is "running") + // OR once maxCrashes is hit. We wait for the initial start round to + // complete so the caller knows daemon-startup is done. + await Promise.all(startPromises); + } + + /** + * Stop every source. Fires abort signals, calls source.stop(), waits up + * to `graceMs` for each to drain. Force-marks any source that overruns. + */ + async stop(graceMs = 5000): Promise { + if (!this._running) return; + this._stopping = true; + + const stopPromises: Promise[] = []; + for (const [id, state] of this.sources) { + state.abortController.abort(); + if (!state.started) continue; + const p = (async () => { + try { + await raceWithTimeout( + state.registration.source.stop(), + graceMs, + `source '${id}' did not stop within ${graceMs}ms`, + ); + } catch (err) { + this.opts.logger.warn( + `[ingestion] source '${id}' stop failed: ${err instanceof Error ? err.message : String(err)}`, + ); + } + })(); + stopPromises.push(p); + } + + await Promise.all(stopPromises); + this._running = false; + this._stopping = false; + } + + /** Aggregate health report for `gbrain doctor ingestion_health`. */ + async healthCheck(): Promise { + const perSource: DaemonHealth['sources'] = []; + let aggregateStatus: 'ok' | 'warn' | 'fail' = 'ok'; + + for (const [id, state] of this.sources) { + let status: 'ok' | 'warn' | 'fail'; + let message: string | undefined; + + if (state.exhausted) { + status = 'fail'; + message = state.lastError ?? 'source exceeded maxCrashes'; + } else if (state.crashCount > 0) { + status = 'warn'; + message = `${state.crashCount} crash(es) since last stable run${state.lastError ? `: ${state.lastError}` : ''}`; + } else if (state.started && state.registration.source.healthCheck) { + try { + const result = await raceWithTimeout( + state.registration.source.healthCheck(), + 5_000, + `source '${id}' healthCheck() timed out`, + ); + status = result.status; + message = result.message; + } catch (err) { + status = 'warn'; + message = err instanceof Error ? err.message : String(err); + } + } else { + status = state.started ? 'ok' : 'warn'; + if (!state.started) message = 'not started yet'; + } + + perSource.push({ + id, + kind: state.registration.source.kind, + status, + message, + crashCount: state.crashCount, + eventCount: state.eventCount, + rateLimitHits: state.rateLimitHits, + }); + + // Worst-case wins. + if (status === 'fail' || (status === 'warn' && aggregateStatus === 'ok')) { + aggregateStatus = status; + } + } + + return { + status: aggregateStatus, + sources: perSource, + dedup: this.dedup.stats(), + }; + } + + /** + * Per-source supervision loop. Mirrors the ChildWorkerSupervisor pattern + * (crash counter + stable-run reset + exponential backoff + max crashes) + * adapted for in-process JS modules. + */ + private async superviseSource(id: string): Promise { + const state = this.sources.get(id); + if (!state) return; + const source = state.registration.source; + + while (!this._stopping && state.crashCount < this.supervision.maxCrashes) { + state.lastStartTime = this.now(); + state.abortController = new AbortController(); + + const ctx = this.buildContext(state); + + try { + await source.start(ctx); + state.started = true; + state.lastError = null; + // Success — exit the supervisor loop. The source is now running + // in the background, emitting events via ctx.emit. The daemon + // does NOT call start() repeatedly; only on crash. + return; + } catch (err) { + state.started = false; + const errMsg = err instanceof Error ? err.message : String(err); + state.lastError = errMsg; + const runDuration = this.now() - state.lastStartTime; + + if (runDuration > this.supervision.stableRunResetMs) { + state.crashCount = 1; // stable-run reset + } else { + state.crashCount++; + } + + this.opts.logger.warn( + `[ingestion] source '${id}' start failed (crash ${state.crashCount}/${this.supervision.maxCrashes}): ${errMsg}`, + ); + + if (state.crashCount >= this.supervision.maxCrashes) { + state.exhausted = true; + this.opts.logger.error( + `[ingestion] source '${id}' exhausted maxCrashes=${this.supervision.maxCrashes}; ` + + `giving up. Last error: ${errMsg}`, + ); + return; + } + + // Exponential backoff: 1s, 2s, 4s, ... capped at maxBackoffMs. + const backoff = Math.min( + this.supervision.initialBackoffMs * 2 ** (state.crashCount - 1), + this.supervision.maxBackoffMs, + ); + await this.sleep(backoff); + } + } + } + + /** + * Build the IngestionSourceContext for a given source. Each emit goes + * through the daemon's validate → dedup → rate-limit → dispatch pipeline. + */ + private buildContext(state: SourceState): IngestionSourceContext { + const daemon = this; + const sourceId = state.registration.source.id; + return { + emit(event: IngestionEvent): void { + // Schedule via microtask so the source's emit() returns synchronously + // (publishers expect emit to be fire-and-forget). Errors in the + // pipeline log but don't propagate back to the source. + Promise.resolve().then(() => daemon.handleEmit(state, event)).catch((err) => { + daemon.opts.logger.error( + `[ingestion] source '${sourceId}' dispatch error: ${err instanceof Error ? err.message : String(err)}`, + ); + }); + }, + engine: daemon.opts.engine, + logger: daemon.wrapLogger(sourceId), + abortSignal: state.abortController.signal, + config: state.registration.config ?? {}, + }; + } + + /** + * Process a single emit through the daemon pipeline: + * 1. Validate event shape — drop on failure. + * 2. Check dedup window — drop on hit. + * 3. Check rate limit — drop on bucket exhausted. + * 4. Dispatch to Minion queue via opts.dispatch. + */ + private async handleEmit(state: SourceState, event: IngestionEvent): Promise { + const sourceId = state.registration.source.id; + const sourceKind = state.registration.source.kind; + + // 1. Validate. + const validationErr = validateIngestionEvent(event); + if (validationErr) { + this.opts.logger.warn( + `[ingestion] source '${sourceId}' emitted invalid event: ${validationErr.message}`, + ); + return; + } + + // Defense in depth: validated event but kind mismatch means the source + // is lying about its identity. Trust source.kind over event.source_kind. + const effectiveEvent: IngestionEvent = { ...event, source_kind: sourceKind, source_id: sourceId }; + + // 2. Dedup. + const isNew = this.dedup.mark(sourceKind, effectiveEvent.content_hash); + if (!isNew) { + // Silent dedup hit. dedup.hits counter already incremented. + return; + } + + // 3. Rate limit (token-bucket-ish: count events in trailing window). + const nowMs = this.now(); + const windowStart = nowMs - this.rateLimit.windowMs; + state.rateLimitTimestamps = state.rateLimitTimestamps.filter((t) => t > windowStart); + if (state.rateLimitTimestamps.length >= this.rateLimit.capacity) { + state.rateLimitHits++; + this.opts.logger.warn( + `[ingestion] source '${sourceId}' rate limit hit ` + + `(${this.rateLimit.capacity} events / ${this.rateLimit.windowMs}ms); dropping event`, + ); + return; + } + state.rateLimitTimestamps.push(nowMs); + state.eventCount++; + + // 4. Dispatch. + try { + const outcome = await this.opts.dispatch(effectiveEvent); + if (outcome.kind === 'failed') { + this.opts.logger.warn( + `[ingestion] source '${sourceId}' dispatch failed: ${outcome.error}`, + ); + } + } catch (err) { + this.opts.logger.error( + `[ingestion] source '${sourceId}' dispatcher threw: ${err instanceof Error ? err.message : String(err)}`, + ); + } + } + + /** Wrap the daemon logger with a per-source prefix. */ + private wrapLogger(sourceId: string): Logger { + const baseLogger = this.opts.logger; + return { + info(msg: string) { baseLogger.info(`[ingestion:${sourceId}] ${msg}`); }, + warn(msg: string) { baseLogger.warn(`[ingestion:${sourceId}] ${msg}`); }, + error(msg: string) { baseLogger.error(`[ingestion:${sourceId}] ${msg}`); }, + }; + } + + private now(): number { + return this.opts._now ? this.opts._now() : Date.now(); + } + + private sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); + } + + /** Internal access for tests. @internal */ + _stateForTest(id: string): Readonly | undefined { + return this.sources.get(id); + } +} + +/** + * Race a promise against a timeout. Rejects with the timeout message if + * the promise hasn't settled in `ms`. Caller is responsible for cleanup + * — used by stop() drains and healthCheck() probes. + */ +function raceWithTimeout(p: Promise, ms: number, timeoutMsg: string): Promise { + return new Promise((resolve, reject) => { + let settled = false; + const t = setTimeout(() => { + if (settled) return; + settled = true; + reject(new Error(timeoutMsg)); + }, ms); + p.then( + (v) => { + if (settled) return; + settled = true; + clearTimeout(t); + resolve(v); + }, + (err) => { + if (settled) return; + settled = true; + clearTimeout(t); + reject(err); + }, + ); + }); +} diff --git a/src/core/ingestion/dedup.ts b/src/core/ingestion/dedup.ts new file mode 100644 index 000000000..f4d00afdf --- /dev/null +++ b/src/core/ingestion/dedup.ts @@ -0,0 +1,169 @@ +/** + * 24-hour content-hash dedup window for ingestion events. + * + * Daemon-side defense against duplicate events. Two scenarios in practice: + * + * 1. Overlapping sources: a file-watcher and an inbox-folder source both + * observe the same path (the inbox dir is inside the brain repo). Both + * emit. Dedup catches the second. + * + * 2. At-least-once delivery: a source emits, the daemon hasn't acked the + * Minion queue insert when the source crashes, source restarts and + * re-emits the same content. Dedup catches the replay. + * + * Design constraints: + * + * - In-memory only. Surviving daemon restart is NOT a goal — at-least-once + * across daemon restarts is the existing Minion queue's job + * (idempotency_key). This LRU only catches same-process duplicates. + * + * - Bounded by entry count, not memory. We cap at MAX_ENTRIES so a runaway + * source can't OOM the daemon. Each entry is ~96 bytes + * (source_kind + 64-hex hash + Date ts), so the worst case is ~480KB at + * 5000 entries. + * + * - 24h TTL. Pages don't get rewritten more than once per 24h in normal + * operation. Beyond that window, treat the same content as a new event + * (probably a re-import or a user explicitly re-saving). + * + * - Pure functions + a class for the stateful instance. Tests inject the + * clock via the constructor. + * + * Concurrency: the daemon runs single-threaded JS, so we don't need locks. + * If we ever move source emit to a worker thread, this needs revisiting. + */ + +const DEFAULT_TTL_MS = 24 * 60 * 60 * 1000; // 24 hours +const DEFAULT_MAX_ENTRIES = 5000; + +export interface DedupOpts { + /** Time-to-live for each entry. Default 24 hours. */ + ttlMs?: number; + /** Max entries before LRU eviction. Default 5000. */ + maxEntries?: number; + /** Test seam for the clock. Defaults to Date.now. */ + _now?: () => number; +} + +export interface DedupStats { + /** Total events seen since daemon start. */ + total: number; + /** Number that were dedup hits (silent drop). */ + hits: number; + /** Number of LRU evictions performed. */ + evictions: number; + /** Current cache size. */ + size: number; +} + +/** + * Bounded LRU keyed on `${source_kind}:${content_hash}`. Insertion-ordered + * Map preserves LRU semantics: re-touching an existing key moves it to the + * back via delete-then-set so eviction picks the oldest entry. + * + * Returns: + * - `mark(kind, hash)` → true if the key is new (proceed with emit), + * false if already seen (silent dedup). + * - `prune(now?)` → removes entries past TTL. + * - `stats()` → live counters. + */ +export class DedupWindow { + private readonly ttlMs: number; + private readonly maxEntries: number; + private readonly now: () => number; + private readonly entries: Map = new Map(); + private _total = 0; + private _hits = 0; + private _evictions = 0; + + constructor(opts: DedupOpts = {}) { + this.ttlMs = opts.ttlMs ?? DEFAULT_TTL_MS; + this.maxEntries = opts.maxEntries ?? DEFAULT_MAX_ENTRIES; + this.now = opts._now ?? Date.now; + } + + /** + * Probe-and-mark the dedup window. Returns true if the event is new + * (caller should proceed); false if this content was seen within the + * TTL (caller drops silently). + * + * Internally this also performs lazy TTL pruning on the touched key and + * LRU eviction when the cap is exceeded. Pruning the entire window every + * call would be O(n) per emit; we let the window grow up to maxEntries + * and prune in batches via the explicit `prune()` method. + */ + mark(source_kind: string, content_hash: string): boolean { + this._total++; + const key = `${source_kind}:${content_hash}`; + const ts = this.now(); + + const existing = this.entries.get(key); + if (existing !== undefined) { + // Within TTL? Hit. + if (ts - existing < this.ttlMs) { + this._hits++; + // Re-set the entry to bump it to the back (LRU touch). The hit's + // timestamp doesn't matter for the dedup decision, but does for + // eviction order. + this.entries.delete(key); + this.entries.set(key, ts); + return false; + } + // Outside TTL: treat as new. Update the timestamp by setting fresh. + this.entries.delete(key); + } + + // New entry. Enforce cap before insert so we never exceed maxEntries + // even transiently. Map.keys() returns keys in insertion order, so the + // first one is the oldest. + while (this.entries.size >= this.maxEntries) { + const oldestKey = this.entries.keys().next().value; + if (oldestKey === undefined) break; + this.entries.delete(oldestKey); + this._evictions++; + } + + this.entries.set(key, ts); + return true; + } + + /** + * Sweep entries past their TTL. Caller decides when to run this. The + * daemon calls it every ~5 minutes as part of the supervisor tick. + * Returns the count of entries removed for the audit log. + */ + prune(now?: number): number { + const cutoff = (now ?? this.now()) - this.ttlMs; + let removed = 0; + for (const [key, ts] of this.entries) { + if (ts < cutoff) { + this.entries.delete(key); + removed++; + } else { + // Entries are in insertion order; the first non-expired one means + // everything after it is also non-expired. Bail early. The Map + // iteration spec guarantees insertion order so this short-circuit is + // safe. + break; + } + } + return removed; + } + + stats(): DedupStats { + return { + total: this._total, + hits: this._hits, + evictions: this._evictions, + size: this.entries.size, + }; + } + + /** Test seam: reset all state. Never call from production. */ + _resetForTest(): void { + this.entries.clear(); + this._total = 0; + this._hits = 0; + this._evictions = 0; + } +} diff --git a/src/core/ingestion/index.ts b/src/core/ingestion/index.ts new file mode 100644 index 000000000..0424c5593 --- /dev/null +++ b/src/core/ingestion/index.ts @@ -0,0 +1,33 @@ +/** + * Public barrel for the gbrain/ingestion subpath. + * + * Skillpack publishers import from here: + * + * import { IngestionSource, IngestionEvent, computeContentHash } from 'gbrain/ingestion'; + * + * Treat this surface as a versioned public API. Adding exports is a minor + * release; removing or breaking-changing them is a major. Pinned by + * test/public-exports.test.ts. + * + * The daemon itself is intentionally NOT exported — it's gbrain-internal. + * Publishers run their sources via either: + * - the test harness (gbrain/ingestion/test-harness, for unit tests) + * - the CLI (`gbrain ingest test`, for hot-iteration dry-run) + * - the production daemon (`gbrain ingest`, which composes everything) + */ + +export type { + IngestionContentType, + IngestionEvent, + IngestionSource, + IngestionSourceContext, + IngestionSourceHealth, +} from './types.ts'; + +export { + INGESTION_CONTENT_TYPES, + INGESTION_SOURCE_API_VERSION, + IngestionEventError, + computeContentHash, + validateIngestionEvent, +} from './types.ts'; diff --git a/src/core/ingestion/skillpack-load.ts b/src/core/ingestion/skillpack-load.ts new file mode 100644 index 000000000..dea5505ac --- /dev/null +++ b/src/core/ingestion/skillpack-load.ts @@ -0,0 +1,345 @@ +/** + * Skillpack-distributed IngestionSource loader. Sibling to plugin-loader.ts + * (which loads subagent definitions); shares the same GBRAIN_PLUGIN_PATH + * discovery mechanism and gbrain.plugin.json manifest format, but reads a + * different optional field (`ingestion_sources`) and produces a different + * shape (factory functions, not subagent definitions). + * + * A skillpack that ships an ingestion source adds to its gbrain.plugin.json: + * + * { + * "name": "granola-source", + * "plugin_version": "gbrain-plugin-v1", + * "ingestion_sources": [ + * { + * "kind": "voice-granola", + * "module": "./dist/source.js", + * "api_version": "gbrain-ingestion-source-v1", + * "default_config": { "transcription_model": "whisper-1" }, + * "permissions": ["network"] + * } + * ] + * } + * + * The module's default export MUST be a factory: + * + * export default function createSource(config: Record): + * IngestionSource { return { id, kind, start, stop, healthCheck? }; } + * + * Trust model (v1): sources are in-process, evaluated as TS/JS modules in + * the daemon. The TOFU prompt during `gbrain skillpack scaffold` is the user + * acknowledging they trust the source's code. Subprocess / VM isolation is + * a v2 hardening wave — see TODOS.md. + * + * Permissions are display-only in v1 (informational during install). The + * field exists so future v2 isolation can enforce them at runtime without a + * breaking manifest change. + */ + +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import type { IngestionSource } from './types.ts'; +import { INGESTION_SOURCE_API_VERSION } from './types.ts'; + +/** Currently-supported api_version values. Reverse aliases for older + * versions live here when we ship contract v2 — until then there's just + * the canonical v1. */ +const COMPATIBLE_API_VERSIONS: ReadonlySet = new Set([ + INGESTION_SOURCE_API_VERSION, +]); + +const SUPPORTED_PLUGIN_VERSION = 'gbrain-plugin-v1'; + +export interface IngestionSourceDeclaration { + /** Source kind taxonomy. Must be unique across all loaded sources. */ + kind: string; + /** Relative path within the skillpack root pointing at the factory module. */ + module: string; + /** Contract version the source was built against. Must match + * INGESTION_SOURCE_API_VERSION (or a known back-compat alias). */ + api_version: string; + /** Default config merged with per-install overrides before passing to + * ctx.config in source.start(). Optional. */ + default_config?: Record; + /** Display-only in v1; declares the runtime capabilities the source + * expects (network, filesystem, etc.) for the TOFU trust prompt. */ + permissions?: string[]; +} + +interface IngestionSourceManifestSlice { + name: string; + plugin_version: string; + ingestion_sources?: IngestionSourceDeclaration[]; +} + +/** Factory contract — default export shape every skillpack source module + * must conform to. */ +export type IngestionSourceFactory = + (config: Record) => IngestionSource; + +export interface LoadedIngestionSource { + /** The plugin/skillpack that shipped this source. For error messages + * and the doctor surface. */ + plugin_name: string; + /** Source declaration from the manifest. */ + declaration: IngestionSourceDeclaration; + /** Resolved factory function. Daemon invokes with config to instantiate. */ + factory: IngestionSourceFactory; + /** Absolute path to the loaded module file, for debug surfaces. */ + module_path: string; + /** The plugin root dir the source was loaded from. */ + plugin_root: string; +} + +export interface SkillpackSourceLoadResult { + /** Successfully loaded source factories. */ + sources: LoadedIngestionSource[]; + /** Per-path warnings (rejected, missing, malformed) for non-fatal cases + * the doctor surfaces. */ + warnings: string[]; +} + +export interface LoadSkillpackSourcesOpts { + /** Override the GBRAIN_PLUGIN_PATH env (for tests). */ + envPath?: string; + /** Test seam: alternative import() function for stubbing module loads. */ + _import?: (specifier: string) => Promise; +} + +/** + * Discover and load every IngestionSource from GBRAIN_PLUGIN_PATH. Iteration + * order follows the path list (left-to-right); collisions on `kind` are + * surfaced as warnings and the later one is skipped. + * + * Non-fatal failures (missing path, malformed manifest, module load error) + * land in `warnings` and the offending plugin is skipped. Fatal failures + * (api_version mismatch on a declared source) abort that plugin's source + * loading entirely so the user sees the loud-fail message in doctor. + */ +export async function loadSkillpackSources( + opts: LoadSkillpackSourcesOpts = {}, +): Promise { + const raw = opts.envPath ?? process.env.GBRAIN_PLUGIN_PATH ?? ''; + const paths = raw.split(':').map((s) => s.trim()).filter(Boolean); + const result: SkillpackSourceLoadResult = { sources: [], warnings: [] }; + + // Left-wins collision tracking on `kind`. Two skillpacks declaring the + // same kind is a real problem — sources are identified by kind in + // gbrain.yml — and we want the warning to name both sides so the user + // can pick. + const kindByPlugin = new Map(); + + for (const p of paths) { + const rejection = rejectIfNotAbsolute(p); + if (rejection) { + result.warnings.push(rejection); + continue; + } + if (!fs.existsSync(p)) { + result.warnings.push(`[ingestion-load] path does not exist, skipping: ${p}`); + continue; + } + if (!fs.statSync(p).isDirectory()) { + result.warnings.push(`[ingestion-load] not a directory, skipping: ${p}`); + continue; + } + + const manifestPath = path.join(p, 'gbrain.plugin.json'); + if (!fs.existsSync(manifestPath)) { + // Not an error — many plugins ship only subagents and skip the + // ingestion_sources field. Silently move on. + continue; + } + + let manifest: IngestionSourceManifestSlice; + try { + manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8')); + } catch (e) { + result.warnings.push( + `[ingestion-load] invalid manifest JSON at ${manifestPath}: ${e instanceof Error ? e.message : String(e)}`, + ); + continue; + } + + if (manifest.plugin_version !== SUPPORTED_PLUGIN_VERSION) { + result.warnings.push( + `[ingestion-load] unsupported plugin_version '${manifest.plugin_version}' at ${manifestPath} ` + + `(gbrain supports '${SUPPORTED_PLUGIN_VERSION}')`, + ); + continue; + } + + if (!manifest.ingestion_sources || manifest.ingestion_sources.length === 0) { + // No sources declared — fine, the plugin may ship other artifacts. + continue; + } + + if (typeof manifest.name !== 'string' || manifest.name.length === 0) { + result.warnings.push( + `[ingestion-load] manifest at ${manifestPath} missing required "name" field; skipping`, + ); + continue; + } + + for (const decl of manifest.ingestion_sources) { + const declErr = validateDeclaration(decl); + if (declErr) { + result.warnings.push( + `[ingestion-load] ${manifest.name} declaration rejected: ${declErr}`, + ); + continue; + } + + // api_version compatibility check. Loud-fail with paste-ready upgrade + // hint so users know exactly what to do. + if (!COMPATIBLE_API_VERSIONS.has(decl.api_version)) { + result.warnings.push( + `[ingestion-load] ${manifest.name} source '${decl.kind}' declares ` + + `api_version='${decl.api_version}' but gbrain expects ` + + `'${INGESTION_SOURCE_API_VERSION}'. The skillpack was built ` + + `against a different contract version. Fix: upgrade the ` + + `skillpack (publisher needs to rebuild against the new ` + + `IngestionSource contract) OR downgrade gbrain. Skillpack docs: ` + + `https://github.com/garrytan/gbrain/blob/master/docs/ingestion-source-skillpack.md`, + ); + continue; + } + + const prior = kindByPlugin.get(decl.kind); + if (prior) { + result.warnings.push( + `[ingestion-load] kind collision: source '${decl.kind}' from ` + + `'${manifest.name}' at ${p} is shadowed by earlier ` + + `'${prior.pluginName}' at ${prior.pluginRoot} (first wins)`, + ); + continue; + } + + // Resolve the module path inside the plugin root. Prevent ../ escape. + const modulePath = path.resolve(p, decl.module); + if (!modulePath.startsWith(p + path.sep) && modulePath !== p) { + result.warnings.push( + `[ingestion-load] ${manifest.name} source '${decl.kind}' module ` + + `path '${decl.module}' escapes plugin root; rejected`, + ); + continue; + } + if (!fs.existsSync(modulePath)) { + result.warnings.push( + `[ingestion-load] ${manifest.name} source '${decl.kind}' module ` + + `not found at ${modulePath}; rejected`, + ); + continue; + } + + // Dynamic import. Errors here are typically syntax errors or missing + // peer deps in the skillpack — surface them with the file path. + const importer = opts._import ?? ((spec: string) => import(spec)); + let mod: unknown; + try { + mod = await importer(modulePath); + } catch (e) { + result.warnings.push( + `[ingestion-load] ${manifest.name} source '${decl.kind}' failed ` + + `to import: ${e instanceof Error ? e.message : String(e)}`, + ); + continue; + } + + // Resolve factory from default export. We accept either: + // - `export default function(config) { ... }` (ESM default) + // - `module.exports = function(config) { ... }` (CJS default + // when interop'd by Bun) + const factoryCandidate = extractFactory(mod); + if (!factoryCandidate) { + result.warnings.push( + `[ingestion-load] ${manifest.name} source '${decl.kind}' module ` + + `${modulePath} does not export a factory function as its default. ` + + `Expected: \`export default function createSource(config) { return { id, kind, start, stop }; }\``, + ); + continue; + } + + kindByPlugin.set(decl.kind, { pluginName: manifest.name, pluginRoot: p }); + result.sources.push({ + plugin_name: manifest.name, + declaration: decl, + factory: factoryCandidate, + module_path: modulePath, + plugin_root: p, + }); + } + } + + return result; +} + +/** + * Validate a single source declaration. Returns null on success; error + * string on failure. Pure function, no I/O. + */ +function validateDeclaration(decl: unknown): string | null { + if (decl === null || typeof decl !== 'object') { + return 'declaration must be an object'; + } + const d = decl as Record; + if (typeof d.kind !== 'string' || d.kind.length === 0) { + return 'kind must be a non-empty string'; + } + if (typeof d.module !== 'string' || d.module.length === 0) { + return `source '${d.kind}': module must be a non-empty string`; + } + if (typeof d.api_version !== 'string' || d.api_version.length === 0) { + return `source '${d.kind}': api_version must be a non-empty string`; + } + if (d.default_config !== undefined) { + if (d.default_config === null || typeof d.default_config !== 'object' || Array.isArray(d.default_config)) { + return `source '${d.kind}': default_config must be a plain object when present`; + } + } + if (d.permissions !== undefined) { + if (!Array.isArray(d.permissions) || !d.permissions.every((p) => typeof p === 'string')) { + return `source '${d.kind}': permissions must be an array of strings when present`; + } + } + return null; +} + +/** Extract a factory function from a loaded module (ESM or CJS-interop). */ +function extractFactory(mod: unknown): IngestionSourceFactory | null { + if (mod === null || typeof mod !== 'object') return null; + const m = mod as Record; + // ESM default export: `import` produces an object whose `default` + // property is the value we want. + if (typeof m.default === 'function') { + return m.default as IngestionSourceFactory; + } + // Some CJS-interop modules surface their default directly as the + // exports object. Treat the module itself as the factory if callable. + if (typeof mod === 'function') { + return mod as IngestionSourceFactory; + } + return null; +} + +function rejectIfNotAbsolute(p: string): string | null { + if (/^[a-z][a-z0-9+.-]*:\/\//i.test(p)) { + return `[ingestion-load] remote URL rejected: ${p}`; + } + if (p.startsWith('~')) { + return `[ingestion-load] ~-prefixed path rejected (expand explicitly): ${p}`; + } + if (!path.isAbsolute(p)) { + return `[ingestion-load] relative path rejected: ${p}`; + } + return null; +} + +/** Testing surface. */ +export const __testing = { + COMPATIBLE_API_VERSIONS, + SUPPORTED_PLUGIN_VERSION, + validateDeclaration, + extractFactory, + rejectIfNotAbsolute, +}; diff --git a/src/core/ingestion/sources/file-watcher.ts b/src/core/ingestion/sources/file-watcher.ts new file mode 100644 index 000000000..d09a6055e --- /dev/null +++ b/src/core/ingestion/sources/file-watcher.ts @@ -0,0 +1,309 @@ +/** + * FileWatcherSource — chokidar-based ingestion source for the brain repo. + * + * Replaces the v0.37 autopilot's 300s poll loop. When the user (or an + * editor, or a git pull, or rsync) writes a markdown file inside the + * brain repo, this source emits an IngestionEvent within ~1s of the write + * settling. The daemon dedups overlapping events with the inbox-folder + * source via the 24h content-hash window. + * + * Cross-platform via chokidar (macOS FSEvents, Linux inotify, Windows + * ReadDirectoryChangesW). v4 uses native fs.watch by default. Pinned + * version: ^4.0.3 in package.json — bumps require re-validating the + * `add` / `change` / `unlink` event semantics. + * + * Linux scale note: chokidar inherits inotify's limits. On a 10K+ file + * brain repo, the system `fs.inotify.max_user_watches` (default 8192 on + * many distros) is the bottleneck. `gbrain doctor`'s inotify_limit + * probe surfaces this with a paste-ready sysctl hint — see E3 in the + * eng review plan. + * + * Design constraints: + * + * - Atomic writes (editor's .swp → rename) emit ONE event on the final + * path, not two intermediate events. chokidar's awaitWriteFinish + * handles this for us. + * + * - Symlinks are NOT followed (security: a symlink into the brain repo + * could let a misbehaving source ingest arbitrary filesystem paths). + * Hard-coded `followSymlinks: false`. + * + * - Honors pruneDir from src/core/sync.ts so the file-watcher and the + * sync command agree on what counts as "in the brain." Single source + * of truth for excluded directories. + * + * - Markdown only by default. Sources for image/audio/PDF assets are + * separate skillpacks; this source emits text/markdown events for + * `.md` and `.markdown` files only. + * + * - 1s debounce coalesces editor save-storms (vim's "save every 5s + * while typing", VS Code's auto-save). + */ + +import { watch, type FSWatcher, type ChokidarOptions } from 'chokidar'; +import { stat, readFile } from 'node:fs/promises'; +import { resolve, relative, basename, dirname, sep } from 'node:path'; +import { + computeContentHash, + type IngestionEvent, + type IngestionSource, + type IngestionSourceContext, +} from '../types.ts'; +import { pruneDir } from '../../sync.ts'; + +export interface FileWatcherSourceOpts { + /** Source instance id. Defaults to 'file-watcher'. Use distinct ids when + * watching multiple directories (e.g. brain + secondary mount). */ + id?: string; + /** Directory to watch. Required; usually the resolved sync.repo_path. */ + brainDir: string; + /** File extensions to include. Defaults to ['.md', '.markdown']. */ + includeExtensions?: string[]; + /** Debounce window in ms for coalescing rapid writes. Default 1000. */ + debounceMs?: number; + /** chokidar awaitWriteFinish stability threshold. Default 1000. */ + awaitStabilityMs?: number; + /** Test seam: alternative chokidar factory. */ + _watchFactory?: (paths: string, opts: ChokidarOptions) => FSWatcher; +} + +const DEFAULT_INCLUDE_EXTENSIONS = ['.md', '.markdown']; + +/** State per watched path: pending debounce timer + buffered event source. */ +interface PendingEntry { + timer: NodeJS.Timeout; + /** The last seen chokidar event for this path — drives the IngestionEvent + * we eventually emit (add vs change handled identically; unlink is + * separate; the daemon's dedup catches replays). */ + eventName: 'add' | 'change'; +} + +export function createFileWatcherSource(opts: FileWatcherSourceOpts): IngestionSource { + if (!opts.brainDir || typeof opts.brainDir !== 'string') { + throw new Error('FileWatcherSource: brainDir is required (typically sync.repo_path)'); + } + const id = opts.id ?? 'file-watcher'; + const kind = 'file-watcher'; + const includeExtensions = (opts.includeExtensions ?? DEFAULT_INCLUDE_EXTENSIONS).map((e) => + e.startsWith('.') ? e.toLowerCase() : `.${e.toLowerCase()}`, + ); + const debounceMs = opts.debounceMs ?? 1000; + const awaitStabilityMs = opts.awaitStabilityMs ?? 1000; + const brainDirAbs = resolve(opts.brainDir); + + let watcher: FSWatcher | null = null; + const pending: Map = new Map(); + + /** Pruning function passed to chokidar. Returns true if path should be + * IGNORED (chokidar's convention). Pre-computed brainDir-relative path + * segments fed through `pruneDir`. */ + function shouldIgnore(absPath: string): boolean { + // Files outside the brain dir shouldn't exist in events from chokidar + // (we passed brainDirAbs as the watch root), but defensive guard. + if (!absPath.startsWith(brainDirAbs)) return true; + const rel = relative(brainDirAbs, absPath); + if (!rel) return false; // the brainDir itself + const segments = rel.split(sep); + // pruneDir says false = exclude. We check every intermediate segment. + // The leaf can be a filename, so we only apply pruneDir to directory + // segments — chokidar gives us files AND directories in this callback, + // so the most general check is: any segment that pruneDir vetoes + // means the whole path is excluded. + for (const seg of segments) { + if (!pruneDir(seg)) return true; + } + return false; + } + + /** Filter for whether to emit on a given path's extension. */ + function isIncludedFile(absPath: string): boolean { + const name = basename(absPath).toLowerCase(); + return includeExtensions.some((ext) => name.endsWith(ext)); + } + + function flushPending(absPath: string, ctx: IngestionSourceContext): void { + const entry = pending.get(absPath); + if (!entry) return; + pending.delete(absPath); + // Read the file at flush time so we capture the post-debounce content. + // Read failures (file deleted between debounce-fire and read) become + // silent skips — chokidar will fire 'unlink' separately. + readFile(absPath, 'utf8').then( + (content) => { + const nowIso = new Date().toISOString(); + const ev: IngestionEvent = { + source_id: id, + source_kind: kind, + source_uri: absPath, + received_at: nowIso, + content_type: 'text/markdown', + content, + content_hash: computeContentHash(content), + metadata: { + event: entry.eventName, + extension: includeExtensions.find((e) => absPath.toLowerCase().endsWith(e)) ?? '', + }, + }; + ctx.emit(ev); + }, + (err: unknown) => { + ctx.logger.warn( + `file-watcher: failed to read ${absPath}: ${err instanceof Error ? err.message : String(err)}`, + ); + }, + ); + } + + function scheduleFlush(absPath: string, eventName: 'add' | 'change', ctx: IngestionSourceContext): void { + const existing = pending.get(absPath); + if (existing) { + clearTimeout(existing.timer); + } + const timer = setTimeout(() => flushPending(absPath, ctx), debounceMs); + // Don't keep the process alive solely because of a pending flush. + if (typeof (timer as { unref?: () => void }).unref === 'function') { + (timer as { unref?: () => void }).unref!(); + } + pending.set(absPath, { timer, eventName }); + } + + return { + id, + kind, + async start(ctx: IngestionSourceContext): Promise { + const stats = await stat(brainDirAbs).catch(() => null); + if (!stats || !stats.isDirectory()) { + throw new Error( + `file-watcher: brainDir does not exist or is not a directory: ${brainDirAbs}`, + ); + } + + const factory = opts._watchFactory ?? watch; + const chokidarOpts: ChokidarOptions = { + persistent: true, + ignoreInitial: true, + followSymlinks: false, + // ignored: function matcher. chokidar v4 calls this for each path + // candidate during walking. Wrapping pruneDir gives us the + // single-source-of-truth invariant from src/core/sync.ts. + ignored: (p: string) => shouldIgnore(p), + // awaitWriteFinish coalesces atomic writes (vim's .swp → rename + // sequence, IDEs that write-then-rename) into a single 'change' + // event on the final path. + awaitWriteFinish: { + stabilityThreshold: awaitStabilityMs, + pollInterval: 100, + }, + }; + + const w = factory(brainDirAbs, chokidarOpts); + watcher = w; + + // Error events are not fatal — log and continue. chokidar surfaces + // platform issues (EBUSY on Windows, EACCES on locked files, ENOSPC + // when inotify limit is exhausted on Linux) via this channel. + w.on('error', (err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + ctx.logger.warn(`file-watcher: chokidar error: ${msg}`); + // Surface inotify exhaustion in particular — it's the most common + // operational footgun on Linux at scale. Doctor's inotify probe + // catches the static case; this is the runtime detection. + if (msg.includes('ENOSPC')) { + ctx.logger.error( + `file-watcher: inotify watch limit exceeded. Raise the kernel limit: ` + + `sudo sysctl fs.inotify.max_user_watches=524288 ` + + `(persist by adding to /etc/sysctl.conf)`, + ); + } + }); + + w.on('add', (path: string) => { + if (!isIncludedFile(path)) return; + scheduleFlush(path, 'add', ctx); + }); + w.on('change', (path: string) => { + if (!isIncludedFile(path)) return; + scheduleFlush(path, 'change', ctx); + }); + // Unlink events: we don't emit IngestionEvents for deletions in v1. + // The reconcile-on-sync path handles tombstones via the broader sync + // command — that's the canonical write-side delete signal. The daemon + // is for additive ingestion. + + // Cooperate with daemon shutdown. + ctx.abortSignal.addEventListener('abort', () => { + // Stop processing pending flushes — they may never settle if the + // daemon is shutting down. + for (const entry of pending.values()) { + clearTimeout(entry.timer); + } + pending.clear(); + }); + + // chokidar fires 'ready' once the initial scan is complete. Resolve + // start() at that point so the daemon supervisor knows we're live. + await new Promise((resolveReady, rejectReady) => { + let settled = false; + w.once('ready', () => { + if (settled) return; + settled = true; + ctx.logger.info(`file-watcher: ready, watching ${brainDirAbs}`); + resolveReady(); + }); + w.once('error', (err: unknown) => { + if (settled) return; + settled = true; + rejectReady(err instanceof Error ? err : new Error(String(err))); + }); + // Safety timeout: a 30K-file brain on slow disks could take a while, + // but if 60s passes without 'ready', something is wrong. Reject so + // the supervisor can apply backoff. + const safetyTimer = setTimeout(() => { + if (settled) return; + settled = true; + rejectReady(new Error(`file-watcher: chokidar did not emit 'ready' within 60s for ${brainDirAbs}`)); + }, 60_000); + if (typeof (safetyTimer as { unref?: () => void }).unref === 'function') { + (safetyTimer as { unref?: () => void }).unref!(); + } + }); + }, + + async stop(): Promise { + // Flush nothing — pending writes are dropped on shutdown. The next + // daemon startup will rescan and emit any markdown that was changed + // (chokidar's ignoreInitial: true means rescans don't re-emit; but + // file changes after the last sync's content_hash diff will appear + // via the v0.37 sync command before the daemon takes over). + for (const entry of pending.values()) { + clearTimeout(entry.timer); + } + pending.clear(); + if (watcher) { + await watcher.close(); + watcher = null; + } + }, + + async healthCheck() { + if (!watcher) { + return { status: 'fail', message: 'watcher not initialized' }; + } + // chokidar exposes getWatched() which returns the directories under + // watch. An empty map means inotify is exhausted or the brain dir + // disappeared — surface as warn. + const watched = watcher.getWatched(); + const totalDirs = Object.keys(watched).length; + if (totalDirs === 0) { + return { status: 'warn', message: 'no directories under watch' }; + } + return { status: 'ok' }; + }, + }; +} + +/** For tests that want to construct events with the same shape the source + * emits, without spinning up chokidar. */ +export const __testing = { + DEFAULT_INCLUDE_EXTENSIONS, +}; diff --git a/src/core/ingestion/sources/inbox-folder.ts b/src/core/ingestion/sources/inbox-folder.ts new file mode 100644 index 000000000..ec0b1198d --- /dev/null +++ b/src/core/ingestion/sources/inbox-folder.ts @@ -0,0 +1,367 @@ +/** + * InboxFolderSource — drop-in capture target for Shortcuts / AirDrop / Drafts. + * + * Watches `~/.gbrain/inbox/` by default. When a file appears (anyone can + * drop one — iOS Shortcuts share-extension, macOS AirDrop, Drafts export, + * Finder drag), the source emits an IngestionEvent then moves the file to + * `~/.gbrain/inbox/.archived/YYYY-MM-DD/` so the user has a + * visible audit trail of what was captured AND the inbox dir stays + * uncluttered. + * + * This is the magical-moment ingestion path for mobile capture without + * building a mobile app: any iOS Shortcut that writes to your synced + * iCloud folder pointed at the brain's inbox lands as a brain page within + * seconds. + * + * Design constraints: + * + * - Symlinks are NOT followed at the dir level (chokidar followSymlinks + * false). Per-file symlink rejection at emit time is layered defense + * against a user dragging a symlink in directly. + * + * - World-writable inbox dir warning at startup. The inbox accepts ANY + * file from ANY local process; if the dir is world-writable then any + * unprivileged process on the host can plant content. Warn loud so + * the user can tighten permissions. + * + * - Archive dir is local-only — no DB write. The IngestionEvent flowing + * through put_page is the canonical brain record; the archive copy is + * just for the user's "did this get captured?" audit. + * + * - Content-type detection by extension. Most drops will be `.md` or + * `.txt`; PDFs and images route to the appropriate content processor + * in the daemon's dispatch pipeline (E2 hybrid model — a wave-3 + * concern, the inbox source just labels the event). + * + * - One event per file, even on directory drops. chokidar fires `add` + * per file as it walks the new directory; we don't have to do + * anything special. + */ + +import { watch, type FSWatcher, type ChokidarOptions } from 'chokidar'; +import { mkdir, rename, readFile, stat, lstat } from 'node:fs/promises'; +import { existsSync, statSync } from 'node:fs'; +import { resolve, basename, join, dirname, extname } from 'node:path'; +import { + computeContentHash, + type IngestionContentType, + type IngestionEvent, + type IngestionSource, + type IngestionSourceContext, +} from '../types.ts'; + +const DEFAULT_DEBOUNCE_MS = 500; +const DEFAULT_STABILITY_MS = 1000; + +export interface InboxFolderSourceOpts { + /** Source instance id. Defaults to 'inbox-folder'. */ + id?: string; + /** Inbox directory to watch. Required. Typical: ~/.gbrain/inbox/. */ + inboxDir: string; + /** Archive subdir name (relative to inbox). Default '.archived'. */ + archiveSubdir?: string; + /** Set to false to skip the post-emit archive move (debugging only). */ + archiveAfterEmit?: boolean; + /** chokidar awaitWriteFinish stability threshold. Default 1000. */ + awaitStabilityMs?: number; + /** Debounce between identical-path events. Default 500. */ + debounceMs?: number; + /** Test seam. */ + _watchFactory?: (paths: string, opts: ChokidarOptions) => FSWatcher; +} + +/** Map extension → content_type. Unknown extensions become 'unknown' and + * pass through unchanged; the put_page handler treats them as opaque. */ +function detectContentType(filename: string): IngestionContentType { + const ext = extname(filename).toLowerCase(); + switch (ext) { + case '.md': + case '.markdown': + return 'text/markdown'; + case '.txt': + return 'text/plain'; + case '.html': + case '.htm': + return 'text/html'; + case '.json': + return 'application/json'; + case '.pdf': + return 'application/pdf'; + case '.png': + case '.jpg': + case '.jpeg': + case '.webp': + case '.gif': + case '.bmp': + case '.tiff': + return 'image/*'; + case '.mp3': + case '.m4a': + case '.wav': + case '.ogg': + case '.flac': + return 'audio/*'; + case '.mp4': + case '.mov': + case '.webm': + case '.mkv': + return 'video/*'; + default: + return 'unknown'; + } +} + +/** Format a UTC date as YYYY-MM-DD for the archive subdir. */ +function archiveDateFolder(d: Date): string { + const y = d.getUTCFullYear(); + const m = String(d.getUTCMonth() + 1).padStart(2, '0'); + const day = String(d.getUTCDate()).padStart(2, '0'); + return `${y}-${m}-${day}`; +} + +/** + * Try to acquire a unique destination path in the archive dir. If the + * filename already exists (two captures of the same name on the same + * day), suffix with `-1`, `-2`, ... before the extension. + */ +function uniqueArchivePath(dir: string, filename: string): string { + const candidate = join(dir, filename); + if (!existsSync(candidate)) return candidate; + const ext = extname(filename); + const stem = filename.slice(0, filename.length - ext.length); + for (let i = 1; i < 1000; i++) { + const next = join(dir, `${stem}-${i}${ext}`); + if (!existsSync(next)) return next; + } + // Last-resort: timestamp suffix. + return join(dir, `${stem}-${Date.now()}${ext}`); +} + +export function createInboxFolderSource(opts: InboxFolderSourceOpts): IngestionSource { + if (!opts.inboxDir || typeof opts.inboxDir !== 'string') { + throw new Error('InboxFolderSource: inboxDir is required (typical: ~/.gbrain/inbox)'); + } + const id = opts.id ?? 'inbox-folder'; + const kind = 'inbox-folder'; + const inboxDirAbs = resolve(opts.inboxDir); + const archiveSubdir = opts.archiveSubdir ?? '.archived'; + const archiveDirAbs = join(inboxDirAbs, archiveSubdir); + const archiveAfterEmit = opts.archiveAfterEmit ?? true; + const awaitStabilityMs = opts.awaitStabilityMs ?? DEFAULT_STABILITY_MS; + const debounceMs = opts.debounceMs ?? DEFAULT_DEBOUNCE_MS; + + let watcher: FSWatcher | null = null; + /** Per-path debounce timers. Coalesces rapid create+rename events from + * Finder copy operations. */ + const pending: Map = new Map(); + + function isUnderArchive(absPath: string): boolean { + return absPath.startsWith(archiveDirAbs + '/') || absPath === archiveDirAbs; + } + + async function handleAdd(absPath: string, ctx: IngestionSourceContext): Promise { + // Layered defense against the file becoming a symlink between + // chokidar discovery and our read. chokidar's followSymlinks=false + // already filters at the dir level. + let info; + try { + info = await lstat(absPath); + } catch (err) { + ctx.logger.warn( + `inbox-folder: failed to lstat ${absPath}: ${err instanceof Error ? err.message : String(err)}`, + ); + return; + } + if (info.isSymbolicLink()) { + ctx.logger.warn(`inbox-folder: rejected symlink ${absPath} (security)`); + return; + } + if (!info.isFile()) { + // Directories and other types are ignored — we only ingest files. + return; + } + + const filename = basename(absPath); + const contentType = detectContentType(filename); + + // For text-typed content we read and emit the bytes inline. For binary + // types (image/audio/video/pdf) we emit the absolute path as the content + // and let the daemon's processor pipeline handle extraction. Path-only + // emit for binary keeps the daemon's hot path lean — a 200MB video + // shouldn't be read into memory just to compute content_hash. + const isText = contentType === 'text/markdown' || contentType === 'text/plain' || + contentType === 'text/html' || contentType === 'application/json'; + + let content: string; + let contentHashSource: string; + if (isText) { + try { + content = await readFile(absPath, 'utf8'); + } catch (err) { + ctx.logger.warn( + `inbox-folder: failed to read ${absPath}: ${err instanceof Error ? err.message : String(err)}`, + ); + return; + } + contentHashSource = content; + } else { + // For binary, content_hash is over the absolute path + size + mtime + // so repeat-drops of the SAME file dedup but different files with + // matching name (rare) get different hashes. Reading the bytes to + // hash them would be O(file size); the path+stat shortcut is O(1). + content = absPath; + contentHashSource = `${absPath}|${info.size}|${info.mtimeMs}`; + } + + const ev: IngestionEvent = { + source_id: id, + source_kind: kind, + source_uri: absPath, + received_at: new Date().toISOString(), + content_type: contentType, + content, + content_hash: computeContentHash(contentHashSource), + metadata: { + original_filename: filename, + size_bytes: info.size, + is_text: isText, + }, + }; + + ctx.emit(ev); + + // Move to .archived/YYYY-MM-DD/ so the user has a visible + // record of what was captured AND the inbox stays uncluttered. Failure + // is non-fatal — log and leave the file in place. Better to risk a + // re-emit (caught by the daemon's 24h dedup) than to silently lose + // the user's capture by failing the move and crashing. + if (archiveAfterEmit) { + try { + const todayDir = join(archiveDirAbs, archiveDateFolder(new Date())); + await mkdir(todayDir, { recursive: true }); + const dest = uniqueArchivePath(todayDir, filename); + await rename(absPath, dest); + } catch (err) { + ctx.logger.warn( + `inbox-folder: failed to archive ${absPath}: ${err instanceof Error ? err.message : String(err)}`, + ); + } + } + } + + function scheduleHandle(absPath: string, ctx: IngestionSourceContext): void { + const existing = pending.get(absPath); + if (existing) clearTimeout(existing); + const timer = setTimeout(() => { + pending.delete(absPath); + void handleAdd(absPath, ctx); + }, debounceMs); + if (typeof (timer as { unref?: () => void }).unref === 'function') { + (timer as { unref?: () => void }).unref!(); + } + pending.set(absPath, timer); + } + + return { + id, + kind, + async start(ctx: IngestionSourceContext): Promise { + // Ensure the inbox dir exists. Create it if missing so the user can + // start dropping files immediately — this is a magical-moment + // affordance and we don't want to fail on a missing directory. + await mkdir(inboxDirAbs, { recursive: true }); + await mkdir(archiveDirAbs, { recursive: true }); + + // World-writable warning. mode & 0o002 = world-write bit set. + try { + const info = statSync(inboxDirAbs); + if ((info.mode & 0o002) !== 0) { + ctx.logger.warn( + `inbox-folder: ${inboxDirAbs} is world-writable. Any unprivileged ` + + `process on this host can plant content. Tighten with: ` + + `chmod 700 ${inboxDirAbs}`, + ); + } + } catch (_err) { + // stat shouldn't fail (we just mkdir'd), but defensive. + } + + const factory = opts._watchFactory ?? watch; + const chokidarOpts: ChokidarOptions = { + persistent: true, + // ignoreInitial: false — we WANT to ingest files that were + // dropped while the daemon was offline. The 24h content-hash + // dedup catches repeat-startup of the same file. + ignoreInitial: false, + followSymlinks: false, + ignored: (p: string) => isUnderArchive(p), + awaitWriteFinish: { + stabilityThreshold: awaitStabilityMs, + pollInterval: 100, + }, + }; + + const w = factory(inboxDirAbs, chokidarOpts); + watcher = w; + + w.on('error', (err: unknown) => { + const msg = err instanceof Error ? err.message : String(err); + ctx.logger.warn(`inbox-folder: chokidar error: ${msg}`); + }); + + w.on('add', (path: string) => { + if (isUnderArchive(path)) return; // belt + suspenders + scheduleHandle(path, ctx); + }); + + ctx.abortSignal.addEventListener('abort', () => { + for (const timer of pending.values()) clearTimeout(timer); + pending.clear(); + }); + + await new Promise((resolveReady, rejectReady) => { + let settled = false; + w.once('ready', () => { + if (settled) return; + settled = true; + ctx.logger.info(`inbox-folder: ready, watching ${inboxDirAbs}`); + resolveReady(); + }); + w.once('error', (err: unknown) => { + if (settled) return; + settled = true; + rejectReady(err instanceof Error ? err : new Error(String(err))); + }); + const safetyTimer = setTimeout(() => { + if (settled) return; + settled = true; + rejectReady(new Error(`inbox-folder: chokidar did not emit 'ready' within 30s for ${inboxDirAbs}`)); + }, 30_000); + if (typeof (safetyTimer as { unref?: () => void }).unref === 'function') { + (safetyTimer as { unref?: () => void }).unref!(); + } + }); + }, + + async stop(): Promise { + for (const timer of pending.values()) clearTimeout(timer); + pending.clear(); + if (watcher) { + await watcher.close(); + watcher = null; + } + }, + + async healthCheck() { + if (!watcher) return { status: 'fail', message: 'watcher not initialized' }; + return { status: 'ok' }; + }, + }; +} + +/** For tests. */ +export const __testing = { + detectContentType, + archiveDateFolder, + uniqueArchivePath, +}; diff --git a/src/core/ingestion/test-harness.ts b/src/core/ingestion/test-harness.ts new file mode 100644 index 000000000..f104eae43 --- /dev/null +++ b/src/core/ingestion/test-harness.ts @@ -0,0 +1,323 @@ +/** + * IngestionTestHarness — Publisher-facing test utility for IngestionSource authors. + * + * Exported as a public subpath (gbrain/ingestion/test-harness) and pinned by + * test/public-exports.test.ts so it stays a versioned API for skillpack + * publishers. Treat as the equivalent of @testing-library for browser apps: + * downstream code depends on this surface, breaking it requires a major bump. + * + * Typical usage: + * + * import { IngestionTestHarness, expectEvent } from 'gbrain/ingestion/test-harness'; + * import { GranolaSource } from './source'; + * + * test('emits one event per transcript', async () => { + * const harness = new IngestionTestHarness({ config: { apiKey: 'fake' } }); + * const source = new GranolaSource(); + * await harness.run(source); + * await harness.advance(60_000); // fake clock advance + * expect(harness.events).toHaveLength(1); + * expectEvent(harness.events[0]).toHaveKind('voice-whisper'); + * expectEvent(harness.events[0]).toHaveContentHash(); + * await harness.stop(); + * }); + * + * The harness deliberately does NOT spin up a PGLite brain — that's what + * `gbrain ingest test --watch` is for (CLI-side ephemeral brain for the + * full daemon-roundtrip iteration loop). The harness is for fast unit + * tests against the source contract in isolation. + */ + +import type { + IngestionEvent, + IngestionSource, + IngestionSourceContext, + IngestionSourceHealth, +} from './types.ts'; +import { validateIngestionEvent } from './types.ts'; +import type { BrainEngine } from '../engine.ts'; +import type { Logger } from '../operations.ts'; + +export interface IngestionTestHarnessOpts { + /** Source-specific config passed to source.start(ctx). Defaults to {}. */ + config?: Record; + /** Bring-your-own engine. When omitted, ctx.engine is a Proxy that throws + * on any property access — sources that touch the engine fail loud in + * tests, prompting the publisher to either pass a real engine here or + * refactor the source to not depend on it. */ + engine?: BrainEngine; + /** Custom logger. Defaults to capturing logs into harness.logs. */ + logger?: Logger; + /** Initial fake clock value (ms since epoch). Defaults to a fixed + * deterministic point in 2026 so tests don't flake on real clock drift. */ + startTime?: number; +} + +/** + * Single-source test harness. Construct one per test; not designed to host + * multiple sources simultaneously (that's the daemon's job; tests should + * exercise sources in isolation). + */ +export class IngestionTestHarness { + private readonly opts: IngestionTestHarnessOpts; + private readonly _events: IngestionEvent[] = []; + private readonly _logs: Array<{ level: 'info' | 'warn' | 'error'; msg: string }> = []; + private readonly _validationErrors: string[] = []; + private readonly _abortController = new AbortController(); + private _currentTime: number; + private _activeSource: IngestionSource | null = null; + private _running = false; + + constructor(opts: IngestionTestHarnessOpts = {}) { + this.opts = opts; + // Deterministic default: 2026-05-20T12:00:00Z — chosen far enough into + // the future that any historical date tests of the source (e.g. a + // file-watcher checking mtime) don't get confused by Date.now() drift. + this._currentTime = opts.startTime ?? 1747742400000; + } + + /** Recorded events from the source so far, in emit order. */ + get events(): readonly IngestionEvent[] { + return this._events; + } + + /** Recorded log calls from the source. */ + get logs(): readonly { level: 'info' | 'warn' | 'error'; msg: string }[] { + return this._logs; + } + + /** Validation errors observed at the harness boundary. Real daemon + * rejects malformed events; harness collects them so tests can + * assert "my source never emits a bad event". */ + get validationErrors(): readonly string[] { + return this._validationErrors; + } + + /** Whether the source is currently in its `start`/`emit` lifecycle. */ + get running(): boolean { + return this._running; + } + + /** Build the context the daemon would pass to the source. */ + private buildContext(): IngestionSourceContext { + const harness = this; + return { + emit(event: IngestionEvent): void { + const err = validateIngestionEvent(event); + if (err) { + harness._validationErrors.push(`${err.field}: ${err.reason}`); + return; + } + harness._events.push(event); + }, + engine: this.opts.engine ?? createThrowingEngineProxy(), + logger: this.opts.logger ?? this.buildCaptureLogger(), + abortSignal: this._abortController.signal, + config: this.opts.config ?? {}, + }; + } + + private buildCaptureLogger(): Logger { + const logs = this._logs; + return { + info(msg: string) { logs.push({ level: 'info', msg }); }, + warn(msg: string) { logs.push({ level: 'warn', msg }); }, + error(msg: string) { logs.push({ level: 'error', msg }); }, + }; + } + + /** + * Run the source's start lifecycle. Returns once `source.start(ctx)` + * resolves — note that for sources that watch indefinitely (file + * watcher, scheduler), `start` typically resolves quickly with the + * source running in the background. The harness keeps the abort + * signal alive until `stop()` is called. + */ + async run(source: IngestionSource): Promise { + if (this._running) { + throw new Error('IngestionTestHarness already running a source; construct a new harness per test'); + } + this._activeSource = source; + this._running = true; + const ctx = this.buildContext(); + await source.start(ctx); + } + + /** + * Advance the harness clock by `ms` milliseconds. Sources that schedule + * timers via the standard setTimeout will fire if the test suite is also + * using Bun's fake timers; this helper just bumps the harness's own clock + * reading so log timestamps and dedup windows reflect the test's time. + * + * Note: this does NOT advance global setTimeout — use `Bun.sleep(ms)` or + * `await new Promise(r => setTimeout(r, 0))` in test code if you need to + * yield to source's pending microtasks. + */ + advance(ms: number): void { + if (ms < 0) { + throw new Error('IngestionTestHarness.advance(ms): ms must be >= 0'); + } + this._currentTime += ms; + } + + /** Current fake clock value. Sources can read via ctx.config or the + * harness can be queried by tests asserting on time. */ + now(): number { + return this._currentTime; + } + + /** + * Call source.healthCheck() if defined and return the result. Sources + * without a healthCheck implementation surface as `ok`. + */ + async healthCheck(): Promise { + if (!this._activeSource) { + throw new Error('IngestionTestHarness.healthCheck(): no source started'); + } + if (this._activeSource.healthCheck) { + return this._activeSource.healthCheck(); + } + return { status: 'ok' }; + } + + /** + * Stop the active source. Fires the abort signal, then calls source.stop(). + * Source MUST cooperate within a bounded grace window (default 5 seconds). + */ + async stop(graceMs = 5000): Promise { + if (!this._running || !this._activeSource) return; + this._abortController.abort(); + const stopPromise = this._activeSource.stop(); + const timeoutPromise = new Promise((_, reject) => { + const t = setTimeout( + () => reject(new Error(`IngestionTestHarness.stop(): source did not stop within ${graceMs}ms`)), + graceMs, + ); + // Clear the timeout if stop resolves first so the test process doesn't + // hang on a pending timer. + stopPromise.finally(() => clearTimeout(t)).catch(() => {}); + }); + try { + await Promise.race([stopPromise, timeoutPromise]); + } finally { + this._running = false; + this._activeSource = null; + } + } + + /** Clear recorded events + logs without resetting the clock or running + * state. Useful between phases of a multi-step test. */ + clearRecorded(): void { + this._events.length = 0; + this._logs.length = 0; + this._validationErrors.length = 0; + } +} + +/** + * Fluent expectation helper. Pairs with bun:test / Vitest / Jest expect() + * without depending on any of them — pure value checks that throw on + * mismatch. + * + * expectEvent(events[0]).toHaveKind('voice-whisper'); + * expectEvent(events[0]).toHaveContentHash(); + * expectEvent(events[0]).toHaveSourceUri(matching: /\.granola$/); + */ +export function expectEvent(event: IngestionEvent | undefined) { + function fail(msg: string): never { + const got = event === undefined ? 'undefined' : JSON.stringify(event, null, 2); + throw new Error(`expectEvent: ${msg}\nGot: ${got}`); + } + + return { + toExist(): IngestionEvent { + if (event === undefined) fail('event was undefined'); + return event as IngestionEvent; + }, + toHaveKind(kind: string): void { + if (!event) fail(`expected kind '${kind}' but event was undefined`); + if (event.source_kind !== kind) { + fail(`expected source_kind '${kind}', got '${event.source_kind}'`); + } + }, + toHaveSourceId(id: string): void { + if (!event) fail(`expected source_id '${id}' but event was undefined`); + if (event.source_id !== id) { + fail(`expected source_id '${id}', got '${event.source_id}'`); + } + }, + toHaveSourceUri(matching: string | RegExp): void { + if (!event) fail('event was undefined'); + const uri = event.source_uri; + if (matching instanceof RegExp) { + if (!matching.test(uri)) fail(`source_uri '${uri}' did not match ${matching}`); + } else if (uri !== matching) { + fail(`expected source_uri '${matching}', got '${uri}'`); + } + }, + toHaveContentType(type: string): void { + if (!event) fail(`expected content_type '${type}' but event was undefined`); + if (event.content_type !== type) { + fail(`expected content_type '${type}', got '${event.content_type}'`); + } + }, + toHaveContentHash(hash?: string): void { + if (!event) fail('event was undefined'); + if (!/^[0-9a-f]{64}$/i.test(event.content_hash)) { + fail(`content_hash '${event.content_hash}' is not a valid SHA-256 hex string`); + } + if (hash !== undefined && event.content_hash !== hash) { + fail(`expected content_hash '${hash}', got '${event.content_hash}'`); + } + }, + toBeUntrusted(): void { + if (!event) fail('event was undefined'); + if (event.untrusted_payload !== true) { + fail(`expected untrusted_payload: true, got ${event.untrusted_payload}`); + } + }, + toBeTrusted(): void { + if (!event) fail('event was undefined'); + if (event.untrusted_payload === true) { + fail('expected event to be trusted, but untrusted_payload was true'); + } + }, + toHaveMetadata(matcher: Record): void { + if (!event) fail('event was undefined'); + const md = event.metadata ?? {}; + for (const [k, v] of Object.entries(matcher)) { + if ((md as Record)[k] !== v) { + fail(`expected metadata.${k}=${JSON.stringify(v)}, got ${JSON.stringify((md as Record)[k])}`); + } + } + }, + }; +} + +/** + * Construct a BrainEngine proxy that throws on every property access. Used + * as the default ctx.engine in the harness so sources that touch the engine + * fail loudly in tests with a paste-ready hint to provide a real engine. + * + * Returning a no-op stub instead would let bugs through — a source that + * silently swallows engine calls in tests would pass and then crash in + * production. + */ +function createThrowingEngineProxy(): BrainEngine { + return new Proxy({} as BrainEngine, { + get(_target, prop) { + // Some runtime probes happen during Promise unwrapping etc. — let + // these pass without throwing so the harness doesn't blow up on + // type introspection. The named methods are what trigger the error. + if (prop === 'then' || prop === Symbol.toPrimitive || prop === Symbol.toStringTag) { + return undefined; + } + throw new Error( + `IngestionTestHarness: source attempted to access BrainEngine.${String(prop)} ` + + `but no engine was provided. Pass { engine } to new IngestionTestHarness({...}) ` + + `with a real engine (e.g. new PGLiteEngine()) if the source needs DB access, ` + + `OR refactor the source so it doesn't touch the engine and only emits events.`, + ); + }, + }); +} diff --git a/src/core/ingestion/types.ts b/src/core/ingestion/types.ts new file mode 100644 index 000000000..e013d6d32 --- /dev/null +++ b/src/core/ingestion/types.ts @@ -0,0 +1,318 @@ +/** + * Ingestion contract — IngestionSource, IngestionEvent, IngestionSourceContext. + * + * The locked public API surface for third-party skillpack publishers. Once a + * skillpack ships against this contract, breaking it requires a major bump and + * shows up in test/public-exports.test.ts. Treat these types as a versioned + * public API the same way BrainEngine is. + * + * Source contract design decisions (locked in /plan-ceo-review + /plan-eng-review): + * + * - Sources are dumb emitters; daemon owns supervision (SourceSupervisor + * mirrors the v0.34.3.0 ChildWorkerSupervisor pattern for in-process + * modules — see daemon.ts). Sources THROW exceptions on failure; daemon + * catches and applies exponential backoff per the crash-counter rule. + * + * - IngestionEvent.content_type taxonomy drives daemon-side hybrid routing + * (E2 eng-review decision): content under 1MB is processed inline before + * queue submission; content over 1MB submits a separate process_audio / + * process_video Minion handler chain. Sources can opt out per-event by + * pre-emitting content_type: 'text/markdown' with already-extracted text. + * + * - IngestionEvent.untrusted_payload flag round-trips to the put_page + * handler. Set by the webhook source (network input) and skillpack + * sources that fetch URLs. When true, put_page skips auto-link and + * applies the slug-allowlist gate. Untrusted in-process callers (CLI + * `gbrain capture`) leave it false. + * + * - The api_version constant on the skillpack manifest decouples the + * contract from skillpack release cadence. v1 sources fail loudly with a + * paste-ready upgrade hint when the daemon loads against contract v2. + */ + +import type { BrainEngine } from '../engine.ts'; +import type { Logger } from '../operations.ts'; + +/** + * Contract version stamped on every gbrain.plugin.json that ships an + * IngestionSource. Bumped only when the IngestionSource / IngestionEvent + * shape changes incompatibly. Reverse aliases for prior versions live in the + * skillpack-load module so existing packs continue to work across a + * deprecation window. + */ +export const INGESTION_SOURCE_API_VERSION = 'gbrain-ingestion-source-v1'; + +/** + * Canonical taxonomy of content types the daemon recognizes. The router + * dispatches on these values; unknown types pass through unchanged and the + * pipeline treats them as opaque text/markdown for indexing purposes. + * + * `image/*`, `audio/*`, `video/*` are deliberately the only wildcard forms. + * Subtypes are encoded in IngestionEvent.metadata when needed (e.g. + * `{format: 'png'}`). Wildcards keep the router map small while preserving + * provenance fidelity. + */ +export const INGESTION_CONTENT_TYPES = [ + 'text/markdown', + 'text/plain', + 'text/html', + 'application/pdf', + 'application/json', + 'image/*', + 'audio/*', + 'video/*', + 'unknown', +] as const; + +export type IngestionContentType = typeof INGESTION_CONTENT_TYPES[number]; + +/** + * Stable event the daemon receives from every source. Carries enough + * identity for content-hash dedup at the daemon layer and enough provenance + * for the put_page handler to stamp frontmatter without re-deriving fields. + * + * Sources MUST populate every required field. The daemon validates at the + * boundary via `validateIngestionEvent`; malformed events are rejected with + * a logged error rather than crashing the source. + */ +export interface IngestionEvent { + /** Source instance id. Matches the IngestionSource.id of the emitter. */ + source_id: string; + /** Source kind taxonomy (file-watcher | inbox-folder | webhook | ). */ + source_kind: string; + /** Original URI of the content (file path, mail message-id, URL, etc.). */ + source_uri: string; + /** UTC ISO timestamp the source observed the event. */ + received_at: string; + /** Detected content type. Drives daemon-side routing per E2 hybrid model. */ + content_type: IngestionContentType; + /** Primary content body. For text/* types this is the markdown/text payload. + * For binary types (image/audio/video/pdf), this is an absolute path or + * a data URI; the processor reads from there. */ + content: string; + /** SHA-256 hex of `content`. Daemon dedups on (source_kind, content_hash) + * within a 24h window before queueing. Computing this is the source's + * responsibility because the source knows whether content is text or + * a path-pointer. */ + content_hash: string; + /** + * Trust tag. Set to true by sources that receive input from untrusted + * channels (webhook, future URL fetcher sources). The downstream put_page + * handler honors this flag: skips auto-link entity extraction and applies + * the slug-allowlist gate. Local in-process callers (CLI capture, file + * watcher reading the user's own brain repo) MUST leave this false. + */ + untrusted_payload?: boolean; + /** Optional source-specific metadata. Free-form. Persisted into the page's + * frontmatter under `ingestion_metadata` when present. */ + metadata?: Record; +} + +/** + * Health probe surface for sources that want to expose state to + * `gbrain doctor ingestion_health`. Optional — sources that don't implement + * it surface as `ok` from the daemon side (no signal == healthy assumption). + */ +export interface IngestionSourceHealth { + status: 'ok' | 'warn' | 'fail'; + message?: string; +} + +/** + * Pluggable ingestion source. Built-in sources (file-watcher, inbox-folder, + * cron-scheduler) and skillpack-distributed sources implement the same + * interface — there are no special code paths for built-ins. + * + * Lifecycle: + * 1. Daemon constructs the source via the skillpack-declared factory. + * 2. Daemon calls `start(ctx)`. MUST resolve when source is ready to emit. + * MAY throw — the SourceSupervisor catches and applies backoff. + * 3. Source emits events via `ctx.emit(event)` until shutdown. + * 4. Daemon calls `stop()`. MUST drain any in-flight emission within a + * bounded grace window (default 5 seconds; configurable via + * `ingestion.shutdown_grace_ms`). + * 5. Daemon may call `healthCheck()` periodically (default every 60s) + * for the doctor surface. + * + * Error model (locked /plan-devex-review D1): exceptions thrown from + * `start` / `stop` / inside an `onEvent` callback bubble to the daemon. + * The SourceSupervisor catches them, increments the crash counter, + * applies exponential backoff, and restarts (up to maxCrashes). Sources + * that need richer semantics (transient vs fatal) are a v2 concern; for + * v1, "throw to fail" is the entire contract. + */ +export interface IngestionSource { + /** Unique source instance id. Two file-watcher sources pointing at + * different directories MUST have different ids. The daemon dedups + * events on (source_kind, content_hash); id is for provenance and + * health reporting. */ + readonly id: string; + /** Source kind taxonomy. The router uses this to look up processors + * and the dedup window to scope content-hash keys. */ + readonly kind: string; + /** + * Begin emitting events. MUST resolve when the source is ready to emit; + * MAY throw on unrecoverable startup failure. The daemon catches throws + * and applies the supervisor backoff policy. + */ + start(ctx: IngestionSourceContext): Promise; + /** + * Stop emitting and drain in-flight work. The daemon will wait up to the + * configured grace window before forcing shutdown. Sources MUST cooperate + * with `ctx.abortSignal` — long-running waits should be `Promise.race`-d + * against the signal. + */ + stop(): Promise; + /** Optional health probe. Fired by the daemon every ~60s for the doctor + * surface. When omitted, the source is assumed healthy unless it has + * crashed recently. */ + healthCheck?(): Promise; +} + +/** + * Context the daemon passes to every source's `start()` call. Sources + * interact with the daemon exclusively through this shape — they do not + * touch the Minion queue, the engine, or the audit log directly. + */ +export interface IngestionSourceContext { + /** + * Pure event-emit. The daemon dedups, applies the per-source rate limit, + * and dispatches the event to the Minion queue. Synchronous from the + * source's perspective — emit returns immediately whether the daemon + * accepted, dropped (dedup hit), or rate-limited the event. + */ + emit(event: IngestionEvent): void; + /** + * Read-only engine handle for sources that need to consult the existing + * brain (e.g. a future dedup-aware source that checks for an existing + * page before emitting). Sources MUST NOT write directly — emit an event + * and let the daemon route it through put_page. + */ + engine: BrainEngine; + /** Daemon-provided logger. Sources log here, not to console.log. */ + logger: Logger; + /** Fires when the daemon is shutting down. Sources MUST cooperate by + * exiting any pending operations within the grace window. Long-running + * watches should `Promise.race(..., new Promise(r => signal.addEventListener('abort', r)))`. */ + abortSignal: AbortSignal; + /** Source-specific config resolved at daemon startup from gbrain.yml + * (built-in sources) or gbrain.plugin.json default_config + per-install + * overrides (skillpack sources). Free-form JSON-serializable. */ + config: Record; +} + +/** + * Validation error raised by `validateIngestionEvent`. Carries the field + * that failed and a human-readable reason. The daemon logs and rejects; + * the source's emit returns silently (the source already moved on). + */ +export class IngestionEventError extends Error { + constructor( + public readonly field: string, + public readonly reason: string, + public readonly event: Partial, + ) { + super(`IngestionEvent.${field}: ${reason}`); + this.name = 'IngestionEventError'; + } +} + +/** + * Boundary validator. Daemon runs this on every emit before queueing. Returns + * null on success; an IngestionEventError on the first failed field. + * + * Deliberately structural — we don't validate content_hash matches the SHA-256 + * of content here because (a) the source computed it; (b) recomputing on + * every emit would double the CPU cost on the hot path. The dedup layer is + * tolerant of bad hashes — a bad hash just means dedup misses, not corruption. + */ +export function validateIngestionEvent(event: unknown): IngestionEventError | null { + if (event === null || typeof event !== 'object') { + return new IngestionEventError('root', 'must be an object', {}); + } + const e = event as Record; + + // Required strings. + for (const field of [ + 'source_id', + 'source_kind', + 'source_uri', + 'received_at', + 'content', + 'content_hash', + ] as const) { + if (typeof e[field] !== 'string' || (e[field] as string).length === 0) { + return new IngestionEventError(field, 'must be a non-empty string', e as Partial); + } + } + + // Content type from the closed taxonomy. + if (typeof e.content_type !== 'string') { + return new IngestionEventError('content_type', 'must be a string', e as Partial); + } + if (!INGESTION_CONTENT_TYPES.includes(e.content_type as IngestionContentType)) { + return new IngestionEventError( + 'content_type', + `must be one of ${INGESTION_CONTENT_TYPES.join(', ')}; got '${e.content_type}'`, + e as Partial, + ); + } + + // received_at must parse as an ISO timestamp. Reject malformed without trying + // to be clever about formats — sources should emit Date.prototype.toISOString(). + const parsed = Date.parse(e.received_at as string); + if (!Number.isFinite(parsed)) { + return new IngestionEventError( + 'received_at', + `must be an ISO 8601 timestamp; got '${e.received_at}'`, + e as Partial, + ); + } + + // content_hash should look like a SHA-256 hex string. We don't recompute and + // verify (CPU cost), but we reject obviously bogus values that would create + // hash-key chaos at the dedup layer. + const hash = e.content_hash as string; + if (!/^[0-9a-f]{64}$/i.test(hash)) { + return new IngestionEventError( + 'content_hash', + `must be 64 lowercase hex characters (SHA-256); got '${hash.slice(0, 16)}...'`, + e as Partial, + ); + } + + // untrusted_payload is optional but must be boolean if present. + if (e.untrusted_payload !== undefined && typeof e.untrusted_payload !== 'boolean') { + return new IngestionEventError( + 'untrusted_payload', + `must be boolean when present; got ${typeof e.untrusted_payload}`, + e as Partial, + ); + } + + // metadata is optional but must be a plain object if present. + if (e.metadata !== undefined) { + if (e.metadata === null || typeof e.metadata !== 'object' || Array.isArray(e.metadata)) { + return new IngestionEventError( + 'metadata', + 'must be a plain object when present', + e as Partial, + ); + } + } + + return null; +} + +/** + * Compute SHA-256 hex of a string. Helper for source authors so they don't + * each invent their own hashing. Sources can also pre-hash binary content + * separately (e.g. file-watcher hashes the file bytes, not the path). + */ +export function computeContentHash(content: string): string { + // Bun's built-in crypto returns hex directly. We don't import Node's + // 'node:crypto' because the conditional types diverge in the Bun runtime. + const hasher = new Bun.CryptoHasher('sha256'); + hasher.update(content); + return hasher.digest('hex'); +} diff --git a/src/core/markdown.ts b/src/core/markdown.ts index d0f61ef4b..442c22d00 100644 --- a/src/core/markdown.ts +++ b/src/core/markdown.ts @@ -1,6 +1,6 @@ import matter from 'gray-matter'; import { safeLoad as yamlSafeLoad } from 'js-yaml'; -import type { PageType } from './types.ts'; +import type { Page, PageType } from './types.ts'; import { slugifyPath } from './sync.ts'; export type ParseValidationCode = @@ -415,3 +415,83 @@ function extractTags(frontmatter: Record): string[] { if (typeof tags === 'string') return tags.split(',').map(t => t.trim()).filter(Boolean); return []; } + +// --------------------------------------------------------------------------- +// Page -> markdown serialization helpers (v0.38 DRY extract per eng review) +// +// Pre-v0.38 the dream cycle's reverse-render at src/core/cycle/synthesize.ts +// and the planned v0.38 put_page write-through path were going to have +// near-identical 15-line bodies that differed only in their frontmatter +// stamps. This extract is the single source of truth. +// --------------------------------------------------------------------------- + +import { join } from 'node:path'; + +/** Options for serializePageToMarkdown. */ +export interface SerializePageOpts { + /** Frontmatter fields merged on top of page.frontmatter at render time. + * Use this to stamp provenance (`ingested_via: 'webhook'`), identity + * markers (`dream_generated: true`), or any caller-specific extra + * fields. Original page.frontmatter keys win unless explicitly + * overridden. */ + frontmatterOverrides?: Record; +} + +/** + * Render a Page row to its canonical on-disk markdown form. Sibling to + * `serializeMarkdown` (which takes the underlying primitives); this version + * pulls everything from a `Page` object so callers don't have to destructure + * compiled_truth / timeline / tags / frontmatter at every site. + * + * - Frontmatter: starts from `page.frontmatter`, merged with optional + * `opts.frontmatterOverrides`. Useful for stamping `dream_generated`, + * `ingested_via`, etc. + * - Type / title: pulled from the Page columns; falls back to 'note' / + * empty string when absent. + * - Tags: passed separately so callers don't need to query engine.getTags + * if they already have them in hand. + */ +export function serializePageToMarkdown( + page: Page, + tags: string[], + opts: SerializePageOpts = {}, +): string { + const frontmatter: Record = { + ...((page.frontmatter ?? {}) as Record), + ...(opts.frontmatterOverrides ?? {}), + }; + return serializeMarkdown( + frontmatter, + page.compiled_truth ?? '', + page.timeline ?? '', + { + type: (page.type as PageType) ?? 'note', + title: page.title ?? '', + tags, + }, + ); +} + +/** + * Compute the on-disk path for a (brainDir, slug, source_id) tuple per + * the v0.32.8 multi-source filing layout: + * - Default source: `/.md` + * - Non-default source: `/.sources//.md` + * + * Shared by the dream-cycle reverse-render (`reverseWriteRefs` in + * synthesize.ts) and the v0.38 put_page write-through path so both + * sites compute the same path for the same row. + * + * NOTE: caller is responsible for validating `source_id` against path- + * traversal attacks via `validateSourceId` (src/core/utils.ts) BEFORE + * passing it here. This helper does the filename math only. + */ +export function resolvePageFilePath( + brainDir: string, + slug: string, + sourceId: string, +): string { + return sourceId === 'default' + ? join(brainDir, `${slug}.md`) + : join(brainDir, '.sources', sourceId, `${slug}.md`); +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index ad6371a35..7059d253a 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -3766,6 +3766,54 @@ export const MIGRATIONS: Migration[] = [ ); `, }, + { + version: 81, + name: 'pages_provenance_columns', + // v0.38 ingestion cathedral (eng review E4): + // Adds four nullable provenance columns to `pages` so every ingested + // page carries a record of WHERE it came from. The columns are + // populated by the ingest_capture Minion handler (via the put_page + // write-through path landing in a sibling commit). NULL is the + // historical-page default — pre-v0.38 pages never had provenance. + // + // - ingested_via TEXT — source kind taxonomy + // (file-watcher | inbox-folder | webhook | + // cron-scheduler | capture-cli | + // ) + // - ingested_at TIMESTAMPTZ — UTC time the ingestion daemon + // accepted the event + // - source_uri TEXT — original URI/path/message-id the event + // carried (file path, mail message-id, URL) + // - source_kind TEXT — duplicates ingested_via for indexed + // filtering convenience (one column for + // "type of source", one for richer label + // — kept narrow + indexable separately) + // + // ADD COLUMN with NULL default is metadata-only on Postgres 11+ and + // PGLite 17.5 — instant on tables of any size. + // + // No index: provenance queries are admin-surface only (admin SPA + // Sources tab + gbrain doctor ingestion_health). Throwing an index on + // a low-cardinality TEXT column would inflate the brain repo for + // negligible read benefit. + // + // Forward-reference bootstrap: every brain that upgrades through this + // version needs the columns visible to the embedded SCHEMA_SQL replay + // BEFORE migrations run. applyForwardReferenceBootstrap on both + // engines is extended in this commit; the matching entries in + // test/schema-bootstrap-coverage.test.ts REQUIRED_BOOTSTRAP_COVERAGE + // pin the contract. + // + // Renumbered v80→v81 during master merge: master's v0.37.2.0 + // takes_unresolvable_quality hotfix claimed v80 first. + idempotent: true, + sql: ` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS ingested_via TEXT NULL; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS ingested_at TIMESTAMPTZ NULL; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_uri TEXT NULL; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_kind TEXT NULL; + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/minions/handlers/ingest-capture.ts b/src/core/minions/handlers/ingest-capture.ts new file mode 100644 index 000000000..9cee1c161 --- /dev/null +++ b/src/core/minions/handlers/ingest-capture.ts @@ -0,0 +1,127 @@ +/** + * `ingest_capture` Minion job handler. Receives an IngestionEvent payload + * from the daemon's dispatcher (or the webhook source's POST /ingest + * handler) and routes it through `importFromContent` to land as a brain + * page. + * + * Trust posture (E1 + eng-review decisions): + * - The event's `untrusted_payload` flag is preserved on the job's + * result for audit, but does NOT change the importFromContent call + * itself — auto-link runs at the put_page operation layer, which we + * deliberately bypass here. The handler calls importFromContent + * directly. v1 path: webhook OAuth gate is the trust boundary; the + * handler trusts the event-shape but treats content as user-authored + * markdown. + * - Auto-link integration with the untrusted_payload tag is a v2 + * improvement (would require routing through the put_page op AND + * extending OperationContext with the trust tag). See TODOs in the + * plan. + * + * Slug resolution (in order): + * 1. `job.data.slug` if caller provided one + * 2. `job.data.metadata.slug` if event metadata carried one + * 3. Generated default: `inbox/YYYY-MM-DD-` using the event's + * content_hash prefix. Stable for the same content. + * + * The default slug deliberately lives under `inbox/` — that's the + * triage convention the user will discover when reviewing recent + * captures. A downstream skill (post-capture-triage) can promote inbox + * pages to canonical homes later. + */ + +import type { MinionJobContext } from '../types.ts'; +import type { BrainEngine } from '../../engine.ts'; +import type { IngestionEvent } from '../../ingestion/types.ts'; +import { validateIngestionEvent } from '../../ingestion/types.ts'; +import { importFromContent } from '../../import-file.ts'; + +export interface IngestCaptureResult { + slug: string; + status: 'imported' | 'skipped' | 'error'; + chunks: number; + untrusted_payload: boolean; + source_kind: string; + source_uri: string; +} + +/** Builds the default slug for an event when the caller didn't provide one. */ +export function defaultSlugForEvent(event: IngestionEvent, now: Date = new Date()): string { + const y = now.getUTCFullYear(); + const m = String(now.getUTCMonth() + 1).padStart(2, '0'); + const d = String(now.getUTCDate()).padStart(2, '0'); + const hashPrefix = event.content_hash.slice(0, 6); + return `inbox/${y}-${m}-${d}-${hashPrefix}`; +} + +export function makeIngestCaptureHandler(engine: BrainEngine) { + return async function ingestCaptureHandler(job: MinionJobContext): Promise { + const data = job.data as { event?: unknown; slug?: unknown }; + const event = data.event as IngestionEvent | undefined; + if (!event) { + throw new Error('ingest_capture: job.data.event is required'); + } + const validationErr = validateIngestionEvent(event); + if (validationErr) { + throw new Error(`ingest_capture: invalid event payload: ${validationErr.message}`); + } + + // Slug resolution. + let slug: string; + if (typeof data.slug === 'string' && data.slug.length > 0) { + slug = data.slug; + } else if ( + event.metadata && + typeof (event.metadata as Record).slug === 'string' + ) { + slug = (event.metadata as Record).slug as string; + } else { + slug = defaultSlugForEvent(event); + } + + // Untrusted-payload posture. For v1, the flag is propagated for audit + // but not enforced at this layer (see file header). Future v2 wiring + // through put_page will use this flag. + const untrustedPayload = event.untrusted_payload === true; + + // For text-typed events, content is the inline markdown/text. For + // binary types (image/audio/video/pdf), content is a path-or-URI that + // the content-type processor pipeline transforms. The v1 wave lands + // the text path; processors arrive in subsequent commits. + const isText = + event.content_type === 'text/markdown' || + event.content_type === 'text/plain' || + event.content_type === 'text/html' || + event.content_type === 'application/json' || + event.content_type === 'unknown'; + + if (!isText) { + // Binary content without a processor would land as a path-string + // page, which isn't useful. Surface as job-level error so the + // operator sees the gap in `gbrain doctor` and can decide whether + // to install the appropriate skillpack-distributed processor. + throw new Error( + `ingest_capture: content_type '${event.content_type}' requires a content-type ` + + `processor that is not yet installed. Install a processor skillpack ` + + `(e.g. gbrain-audio-transcribe, gbrain-image-ocr) or pre-extract the ` + + `content to text/markdown before emitting.`, + ); + } + + // noEmbed defaults to true. Mirrors the sync handler's pattern: + // embed runs as a separate Minion job (autopilot's embed phase OR an + // explicit `gbrain embed --stale`). Callers can opt in to inline embed + // by passing { noEmbed: false } in job.data. + const noEmbed = (data as { noEmbed?: unknown }).noEmbed !== false; + + const result = await importFromContent(engine, slug, event.content, { noEmbed }); + + return { + slug, + status: result.status, + chunks: result.chunks, + untrusted_payload: untrustedPayload, + source_kind: event.source_kind, + source_uri: event.source_uri, + }; + }; +} diff --git a/src/core/operations.ts b/src/core/operations.ts index eb733ea67..bc013227f 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -10,6 +10,9 @@ import { clampSearchLimit } from './engine.ts'; import type { GBrainConfig } from './config.ts'; import type { PageType } from './types.ts'; import { importFromContent } from './import-file.ts'; +import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts'; +import { mkdirSync, writeFileSync, existsSync, statSync } from 'fs'; +import { dirname } from 'path'; import { hybridSearch, hybridSearchCached } from './search/hybrid.ts'; import { expandQuery } from './search/expansion.ts'; import { dedupResults } from './search/dedup.ts'; @@ -590,6 +593,76 @@ const put_page: Operation = { ...(ctx.sourceId ? { sourceId: ctx.sourceId } : {}), }); + // v0.38 put_page write-through: + // After importFromContent succeeds, if `sync.repo_path` resolves to a + // real directory, persist the markdown file to disk alongside the DB + // row. Closes the drift class where DB and file diverged after every + // put_page call (the v0.35.6.0 phantom-redirect pass exists because + // of this drift). Failures are non-fatal — log but don't roll back + // the DB write, which is the durable record. Subsequent sync runs + // reconcile if needed. + // + // Trust gating: + // - Subagent sandbox (viaSubagent without allowedSlugPrefixes) → DB-only. + // Sandbox writes live in wiki/agents// and don't earn a file slot. + // - All other writes (local CLI, MCP write-scope agents, trusted + // workspace subagents) → write-through. + // + // The trusted-workspace path (synthesize/patterns) also runs its own + // reverseWriteRefs in synthesize.ts:reverseWriteRefs as part of the + // cycle phase. Both paths writing the same file is idempotent — the + // second writeFileSync overwrites with byte-identical content. + let writeThrough: { written: boolean; path?: string; skipped?: string; error?: string } | undefined; + const isSandboxSubagent = ctx.viaSubagent === true + && !(Array.isArray(ctx.allowedSlugPrefixes) && ctx.allowedSlugPrefixes.length > 0); + if (!ctx.dryRun && result.status !== 'error' && !isSandboxSubagent) { + try { + const repoPath = await ctx.engine.getConfig('sync.repo_path'); + if (!repoPath) { + writeThrough = { written: false, skipped: 'no_repo_configured' }; + } else if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) { + writeThrough = { written: false, skipped: 'repo_not_found' }; + } else { + // Pull the freshly-written page + tags from the engine so the + // markdown we serialize reflects the post-import state, not the + // pre-import parsedPage view (which lacks DB-derived fields like + // updated_at metadata). + const sourceId = ctx.sourceId ?? 'default'; + const writtenPage = await ctx.engine.getPage(result.slug, { sourceId }); + if (writtenPage) { + const tags = await ctx.engine.getTags(result.slug, { sourceId }); + // Provenance stamp on the frontmatter so future sync round-trips + // know where this page came from. Local CLI writes get + // ingested_via='put_page'; MCP writes get 'mcp:put_page'. + const provenanceVia = ctx.remote === false ? 'put_page' : 'mcp:put_page'; + const md = serializePageToMarkdown(writtenPage, tags, { + frontmatterOverrides: { + ingested_via: provenanceVia, + ingested_at: new Date().toISOString(), + source_kind: provenanceVia, + }, + }); + const filePath = resolvePageFilePath(repoPath as string, result.slug, sourceId); + mkdirSync(dirname(filePath), { recursive: true }); + writeFileSync(filePath, md, 'utf8'); + writeThrough = { written: true, path: filePath }; + } else { + writeThrough = { written: false, skipped: 'page_not_found_after_write' }; + } + } + } catch (e) { + // Loud log; DB write NOT rolled back. The phantom-redirect pass + // catches lingering drift. + const msg = e instanceof Error ? e.message : String(e); + ctx.logger.warn(`[put_page] write-through failed for ${result.slug}: ${msg}`); + writeThrough = { written: false, error: msg }; + } + } else if (isSandboxSubagent) { + writeThrough = { written: false, skipped: 'subagent_sandbox' }; + } else if (ctx.dryRun) { + writeThrough = { written: false, skipped: 'dry_run' }; + } + // Auto-link post-hook: runs AFTER importFromContent (which is its own // transaction). Runs even on status='skipped' so reconciliation catches drift // between the page text and the links table. Failures are non-blocking. @@ -729,6 +802,7 @@ const put_page: Operation = { ...(autoTimeline ? { auto_timeline: autoTimeline } : {}), ...(writerLint ? { writer_lint: writerLint } : {}), ...(factsQueued ? { facts_backstop: factsQueued } : {}), + ...(writeThrough ? { write_through: writeThrough } : {}), }; }, cliHints: { name: 'put', positional: ['slug'], stdin: 'content' }, diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 50791d38c..fbb63b494 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -324,7 +324,15 @@ export class PGLiteEngine implements BrainEngine { EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema='public' AND table_name='sources' AND column_name='archive_expires_at') AS sources_archive_expires_at_exists, EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_schema='public' AND table_name='pages' AND column_name='last_retrieved_at') AS pages_last_retrieved_at_exists + WHERE table_schema='public' AND table_name='pages' AND column_name='last_retrieved_at') AS pages_last_retrieved_at_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='pages' AND column_name='ingested_via') AS pages_ingested_via_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='pages' AND column_name='ingested_at') AS pages_ingested_at_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='pages' AND column_name='source_uri') AS pages_source_uri_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='pages' AND column_name='source_kind') AS pages_source_kind_exists `); const probe = rows[0] as { pages_exists: boolean; @@ -356,6 +364,10 @@ export class PGLiteEngine implements BrainEngine { sources_archived_at_exists: boolean; sources_archive_expires_at_exists: boolean; pages_last_retrieved_at_exists: boolean; + pages_ingested_via_exists: boolean; + pages_ingested_at_exists: boolean; + pages_source_uri_exists: boolean; + pages_source_kind_exists: boolean; }; const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists; @@ -401,6 +413,14 @@ export class PGLiteEngine implements BrainEngine { // v0.37.0 (v79): pages_last_retrieved_at_idx in PGLITE_SCHEMA_SQL // references last_retrieved_at. Pre-v79 brains crash without the column. const needsPagesLastRetrievedAt = probe.pages_exists && !probe.pages_last_retrieved_at_exists; + // v0.38.0 (v80): provenance columns on pages. Not referenced by any + // SCHEMA_SQL index or FK today, but added defense-in-depth so future + // schema work that references them doesn't wedge pre-v80 brains. + const needsPagesProvenance = probe.pages_exists + && (!probe.pages_ingested_via_exists + || !probe.pages_ingested_at_exists + || !probe.pages_source_uri_exists + || !probe.pages_source_kind_exists); // Fresh installs (no tables yet) and modern brains both no-op. if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap @@ -408,7 +428,8 @@ export class PGLiteEngine implements BrainEngine { && !needsMcpLogBootstrap && !needsSubagentProviderId && !needsPagesRecency && !needsIngestLogSourceId && !needsFilesBootstrap && !needsOauthClientsBootstrap - && !needsSourcesArchive && !needsPagesLastRetrievedAt) return; + && !needsSourcesArchive && !needsPagesLastRetrievedAt + && !needsPagesProvenance) return; console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap'); @@ -597,6 +618,20 @@ export class PGLiteEngine implements BrainEngine { ALTER TABLE pages ADD COLUMN IF NOT EXISTS last_retrieved_at TIMESTAMPTZ; `); } + + if (needsPagesProvenance) { + // v81 (pages_provenance_columns): four nullable columns added by the + // v0.38 ingestion cathedral. No SCHEMA_SQL index or FK references + // them today, but bootstrap probes cover the column-only forward- + // reference class defense-in-depth so future schema work doesn't + // wedge pre-v81 brains. + await this.db.exec(` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS ingested_via TEXT; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS ingested_at TIMESTAMPTZ; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_uri TEXT; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_kind TEXT; + `); + } } async withReservedConnection(fn: (conn: ReservedConnection) => Promise): Promise { diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 7210bf00b..769d9b11d 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -415,7 +415,15 @@ export class PostgresEngine implements BrainEngine { EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'archive_expires_at') AS sources_archive_expires_at_exists, EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'last_retrieved_at') AS pages_last_retrieved_at_exists + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'last_retrieved_at') AS pages_last_retrieved_at_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'ingested_via') AS pages_ingested_via_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'ingested_at') AS pages_ingested_at_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_uri') AS pages_source_uri_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_kind') AS pages_source_kind_exists `; const probe = probeRows[0]!; @@ -468,13 +476,28 @@ export class PostgresEngine implements BrainEngine { // adds it before SCHEMA_SQL replay creates the index. v79 runs later // via runMigrations and is idempotent. const needsPagesLastRetrievedAt = probe.pages_exists && !(probe as { pages_last_retrieved_at_exists?: boolean }).pages_last_retrieved_at_exists; + // v0.38.0 (v80): provenance columns. Not referenced by any SCHEMA_SQL + // index/FK today; bootstrap exists for the column-only forward- + // reference class defense-in-depth. + const probeProv = probe as { + pages_ingested_via_exists?: boolean; + pages_ingested_at_exists?: boolean; + pages_source_uri_exists?: boolean; + pages_source_kind_exists?: boolean; + }; + const needsPagesProvenance = probe.pages_exists + && (!probeProv.pages_ingested_via_exists + || !probeProv.pages_ingested_at_exists + || !probeProv.pages_source_uri_exists + || !probeProv.pages_source_kind_exists); if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt && !needsMcpLogBootstrap && !needsSubagentProviderId && !needsChunksEmbeddingImage && !needsPagesRecency && !needsIngestLogSourceId && !needsFilesBootstrap && !needsOauthClientsBootstrap && !needsSourcesArchive - && !needsPagesLastRetrievedAt) return; + && !needsPagesLastRetrievedAt + && !needsPagesProvenance) return; console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap'); @@ -663,6 +686,19 @@ export class PostgresEngine implements BrainEngine { ALTER TABLE pages ADD COLUMN IF NOT EXISTS last_retrieved_at TIMESTAMPTZ; `); } + + if (needsPagesProvenance) { + // v81 (pages_provenance_columns): four nullable columns added by the + // v0.38 ingestion cathedral. No SCHEMA_SQL index/FK references them + // today; bootstrap exists defense-in-depth so future schema work that + // does reference them doesn't wedge pre-v81 brains. + await conn.unsafe(` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS ingested_via TEXT; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS ingested_at TIMESTAMPTZ; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_uri TEXT; + ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_kind TEXT; + `); + } } async transaction(fn: (engine: BrainEngine) => Promise): Promise { diff --git a/test/commands/capture.test.ts b/test/commands/capture.test.ts new file mode 100644 index 000000000..311e28a0d --- /dev/null +++ b/test/commands/capture.test.ts @@ -0,0 +1,230 @@ +/** + * gbrain capture CLI verb tests. + * + * Pure helper coverage (parseArgs, defaultSlug, buildContent) + an + * integration smoke that the local-install path lands a page when the + * engine is connected. Thin-client routing isn't exercised here (that's + * an e2e concern; the local path uses the same put_page operation and + * gets the same write-through plumbing). + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { runCapture, __testing } from '../../src/commands/capture.ts'; + +let engine: PGLiteEngine; +let tmpRoot: string; +let brainDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-capture-')); + brainDir = path.join(tmpRoot, 'brain'); + fs.mkdirSync(brainDir, { recursive: true }); + await engine.setConfig('sync.repo_path', brainDir); +}); + +describe('capture — defaultSlug helper', () => { + test('builds inbox/YYYY-MM-DD- shape', () => { + const slug = __testing.defaultSlug('hello world', new Date('2026-05-20T12:00:00Z')); + expect(slug).toMatch(/^inbox\/2026-05-20-[a-f0-9]{8}$/); + }); + + test('deterministic for same content', () => { + const date = new Date('2026-05-20T00:00:00Z'); + expect(__testing.defaultSlug('thought A', date)).toBe(__testing.defaultSlug('thought A', date)); + }); + + test('different content -> different slug', () => { + const date = new Date('2026-05-20T00:00:00Z'); + expect(__testing.defaultSlug('thought A', date)).not.toBe(__testing.defaultSlug('thought B', date)); + }); + + test('UTC math (no tz drift)', () => { + const slug = __testing.defaultSlug('x', new Date('2026-01-05T23:59:59Z')); + expect(slug).toMatch(/^inbox\/2026-01-05-/); + }); +}); + +describe('capture — parseArgs', () => { + test('inline content goes into opts.content', () => { + const r = __testing.parseArgs(['hello world']); + expect('help' in r).toBe(false); + if (!('help' in r)) expect(r.content).toBe('hello world'); + }); + + test('multi-token inline content is joined with spaces', () => { + const r = __testing.parseArgs(['some', 'tokens', 'here']); + if (!('help' in r)) expect(r.content).toBe('some tokens here'); + }); + + test('flags are extracted out of positional', () => { + const r = __testing.parseArgs(['--slug', 'inbox/x', 'the', 'thought']); + if (!('help' in r)) { + expect(r.slug).toBe('inbox/x'); + expect(r.content).toBe('the thought'); + } + }); + + test('--file is captured', () => { + const r = __testing.parseArgs(['--file', '/tmp/note.md']); + if (!('help' in r)) expect(r.filePath).toBe('/tmp/note.md'); + }); + + test('--stdin flag', () => { + const r = __testing.parseArgs(['--stdin']); + if (!('help' in r)) expect(r.stdin).toBe(true); + }); + + test('--quiet, --json, --help', () => { + const r1 = __testing.parseArgs(['--quiet']); + if (!('help' in r1)) expect(r1.quiet).toBe(true); + const r2 = __testing.parseArgs(['--json']); + if (!('help' in r2)) expect(r2.json).toBe(true); + const r3 = __testing.parseArgs(['--help']); + expect('help' in r3).toBe(true); + }); +}); + +describe('capture — buildContent', () => { + test('wraps plain prose in frontmatter + heading', () => { + const result = __testing.buildContent('a simple thought', {}); + expect(result).toContain('---'); + expect(result).toContain('type: note'); + expect(result).toContain('captured_via: capture-cli'); + expect(result).toContain('# a simple thought'); + }); + + test('honors --type override', () => { + const result = __testing.buildContent('body', { type: 'idea' }); + expect(result).toContain('type: idea'); + }); + + test('does not double-wrap content that already has frontmatter', () => { + const result = __testing.buildContent('---\ntitle: pre-wrapped\n---\n\nbody', {}); + // The result has our captured-via frontmatter PLUS the body that + // included its own frontmatter (raw). Importantly we don't prepend a + // second `# ...` heading. + const headingCount = (result.match(/^# /gm) ?? []).length; + expect(headingCount).toBe(0); + }); + + test('uses first non-empty line as the title', () => { + const result = __testing.buildContent('\n \nReal first line\nmore', {}); + expect(result).toContain('title: "Real first line"'); + }); + + test('caps title at 80 chars', () => { + const longLine = 'x'.repeat(200); + const result = __testing.buildContent(longLine, {}); + const titleMatch = result.match(/title: "([^"]*)"/); + expect(titleMatch).not.toBeNull(); + expect(titleMatch![1].length).toBeLessThanOrEqual(80); + }); + + test('honors --source via captured_via', () => { + const result = __testing.buildContent('body', { source: 'voice-whisper' }); + expect(result).toContain('captured_via: voice-whisper'); + }); +}); + +describe('capture — local install integration', () => { + test('inline content lands as a page + a file on disk', async () => { + // Capture writes to stdout/stderr; redirect those to nullsinks so the + // test runner doesn't show the receipt block. + const logCaptured: string[] = []; + const origLog = console.log; + console.log = (...args: unknown[]) => logCaptured.push(args.map(String).join(' ')); + try { + await runCapture(engine, ['--slug', 'inbox/test-capture-1', '--quiet', 'A captured thought']); + } finally { + console.log = origLog; + } + + // --quiet should have printed just the slug. + expect(logCaptured.length).toBeGreaterThan(0); + expect(logCaptured[0]).toBe('inbox/test-capture-1'); + + // Page exists in the DB. + const page = await engine.getPage('inbox/test-capture-1'); + expect(page).not.toBeNull(); + expect(page?.compiled_truth).toContain('A captured thought'); + + // File written to disk via write-through. + const onDisk = path.join(brainDir, 'inbox/test-capture-1.md'); + expect(fs.existsSync(onDisk)).toBe(true); + }); + + test('default slug is inbox/YYYY-MM-DD- when --slug not provided', async () => { + const logCaptured: string[] = []; + const origLog = console.log; + console.log = (...args: unknown[]) => logCaptured.push(args.map(String).join(' ')); + try { + await runCapture(engine, ['--quiet', 'auto-slugged']); + } finally { + console.log = origLog; + } + const printedSlug = logCaptured[0]; + expect(printedSlug).toMatch(/^inbox\/\d{4}-\d{2}-\d{2}-[a-f0-9]{8}$/); + }); + + test('--file reads content from disk', async () => { + const file = path.join(tmpRoot, 'note.md'); + fs.writeFileSync(file, '# from a file\n\nbody content here'); + const logCaptured: string[] = []; + const origLog = console.log; + console.log = (...args: unknown[]) => logCaptured.push(args.map(String).join(' ')); + try { + await runCapture(engine, ['--file', file, '--slug', 'inbox/from-file', '--quiet']); + } finally { + console.log = origLog; + } + const page = await engine.getPage('inbox/from-file'); + expect(page).not.toBeNull(); + expect(page?.compiled_truth).toContain('body content here'); + }); + + test('--json emits structured output', async () => { + const logCaptured: string[] = []; + const origLog = console.log; + console.log = (...args: unknown[]) => logCaptured.push(args.map(String).join(' ')); + try { + await runCapture(engine, ['--json', '--slug', 'inbox/json-out', 'jsonny']); + } finally { + console.log = origLog; + } + expect(logCaptured.length).toBeGreaterThan(0); + const json = JSON.parse(logCaptured.join('\n')); + expect(json.slug).toBe('inbox/json-out'); + expect(json.content_hash).toMatch(/^[a-f0-9]{64}$/); + expect(json.captured_at).toMatch(/^\d{4}-\d{2}-\d{2}T/); + }); +}); + +describe('capture — help', () => { + test('--help prints usage and returns without engine roundtrip', async () => { + const logCaptured: string[] = []; + const origLog = console.log; + console.log = (...args: unknown[]) => logCaptured.push(args.map(String).join(' ')); + try { + await runCapture(null, ['--help']); + } finally { + console.log = origLog; + } + expect(logCaptured.join('\n')).toContain('Usage: gbrain capture'); + }); +}); diff --git a/test/e2e/ingestion-roundtrip.test.ts b/test/e2e/ingestion-roundtrip.test.ts new file mode 100644 index 000000000..705c7c59f --- /dev/null +++ b/test/e2e/ingestion-roundtrip.test.ts @@ -0,0 +1,253 @@ +/** + * E2E roundtrip for the v0.38 ingestion substrate. + * + * Wires the daemon + inbox-folder source + dispatcher together against + * an in-memory PGLite (no DATABASE_URL needed) and verifies: + * 1. A file dropped into the inbox dir produces an IngestionEvent + * that flows through the daemon's validate → dedup → rate-limit + * → dispatch pipeline. + * 2. The dispatched event reaches the ingest_capture Minion handler. + * 3. The handler routes through importFromContent and lands a page + * in the DB with the expected slug + content. + * 4. The page is queryable by content via getPage. + * 5. Dedup catches a repeat-drop of the same file content within the + * 24h window. + * + * This is the substrate-level e2e — it's the unit-test concept lifted + * one layer up to the daemon+handler wiring. The full server-process + * e2e (gbrain serve --http + POST /ingest + real OAuth) is a separate + * test that requires DATABASE_URL. + */ + +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { IngestionDaemon } from '../../src/core/ingestion/daemon.ts'; +import { createInboxFolderSource } from '../../src/core/ingestion/sources/inbox-folder.ts'; +import { makeIngestCaptureHandler } from '../../src/core/minions/handlers/ingest-capture.ts'; +import type { IngestionEvent } from '../../src/core/ingestion/types.ts'; +import type { MinionJobContext } from '../../src/core/minions/types.ts'; + +// Fake job-context constructor so we can drive the handler directly +// from the dispatcher without spinning up a Minion worker. +function makeFakeJobCtx(data: Record): MinionJobContext { + return { + id: Math.floor(Math.random() * 1_000_000), + name: 'ingest_capture', + data, + attempts_made: 1, + signal: new AbortController().signal, + shutdownSignal: new AbortController().signal, + updateProgress: async () => {}, + updateTokens: async () => {}, + log: async () => {}, + isActive: async () => true, + readInbox: async () => [], + }; +} + +let engine: PGLiteEngine; +let tmpRoot: string; +let inboxDir: string; +let brainDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-e2e-roundtrip-')); + inboxDir = path.join(tmpRoot, 'inbox'); + brainDir = path.join(tmpRoot, 'brain'); + fs.mkdirSync(brainDir, { recursive: true }); + await engine.setConfig('sync.repo_path', brainDir); +}); + +afterEach(() => { + fs.rmSync(tmpRoot, { recursive: true, force: true }); +}); + +const captureLogger = () => { + const messages: Array<{ level: string; msg: string }> = []; + return { + logger: { + info: (msg: string) => { messages.push({ level: 'info', msg }); }, + warn: (msg: string) => { messages.push({ level: 'warn', msg }); }, + error: (msg: string) => { messages.push({ level: 'error', msg }); }, + }, + messages, + }; +}; + +async function waitFor(predicate: () => boolean, timeoutMs = 5000): Promise { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + if (predicate()) return; + await new Promise((r) => setTimeout(r, 25)); + } + throw new Error(`waitFor: predicate did not become true within ${timeoutMs}ms`); +} + +describe('ingestion roundtrip — inbox-folder → daemon → ingest_capture → DB', () => { + test('full pipeline: file drop → page in DB', async () => { + const { logger } = captureLogger(); + const handler = makeIngestCaptureHandler(engine); + const dispatchedEvents: IngestionEvent[] = []; + + const daemon = new IngestionDaemon({ + engine, + logger, + dispatch: async (event) => { + dispatchedEvents.push(event); + // Route the event into the handler directly. In production the + // daemon would submit a Minion job and the worker would invoke + // the handler; here we collapse that for test-loop efficiency. + await handler(makeFakeJobCtx({ event })); + return { kind: 'queued' }; + }, + }); + + const source = createInboxFolderSource({ + inboxDir, + debounceMs: 50, + awaitStabilityMs: 100, + }); + daemon.register({ source }); + await daemon.start(); + + // Drop a file into the inbox. + fs.mkdirSync(inboxDir, { recursive: true }); + const captured = path.join(inboxDir, 'roundtrip.md'); + fs.writeFileSync(captured, '---\ntitle: Roundtrip\n---\n\nfull e2e flow'); + + // Wait for the daemon to pick it up + dispatch + handler to write. + await waitFor(() => dispatchedEvents.length === 1, 6000); + + // Page is in the DB. + const page = await engine.getPage(dispatchedEvents[0]!.metadata!.slug as string ?? + `inbox/${new Date().toISOString().slice(0, 10)}-${dispatchedEvents[0]!.content_hash.slice(0, 6)}`); + // The handler defaults to inbox/- if no slug provided by + // the source. inbox-folder source doesn't set metadata.slug so the + // handler computes the default. + const expectedSlug = `inbox/${new Date().toISOString().slice(0, 10)}-${dispatchedEvents[0]!.content_hash.slice(0, 6)}`; + const fetched = await engine.getPage(expectedSlug); + expect(fetched).not.toBeNull(); + expect(fetched?.compiled_truth).toContain('full e2e flow'); + + // File was archived after ingestion (the inbox-folder source's + // post-emit archive step). + expect(fs.existsSync(captured)).toBe(false); + const archiveDate = new Date().toISOString().slice(0, 10); + expect(fs.existsSync(path.join(inboxDir, '.archived', archiveDate, 'roundtrip.md'))).toBe(true); + + await daemon.stop(); + }, 15_000); + + test('repeat drop of same content dedups silently', async () => { + const { logger } = captureLogger(); + const handler = makeIngestCaptureHandler(engine); + const dispatchedEvents: IngestionEvent[] = []; + + const daemon = new IngestionDaemon({ + engine, + logger, + dispatch: async (event) => { + dispatchedEvents.push(event); + await handler(makeFakeJobCtx({ event })); + return { kind: 'queued' }; + }, + }); + + const source = createInboxFolderSource({ + inboxDir, + debounceMs: 50, + awaitStabilityMs: 100, + }); + daemon.register({ source }); + await daemon.start(); + + fs.mkdirSync(inboxDir, { recursive: true }); + + // Drop file 1 + const drop1 = path.join(inboxDir, 'dup-1.md'); + fs.writeFileSync(drop1, '# duplicate content\n\nidentical body'); + await waitFor(() => dispatchedEvents.length === 1, 6000); + + // Drop file 2 with byte-identical content (different filename). + const drop2 = path.join(inboxDir, 'dup-2.md'); + fs.writeFileSync(drop2, '# duplicate content\n\nidentical body'); + // chokidar.archive moves drop2, but the dedup should catch the event + // BEFORE the handler runs. Let chokidar process a bit then check. + await new Promise((r) => setTimeout(r, 600)); + + // Only ONE event made it through dispatch (dedup intercepted the second). + expect(dispatchedEvents).toHaveLength(1); + + // The daemon's dedup stats reflect a hit. + const health = await daemon.healthCheck(); + expect(health.dedup.hits).toBeGreaterThanOrEqual(1); + + await daemon.stop(); + }, 15_000); +}); + +describe('ingestion roundtrip — multi-source coordination', () => { + test('two sources see different content; daemon ingests both', async () => { + const { logger } = captureLogger(); + const handler = makeIngestCaptureHandler(engine); + const dispatchedEvents: IngestionEvent[] = []; + + const daemon = new IngestionDaemon({ + engine, + logger, + dispatch: async (event) => { + dispatchedEvents.push(event); + await handler(makeFakeJobCtx({ event })); + return { kind: 'queued' }; + }, + }); + + // Two distinct inbox dirs, two sources. + const inboxA = path.join(tmpRoot, 'inbox-a'); + const inboxB = path.join(tmpRoot, 'inbox-b'); + const sourceA = createInboxFolderSource({ + id: 'inbox-a', + inboxDir: inboxA, + debounceMs: 50, + awaitStabilityMs: 100, + }); + const sourceB = createInboxFolderSource({ + id: 'inbox-b', + inboxDir: inboxB, + debounceMs: 50, + awaitStabilityMs: 100, + }); + daemon.register({ source: sourceA }); + daemon.register({ source: sourceB }); + await daemon.start(); + + fs.writeFileSync(path.join(inboxA, 'from-a.md'), 'content from A'); + fs.writeFileSync(path.join(inboxB, 'from-b.md'), 'content from B'); + + await waitFor(() => dispatchedEvents.length === 2, 6000); + + const fromA = dispatchedEvents.find((e) => e.source_id === 'inbox-a'); + const fromB = dispatchedEvents.find((e) => e.source_id === 'inbox-b'); + expect(fromA).toBeDefined(); + expect(fromB).toBeDefined(); + expect(fromA?.content).toContain('content from A'); + expect(fromB?.content).toContain('content from B'); + + await daemon.stop(); + }, 15_000); +}); diff --git a/test/e2e/serve-http-ingest-webhook.test.ts b/test/e2e/serve-http-ingest-webhook.test.ts new file mode 100644 index 000000000..b0cd819b6 --- /dev/null +++ b/test/e2e/serve-http-ingest-webhook.test.ts @@ -0,0 +1,365 @@ +/** + * v0.38 — E2E HTTP contract tests for POST /ingest, the webhook ingestion + * source registered inside `gbrain serve --http` per the plan-eng-review E1 + * decision (webhook source lives IN serve --http, NOT in the ingestion + * daemon; uses Minion queue as the cross-process sync primitive). + * + * The pre-existing `test/e2e/ingestion-roundtrip.test.ts` covers the + * end-to-end pipeline (event → daemon → ingest_capture → DB) using + * in-process simulation; what it explicitly does NOT cover is "the real + * HTTP route with real OAuth." This file fills that gap. + * + * Spawns a real `gbrain serve --http` against real Postgres, mints OAuth + * tokens with various scopes, and exercises every documented + * status-code branch of the route: + * + * 1. Auth: missing token → 401; read-only token → 403 (write scope + * required by the route) + * 2. Body validation: empty body → 400 with `error: empty_body` + * 3. Content-type allowlist: image/png → 415 with paste-ready + * processor-skillpack hint + * 4. Happy path: text/markdown → 200/202 with job_id in response + * 5. Header overrides: X-Gbrain-Slug is forwarded; X-Gbrain-Source-Id + * tags the event + * 6. Idempotency: same content + same client → job_id returned twice + * should match (queue dedup on (client_id, content_hash)) + * + * Mirrors the spawn + mint pattern from test/e2e/serve-http-oauth.test.ts + * exactly so future maintainers see one pattern, not two. + * + * Run: GBRAIN_DATABASE_URL=... bun test test/e2e/serve-http-ingest-webhook.test.ts + */ + +import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import { hasDatabase } from './helpers.ts'; + +const skip = !hasDatabase(); +const describeE2E = skip ? describe.skip : describe; + +if (skip) { + console.log('Skipping E2E serve-http-ingest-webhook tests (DATABASE_URL not set)'); +} + +const PORT = 19138; // Distinct from sibling E2Es to avoid collision +const BASE = `http://localhost:${PORT}`; + +describeE2E('serve-http POST /ingest webhook (v0.38)', () => { + let serverProcess: ReturnType | null = null; + let clientId: string | undefined; + let clientSecret: string | undefined; + + beforeAll(async () => { + const { execSync, spawn } = await import('child_process'); + + // Register a confidential client with both read and write scopes. + // The write scope is what POST /ingest gates on. + const regOutput = execSync( + 'bun run src/cli.ts auth register-client e2e-webhook-test --grant-types client_credentials --scopes "read write"', + { cwd: process.cwd(), encoding: 'utf8', env: { ...process.env } }, + ); + const idMatch = regOutput.match(/Client ID:\s+(gbrain_cl_\S+)/); + const secretMatch = regOutput.match(/Client Secret:\s+(gbrain_cs_\S+)/); + if (!idMatch || !secretMatch) { + throw new Error('Failed to register webhook test client:\n' + regOutput); + } + clientId = idMatch[1]; + clientSecret = secretMatch[1]; + + serverProcess = spawn( + 'bun', + [ + 'run', + 'src/cli.ts', + 'serve', + '--http', + '--port', + String(PORT), + '--public-url', + `http://localhost:${PORT}`, + ], + { + cwd: process.cwd(), + env: process.env, + stdio: ['ignore', 'pipe', 'pipe'], + }, + ); + + let stderr = ''; + serverProcess.stderr?.on('data', (d: Buffer) => { + stderr += d.toString(); + }); + + // Wait for /health to respond. Up to 15s. + let ready = false; + for (let i = 0; i < 30; i++) { + try { + const res = await fetch(`${BASE}/health`); + if (res.ok) { + ready = true; + break; + } + } catch { + /* not ready yet */ + } + await new Promise(r => setTimeout(r, 500)); + } + if (!ready) { + throw new Error('Webhook E2E server failed to start within 15s.\nstderr: ' + stderr.slice(-500)); + } + }, 30_000); + + afterAll(async () => { + if (serverProcess) { + serverProcess.kill('SIGTERM'); + await new Promise(r => setTimeout(r, 1000)); + if (!serverProcess.killed) serverProcess.kill('SIGKILL'); + } + if (clientId) { + try { + const { execSync } = await import('child_process'); + execSync(`bun run src/cli.ts auth revoke-client "${clientId}"`, { + cwd: process.cwd(), + encoding: 'utf8', + env: { ...process.env }, + }); + } catch (e) { + // eslint-disable-next-line no-console + console.error(`[afterAll] revoke-client cleanup failed: ${(e as Error).message}`); + } + } + }, 30_000); + + // Helper — mint a token with a specific scope subset. + async function mintToken(scope = 'read write'): Promise { + const res = await fetch(`${BASE}/token`, { + method: 'POST', + headers: { 'Content-Type': 'application/x-www-form-urlencoded' }, + body: `grant_type=client_credentials&client_id=${clientId}&client_secret=${clientSecret}&scope=${encodeURIComponent(scope)}`, + }); + expect(res.ok).toBe(true); + const data = (await res.json()) as { access_token: string }; + return data.access_token; + } + + // Helper — POST to /ingest with the given Authorization + Content-Type. + async function postIngest( + token: string | null, + contentType: string, + body: string | Uint8Array, + extraHeaders: Record = {}, + ): Promise { + const headers: Record = { + 'Content-Type': contentType, + ...extraHeaders, + }; + if (token !== null) headers.Authorization = `Bearer ${token}`; + return fetch(`${BASE}/ingest`, { + method: 'POST', + headers, + body: body as BodyInit, + }); + } + + // ========================================================================= + // Auth gate + // ========================================================================= + + test('missing Authorization header → 401 (route is OAuth-gated)', async () => { + const res = await postIngest(null, 'text/markdown', '# unauth attempt'); + expect(res.status).toBe(401); + }); + + test('read-only token → 403 (route requires write scope)', async () => { + const readToken = await mintToken('read'); + const res = await postIngest(readToken, 'text/markdown', '# read-only attempt'); + // Spec: requireBearerAuth with requiredScopes=['write'] returns 403 + // when the bearer scope set lacks write. SDK may return 401 or 403 + // depending on version; either is a refusal. + expect([401, 403]).toContain(res.status); + const body = await res.text(); + // Successful ingest would carry job_id; failure must not. + expect(body).not.toMatch(/"job_id"\s*:\s*"?\d+/); + }); + + test('valid write-scope token accepts text/markdown → 200/202 with job_id', async () => { + const token = await mintToken('read write'); + const res = await postIngest( + token, + 'text/markdown', + `# webhook happy path\n\nIngested at ${new Date().toISOString()}`, + ); + expect([200, 202]).toContain(res.status); + const body = (await res.json()) as { job_id?: number | string; ok?: boolean }; + expect(body.job_id).toBeDefined(); + }); + + // ========================================================================= + // Body validation + // ========================================================================= + + test('empty body → 400 with error: empty_body', async () => { + const token = await mintToken('read write'); + const res = await postIngest(token, 'text/markdown', ''); + expect(res.status).toBe(400); + const body = (await res.json()) as { error?: string; message?: string }; + expect(body.error).toBe('empty_body'); + expect(body.message?.toLowerCase()).toContain('non-empty'); + }); + + // ========================================================================= + // Content-type allowlist (the v0.38 webhook taxonomy) + // ========================================================================= + + test('binary image/png → 415 with paste-ready processor-skillpack hint', async () => { + const token = await mintToken('read write'); + // PNG magic bytes — a real (tiny) PNG header + const png = new Uint8Array([ + 0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0x00, 0x00, 0x00, 0x0d, + ]); + const res = await postIngest(token, 'image/png', png); + expect(res.status).toBe(415); + const body = (await res.json()) as { error?: string; message?: string }; + expect(body.error).toBe('unsupported_content_type'); + // The hint should mention the path forward (skillpack processor). + expect(body.message?.toLowerCase()).toMatch(/skillpack|processor|not yet supported/); + }); + + test('application/pdf → 415 (binary processor deferred)', async () => { + const token = await mintToken('read write'); + const pdfMagic = new Uint8Array([0x25, 0x50, 0x44, 0x46]); // %PDF + const res = await postIngest(token, 'application/pdf', pdfMagic); + expect(res.status).toBe(415); + }); + + test('text/plain accepted (in the v1 allowlist)', async () => { + const token = await mintToken('read write'); + const res = await postIngest( + token, + 'text/plain', + `plain text webhook ${Date.now()}`, + ); + expect([200, 202]).toContain(res.status); + const body = (await res.json()) as { job_id?: number | string }; + expect(body.job_id).toBeDefined(); + }); + + test('application/json accepted (in the v1 allowlist)', async () => { + const token = await mintToken('read write'); + const res = await postIngest( + token, + 'application/json', + JSON.stringify({ kind: 'webhook-event', when: Date.now() }), + ); + expect([200, 202]).toContain(res.status); + }); + + test('text/html accepted (in the v1 allowlist)', async () => { + const token = await mintToken('read write'); + const res = await postIngest( + token, + 'text/html', + `

html webhook ${Date.now()}

`, + ); + expect([200, 202]).toContain(res.status); + }); + + test('unknown text/* sub-type passes through as text/plain', async () => { + const token = await mintToken('read write'); + const res = await postIngest(token, 'text/x-custom', 'unknown text variant'); + // The route maps unknown text/* to text/plain rather than 415. + expect([200, 202]).toContain(res.status); + }); + + test('X-Gbrain-Content-Type header overrides request Content-Type', async () => { + const token = await mintToken('read write'); + // Send as application/octet-stream (would 415) but override to text/markdown. + const res = await postIngest( + token, + 'application/octet-stream', + '# override via header', + { 'X-Gbrain-Content-Type': 'text/markdown' }, + ); + // With override: route should accept as markdown. + expect([200, 202]).toContain(res.status); + }); + + // ========================================================================= + // Header overrides + // ========================================================================= + + test('X-Gbrain-Slug header is accepted (job receives the slug hint)', async () => { + const token = await mintToken('read write'); + const slug = `webhook/test/header-${Date.now()}`; + const res = await postIngest( + token, + 'text/markdown', + '# slug header test', + { 'X-Gbrain-Slug': slug }, + ); + expect([200, 202]).toContain(res.status); + // The route should accept the header without rejecting — actual slug + // application happens inside the ingest_capture handler (covered by + // test/ingestion/ingest-capture.test.ts). + }); + + test('X-Gbrain-Source-Id header is accepted', async () => { + const token = await mintToken('read write'); + const res = await postIngest( + token, + 'text/markdown', + '# source-id header test', + { 'X-Gbrain-Source-Id': 'zapier-webhook' }, + ); + expect([200, 202]).toContain(res.status); + }); + + test('X-Gbrain-Source-Uri header is accepted', async () => { + const token = await mintToken('read write'); + const res = await postIngest( + token, + 'text/markdown', + '# source-uri header test', + { 'X-Gbrain-Source-Uri': 'https://example.com/issue/123' }, + ); + expect([200, 202]).toContain(res.status); + }); + + // ========================================================================= + // Idempotency + // ========================================================================= + + test('same content from same client → identical job_id (queue dedup on content_hash)', async () => { + const token = await mintToken('read write'); + const content = `# idempotency test ${Math.random()}`; + const first = await postIngest(token, 'text/markdown', content); + expect([200, 202]).toContain(first.status); + const firstBody = (await first.json()) as { job_id?: number | string }; + + const second = await postIngest(token, 'text/markdown', content); + expect([200, 202]).toContain(second.status); + const secondBody = (await second.json()) as { job_id?: number | string }; + + // Queue idempotency_key: `ingest:webhook:${clientId}:${contentHash}` — + // same input, same key, MinionQueue.add returns the existing job. + expect(secondBody.job_id).toBe(firstBody.job_id!); + }); + + test('different content from same client → different job_id', async () => { + const token = await mintToken('read write'); + const first = await postIngest( + token, + 'text/markdown', + `# distinct A ${Date.now()}`, + ); + const second = await postIngest( + token, + 'text/markdown', + `# distinct B ${Date.now()}`, + ); + const firstBody = (await first.json()) as { job_id?: number | string }; + const secondBody = (await second.json()) as { job_id?: number | string }; + expect(firstBody.job_id).toBeDefined(); + expect(secondBody.job_id).toBeDefined(); + expect(secondBody.job_id).not.toBe(firstBody.job_id); + }); +}); diff --git a/test/ingestion/daemon.test.ts b/test/ingestion/daemon.test.ts new file mode 100644 index 000000000..a9612de96 --- /dev/null +++ b/test/ingestion/daemon.test.ts @@ -0,0 +1,513 @@ +/** + * IngestionDaemon tests — supervision + dispatch + dedup + rate-limit. + * + * The daemon composes many primitives (validation, dedup, rate-limit, + * supervision, dispatcher). These tests exercise each branch in isolation + * by injecting fake sources, a fake dispatcher, and a fake clock. + * + * Engine is the throwing-proxy from test-harness (sources don't touch it + * in these tests; if any does, fail loud). + */ + +import { describe, expect, test } from 'bun:test'; +import { + IngestionDaemon, + type DispatchOutcome, + type IngestionDispatcher, +} from '../../src/core/ingestion/daemon.ts'; +import { + computeContentHash, + type IngestionEvent, + type IngestionSource, + type IngestionSourceContext, +} from '../../src/core/ingestion/types.ts'; +import type { BrainEngine } from '../../src/core/engine.ts'; +import type { Logger } from '../../src/core/operations.ts'; + +// Engine proxy that throws on access. The daemon doesn't dereference it +// (only passes it through to source ctx); sources in these tests are +// minimal and don't touch it. +function makeThrowingEngine(): BrainEngine { + return new Proxy({} as BrainEngine, { + get(_target, prop) { + if (prop === 'then' || prop === Symbol.toPrimitive || prop === Symbol.toStringTag) { + return undefined; + } + throw new Error(`Engine access not allowed in test: ${String(prop)}`); + }, + }); +} + +function makeCaptureLogger(): { logger: Logger; messages: Array<{ level: string; msg: string }> } { + const messages: Array<{ level: string; msg: string }> = []; + return { + logger: { + info: (msg) => messages.push({ level: 'info', msg }), + warn: (msg) => messages.push({ level: 'warn', msg }), + error: (msg) => messages.push({ level: 'error', msg }), + }, + messages, + }; +} + +function makeRecordingDispatcher(): { + dispatch: IngestionDispatcher; + recorded: IngestionEvent[]; +} { + const recorded: IngestionEvent[] = []; + return { + dispatch: async (event) => { + recorded.push(event); + return { kind: 'queued' as const, jobId: recorded.length }; + }, + recorded, + }; +} + +function makeEvent(overrides: Partial = {}): IngestionEvent { + const content = overrides.content ?? '# default'; + return { + source_id: 'src-1', + source_kind: 'mock', + source_uri: '/tmp/x.md', + received_at: new Date('2026-05-20T12:00:00Z').toISOString(), + content_type: 'text/markdown', + content, + content_hash: overrides.content_hash ?? computeContentHash(content), + ...overrides, + }; +} + +/** Build a mock source that records the ctx it receives and exposes a + * push() method for the test to emit events through. */ +function makeControllableSource(id = 'src-1', kind = 'mock'): { + source: IngestionSource; + ctx: { current: IngestionSourceContext | null }; + startedAt: { current: number }; + push: (event: IngestionEvent) => void; + stopped: { current: boolean }; +} { + const ctxRef: { current: IngestionSourceContext | null } = { current: null }; + const startedAt = { current: 0 }; + const stopped = { current: false }; + + const source: IngestionSource = { + id, + kind, + async start(ctx) { + ctxRef.current = ctx; + startedAt.current = Date.now(); + }, + async stop() { + stopped.current = true; + }, + }; + + return { + source, + ctx: ctxRef, + startedAt, + stopped, + push: (event) => { + if (!ctxRef.current) throw new Error('source not started yet'); + ctxRef.current.emit(event); + }, + }; +} + +async function flushMicrotasks(times = 3): Promise { + // The daemon dispatches events via Promise.resolve().then(handleEmit). + // Yield to the microtask queue several times to ensure handleEmit + any + // async work inside it (dispatcher promise) has settled. + for (let i = 0; i < times; i++) { + await Promise.resolve(); + await new Promise((r) => setTimeout(r, 0)); + } +} + +describe('IngestionDaemon — registration', () => { + test('register adds a source', () => { + const { logger } = makeCaptureLogger(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + const { source } = makeControllableSource('s1'); + daemon.register({ source }); + expect(daemon._stateForTest('s1')?.registration.source).toBe(source); + }); + + test('register rejects duplicate id', () => { + const { logger } = makeCaptureLogger(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + daemon.register({ source: makeControllableSource('dup').source }); + expect(() => + daemon.register({ source: makeControllableSource('dup').source }), + ).toThrow(/duplicate source id/); + }); + + test('register rejects after start()', async () => { + const { logger } = makeCaptureLogger(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + daemon.register({ source: makeControllableSource('a').source }); + await daemon.start(); + expect(() => + daemon.register({ source: makeControllableSource('b').source }), + ).toThrow(/after daemon has started/); + await daemon.stop(); + }); +}); + +describe('IngestionDaemon — lifecycle', () => { + test('start() calls source.start with the daemon ctx', async () => { + const { logger } = makeCaptureLogger(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + const { source, ctx } = makeControllableSource('s1'); + daemon.register({ source }); + await daemon.start(); + expect(ctx.current).not.toBeNull(); + expect(daemon.running).toBe(true); + await daemon.stop(); + }); + + test('stop() fires abort signal and calls source.stop', async () => { + const { logger } = makeCaptureLogger(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + const { source, ctx, stopped } = makeControllableSource('s1'); + daemon.register({ source }); + await daemon.start(); + expect(ctx.current?.abortSignal.aborted).toBe(false); + await daemon.stop(); + expect(ctx.current?.abortSignal.aborted).toBe(true); + expect(stopped.current).toBe(true); + expect(daemon.running).toBe(false); + }); + + test('start() throwing source enters supervisor retry loop and respects maxCrashes', async () => { + const { logger, messages } = makeCaptureLogger(); + let starts = 0; + const source: IngestionSource = { + id: 'crashy', + kind: 'mock', + async start() { + starts++; + throw new Error('boom'); + }, + async stop() {}, + }; + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + supervision: { maxCrashes: 3, initialBackoffMs: 1, maxBackoffMs: 10, stableRunResetMs: 60_000 }, + }); + daemon.register({ source }); + await daemon.start(); + expect(starts).toBe(3); // crashed maxCrashes times + const state = daemon._stateForTest('crashy'); + expect(state?.exhausted).toBe(true); + expect(state?.lastError).toBe('boom'); + // Three warns (one per crash) + one error when exhausted. + expect(messages.filter((m) => m.level === 'warn').length).toBeGreaterThanOrEqual(3); + expect(messages.some((m) => m.level === 'error' && m.msg.includes('exhausted'))).toBe(true); + await daemon.stop(); + }); + + test('multiple sources start independently; one crashing doesn\'t block others', async () => { + const { logger } = makeCaptureLogger(); + const goodSource = makeControllableSource('good', 'mock-a'); + const badSource: IngestionSource = { + id: 'bad', + kind: 'mock-b', + async start() { throw new Error('bad'); }, + async stop() {}, + }; + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + supervision: { maxCrashes: 2, initialBackoffMs: 1, maxBackoffMs: 10, stableRunResetMs: 60_000 }, + }); + daemon.register({ source: goodSource.source }); + daemon.register({ source: badSource }); + await daemon.start(); + expect(goodSource.ctx.current).not.toBeNull(); // good started + expect(daemon._stateForTest('bad')?.exhausted).toBe(true); + expect(daemon._stateForTest('good')?.started).toBe(true); + await daemon.stop(); + }); +}); + +describe('IngestionDaemon — dispatch pipeline', () => { + test('valid emit reaches dispatcher with source_id/source_kind normalized to source identity', async () => { + const { logger } = makeCaptureLogger(); + const { dispatch, recorded } = makeRecordingDispatcher(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch, + }); + const { source, push } = makeControllableSource('expected-id', 'expected-kind'); + daemon.register({ source }); + await daemon.start(); + // Source emits with WRONG identity fields — daemon overrides them. + push(makeEvent({ source_id: 'WRONG', source_kind: 'WRONG' })); + await flushMicrotasks(); + expect(recorded).toHaveLength(1); + expect(recorded[0]?.source_id).toBe('expected-id'); + expect(recorded[0]?.source_kind).toBe('expected-kind'); + await daemon.stop(); + }); + + test('invalid event is dropped at validation, logged as warn', async () => { + const { logger, messages } = makeCaptureLogger(); + const { dispatch, recorded } = makeRecordingDispatcher(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch, + }); + const { source, push } = makeControllableSource('s', 'mock'); + daemon.register({ source }); + await daemon.start(); + push({ ...makeEvent(), content_hash: 'short' }); + await flushMicrotasks(); + expect(recorded).toHaveLength(0); + expect(messages.some((m) => m.level === 'warn' && m.msg.includes('invalid event'))).toBe(true); + await daemon.stop(); + }); + + test('duplicate content_hash within window is silent-dedup\'d', async () => { + const { logger } = makeCaptureLogger(); + const { dispatch, recorded } = makeRecordingDispatcher(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch, + }); + const { source, push } = makeControllableSource('s', 'mock'); + daemon.register({ source }); + await daemon.start(); + const ev = makeEvent({ content: 'dup-content' }); + push(ev); + push(ev); + push(ev); + await flushMicrotasks(); + expect(recorded).toHaveLength(1); + const h = await daemon.healthCheck(); + expect(h.dedup.hits).toBeGreaterThanOrEqual(2); + await daemon.stop(); + }); + + test('rate limit drops events past capacity within the window', async () => { + const { logger, messages } = makeCaptureLogger(); + const { dispatch, recorded } = makeRecordingDispatcher(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch, + rateLimit: { capacity: 3, windowMs: 1000 }, + _now: () => 1000, + }); + const { source, push } = makeControllableSource('s', 'mock'); + daemon.register({ source }); + await daemon.start(); + // Emit 5 unique events; only the first 3 should make it through. + for (let i = 0; i < 5; i++) { + push(makeEvent({ content: `unique-${i}` })); + } + await flushMicrotasks(); + expect(recorded).toHaveLength(3); + expect(daemon._stateForTest('s')?.rateLimitHits).toBe(2); + expect(messages.some((m) => m.level === 'warn' && m.msg.includes('rate limit hit'))).toBe(true); + await daemon.stop(); + }); + + test('dispatcher failure is logged but does not crash the daemon', async () => { + const { logger, messages } = makeCaptureLogger(); + let calls = 0; + const dispatch: IngestionDispatcher = async () => { + calls++; + return { kind: 'failed', error: 'queue write failed' } satisfies DispatchOutcome; + }; + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch, + }); + const { source, push } = makeControllableSource('s', 'mock'); + daemon.register({ source }); + await daemon.start(); + push(makeEvent({ content: 'one' })); + await flushMicrotasks(); + expect(calls).toBe(1); + expect(messages.some((m) => m.level === 'warn' && m.msg.includes('dispatch failed'))).toBe(true); + expect(daemon.running).toBe(true); + await daemon.stop(); + }); + + test('dispatcher throwing is caught and logged as error', async () => { + const { logger, messages } = makeCaptureLogger(); + const dispatch: IngestionDispatcher = async () => { + throw new Error('rogue dispatcher'); + }; + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch, + }); + const { source, push } = makeControllableSource('s', 'mock'); + daemon.register({ source }); + await daemon.start(); + push(makeEvent()); + await flushMicrotasks(); + expect(messages.some((m) => m.level === 'error' && m.msg.includes('rogue dispatcher'))).toBe(true); + expect(daemon.running).toBe(true); + await daemon.stop(); + }); +}); + +describe('IngestionDaemon — health check', () => { + test('reports ok for a clean source', async () => { + const { logger } = makeCaptureLogger(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + daemon.register({ source: makeControllableSource('s').source }); + await daemon.start(); + const h = await daemon.healthCheck(); + expect(h.status).toBe('ok'); + expect(h.sources).toHaveLength(1); + expect(h.sources[0]?.status).toBe('ok'); + await daemon.stop(); + }); + + test('reports fail when a source exhausted maxCrashes', async () => { + const { logger } = makeCaptureLogger(); + const crashy: IngestionSource = { + id: 'crashy', kind: 'mock', + async start() { throw new Error('always'); }, + async stop() {}, + }; + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + supervision: { maxCrashes: 2, initialBackoffMs: 1, maxBackoffMs: 5, stableRunResetMs: 60_000 }, + }); + daemon.register({ source: crashy }); + await daemon.start(); + const h = await daemon.healthCheck(); + expect(h.status).toBe('fail'); + expect(h.sources[0]?.status).toBe('fail'); + expect(h.sources[0]?.message).toContain('always'); + await daemon.stop(); + }); + + test('forwards source healthCheck() result', async () => { + const { logger } = makeCaptureLogger(); + const source: IngestionSource = { + id: 's', kind: 'mock', + async start() {}, + async stop() {}, + async healthCheck() { + return { status: 'warn' as const, message: 'API near rate limit' }; + }, + }; + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + daemon.register({ source }); + await daemon.start(); + const h = await daemon.healthCheck(); + expect(h.status).toBe('warn'); + expect(h.sources[0]?.status).toBe('warn'); + expect(h.sources[0]?.message).toBe('API near rate limit'); + await daemon.stop(); + }); + + test('aggregates dedup stats', async () => { + const { logger } = makeCaptureLogger(); + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + const { source, push } = makeControllableSource('s', 'mock'); + daemon.register({ source }); + await daemon.start(); + const ev = makeEvent({ content: 'one' }); + push(ev); + push(ev); // dedup + await flushMicrotasks(); + const h = await daemon.healthCheck(); + expect(h.dedup.total).toBeGreaterThan(0); + expect(h.dedup.hits).toBeGreaterThanOrEqual(1); + await daemon.stop(); + }); +}); + +describe('IngestionDaemon — config passthrough', () => { + test('source receives the config registered with it', async () => { + const { logger } = makeCaptureLogger(); + let observedConfig: Record | null = null; + const source: IngestionSource = { + id: 's', kind: 'mock', + async start(ctx) { + observedConfig = ctx.config; + }, + async stop() {}, + }; + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + daemon.register({ source, config: { apiKey: 'test', threshold: 42 } }); + await daemon.start(); + expect(observedConfig).not.toBeNull(); + expect(observedConfig!).toEqual({ apiKey: 'test', threshold: 42 }); + await daemon.stop(); + }); + + test('source logger is wrapped with per-source prefix', async () => { + const { logger, messages } = makeCaptureLogger(); + const source: IngestionSource = { + id: 'named-src', kind: 'mock', + async start(ctx) { + ctx.logger.info('hello from inside'); + }, + async stop() {}, + }; + const daemon = new IngestionDaemon({ + engine: makeThrowingEngine(), + logger, + dispatch: async () => ({ kind: 'queued' }), + }); + daemon.register({ source }); + await daemon.start(); + expect(messages.some((m) => m.level === 'info' && m.msg === '[ingestion:named-src] hello from inside')).toBe(true); + await daemon.stop(); + }); +}); diff --git a/test/ingestion/dedup.test.ts b/test/ingestion/dedup.test.ts new file mode 100644 index 000000000..62c73d508 --- /dev/null +++ b/test/ingestion/dedup.test.ts @@ -0,0 +1,160 @@ +/** + * DedupWindow tests — 24h content-hash LRU. + * + * Daemon's defense against duplicate events from overlapping sources and + * at-least-once-emit semantics from the source supervisor. + */ + +import { describe, expect, test } from 'bun:test'; +import { DedupWindow } from '../../src/core/ingestion/dedup.ts'; + +const HASH_A = 'a'.repeat(64); +const HASH_B = 'b'.repeat(64); +const HASH_C = 'c'.repeat(64); + +describe('DedupWindow.mark — basic dedup', () => { + test('first sight of a key returns true', () => { + const w = new DedupWindow({ _now: () => 1000 }); + expect(w.mark('file-watcher', HASH_A)).toBe(true); + }); + + test('repeat of the same key within TTL returns false', () => { + let t = 1000; + const w = new DedupWindow({ _now: () => t }); + expect(w.mark('file-watcher', HASH_A)).toBe(true); + t = 2000; + expect(w.mark('file-watcher', HASH_A)).toBe(false); + }); + + test('same hash from different source kinds are independent keys', () => { + const w = new DedupWindow({ _now: () => 1000 }); + expect(w.mark('file-watcher', HASH_A)).toBe(true); + expect(w.mark('inbox-folder', HASH_A)).toBe(true); + expect(w.mark('file-watcher', HASH_A)).toBe(false); + }); + + test('past-TTL repeat is treated as new', () => { + let t = 1000; + const w = new DedupWindow({ ttlMs: 1000, _now: () => t }); + expect(w.mark('x', HASH_A)).toBe(true); + t = 5000; // far past 1s TTL + expect(w.mark('x', HASH_A)).toBe(true); + }); + + test('within-TTL hit at boundary still dedups', () => { + let t = 1000; + const w = new DedupWindow({ ttlMs: 1000, _now: () => t }); + expect(w.mark('x', HASH_A)).toBe(true); + t = 1999; // 1ms before TTL expiry + expect(w.mark('x', HASH_A)).toBe(false); + }); +}); + +describe('DedupWindow.mark — LRU eviction', () => { + test('exceeding maxEntries evicts the oldest', () => { + const w = new DedupWindow({ maxEntries: 3, _now: () => 1000 }); + w.mark('x', HASH_A); // entry 1 + w.mark('x', HASH_B); // entry 2 + w.mark('x', HASH_C); // entry 3 — at capacity + w.mark('y', HASH_A); // entry 4 — should evict 'x:HASH_A' + + // HASH_A under kind 'x' should now be treated as new because it was + // evicted, not because TTL expired. + expect(w.mark('x', HASH_A)).toBe(true); + expect(w.stats().evictions).toBeGreaterThan(0); + }); + + test('touching an existing key moves it to MRU position', () => { + const w = new DedupWindow({ maxEntries: 3, _now: () => 1000 }); + w.mark('x', HASH_A); // entry 1 (oldest) + w.mark('x', HASH_B); + w.mark('x', HASH_C); + + // Re-touch HASH_A — should move it to MRU. + expect(w.mark('x', HASH_A)).toBe(false); // dedup hit + + // Now insert a new entry. The eviction should pick HASH_B (now oldest), not HASH_A. + w.mark('y', HASH_A); + expect(w.mark('x', HASH_B)).toBe(true); // HASH_B was evicted + expect(w.mark('x', HASH_A)).toBe(false); // HASH_A still in cache (recently touched) + }); +}); + +describe('DedupWindow.prune', () => { + test('removes entries older than TTL', () => { + let t = 1000; + const w = new DedupWindow({ ttlMs: 1000, _now: () => t }); + w.mark('x', HASH_A); + w.mark('x', HASH_B); + + t = 3000; // both entries should be expired + const removed = w.prune(); + expect(removed).toBe(2); + expect(w.stats().size).toBe(0); + }); + + test('preserves entries within TTL', () => { + let t = 1000; + const w = new DedupWindow({ ttlMs: 1000, _now: () => t }); + w.mark('x', HASH_A); // inserted at t=1000 + t = 1500; + w.mark('x', HASH_B); // inserted at t=1500 + t = 2200; // HASH_A is past TTL (1200 cutoff), HASH_B still in window + + const removed = w.prune(); + expect(removed).toBe(1); + expect(w.stats().size).toBe(1); + }); + + test('explicit now parameter overrides clock', () => { + const w = new DedupWindow({ ttlMs: 1000, _now: () => 1000 }); + w.mark('x', HASH_A); + expect(w.prune(3000)).toBe(1); + }); +}); + +describe('DedupWindow.stats', () => { + test('total counts every mark call', () => { + const w = new DedupWindow({ _now: () => 1000 }); + w.mark('x', HASH_A); + w.mark('x', HASH_A); + w.mark('x', HASH_B); + expect(w.stats().total).toBe(3); + }); + + test('hits counts dedup hits', () => { + const w = new DedupWindow({ _now: () => 1000 }); + w.mark('x', HASH_A); + w.mark('x', HASH_A); + w.mark('x', HASH_A); + expect(w.stats().hits).toBe(2); + }); + + test('size tracks live entries', () => { + const w = new DedupWindow({ _now: () => 1000 }); + expect(w.stats().size).toBe(0); + w.mark('x', HASH_A); + expect(w.stats().size).toBe(1); + w.mark('x', HASH_B); + expect(w.stats().size).toBe(2); + }); + + test('evictions count LRU evictions, not TTL prunes', () => { + const w = new DedupWindow({ maxEntries: 2, _now: () => 1000 }); + w.mark('x', HASH_A); + w.mark('x', HASH_B); + w.mark('x', HASH_C); // forces eviction of HASH_A + expect(w.stats().evictions).toBe(1); + }); +}); + +describe('DedupWindow._resetForTest', () => { + test('clears all state', () => { + const w = new DedupWindow({ _now: () => 1000 }); + w.mark('x', HASH_A); + w.mark('x', HASH_A); // hit + w._resetForTest(); + expect(w.stats()).toEqual({ total: 0, hits: 0, evictions: 0, size: 0 }); + expect(w.mark('x', HASH_A)).toBe(true); + }); +}); diff --git a/test/ingestion/ingest-capture.test.ts b/test/ingestion/ingest-capture.test.ts new file mode 100644 index 000000000..6bfc89340 --- /dev/null +++ b/test/ingestion/ingest-capture.test.ts @@ -0,0 +1,192 @@ +/** + * ingest_capture Minion handler tests. Exercises the slug-resolution + * fallback chain, content-type gating (binary rejection), validation, + * and the importFromContent integration against an in-memory PGLite. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { + defaultSlugForEvent, + makeIngestCaptureHandler, +} from '../../src/core/minions/handlers/ingest-capture.ts'; +import { + computeContentHash, + type IngestionEvent, +} from '../../src/core/ingestion/types.ts'; +import type { MinionJobContext } from '../../src/core/minions/types.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +function makeEvent(overrides: Partial = {}): IngestionEvent { + const content = overrides.content ?? '# captured thought'; + return { + source_id: 'webhook-test', + source_kind: 'webhook', + source_uri: 'mcp-webhook:client-x:1234', + received_at: new Date('2026-05-20T12:00:00Z').toISOString(), + content_type: 'text/markdown', + content, + content_hash: overrides.content_hash ?? computeContentHash(content), + ...overrides, + }; +} + +function makeJob(data: Record): MinionJobContext { + return { + id: 1, + name: 'ingest_capture', + data, + attempts_made: 1, + signal: new AbortController().signal, + shutdownSignal: new AbortController().signal, + updateProgress: async () => {}, + updateTokens: async () => {}, + log: async () => {}, + isActive: async () => true, + readInbox: async () => [], + }; +} + +describe('defaultSlugForEvent', () => { + test('builds inbox/YYYY-MM-DD- slug', () => { + const ev = makeEvent({ content_hash: 'abcdef1234567890'.padEnd(64, '0') }); + const slug = defaultSlugForEvent(ev, new Date('2026-05-20T00:00:00Z')); + expect(slug).toBe('inbox/2026-05-20-abcdef'); + }); + + test('stable for same content (deterministic hash)', () => { + const ev = makeEvent({ content: 'same thought' }); + const date = new Date('2026-05-20T00:00:00Z'); + expect(defaultSlugForEvent(ev, date)).toBe(defaultSlugForEvent(ev, date)); + }); + + test('UTC date math (no tz drift)', () => { + const ev = makeEvent(); + const slug = defaultSlugForEvent(ev, new Date('2026-01-05T23:59:59Z')); + expect(slug).toMatch(/^inbox\/2026-01-05-/); + }); +}); + +describe('ingest_capture handler — slug resolution', () => { + test('uses caller-provided job.data.slug when present', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ content: 'with explicit slug' }); + const result = await handler(makeJob({ event: ev, slug: 'wiki/specific/page' })); + expect(result.slug).toBe('wiki/specific/page'); + expect(result.status).toBe('imported'); + }); + + test('uses event.metadata.slug when set', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ content: 'metadata slug', metadata: { slug: 'inbox/custom-from-meta' } }); + const result = await handler(makeJob({ event: ev })); + expect(result.slug).toBe('inbox/custom-from-meta'); + }); + + test('falls back to inbox/YYYY-MM-DD- when no slug provided', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ content: 'fallback slug' }); + const result = await handler(makeJob({ event: ev })); + expect(result.slug).toMatch(/^inbox\/\d{4}-\d{2}-\d{2}-[a-f0-9]{6}$/); + }); +}); + +describe('ingest_capture handler — validation + routing', () => { + test('throws when event missing', async () => { + const handler = makeIngestCaptureHandler(engine); + await expect(handler(makeJob({}))).rejects.toThrow(/job.data.event is required/); + }); + + test('throws on invalid event payload (caught at the handler boundary)', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = { ...makeEvent(), content_hash: 'short' }; + await expect(handler(makeJob({ event: ev }))).rejects.toThrow(/invalid event payload/); + }); + + test('rejects binary content_type with helpful message', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ + content_type: 'image/*', + content: '/path/to/screenshot.png', + content_hash: computeContentHash('/path/to/screenshot.png'), + }); + await expect(handler(makeJob({ event: ev }))).rejects.toThrow( + /content_type 'image\/\*' requires a content-type processor/, + ); + }); + + test('untrusted_payload flag round-trips to the result', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ content: 'untrusted', untrusted_payload: true }); + const result = await handler(makeJob({ event: ev })); + expect(result.untrusted_payload).toBe(true); + }); + + test('trusted (default) payload round-trips as false', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ content: 'trusted' }); + const result = await handler(makeJob({ event: ev })); + expect(result.untrusted_payload).toBe(false); + }); + + test('source provenance round-trips into the result', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ + content: 'with provenance', + source_kind: 'inbox-folder', + source_uri: '/Users/test/.gbrain/inbox/note.md', + }); + const result = await handler(makeJob({ event: ev })); + expect(result.source_kind).toBe('inbox-folder'); + expect(result.source_uri).toBe('/Users/test/.gbrain/inbox/note.md'); + }); +}); + +describe('ingest_capture handler — integration with importFromContent', () => { + test('imported event lands as a page in the DB', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ + content: '---\ntitle: Test Page\n---\n\n# E2E import\n\nbody content', + }); + const result = await handler(makeJob({ event: ev, slug: 'wiki/e2e-test' })); + expect(result.status).toBe('imported'); + + const page = await engine.getPage('wiki/e2e-test'); + expect(page).not.toBeNull(); + expect(page?.compiled_truth).toContain('E2E import'); + }); + + test('repeat ingest of same content returns skipped status (content_hash dedup at importFromContent level)', async () => { + const handler = makeIngestCaptureHandler(engine); + const ev = makeEvent({ content: '# stable content' }); + const result1 = await handler(makeJob({ event: ev, slug: 'wiki/stable' })); + expect(result1.status).toBe('imported'); + + const result2 = await handler(makeJob({ event: ev, slug: 'wiki/stable' })); + expect(result2.status).toBe('skipped'); + }); + + test('chunks count is reported on imported events', async () => { + const handler = makeIngestCaptureHandler(engine); + const longContent = '---\ntitle: long\n---\n\n' + 'Paragraph.\n\n'.repeat(50); + const ev = makeEvent({ content: longContent }); + const result = await handler(makeJob({ event: ev, slug: 'wiki/long' })); + expect(result.chunks).toBeGreaterThan(0); + }); +}); diff --git a/test/ingestion/put-page-write-through.test.ts b/test/ingestion/put-page-write-through.test.ts new file mode 100644 index 000000000..37236439f --- /dev/null +++ b/test/ingestion/put-page-write-through.test.ts @@ -0,0 +1,226 @@ +/** + * put_page write-through tests (v0.38). + * + * Verifies that put_page writes the markdown file to disk alongside the + * DB row when sync.repo_path is configured. Trust gating: subagent + * sandbox writes stay DB-only; dry-run stays DB-only; missing-repo + * stays DB-only. + */ + +import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { PGLiteEngine } from '../../src/core/pglite-engine.ts'; +import { resetPgliteState } from '../helpers/reset-pglite.ts'; +import { operations } from '../../src/core/operations.ts'; +import type { OperationContext } from '../../src/core/operations.ts'; + +let engine: PGLiteEngine; +let tmpRoot: string; +let brainDir: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); + tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-wt-')); + brainDir = path.join(tmpRoot, 'brain'); + fs.mkdirSync(brainDir, { recursive: true }); + // Wire sync.repo_path so write-through can find the repo. + await engine.setConfig('sync.repo_path', brainDir); +}); + +afterEach(() => { + fs.rmSync(tmpRoot, { recursive: true, force: true }); +}); + +const captureLogger = () => { + const messages: Array<{ level: string; msg: string }> = []; + return { + logger: { + info: (msg: string) => messages.push({ level: 'info', msg }), + warn: (msg: string) => messages.push({ level: 'warn', msg }), + error: (msg: string) => messages.push({ level: 'error', msg }), + }, + messages, + }; +}; + +function makeCtx(overrides: Partial = {}): OperationContext { + const { logger } = captureLogger(); + return { + engine, + config: { engine: 'pglite' as const }, + logger, + dryRun: false, + remote: false, + sourceId: 'default', + ...overrides, + }; +} + +const putPage = operations.find((o) => o.name === 'put_page')!; + +describe('put_page write-through — happy path', () => { + test('writes the markdown file to disk at brainDir/.md', async () => { + const ctx = makeCtx(); + const content = '---\ntitle: Test\n---\n\n# WT body'; + const result = (await putPage.handler(ctx, { slug: 'inbox/test-wt-1', content })) as { + slug: string; + write_through?: { written: boolean; path?: string }; + }; + expect(result.write_through?.written).toBe(true); + const expectedPath = path.join(brainDir, 'inbox/test-wt-1.md'); + expect(result.write_through?.path).toBe(expectedPath); + expect(fs.existsSync(expectedPath)).toBe(true); + const onDisk = fs.readFileSync(expectedPath, 'utf8'); + expect(onDisk).toContain('WT body'); + }); + + test('stamps provenance frontmatter (ingested_via=put_page for local CLI)', async () => { + const ctx = makeCtx({ remote: false }); + const result = (await putPage.handler(ctx, { + slug: 'inbox/provenance', + content: '---\ntitle: P\n---\n\nbody', + })) as { write_through?: { written: boolean; path?: string } }; + expect(result.write_through?.written).toBe(true); + const onDisk = fs.readFileSync(result.write_through!.path!, 'utf8'); + expect(onDisk).toMatch(/ingested_via:\s*put_page/); + expect(onDisk).toMatch(/ingested_at:/); + }); + + test('MCP/remote callers get ingested_via=mcp:put_page', async () => { + const ctx = makeCtx({ remote: true }); + const result = (await putPage.handler(ctx, { + slug: 'inbox/mcp-prov', + content: '---\ntitle: Q\n---\n\nbody', + })) as { write_through?: { written: boolean; path?: string } }; + expect(result.write_through?.written).toBe(true); + const onDisk = fs.readFileSync(result.write_through!.path!, 'utf8'); + // YAML quotes strings containing `:` so the literal frontmatter line + // is `ingested_via: 'mcp:put_page'`. Match the value substring. + expect(onDisk).toMatch(/ingested_via:\s*['"]?mcp:put_page['"]?/); + }); +}); + +describe('put_page write-through — trust gating', () => { + test('subagent sandbox write (viaSubagent without allowedSlugPrefixes) stays DB-only', async () => { + const ctx = makeCtx({ + remote: true, + viaSubagent: true, + subagentId: 42, + // No allowedSlugPrefixes — sandbox writes only. + }); + const result = (await putPage.handler(ctx, { + slug: 'wiki/agents/42/scratch', + content: '---\ntitle: S\n---\n\nbody', + })) as { write_through?: { written: boolean; skipped?: string } }; + expect(result.write_through?.written).toBe(false); + expect(result.write_through?.skipped).toBe('subagent_sandbox'); + expect(fs.existsSync(path.join(brainDir, 'wiki/agents/42/scratch.md'))).toBe(false); + }); + + test('trusted-workspace subagent (viaSubagent + allowedSlugPrefixes) writes through', async () => { + const ctx = makeCtx({ + remote: true, + viaSubagent: true, + subagentId: 7, + allowedSlugPrefixes: ['wiki/personal/reflections/*'], + }); + const result = (await putPage.handler(ctx, { + slug: 'wiki/personal/reflections/note', + content: '---\ntitle: R\n---\n\nreflection', + })) as { write_through?: { written: boolean; path?: string } }; + expect(result.write_through?.written).toBe(true); + expect(fs.existsSync(result.write_through!.path!)).toBe(true); + }); + + test('dry-run stays DB-only (early-return before importFromContent)', async () => { + const ctx = makeCtx({ dryRun: true }); + const result = (await putPage.handler(ctx, { + slug: 'inbox/dryrun', + content: '---\ntitle: D\n---\n\nbody', + })) as { dry_run?: boolean; write_through?: { skipped?: string } }; + // put_page's existing handler short-circuits on dry-run BEFORE + // importFromContent, so write_through never fires. The legacy dry_run + // contract is what callers see. + expect(result.dry_run).toBe(true); + expect(fs.existsSync(path.join(brainDir, 'inbox/dryrun.md'))).toBe(false); + }); +}); + +describe('put_page write-through — config edge cases', () => { + test('repo not configured → skipped no_repo_configured', async () => { + // No deleteConfig helper; remove via raw SQL. + await engine.executeRaw("DELETE FROM config WHERE key = 'sync.repo_path'"); + const ctx = makeCtx(); + const result = (await putPage.handler(ctx, { + slug: 'inbox/no-repo', + content: '---\ntitle: N\n---\n\nbody', + })) as { write_through?: { skipped?: string } }; + expect(result.write_through?.skipped).toBe('no_repo_configured'); + }); + + test('repo path points at a missing directory → skipped repo_not_found', async () => { + await engine.setConfig('sync.repo_path', path.join(tmpRoot, 'does-not-exist')); + const ctx = makeCtx(); + const result = (await putPage.handler(ctx, { + slug: 'inbox/missing-repo', + content: '---\ntitle: M\n---\n\nbody', + })) as { write_through?: { skipped?: string } }; + expect(result.write_through?.skipped).toBe('repo_not_found'); + }); +}); + +describe('put_page write-through — multi-source filing', () => { + test('non-default source lands at brainDir/.sources//.md', async () => { + // Create a non-default source row first. Schema fields: id (PK), + // name (UNIQUE), plus the v0.26.5 archive columns with defaults. + await engine.executeRaw( + "INSERT INTO sources (id, name) VALUES ('team-x', 'team-x')", + ); + const ctx = makeCtx({ sourceId: 'team-x' }); + const result = (await putPage.handler(ctx, { + slug: 'shared/page', + content: '---\ntitle: X\n---\n\nbody', + })) as { write_through?: { written: boolean; path?: string } }; + expect(result.write_through?.written).toBe(true); + expect(result.write_through?.path).toBe(path.join(brainDir, '.sources/team-x/shared/page.md')); + expect(fs.existsSync(result.write_through!.path!)).toBe(true); + }); +}); + +describe('put_page write-through — failure isolation', () => { + test('disk-write failure does not roll back DB', async () => { + // Point the config at a path that exists but isn't writable so the + // write fails. Best portable trick: a regular file (writeFileSync to + // a path inside a regular file fails with ENOTDIR). + const blockFile = path.join(tmpRoot, 'block'); + fs.writeFileSync(blockFile, 'i am a file, not a dir'); + await engine.setConfig('sync.repo_path', blockFile); + + const ctx = makeCtx(); + const result = (await putPage.handler(ctx, { + slug: 'inbox/fail-isolated', + content: '---\ntitle: F\n---\n\nbody', + })) as { write_through?: { skipped?: string; error?: string } }; + // Either skipped (existsSync sees a file, not a dir) or error during write. + expect( + result.write_through?.skipped === 'repo_not_found' || + typeof result.write_through?.error === 'string', + ).toBe(true); + + // DB write succeeded. + const page = await engine.getPage('inbox/fail-isolated'); + expect(page).not.toBeNull(); + }); +}); diff --git a/test/ingestion/skillpack-load.test.ts b/test/ingestion/skillpack-load.test.ts new file mode 100644 index 000000000..7eb749fb1 --- /dev/null +++ b/test/ingestion/skillpack-load.test.ts @@ -0,0 +1,318 @@ +/** + * Skillpack-distributed IngestionSource discovery tests. + * + * Uses tmp dirs + the _import stub seam so we exercise the manifest + * parsing + path validation + module-load contract without actually + * loading arbitrary code from the filesystem. CLAUDE.md rule R1 + * forbids process.env mutation; we use withEnv from helpers/with-env.ts. + */ + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { withEnv } from '../helpers/with-env.ts'; +import { + loadSkillpackSources, + __testing, +} from '../../src/core/ingestion/skillpack-load.ts'; +import type { IngestionSource } from '../../src/core/ingestion/types.ts'; + +let tmpRoot: string; + +beforeEach(() => { + tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-skillpack-load-')); +}); + +afterEach(() => { + fs.rmSync(tmpRoot, { recursive: true, force: true }); +}); + +function makePluginDir(opts: { + name?: string; + pluginVersion?: string; + sources?: Array<{ + kind: string; + module?: string; + api_version?: string; + default_config?: Record; + permissions?: string[]; + }>; + moduleBody?: string; +}): string { + const dir = fs.mkdtempSync(path.join(tmpRoot, 'plugin-')); + const manifest: Record = { + name: opts.name ?? 'test-plugin', + version: '1.0.0', + plugin_version: opts.pluginVersion ?? 'gbrain-plugin-v1', + }; + if (opts.sources && opts.sources.length > 0) { + manifest.ingestion_sources = opts.sources.map((s) => ({ + kind: s.kind, + module: s.module ?? './source.js', + api_version: s.api_version ?? 'gbrain-ingestion-source-v1', + default_config: s.default_config, + permissions: s.permissions, + })); + } + fs.writeFileSync(path.join(dir, 'gbrain.plugin.json'), JSON.stringify(manifest)); + if (opts.sources && opts.sources.length > 0) { + fs.writeFileSync(path.join(dir, 'source.js'), opts.moduleBody ?? '// noop'); + } + return dir; +} + +function makeFactoryFn(): (config: Record) => IngestionSource { + return (_config) => ({ + id: 'stub-source-1', + kind: 'stub', + async start() {}, + async stop() {}, + }); +} + +describe('loadSkillpackSources — discovery', () => { + test('empty GBRAIN_PLUGIN_PATH returns empty result', async () => { + const result = await loadSkillpackSources({ envPath: '' }); + expect(result.sources).toHaveLength(0); + expect(result.warnings).toHaveLength(0); + }); + + test('relative path is rejected with warning', async () => { + const result = await loadSkillpackSources({ envPath: './relative/path' }); + expect(result.sources).toHaveLength(0); + expect(result.warnings.some((w) => w.includes('relative path rejected'))).toBe(true); + }); + + test('home-prefixed path is rejected with warning', async () => { + const result = await loadSkillpackSources({ envPath: '~/some/path' }); + expect(result.warnings.some((w) => w.includes('~-prefixed'))).toBe(true); + }); + + test('rejectIfNotAbsolute helper rejects remote URLs (defense-in-depth)', () => { + // GBRAIN_PLUGIN_PATH is colon-separated paths ($PATH style), so URL + // values get split into bogus segments before this branch fires. The + // branch exists as defense-in-depth for any future code path that + // hands a raw URL string to the validator (e.g. a future --plugin-dir + // CLI flag that bypasses the split). + expect(__testing.rejectIfNotAbsolute('https://evil.example.com/plugin')).toContain('remote URL'); + expect(__testing.rejectIfNotAbsolute('file:///etc/passwd')).toContain('remote URL'); + }); + + test('missing directory yields a warning, not a throw', async () => { + const result = await loadSkillpackSources({ + envPath: path.join(tmpRoot, 'does-not-exist'), + }); + expect(result.sources).toHaveLength(0); + expect(result.warnings.some((w) => w.includes('does not exist'))).toBe(true); + }); + + test('directory with no gbrain.plugin.json is silently skipped (not a warning)', async () => { + const dir = fs.mkdtempSync(path.join(tmpRoot, 'no-manifest-')); + const result = await loadSkillpackSources({ envPath: dir }); + expect(result.sources).toHaveLength(0); + expect(result.warnings).toHaveLength(0); + }); + + test('directory with manifest but no ingestion_sources is silently skipped', async () => { + const dir = makePluginDir({}); + const result = await loadSkillpackSources({ envPath: dir }); + expect(result.sources).toHaveLength(0); + expect(result.warnings).toHaveLength(0); + }); +}); + +describe('loadSkillpackSources — manifest validation', () => { + test('rejects invalid manifest JSON', async () => { + const dir = fs.mkdtempSync(path.join(tmpRoot, 'bad-json-')); + fs.writeFileSync(path.join(dir, 'gbrain.plugin.json'), '{ not valid'); + const result = await loadSkillpackSources({ envPath: dir }); + expect(result.warnings.some((w) => w.includes('invalid manifest JSON'))).toBe(true); + }); + + test('rejects unsupported plugin_version', async () => { + const dir = makePluginDir({ pluginVersion: 'gbrain-plugin-v99' }); + const result = await loadSkillpackSources({ envPath: dir }); + expect(result.warnings.some((w) => w.includes('unsupported plugin_version'))).toBe(true); + }); + + test('rejects missing name field', async () => { + const dir = fs.mkdtempSync(path.join(tmpRoot, 'no-name-')); + fs.writeFileSync( + path.join(dir, 'gbrain.plugin.json'), + JSON.stringify({ + plugin_version: 'gbrain-plugin-v1', + ingestion_sources: [{ kind: 'x', module: './source.js', api_version: 'gbrain-ingestion-source-v1' }], + }), + ); + const result = await loadSkillpackSources({ envPath: dir }); + expect(result.warnings.some((w) => w.includes('missing required "name"'))).toBe(true); + }); +}); + +describe('loadSkillpackSources — source declaration validation', () => { + test('rejects missing kind field', async () => { + const dir = fs.mkdtempSync(path.join(tmpRoot, 'no-kind-')); + fs.writeFileSync( + path.join(dir, 'gbrain.plugin.json'), + JSON.stringify({ + name: 'p', + plugin_version: 'gbrain-plugin-v1', + ingestion_sources: [{ module: './source.js', api_version: 'gbrain-ingestion-source-v1' }], + }), + ); + const result = await loadSkillpackSources({ envPath: dir }); + expect(result.warnings.some((w) => w.includes('kind must be'))).toBe(true); + }); + + test('rejects array-shaped default_config', async () => { + const err = __testing.validateDeclaration({ + kind: 'x', + module: './source.js', + api_version: 'gbrain-ingestion-source-v1', + default_config: [1, 2, 3], + }); + expect(err).toContain('default_config must be a plain object'); + }); + + test('rejects non-string-array permissions', async () => { + const err = __testing.validateDeclaration({ + kind: 'x', + module: './source.js', + api_version: 'gbrain-ingestion-source-v1', + permissions: [1, 2, 3], + }); + expect(err).toContain('permissions must be an array of strings'); + }); +}); + +describe('loadSkillpackSources — api_version compatibility', () => { + test('current api_version loads successfully', async () => { + const dir = makePluginDir({ sources: [{ kind: 'stub' }] }); + const result = await loadSkillpackSources({ + envPath: dir, + _import: async () => ({ default: makeFactoryFn() }), + }); + expect(result.sources).toHaveLength(1); + expect(result.warnings).toHaveLength(0); + }); + + test('api_version mismatch fails loudly with upgrade hint', async () => { + const dir = makePluginDir({ + sources: [{ kind: 'stub', api_version: 'gbrain-ingestion-source-v99' }], + }); + const result = await loadSkillpackSources({ + envPath: dir, + _import: async () => ({ default: makeFactoryFn() }), + }); + expect(result.sources).toHaveLength(0); + const warning = result.warnings.find((w) => w.includes('api_version')); + expect(warning).toBeDefined(); + expect(warning).toContain('rebuild against the new'); + expect(warning).toContain('docs/ingestion-source-skillpack.md'); + }); +}); + +describe('loadSkillpackSources — module loading', () => { + test('valid module with default export factory is loaded', async () => { + const dir = makePluginDir({ sources: [{ kind: 'stub' }] }); + const result = await loadSkillpackSources({ + envPath: dir, + _import: async () => ({ default: makeFactoryFn() }), + }); + expect(result.sources).toHaveLength(1); + expect(result.sources[0]?.declaration.kind).toBe('stub'); + expect(result.sources[0]?.plugin_name).toBe('test-plugin'); + // Verify the factory is callable. + const source = result.sources[0]?.factory({}); + expect(source?.kind).toBe('stub'); + }); + + test('module load failure produces a warning', async () => { + const dir = makePluginDir({ sources: [{ kind: 'stub' }] }); + const result = await loadSkillpackSources({ + envPath: dir, + _import: async () => { throw new Error('syntax error in source.js'); }, + }); + expect(result.sources).toHaveLength(0); + expect(result.warnings.some((w) => w.includes('failed to import'))).toBe(true); + }); + + test('module without default export factory produces a warning', async () => { + const dir = makePluginDir({ sources: [{ kind: 'stub' }] }); + const result = await loadSkillpackSources({ + envPath: dir, + _import: async () => ({ someOtherExport: 42 }), + }); + expect(result.sources).toHaveLength(0); + expect(result.warnings.some((w) => w.includes('does not export a factory function'))).toBe(true); + }); + + test('missing module file is rejected', async () => { + const dir = makePluginDir({ + sources: [{ kind: 'stub', module: './does-not-exist.js' }], + }); + // Remove the auto-created source.js so we hit the missing-file branch. + fs.rmSync(path.join(dir, 'source.js')); + const result = await loadSkillpackSources({ envPath: dir }); + expect(result.warnings.some((w) => w.includes('module not found'))).toBe(true); + }); + + test('module path escaping the plugin root is rejected', async () => { + const dir = makePluginDir({ + sources: [{ kind: 'stub', module: '../escape.js' }], + }); + const result = await loadSkillpackSources({ + envPath: dir, + _import: async () => ({ default: makeFactoryFn() }), + }); + expect(result.warnings.some((w) => w.includes('escapes plugin root'))).toBe(true); + }); +}); + +describe('loadSkillpackSources — collision policy', () => { + test('two plugins declaring the same kind: first wins with warning', async () => { + const dirA = makePluginDir({ name: 'plugin-a', sources: [{ kind: 'shared' }] }); + const dirB = makePluginDir({ name: 'plugin-b', sources: [{ kind: 'shared' }] }); + const result = await loadSkillpackSources({ + envPath: `${dirA}:${dirB}`, + _import: async () => ({ default: makeFactoryFn() }), + }); + expect(result.sources).toHaveLength(1); + expect(result.sources[0]?.plugin_name).toBe('plugin-a'); // first + expect(result.warnings.some((w) => w.includes('kind collision') && w.includes('shared'))).toBe(true); + }); +}); + +describe('loadSkillpackSources — env var path', () => { + test('reads GBRAIN_PLUGIN_PATH from process.env when envPath is not passed', async () => { + const dir = makePluginDir({ sources: [{ kind: 'env-stub' }] }); + await withEnv({ GBRAIN_PLUGIN_PATH: dir }, async () => { + const result = await loadSkillpackSources({ + _import: async () => ({ default: makeFactoryFn() }), + }); + expect(result.sources).toHaveLength(1); + expect(result.sources[0]?.declaration.kind).toBe('env-stub'); + }); + }); +}); + +describe('__testing helpers', () => { + test('rejectIfNotAbsolute accepts absolute paths', () => { + expect(__testing.rejectIfNotAbsolute('/abs/path')).toBeNull(); + }); + + test('extractFactory pulls from .default of an ESM-shaped module object', () => { + const f = makeFactoryFn(); + expect(__testing.extractFactory({ default: f })).toBe(f); + }); + + test('extractFactory returns null when no default export', () => { + expect(__testing.extractFactory({ named: 'value' })).toBeNull(); + }); + + test('extractFactory returns null for non-object', () => { + expect(__testing.extractFactory(null)).toBeNull(); + expect(__testing.extractFactory('string')).toBeNull(); + }); +}); diff --git a/test/ingestion/sources/file-watcher.test.ts b/test/ingestion/sources/file-watcher.test.ts new file mode 100644 index 000000000..d3d5c83ee --- /dev/null +++ b/test/ingestion/sources/file-watcher.test.ts @@ -0,0 +1,294 @@ +/** + * FileWatcherSource tests — uses the _watchFactory test seam so we can + * exercise the source lifecycle without spinning up real chokidar in + * a tmp dir (timing-dependent, flaky). + * + * The stub chokidar is an EventEmitter that satisfies the FSWatcher + * surface the source actually touches (.on / .once / .close / .getWatched). + */ + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { EventEmitter } from 'node:events'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { createFileWatcherSource } from '../../../src/core/ingestion/sources/file-watcher.ts'; +import { IngestionTestHarness } from '../../../src/core/ingestion/test-harness.ts'; +import type { FSWatcher } from 'chokidar'; + +let tmpRoot: string; + +beforeEach(() => { + tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-fw-test-')); +}); + +afterEach(() => { + fs.rmSync(tmpRoot, { recursive: true, force: true }); +}); + +/** Build a stub FSWatcher we can drive from tests. + * + * Tracks `readyFired` state so that if a test calls `fireReady()` before + * the source has registered its `.once('ready', ...)` listener (race — + * source awaits mkdir/stat before listener registration), the listener + * still fires once registered. Same for error replay. Real chokidar + * doesn't have this race because its own async init buys userland time + * to attach listeners; the stub has to simulate it. */ +function makeStubWatcher(): { + watcher: FSWatcher; + fireAdd: (path: string) => void; + fireChange: (path: string) => void; + fireReady: () => void; + fireError: (err: Error) => void; + closed: { current: boolean }; +} { + const emitter = new EventEmitter(); + const closed = { current: false }; + const state = { readyFired: false, errorFired: null as Error | null }; + const stub = { + on(event: string, handler: (...args: unknown[]) => void) { + if (event === 'ready' && state.readyFired) { + queueMicrotask(() => handler()); + return stub; + } + if (event === 'error' && state.errorFired) { + const err = state.errorFired; + queueMicrotask(() => handler(err)); + return stub; + } + emitter.on(event, handler); + return stub; + }, + once(event: string, handler: (...args: unknown[]) => void) { + if (event === 'ready' && state.readyFired) { + queueMicrotask(() => handler()); + return stub; + } + if (event === 'error' && state.errorFired) { + const err = state.errorFired; + queueMicrotask(() => handler(err)); + return stub; + } + emitter.once(event, handler); + return stub; + }, + off(event: string, handler: (...args: unknown[]) => void) { + emitter.off(event, handler); + return stub; + }, + close: async () => { + closed.current = true; + emitter.removeAllListeners(); + }, + getWatched: () => ({ '/': ['watched'] }), + add: () => {}, + unwatch: () => {}, + } as unknown as FSWatcher; + return { + watcher: stub, + fireAdd: (p: string) => emitter.emit('add', p), + fireChange: (p: string) => emitter.emit('change', p), + fireReady: () => { state.readyFired = true; emitter.emit('ready'); }, + fireError: (err: Error) => { + state.errorFired = err; + // EventEmitter throws on emit('error', ...) when no listeners. Guard. + if (emitter.listenerCount('error') > 0) emitter.emit('error', err); + }, + closed, + }; +} + +async function waitFor(predicate: () => boolean, timeoutMs = 2000): Promise { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + if (predicate()) return; + await new Promise((r) => setTimeout(r, 10)); + } + throw new Error(`waitFor: predicate did not become true within ${timeoutMs}ms`); +} + +describe('FileWatcherSource — startup', () => { + test('requires brainDir', () => { + expect(() => createFileWatcherSource({ brainDir: '' as never })).toThrow(/brainDir is required/); + }); + + test('rejects when brainDir does not exist', async () => { + const source = createFileWatcherSource({ brainDir: '/does/not/exist/xyz' }); + const harness = new IngestionTestHarness(); + await expect(harness.run(source)).rejects.toThrow(/does not exist/); + }); + + test('rejects when brainDir is a file, not a directory', async () => { + const f = path.join(tmpRoot, 'not-a-dir.md'); + fs.writeFileSync(f, 'x'); + const source = createFileWatcherSource({ brainDir: f }); + const harness = new IngestionTestHarness(); + await expect(harness.run(source)).rejects.toThrow(/does not exist or is not a directory/); + }); + + test('resolves start() on chokidar ready', async () => { + const stub = makeStubWatcher(); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + await harness.stop(); + }); + + test('rejects start() on early chokidar error', async () => { + const stub = makeStubWatcher(); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + // Race: error first. + stub.fireError(new Error('chokidar boom')); + await expect(startPromise).rejects.toThrow(/chokidar boom/); + }); + + test('chokidar close() is called on stop()', async () => { + const stub = makeStubWatcher(); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + await harness.stop(); + expect(stub.closed.current).toBe(true); + }); +}); + +describe('FileWatcherSource — event flow', () => { + test('add event for .md file emits IngestionEvent after debounce', async () => { + const stub = makeStubWatcher(); + const f = path.join(tmpRoot, 'note.md'); + fs.writeFileSync(f, '# Hello'); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + debounceMs: 50, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + stub.fireAdd(f); + await waitFor(() => harness.events.length === 1, 1000); + expect(harness.events[0]?.source_kind).toBe('file-watcher'); + expect(harness.events[0]?.source_uri).toBe(f); + expect(harness.events[0]?.content_type).toBe('text/markdown'); + expect(harness.events[0]?.content).toBe('# Hello'); + await harness.stop(); + }); + + test('non-markdown file does NOT emit', async () => { + const stub = makeStubWatcher(); + const f = path.join(tmpRoot, 'note.txt'); + fs.writeFileSync(f, 'plain'); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + debounceMs: 30, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + stub.fireAdd(f); + await new Promise((r) => setTimeout(r, 100)); // past debounce + expect(harness.events).toHaveLength(0); + await harness.stop(); + }); + + test('rapid changes coalesce into one event (debounce)', async () => { + const stub = makeStubWatcher(); + const f = path.join(tmpRoot, 'spam.md'); + fs.writeFileSync(f, '# first'); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + debounceMs: 50, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + // 5 rapid change events on the same path. + for (let i = 0; i < 5; i++) { + stub.fireChange(f); + await new Promise((r) => setTimeout(r, 5)); + } + await waitFor(() => harness.events.length === 1, 500); + expect(harness.events).toHaveLength(1); + await harness.stop(); + }); + + test('read failure between debounce and flush logs warn, no emit', async () => { + const stub = makeStubWatcher(); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + debounceMs: 30, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + // File never existed. + stub.fireAdd(path.join(tmpRoot, 'ghost.md')); + await new Promise((r) => setTimeout(r, 150)); + expect(harness.events).toHaveLength(0); + expect(harness.logs.some((l) => l.level === 'warn' && l.msg.includes('failed to read'))).toBe(true); + await harness.stop(); + }); +}); + +describe('FileWatcherSource — Linux ENOSPC handling', () => { + test('ENOSPC error surfaces a paste-ready sysctl hint', async () => { + const stub = makeStubWatcher(); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + stub.fireError(new Error('ENOSPC: System limit for number of file watchers reached')); + await new Promise((r) => setTimeout(r, 50)); + expect(harness.logs.some((l) => l.level === 'warn' && l.msg.includes('ENOSPC'))).toBe(true); + expect(harness.logs.some((l) => l.level === 'error' && l.msg.includes('fs.inotify.max_user_watches'))).toBe(true); + await harness.stop(); + }); +}); + +describe('FileWatcherSource — healthCheck', () => { + test('returns ok when watcher is alive with watched dirs', async () => { + const stub = makeStubWatcher(); + const source = createFileWatcherSource({ + brainDir: tmpRoot, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + const h = await harness.healthCheck(); + expect(h.status).toBe('ok'); + await harness.stop(); + }); +}); diff --git a/test/ingestion/sources/inbox-folder.test.ts b/test/ingestion/sources/inbox-folder.test.ts new file mode 100644 index 000000000..a83ff2c8c --- /dev/null +++ b/test/ingestion/sources/inbox-folder.test.ts @@ -0,0 +1,358 @@ +/** + * InboxFolderSource tests. Mix of helper-pure tests (no fs / chokidar) + * and stubbed-chokidar lifecycle tests using tmp dirs for the archive + * move semantics. + */ + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { EventEmitter } from 'node:events'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { + createInboxFolderSource, + __testing, +} from '../../../src/core/ingestion/sources/inbox-folder.ts'; +import { IngestionTestHarness } from '../../../src/core/ingestion/test-harness.ts'; +import type { FSWatcher } from 'chokidar'; + +let tmpRoot: string; +let inboxDir: string; + +beforeEach(() => { + tmpRoot = fs.mkdtempSync(path.join(os.tmpdir(), 'gbrain-inbox-test-')); + inboxDir = path.join(tmpRoot, 'inbox'); +}); + +afterEach(() => { + fs.rmSync(tmpRoot, { recursive: true, force: true }); +}); + +/** Stub with deferred-ready replay (see file-watcher.test.ts for design notes). */ +function makeStubWatcher() { + const emitter = new EventEmitter(); + const closed = { current: false }; + const state = { readyFired: false, errorFired: null as Error | null }; + const stub = { + on(event: string, handler: (...args: unknown[]) => void) { + if (event === 'ready' && state.readyFired) { + queueMicrotask(() => handler()); + return stub; + } + if (event === 'error' && state.errorFired) { + const err = state.errorFired; + queueMicrotask(() => handler(err)); + return stub; + } + emitter.on(event, handler); + return stub; + }, + once(event: string, handler: (...args: unknown[]) => void) { + if (event === 'ready' && state.readyFired) { + queueMicrotask(() => handler()); + return stub; + } + if (event === 'error' && state.errorFired) { + const err = state.errorFired; + queueMicrotask(() => handler(err)); + return stub; + } + emitter.once(event, handler); + return stub; + }, + off(event: string, handler: (...args: unknown[]) => void) { emitter.off(event, handler); return stub; }, + close: async () => { closed.current = true; emitter.removeAllListeners(); }, + getWatched: () => ({ '/': ['watched'] }), + add: () => {}, + unwatch: () => {}, + } as unknown as FSWatcher; + return { + watcher: stub, + fireAdd: (p: string) => emitter.emit('add', p), + fireReady: () => { state.readyFired = true; emitter.emit('ready'); }, + closed, + }; +} + +async function waitFor(predicate: () => boolean, timeoutMs = 2000): Promise { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + if (predicate()) return; + await new Promise((r) => setTimeout(r, 10)); + } + throw new Error(`waitFor: predicate did not become true within ${timeoutMs}ms`); +} + +describe('detectContentType (helper)', () => { + test('routes markdown to text/markdown', () => { + expect(__testing.detectContentType('foo.md')).toBe('text/markdown'); + expect(__testing.detectContentType('foo.markdown')).toBe('text/markdown'); + expect(__testing.detectContentType('FOO.MD')).toBe('text/markdown'); + }); + + test('routes text to text/plain', () => { + expect(__testing.detectContentType('note.txt')).toBe('text/plain'); + }); + + test('routes html', () => { + expect(__testing.detectContentType('page.html')).toBe('text/html'); + expect(__testing.detectContentType('page.htm')).toBe('text/html'); + }); + + test('routes images to image/*', () => { + expect(__testing.detectContentType('shot.png')).toBe('image/*'); + expect(__testing.detectContentType('photo.jpeg')).toBe('image/*'); + expect(__testing.detectContentType('anim.gif')).toBe('image/*'); + }); + + test('routes audio to audio/*', () => { + expect(__testing.detectContentType('voice.m4a')).toBe('audio/*'); + expect(__testing.detectContentType('song.mp3')).toBe('audio/*'); + }); + + test('routes video to video/*', () => { + expect(__testing.detectContentType('clip.mp4')).toBe('video/*'); + expect(__testing.detectContentType('movie.mkv')).toBe('video/*'); + }); + + test('routes pdf to application/pdf', () => { + expect(__testing.detectContentType('doc.pdf')).toBe('application/pdf'); + }); + + test('routes json to application/json', () => { + expect(__testing.detectContentType('data.json')).toBe('application/json'); + }); + + test('unknown extension routes to unknown', () => { + expect(__testing.detectContentType('foo.xyz')).toBe('unknown'); + expect(__testing.detectContentType('no-extension')).toBe('unknown'); + }); +}); + +describe('archiveDateFolder (helper)', () => { + test('formats UTC date as YYYY-MM-DD', () => { + expect(__testing.archiveDateFolder(new Date('2026-05-20T12:00:00Z'))).toBe('2026-05-20'); + expect(__testing.archiveDateFolder(new Date('2026-01-05T00:00:00Z'))).toBe('2026-01-05'); + }); +}); + +describe('uniqueArchivePath (helper)', () => { + test('returns the candidate when no collision', () => { + const dir = fs.mkdtempSync(path.join(tmpRoot, 'arch-')); + const result = __testing.uniqueArchivePath(dir, 'note.md'); + expect(result).toBe(path.join(dir, 'note.md')); + }); + + test('suffixes with -1, -2 on collision', () => { + const dir = fs.mkdtempSync(path.join(tmpRoot, 'arch-')); + fs.writeFileSync(path.join(dir, 'note.md'), ''); + fs.writeFileSync(path.join(dir, 'note-1.md'), ''); + const result = __testing.uniqueArchivePath(dir, 'note.md'); + expect(result).toBe(path.join(dir, 'note-2.md')); + }); +}); + +describe('InboxFolderSource — startup', () => { + test('requires inboxDir', () => { + expect(() => createInboxFolderSource({ inboxDir: '' as never })).toThrow(/inboxDir is required/); + }); + + test('creates inbox + archive dirs if missing', async () => { + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + _watchFactory: () => stub.watcher, + }); + expect(fs.existsSync(inboxDir)).toBe(false); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + expect(fs.existsSync(inboxDir)).toBe(true); + expect(fs.existsSync(path.join(inboxDir, '.archived'))).toBe(true); + await harness.stop(); + }); + + test('warns when inbox dir is world-writable', async () => { + fs.mkdirSync(inboxDir, { recursive: true, mode: 0o777 }); + fs.chmodSync(inboxDir, 0o777); + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + expect(harness.logs.some((l) => l.level === 'warn' && l.msg.includes('world-writable'))).toBe(true); + await harness.stop(); + }); +}); + +describe('InboxFolderSource — file ingestion', () => { + test('text file: read content, emit, archive', async () => { + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + debounceMs: 30, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + const f = path.join(inboxDir, 'capture.md'); + fs.writeFileSync(f, '# captured thought'); + stub.fireAdd(f); + + await waitFor(() => harness.events.length === 1, 1500); + expect(harness.events[0]?.content_type).toBe('text/markdown'); + expect(harness.events[0]?.content).toBe('# captured thought'); + expect(harness.events[0]?.source_uri).toBe(f); + // metadata should report it was text + expect(harness.events[0]?.metadata?.is_text).toBe(true); + + // File should have been archived. + await waitFor(() => !fs.existsSync(f), 1500); + const today = __testing.archiveDateFolder(new Date()); + const archived = path.join(inboxDir, '.archived', today, 'capture.md'); + expect(fs.existsSync(archived)).toBe(true); + expect(fs.readFileSync(archived, 'utf8')).toBe('# captured thought'); + await harness.stop(); + }); + + test('binary file: path-only content, hash from path+stat', async () => { + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + debounceMs: 30, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + const f = path.join(inboxDir, 'photo.png'); + fs.writeFileSync(f, Buffer.from([0x89, 0x50, 0x4e, 0x47])); // PNG magic + stub.fireAdd(f); + + await waitFor(() => harness.events.length === 1, 1500); + expect(harness.events[0]?.content_type).toBe('image/*'); + // For binary, content is the absolute path (NOT the bytes). + expect(harness.events[0]?.content).toBe(f); + expect(harness.events[0]?.metadata?.is_text).toBe(false); + await harness.stop(); + }); + + test('symlink is rejected and logged', async () => { + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + debounceMs: 30, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + // Create the target outside the inbox. + const target = path.join(tmpRoot, 'sensitive.md'); + fs.writeFileSync(target, '# secret'); + const link = path.join(inboxDir, 'evil.md'); + fs.symlinkSync(target, link); + + stub.fireAdd(link); + await new Promise((r) => setTimeout(r, 150)); + expect(harness.events).toHaveLength(0); + expect(harness.logs.some((l) => l.level === 'warn' && l.msg.includes('rejected symlink'))).toBe(true); + await harness.stop(); + }); + + test('events under .archived are ignored (no re-ingestion loop)', async () => { + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + debounceMs: 30, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + // Simulate chokidar accidentally firing 'add' for an archived path. + const archivedFile = path.join(inboxDir, '.archived', '2026-05-20', 'old.md'); + fs.mkdirSync(path.dirname(archivedFile), { recursive: true }); + fs.writeFileSync(archivedFile, '# old'); + stub.fireAdd(archivedFile); + + await new Promise((r) => setTimeout(r, 150)); + expect(harness.events).toHaveLength(0); + await harness.stop(); + }); + + test('archiveAfterEmit=false leaves the file in place', async () => { + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + debounceMs: 30, + archiveAfterEmit: false, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + const f = path.join(inboxDir, 'no-archive.md'); + fs.writeFileSync(f, '# stays'); + stub.fireAdd(f); + + await waitFor(() => harness.events.length === 1, 1500); + expect(fs.existsSync(f)).toBe(true); + await harness.stop(); + }); + + test('rapid duplicate add events coalesce via debounce', async () => { + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + debounceMs: 50, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + + const f = path.join(inboxDir, 'dup.md'); + fs.writeFileSync(f, '# once'); + // Fire 5 rapid 'add' events on the same path before debounce settles. + for (let i = 0; i < 5; i++) { + stub.fireAdd(f); + await new Promise((r) => setTimeout(r, 5)); + } + await waitFor(() => harness.events.length === 1, 800); + expect(harness.events).toHaveLength(1); + await harness.stop(); + }); + + test('healthCheck reports ok when watcher is alive', async () => { + const stub = makeStubWatcher(); + const source = createInboxFolderSource({ + inboxDir, + _watchFactory: () => stub.watcher, + }); + const harness = new IngestionTestHarness(); + const startPromise = harness.run(source); + stub.fireReady(); + await startPromise; + const h = await harness.healthCheck(); + expect(h.status).toBe('ok'); + await harness.stop(); + }); +}); diff --git a/test/ingestion/test-harness.test.ts b/test/ingestion/test-harness.test.ts new file mode 100644 index 000000000..625764244 --- /dev/null +++ b/test/ingestion/test-harness.test.ts @@ -0,0 +1,292 @@ +/** + * Tests for IngestionTestHarness — the publisher-facing test utility + * exported at gbrain/ingestion/test-harness. + * + * The harness IS the publisher contract. These tests pin its behavior so + * skillpack authors can rely on it across gbrain minor versions. + */ + +import { describe, expect, test } from 'bun:test'; +import { + IngestionTestHarness, + expectEvent, +} from '../../src/core/ingestion/test-harness.ts'; +import { + computeContentHash, + type IngestionEvent, + type IngestionSource, +} from '../../src/core/ingestion/types.ts'; + +function makeMockSource(opts: { + id?: string; + kind?: string; + emitOnStart?: IngestionEvent[]; + startThrows?: string; + stopThrows?: string; + stopDelayMs?: number; +} = {}): IngestionSource { + return { + id: opts.id ?? 'mock-1', + kind: opts.kind ?? 'mock', + async start(ctx) { + if (opts.startThrows) throw new Error(opts.startThrows); + for (const ev of opts.emitOnStart ?? []) ctx.emit(ev); + }, + async stop() { + if (opts.stopDelayMs) { + await new Promise((r) => setTimeout(r, opts.stopDelayMs)); + } + if (opts.stopThrows) throw new Error(opts.stopThrows); + }, + }; +} + +function makeEvent(overrides: Partial = {}): IngestionEvent { + return { + source_id: 'mock-1', + source_kind: 'mock', + source_uri: '/tmp/event.md', + received_at: new Date('2026-05-20T12:00:00Z').toISOString(), + content_type: 'text/markdown', + content: '# test', + content_hash: computeContentHash('# test'), + ...overrides, + }; +} + +describe('IngestionTestHarness lifecycle', () => { + test('run() starts the source and captures emitted events', async () => { + const harness = new IngestionTestHarness(); + const source = makeMockSource({ emitOnStart: [makeEvent()] }); + await harness.run(source); + expect(harness.events).toHaveLength(1); + expect(harness.events[0]?.source_kind).toBe('mock'); + await harness.stop(); + }); + + test('events are captured in emit order', async () => { + const harness = new IngestionTestHarness(); + const e1 = makeEvent({ content: 'one', content_hash: computeContentHash('one') }); + const e2 = makeEvent({ content: 'two', content_hash: computeContentHash('two') }); + const source = makeMockSource({ emitOnStart: [e1, e2] }); + await harness.run(source); + expect(harness.events[0]?.content).toBe('one'); + expect(harness.events[1]?.content).toBe('two'); + await harness.stop(); + }); + + test('source startup error propagates from run()', async () => { + const harness = new IngestionTestHarness(); + const source = makeMockSource({ startThrows: 'boom' }); + await expect(harness.run(source)).rejects.toThrow('boom'); + }); + + test('double run() rejects without leaking state', async () => { + const harness = new IngestionTestHarness(); + await harness.run(makeMockSource()); + await expect(harness.run(makeMockSource())).rejects.toThrow(/already running/); + await harness.stop(); + }); + + test('stop() calls source.stop()', async () => { + let stopped = false; + const source: IngestionSource = { + id: 'mock', kind: 'mock', + async start() {}, + async stop() { stopped = true; }, + }; + const harness = new IngestionTestHarness(); + await harness.run(source); + await harness.stop(); + expect(stopped).toBe(true); + }); + + test('stop() honors grace window when source hangs', async () => { + const harness = new IngestionTestHarness(); + const source = makeMockSource({ stopDelayMs: 200 }); + await harness.run(source); + await expect(harness.stop(50)).rejects.toThrow(/did not stop within 50ms/); + }); + + test('stop() before run() is a no-op', async () => { + const harness = new IngestionTestHarness(); + await harness.stop(); + }); +}); + +describe('IngestionTestHarness validation', () => { + test('invalid events are routed to validationErrors, not events', async () => { + const harness = new IngestionTestHarness(); + const bad = { ...makeEvent(), content_type: 'totally-invalid' as never }; + const source = makeMockSource({ emitOnStart: [bad] }); + await harness.run(source); + expect(harness.events).toHaveLength(0); + expect(harness.validationErrors).toHaveLength(1); + expect(harness.validationErrors[0]).toContain('content_type'); + await harness.stop(); + }); + + test('mix of valid and invalid events: valid land, invalid surface as errors', async () => { + const good = makeEvent({ content: 'good', content_hash: computeContentHash('good') }); + const bad = { ...makeEvent(), source_id: '' }; // empty source_id + const harness = new IngestionTestHarness(); + const source = makeMockSource({ emitOnStart: [good, bad] }); + await harness.run(source); + expect(harness.events).toHaveLength(1); + expect(harness.validationErrors).toHaveLength(1); + await harness.stop(); + }); +}); + +describe('IngestionTestHarness clock', () => { + test('default clock starts at the deterministic seed', () => { + const harness = new IngestionTestHarness(); + expect(harness.now()).toBe(1747742400000); // 2026-05-20T12:00:00Z + }); + + test('explicit startTime is honored', () => { + const harness = new IngestionTestHarness({ startTime: 5000 }); + expect(harness.now()).toBe(5000); + }); + + test('advance(ms) bumps the clock', () => { + const harness = new IngestionTestHarness({ startTime: 1000 }); + harness.advance(500); + expect(harness.now()).toBe(1500); + }); + + test('advance rejects negative deltas', () => { + const harness = new IngestionTestHarness(); + expect(() => harness.advance(-1)).toThrow(/must be >= 0/); + }); +}); + +describe('IngestionTestHarness engine proxy', () => { + test('source touching the engine without an injected one throws with helpful message', async () => { + let caught: string | null = null; + const source: IngestionSource = { + id: 'mock', kind: 'mock', + async start(ctx) { + try { + // Accessing ANY engine property fires the proxy. + (ctx.engine as { getStats: () => unknown }).getStats(); + } catch (e) { + caught = e instanceof Error ? e.message : String(e); + } + }, + async stop() {}, + }; + const harness = new IngestionTestHarness(); + await harness.run(source); + expect(caught).not.toBeNull(); + expect(caught!).toContain('IngestionTestHarness'); + expect(caught!).toContain('BrainEngine'); + expect(caught!).toContain('getStats'); + await harness.stop(); + }); +}); + +describe('IngestionTestHarness healthCheck', () => { + test('returns ok when source has no healthCheck implementation', async () => { + const harness = new IngestionTestHarness(); + await harness.run(makeMockSource()); + const h = await harness.healthCheck(); + expect(h.status).toBe('ok'); + await harness.stop(); + }); + + test('returns the source\'s healthCheck result', async () => { + const source: IngestionSource = { + id: 'mock', kind: 'mock', + async start() {}, + async stop() {}, + async healthCheck() { + return { status: 'warn', message: 'API rate limit close' }; + }, + }; + const harness = new IngestionTestHarness(); + await harness.run(source); + const h = await harness.healthCheck(); + expect(h.status).toBe('warn'); + expect(h.message).toBe('API rate limit close'); + await harness.stop(); + }); + + test('healthCheck before run throws', async () => { + const harness = new IngestionTestHarness(); + await expect(harness.healthCheck()).rejects.toThrow(/no source started/); + }); +}); + +describe('IngestionTestHarness clearRecorded', () => { + test('clears events + logs + validationErrors but keeps running state', async () => { + const harness = new IngestionTestHarness(); + const source = makeMockSource({ emitOnStart: [makeEvent()] }); + await harness.run(source); + expect(harness.events).toHaveLength(1); + expect(harness.running).toBe(true); + harness.clearRecorded(); + expect(harness.events).toHaveLength(0); + expect(harness.running).toBe(true); + await harness.stop(); + }); +}); + +describe('expectEvent matchers', () => { + test('toExist returns the event on success', () => { + const ev = makeEvent(); + expect(expectEvent(ev).toExist()).toBe(ev); + }); + + test('toExist throws on undefined', () => { + expect(() => expectEvent(undefined).toExist()).toThrow(/was undefined/); + }); + + test('toHaveKind passes on match', () => { + expect(() => expectEvent(makeEvent()).toHaveKind('mock')).not.toThrow(); + }); + + test('toHaveKind throws on mismatch', () => { + expect(() => expectEvent(makeEvent()).toHaveKind('wrong')).toThrow(/expected source_kind/); + }); + + test('toHaveSourceUri accepts a regex matcher', () => { + const ev = makeEvent({ source_uri: '/tmp/foo.granola' }); + expect(() => expectEvent(ev).toHaveSourceUri(/\.granola$/)).not.toThrow(); + expect(() => expectEvent(ev).toHaveSourceUri(/\.mp3$/)).toThrow(); + }); + + test('toHaveSourceUri accepts a literal string', () => { + const ev = makeEvent({ source_uri: '/exact' }); + expect(() => expectEvent(ev).toHaveSourceUri('/exact')).not.toThrow(); + expect(() => expectEvent(ev).toHaveSourceUri('/other')).toThrow(); + }); + + test('toHaveContentHash with no arg validates shape only', () => { + expect(() => expectEvent(makeEvent()).toHaveContentHash()).not.toThrow(); + const bad = { ...makeEvent(), content_hash: 'not-hex' }; + expect(() => expectEvent(bad).toHaveContentHash()).toThrow(/not a valid SHA-256/); + }); + + test('toHaveContentHash with arg validates exact match', () => { + const ev = makeEvent({ content_hash: 'a'.repeat(64) }); + expect(() => expectEvent(ev).toHaveContentHash('a'.repeat(64))).not.toThrow(); + expect(() => expectEvent(ev).toHaveContentHash('b'.repeat(64))).toThrow(/expected content_hash/); + }); + + test('toBeUntrusted / toBeTrusted', () => { + const trusted = makeEvent({ untrusted_payload: false }); + const untrusted = makeEvent({ untrusted_payload: true }); + expect(() => expectEvent(trusted).toBeTrusted()).not.toThrow(); + expect(() => expectEvent(untrusted).toBeUntrusted()).not.toThrow(); + expect(() => expectEvent(trusted).toBeUntrusted()).toThrow(); + expect(() => expectEvent(untrusted).toBeTrusted()).toThrow(); + }); + + test('toHaveMetadata matches a subset', () => { + const ev = makeEvent({ metadata: { format: 'png', width: 1024 } }); + expect(() => expectEvent(ev).toHaveMetadata({ format: 'png' })).not.toThrow(); + expect(() => expectEvent(ev).toHaveMetadata({ width: 1024 })).not.toThrow(); + expect(() => expectEvent(ev).toHaveMetadata({ format: 'jpg' })).toThrow(); + }); +}); diff --git a/test/ingestion/types.test.ts b/test/ingestion/types.test.ts new file mode 100644 index 000000000..49d19bc49 --- /dev/null +++ b/test/ingestion/types.test.ts @@ -0,0 +1,195 @@ +/** + * Tests for the IngestionEvent + IngestionSource contract types. + * + * Pinned at the contract level: changing these tests is a public-API change. + * Treat them like test/public-exports.test.ts — they're a regression guard + * for skillpack publishers depending on the surface. + */ + +import { describe, expect, test } from 'bun:test'; +import { + INGESTION_CONTENT_TYPES, + INGESTION_SOURCE_API_VERSION, + IngestionEventError, + computeContentHash, + validateIngestionEvent, + type IngestionEvent, +} from '../../src/core/ingestion/types.ts'; + +const VALID_HASH = 'a'.repeat(64); + +function makeEvent(overrides: Partial = {}): IngestionEvent { + return { + source_id: 'test-source-1', + source_kind: 'file-watcher', + source_uri: '/tmp/test.md', + received_at: new Date('2026-05-20T12:00:00Z').toISOString(), + content_type: 'text/markdown', + content: '# test content', + content_hash: VALID_HASH, + ...overrides, + }; +} + +describe('IngestionSource contract constants', () => { + test('INGESTION_SOURCE_API_VERSION is the v1 string', () => { + expect(INGESTION_SOURCE_API_VERSION).toBe('gbrain-ingestion-source-v1'); + }); + + test('INGESTION_CONTENT_TYPES covers the documented taxonomy', () => { + expect(INGESTION_CONTENT_TYPES).toContain('text/markdown'); + expect(INGESTION_CONTENT_TYPES).toContain('text/plain'); + expect(INGESTION_CONTENT_TYPES).toContain('text/html'); + expect(INGESTION_CONTENT_TYPES).toContain('application/pdf'); + expect(INGESTION_CONTENT_TYPES).toContain('application/json'); + expect(INGESTION_CONTENT_TYPES).toContain('image/*'); + expect(INGESTION_CONTENT_TYPES).toContain('audio/*'); + expect(INGESTION_CONTENT_TYPES).toContain('video/*'); + expect(INGESTION_CONTENT_TYPES).toContain('unknown'); + }); +}); + +describe('computeContentHash', () => { + test('produces a 64-char lowercase hex string', () => { + const h = computeContentHash('hello world'); + expect(h).toMatch(/^[0-9a-f]{64}$/); + }); + + test('deterministic for the same input', () => { + expect(computeContentHash('foo')).toBe(computeContentHash('foo')); + }); + + test('different inputs produce different hashes', () => { + expect(computeContentHash('foo')).not.toBe(computeContentHash('bar')); + }); + + test('empty string is allowed and stable', () => { + const h = computeContentHash(''); + expect(h).toMatch(/^[0-9a-f]{64}$/); + expect(computeContentHash('')).toBe(h); + }); +}); + +describe('validateIngestionEvent — happy path', () => { + test('accepts a well-formed event', () => { + expect(validateIngestionEvent(makeEvent())).toBeNull(); + }); + + test('accepts events with optional metadata', () => { + const ev = makeEvent({ metadata: { format: 'png', width: 1024 } }); + expect(validateIngestionEvent(ev)).toBeNull(); + }); + + test('accepts events with untrusted_payload true', () => { + expect(validateIngestionEvent(makeEvent({ untrusted_payload: true }))).toBeNull(); + }); + + test('accepts events with untrusted_payload false', () => { + expect(validateIngestionEvent(makeEvent({ untrusted_payload: false }))).toBeNull(); + }); + + test('accepts every content_type in the taxonomy', () => { + for (const ct of INGESTION_CONTENT_TYPES) { + const ev = makeEvent({ content_type: ct }); + expect(validateIngestionEvent(ev)).toBeNull(); + } + }); +}); + +describe('validateIngestionEvent — rejection cases', () => { + test('rejects null', () => { + const err = validateIngestionEvent(null); + expect(err).toBeInstanceOf(IngestionEventError); + expect(err?.field).toBe('root'); + }); + + test('rejects non-object', () => { + const err = validateIngestionEvent('not an event'); + expect(err).toBeInstanceOf(IngestionEventError); + }); + + test.each([ + 'source_id', + 'source_kind', + 'source_uri', + 'received_at', + 'content', + 'content_hash', + ] as const)('rejects missing required field: %s', (field) => { + const ev = makeEvent(); + delete (ev as unknown as Record)[field]; + const err = validateIngestionEvent(ev); + expect(err?.field).toBe(field); + }); + + test.each([ + 'source_id', + 'source_kind', + 'source_uri', + 'received_at', + 'content', + 'content_hash', + ] as const)('rejects empty string for required field: %s', (field) => { + const ev = { ...makeEvent(), [field]: '' }; + const err = validateIngestionEvent(ev); + expect(err?.field).toBe(field); + }); + + test('rejects unknown content_type', () => { + const ev = makeEvent({ content_type: 'application/x-malware' as never }); + const err = validateIngestionEvent(ev); + expect(err?.field).toBe('content_type'); + }); + + test('rejects malformed received_at (not parseable)', () => { + const ev = makeEvent({ received_at: 'not a date' }); + const err = validateIngestionEvent(ev); + expect(err?.field).toBe('received_at'); + }); + + test('rejects malformed content_hash (too short)', () => { + const ev = makeEvent({ content_hash: 'abc123' }); + const err = validateIngestionEvent(ev); + expect(err?.field).toBe('content_hash'); + }); + + test('rejects malformed content_hash (non-hex characters)', () => { + const ev = makeEvent({ content_hash: 'Z'.repeat(64) }); + const err = validateIngestionEvent(ev); + expect(err?.field).toBe('content_hash'); + }); + + test('rejects non-boolean untrusted_payload', () => { + const ev = { ...makeEvent(), untrusted_payload: 'yes' }; + const err = validateIngestionEvent(ev); + expect(err?.field).toBe('untrusted_payload'); + }); + + test('rejects null metadata', () => { + const ev = { ...makeEvent(), metadata: null }; + const err = validateIngestionEvent(ev); + expect(err?.field).toBe('metadata'); + }); + + test('rejects array metadata', () => { + const ev = { ...makeEvent(), metadata: [1, 2, 3] }; + const err = validateIngestionEvent(ev); + expect(err?.field).toBe('metadata'); + }); +}); + +describe('IngestionEventError', () => { + test('carries field, reason, and event payload', () => { + const err = new IngestionEventError('content_hash', 'too short', { source_id: 'x' }); + expect(err.field).toBe('content_hash'); + expect(err.reason).toBe('too short'); + expect(err.event).toEqual({ source_id: 'x' }); + expect(err.message).toContain('content_hash'); + expect(err.name).toBe('IngestionEventError'); + }); + + test('is an instance of Error', () => { + const err = new IngestionEventError('source_id', 'missing', {}); + expect(err).toBeInstanceOf(Error); + }); +}); diff --git a/test/markdown-serializer.test.ts b/test/markdown-serializer.test.ts new file mode 100644 index 000000000..23d79deff --- /dev/null +++ b/test/markdown-serializer.test.ts @@ -0,0 +1,221 @@ +/** + * v0.38 — DRY-extract helpers in `src/core/markdown.ts`. Two pure functions: + * + * - `serializePageToMarkdown(page, tags, opts?)` — render a Page row to its + * canonical on-disk markdown form. Consumed by the dream-cycle reverse- + * render in `src/core/cycle/synthesize.ts` AND by the put_page write- + * through path in `src/core/operations.ts`. Both used to inline-duplicate + * ~90% of `serializeMarkdown` setup; the extract collapses them to a + * 4-line wrapper. The `frontmatterOverrides` option is the only thing + * that differs at the two call sites (dream stamps `dream_generated`, + * put_page stamps `ingested_via` + `ingested_at`). + * + * - `resolvePageFilePath(brainDir, slug, sourceId)` — single source of + * truth for the v0.32.8 multi-source filing layout: + * - default source → `/.md` + * - non-default → `/.sources//.md` + * Shared by `reverseWriteRefs` and the put_page write-through path so + * both compute the same path for the same row. Caller is responsible + * for validating `source_id` against path-traversal attacks via + * `validateSourceId` BEFORE passing it here — the helper does filename + * math only. + * + * These pure functions are the DRY extract that the v0.38 plan-eng-review + * locked in. Without focused tests, future schema-shape changes to either + * helper could silently drift the two call sites apart. The test pins: + * - Frontmatter merge precedence (page < overrides < tags) + * - Type / title defaults when Page columns are sparse + * - The empty-overrides happy path (matches pre-v0.38 behavior) + * - Provenance stamping shape (the v0.38 use case) + * - Path composition for both source layouts + * - The exact filename produced for nested slugs (people/alice → file ok) + */ +import { describe, expect, test } from 'bun:test'; +import { join } from 'node:path'; +import type { Page } from '../src/core/types.ts'; +import { + resolvePageFilePath, + serializePageToMarkdown, +} from '../src/core/markdown.ts'; + +function buildPage(overrides: Partial = {}): Page { + return { + id: 1, + slug: 'wiki/people/alice', + type: 'person', + title: 'Alice Example', + compiled_truth: 'Alice is a founder.', + timeline: '', + frontmatter: {}, + created_at: new Date('2026-05-21T00:00:00Z'), + updated_at: new Date('2026-05-21T00:00:00Z'), + source_id: 'default', + ...overrides, + } as Page; +} + +describe('serializePageToMarkdown — DRY extract for dream + put_page write-through', () => { + test('renders a minimal page with no overrides', () => { + const page = buildPage(); + const md = serializePageToMarkdown(page, []); + // Frontmatter open + close fences should bracket the body. + expect(md).toMatch(/^---\n/); + expect(md).toContain('title: Alice Example'); + expect(md).toContain('type: person'); + expect(md).toContain('Alice is a founder.'); + }); + + test('opts.frontmatterOverrides win over page.frontmatter on key collisions', () => { + const page = buildPage({ + frontmatter: { ingested_via: 'original', custom_key: 'kept' }, + }); + const md = serializePageToMarkdown(page, [], { + frontmatterOverrides: { ingested_via: 'put_page' }, + }); + // The override won. + expect(md).toMatch(/ingested_via:\s*put_page/); + // The unrelated key from page.frontmatter survived the merge. + expect(md).toContain('custom_key: kept'); + // The original value is gone. + expect(md).not.toMatch(/ingested_via:\s*original/); + }); + + test('stamps v0.38 provenance frontmatter (the put_page write-through use case)', () => { + const page = buildPage(); + const md = serializePageToMarkdown(page, [], { + frontmatterOverrides: { + ingested_via: 'put_page', + ingested_at: '2026-05-21T04:15:00Z', + source_kind: 'capture-cli', + }, + }); + expect(md).toMatch(/ingested_via:\s*put_page/); + expect(md).toContain('2026-05-21T04:15:00Z'); + expect(md).toContain('capture-cli'); + }); + + test('stamps dream_generated frontmatter (the dream-cycle reverse-render use case)', () => { + const page = buildPage(); + const md = serializePageToMarkdown(page, [], { + frontmatterOverrides: { + dream_generated: true, + dream_cycle_date: '2026-05-21', + }, + }); + expect(md).toMatch(/dream_generated:\s*true/); + expect(md).toContain('2026-05-21'); + }); + + test('falls back to type=note when Page.type is missing', () => { + // Coerce to bypass the strict typed Page interface — exercising the + // defensive default that the helper documents. + const page = buildPage({ type: undefined as unknown as Page['type'] }); + const md = serializePageToMarkdown(page, []); + expect(md).toContain('type: note'); + }); + + test('falls back to empty title when Page.title is missing', () => { + const page = buildPage({ title: undefined as unknown as Page['title'] }); + const md = serializePageToMarkdown(page, []); + // The helper should not crash; the title line should be empty or absent. + // serializeMarkdown emits `title: ` (no value) for empty titles. + expect(md).toMatch(/title:\s*('')?\s*\n/); + }); + + test('preserves timeline section when present', () => { + const page = buildPage({ + timeline: '## Timeline\n\n- 2026-01-01: Founded', + }); + const md = serializePageToMarkdown(page, []); + expect(md).toContain('## Timeline'); + expect(md).toContain('2026-01-01: Founded'); + }); + + test('handles empty compiled_truth + empty timeline (page with frontmatter only)', () => { + const page = buildPage({ compiled_truth: '', timeline: '' }); + const md = serializePageToMarkdown(page, []); + // Frontmatter close fence should still appear. + expect(md).toMatch(/^---\n/); + expect(md.match(/---/g)?.length ?? 0).toBeGreaterThanOrEqual(2); + }); + + test('passes tags through to the underlying serializer', () => { + const page = buildPage(); + const md = serializePageToMarkdown(page, ['yc', 'w2025']); + expect(md).toContain('yc'); + expect(md).toContain('w2025'); + }); + + test('omits opts argument entirely (matches the dream-cycle wrapper signature)', () => { + const page = buildPage(); + // The dream-cycle wrapper called serializePageToMarkdown(page, tags) + // with no opts argument. Pin that this still works post-extract. + const md = serializePageToMarkdown(page, []); + expect(md).toContain('Alice Example'); + }); + + test('null/undefined frontmatter on Page does not crash', () => { + const page = buildPage({ + frontmatter: undefined as unknown as Page['frontmatter'], + }); + const md = serializePageToMarkdown(page, [], { + frontmatterOverrides: { ingested_via: 'put_page' }, + }); + expect(md).toMatch(/ingested_via:\s*put_page/); + }); +}); + +describe('resolvePageFilePath — single source of truth for v0.32.8 filing layout', () => { + test('default source writes to /.md', () => { + const path = resolvePageFilePath('/brain', 'wiki/people/alice', 'default'); + expect(path).toBe(join('/brain', 'wiki/people/alice.md')); + }); + + test('non-default source writes under /.sources//.md', () => { + const path = resolvePageFilePath('/brain', 'wiki/people/alice', 'gstack'); + expect(path).toBe(join('/brain', '.sources', 'gstack', 'wiki/people/alice.md')); + }); + + test('source_id is the literal string "default" — anything else routes to .sources/', () => { + // The discriminator is exactly the string 'default'. Casing matters. + const lower = resolvePageFilePath('/brain', 's', 'default'); + const upper = resolvePageFilePath('/brain', 's', 'Default'); + expect(lower).toBe(join('/brain', 's.md')); + expect(upper).toBe(join('/brain', '.sources', 'Default', 's.md')); + }); + + test('empty string source_id is treated as non-default', () => { + // Regression: '' is not 'default' so it lands in .sources//. This is + // a bug surface — `validateSourceId` is the caller's responsibility to + // reject. Pin the current behavior so a refactor doesn't quietly change it. + const path = resolvePageFilePath('/brain', 's', ''); + expect(path).toBe(join('/brain', '.sources', '', 's.md')); + }); + + test('nested slug paths join cleanly without traversal artifacts', () => { + const path = resolvePageFilePath('/brain', 'wiki/companies/acme-co', 'default'); + expect(path).toBe(join('/brain', 'wiki/companies/acme-co.md')); + expect(path).not.toContain('..'); + }); + + test('non-default source path with nested slug preserves structure', () => { + const path = resolvePageFilePath('/brain', 'inbox/2026-05-21-thought', 'mobile'); + expect(path).toBe(join('/brain', '.sources', 'mobile', 'inbox', '2026-05-21-thought.md')); + }); + + test('absolute and relative brainDir both work (path.join semantics)', () => { + const abs = resolvePageFilePath('/absolute/brain', 's', 'default'); + const rel = resolvePageFilePath('./relative/brain', 's', 'default'); + expect(abs).toBe(join('/absolute/brain', 's.md')); + expect(rel).toBe(join('./relative/brain', 's.md')); + }); + + test('source_id with spaces or special chars passes through (caller validates)', () => { + // Pin that the helper does NOT mutate source_id — validation is the + // caller's job per the helper's documented contract. If this ever + // changes (helper starts validating internally) a real implementation + // change is needed and this test surfaces it. + const path = resolvePageFilePath('/brain', 's', 'has spaces'); + expect(path).toBe(join('/brain', '.sources', 'has spaces', 's.md')); + }); +}); diff --git a/test/migrate.test.ts b/test/migrate.test.ts index d41f28af7..312fed53e 100644 --- a/test/migrate.test.ts +++ b/test/migrate.test.ts @@ -1796,3 +1796,183 @@ describe('migrate v80 — CHECK widening end-to-end on PGLite', () => { expect(typeof scorecard.partial_rate).not.toBe('undefined'); }); }); + +// ─── v0.38.0.0 — v81 pages_provenance_columns ───────────────────────────── +// +// Adds four nullable provenance columns to `pages` so every ingested page +// carries a record of WHERE it came from (capture-cli, webhook, put_page, +// dream, etc.). The columns are populated by the put_page write-through +// path AND by the `ingest_capture` Minion handler. NULL is the +// historical-page default — pre-v0.38 pages never had provenance. +// +// Renumbered v80 → v81 during master merge with v0.37.2.0's +// takes_unresolvable_quality_v0_37_2_0 hotfix (which claimed v80 first). +// +// Structural assertions pin the migration's SQL shape; the PGLite +// round-trip below verifies the columns are actually queryable + nullable +// after `initSchema()`. Schema-bootstrap-coverage covers the forward- +// reference probe contract separately at test/schema-bootstrap-coverage.test.ts. + +describe('migrate v81 — pages_provenance_columns', () => { + const v81 = MIGRATIONS.find(m => m.version === 81); + + test('v81 entry exists with the documented name', () => { + expect(v81).toBeDefined(); + expect(v81!.name).toBe('pages_provenance_columns'); + }); + + test('v81 is marked idempotent so re-runs are safe', () => { + expect(v81!.idempotent).toBe(true); + }); + + test('v81 adds exactly four provenance columns to pages', () => { + const sql = (v81!.sql ?? '').toLowerCase(); + expect(sql).toContain('alter table pages add column if not exists ingested_via text'); + expect(sql).toContain('alter table pages add column if not exists ingested_at timestamptz'); + expect(sql).toContain('alter table pages add column if not exists source_uri text'); + expect(sql).toContain('alter table pages add column if not exists source_kind text'); + }); + + test('v81 uses IF NOT EXISTS for every ALTER — re-run-safe on partial states', () => { + const sql = (v81!.sql ?? '').toLowerCase(); + // Four ADD COLUMN statements, every one guarded. + const guarded = sql.match(/add column if not exists/g) ?? []; + expect(guarded.length).toBe(4); + }); + + test('v81 columns are nullable (no NOT NULL constraint, no DEFAULT)', () => { + const sql = (v81!.sql ?? '').toLowerCase(); + // ADD COLUMN with NULL default is metadata-only on Postgres 11+ and + // PGLite 17.5 — instant on tables of any size. Regression guard: any + // future contributor who adds NOT NULL or DEFAULT must update this + // assertion deliberately, since both flip the migration from O(1) to + // O(N) rewrite on large tables. + expect(sql).not.toMatch(/ingested_via\s+text\s+not\s+null/); + expect(sql).not.toMatch(/ingested_at\s+timestamptz\s+not\s+null/); + expect(sql).not.toMatch(/source_uri\s+text\s+not\s+null/); + expect(sql).not.toMatch(/source_kind\s+text\s+not\s+null/); + expect(sql).not.toMatch(/ingested_via\s+text\s+default/); + }); + + test('v81 does NOT create any index (provenance is admin-surface only)', () => { + const sql = (v81!.sql ?? '').toLowerCase(); + // Documented in the migration comment: provenance queries are admin- + // surface only (admin SPA Sources tab + gbrain doctor + // ingestion_health). Throwing an index on a low-cardinality TEXT + // column would inflate the brain repo for negligible read benefit. + expect(sql).not.toContain('create index'); + }); +}); + +describe('migrate v81 — round-trip on PGLite', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('all four provenance columns exist on pages after initSchema', async () => { + const rows = await engine.executeRaw<{ column_name: string; is_nullable: string; data_type: string }>( + `SELECT column_name, is_nullable, data_type + FROM information_schema.columns + WHERE table_name = 'pages' + AND column_name IN ('ingested_via', 'ingested_at', 'source_uri', 'source_kind') + ORDER BY column_name`, + [], + ); + expect(rows.length).toBe(4); + const byName = new Map(rows.map(r => [r.column_name, r])); + expect(byName.get('ingested_via')?.is_nullable).toBe('YES'); + expect(byName.get('ingested_at')?.is_nullable).toBe('YES'); + expect(byName.get('source_uri')?.is_nullable).toBe('YES'); + expect(byName.get('source_kind')?.is_nullable).toBe('YES'); + // ingested_at must be TIMESTAMPTZ — pin the type so an accidental + // bump to TIMESTAMP (no zone) doesn't slip through. + expect(byName.get('ingested_at')?.data_type.toLowerCase()).toContain('timestamp'); + }); + + test('inserting a page with full provenance round-trips through getPage', async () => { + const slug = `wiki/inbox/v81-provenance-${Date.now()}`; + await engine.putPage(slug, { + type: 'note', + title: 'v81 provenance round-trip', + compiled_truth: 'A note with provenance.', + timeline: '', + frontmatter: { + ingested_via: 'capture-cli', + ingested_at: '2026-05-21T04:15:00Z', + source_uri: 'cli://capture/test', + source_kind: 'capture', + }, + content_hash: `v81-${Math.random()}`, + }); + const page = await engine.getPage(slug); + expect(page).not.toBeNull(); + // The frontmatter columns persist via the JSONB blob, not the + // dedicated provenance columns yet — write paths that target the + // columns directly are the put_page write-through + ingest_capture + // handler covered separately. The point of this test is the schema + // shape is correct so a future direct-column writer can land cleanly. + expect((page!.frontmatter as Record).ingested_via).toBe('capture-cli'); + }); + + test('pre-v0.38 page with NULL provenance columns is queryable', async () => { + // Simulates the historical-page upgrade scenario: a row whose + // provenance columns were never populated. Should not break any SQL + // path that touches `pages`. + const slug = `wiki/legacy/v81-null-prov-${Date.now()}`; + await engine.executeRaw( + `INSERT INTO pages (slug, type, title, compiled_truth, timeline, frontmatter, content_hash, source_id) + VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7, $8)`, + [slug, 'note', 'legacy', 'body', '', '{}', `v81-legacy-${Math.random()}`, 'default'], + ); + const rows = await engine.executeRaw<{ ingested_via: string | null; ingested_at: Date | null }>( + `SELECT ingested_via, ingested_at FROM pages WHERE slug = $1`, + [slug], + ); + expect(rows.length).toBe(1); + expect(rows[0].ingested_via).toBeNull(); + expect(rows[0].ingested_at).toBeNull(); + }); + + test('directly UPDATE-ing the provenance columns succeeds (no constraint blocks)', async () => { + // Pins that nothing on the column shape (e.g. an accidental CHECK) + // would reject a write the put_page write-through path is going to do. + const slug = `wiki/test/v81-update-${Date.now()}`; + await engine.putPage(slug, { + type: 'note', + title: 'v81 update test', + compiled_truth: '', + timeline: '', + frontmatter: {}, + content_hash: `v81-upd-${Math.random()}`, + }); + await engine.executeRaw( + `UPDATE pages + SET ingested_via = $1, + ingested_at = now(), + source_uri = $2, + source_kind = $3 + WHERE slug = $4`, + ['put_page', 'mcp://put_page/test', 'mcp', slug], + ); + const rows = await engine.executeRaw<{ + ingested_via: string | null; + source_uri: string | null; + source_kind: string | null; + }>( + `SELECT ingested_via, source_uri, source_kind FROM pages WHERE slug = $1`, + [slug], + ); + expect(rows[0].ingested_via).toBe('put_page'); + expect(rows[0].source_uri).toBe('mcp://put_page/test'); + expect(rows[0].source_kind).toBe('mcp'); + }); +}); + diff --git a/test/public-exports.test.ts b/test/public-exports.test.ts index bf783f9c6..c911de1f7 100644 --- a/test/public-exports.test.ts +++ b/test/public-exports.test.ts @@ -51,6 +51,8 @@ const EXPECTED_EXPORTS: ExpectedExport[] = [ { subpath: 'gbrain/search/expansion', canary: ['expandQuery'] }, { subpath: 'gbrain/ai/gateway', canary: ['configureGateway', 'embed'] }, { subpath: 'gbrain/extract', canary: [] }, + { subpath: 'gbrain/ingestion', canary: ['INGESTION_SOURCE_API_VERSION', 'validateIngestionEvent', 'computeContentHash'] }, + { subpath: 'gbrain/ingestion/test-harness', canary: ['IngestionTestHarness', 'expectEvent'] }, ]; function readPackageExports(): Record { @@ -66,7 +68,7 @@ describe('public exports — package.json exports map', () => { // Adding new exports: increment this + add to EXPECTED_EXPORTS below. // Removing exports: see CLAUDE.md "Removing any of these is a // breaking change going forward" — bump minor and update this count. - expect(count).toBe(18); + expect(count).toBe(20); }); test('EXPECTED_EXPORTS list matches the exports map exactly (no drift)', () => { diff --git a/test/schema-bootstrap-coverage.test.ts b/test/schema-bootstrap-coverage.test.ts index 4121a27e1..d9d20ac37 100644 --- a/test/schema-bootstrap-coverage.test.ts +++ b/test/schema-bootstrap-coverage.test.ts @@ -134,6 +134,17 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [ // have pages without this column; bootstrap adds it before SCHEMA_SQL // replay creates the index. { kind: 'column', table: 'pages', column: 'last_retrieved_at' }, + // v0.38.0 (v81) — pages_provenance_columns adds four nullable columns + // (ingested_via, ingested_at, source_uri, source_kind) to track WHERE + // every page came from (capture-cli, webhook, put_page, dream, etc.). + // No SCHEMA_SQL index/FK references them today, but bootstrap probes + // are added defense-in-depth so future schema work that does reference + // them doesn't wedge pre-v81 brains. Renumbered v80→v81 during master + // merge with v0.37.2.0 takes_unresolvable_quality hotfix. + { kind: 'column', table: 'pages', column: 'ingested_via' }, + { kind: 'column', table: 'pages', column: 'ingested_at' }, + { kind: 'column', table: 'pages', column: 'source_uri' }, + { kind: 'column', table: 'pages', column: 'source_kind' }, ]; test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => {