diff --git a/CHANGELOG.md b/CHANGELOG.md index 61dd6e4a6..0bc4160ed 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,99 @@ All notable changes to GBrain will be documented in this file. +## [0.35.5.0] - 2026-05-16 + +**Upgrade goes through on stuck Supabase brains. Orphan counts get honest. `gbrain think` over MCP actually answers. Worktrees stop being misclassified. node_modules stops leaking.** + +A correctness wave. Five user-visible bugs closed; one structural CI guard added so the bootstrap class can't bite the same shape again. The shape of the change: `applyForwardReferenceBootstrap` now probes for seven previously-missing columns (files.source_id/page_id, oauth_clients.source_id/federated_read, sources.archived/archived_at/archive_expires_at) AND runs on the same DDL connection that holds the advisory lock, `findOrphanPages` filters soft-deleted pages on BOTH the candidate side AND the link-source side, `runThink` routes through `gateway.chat()` so the API key configured via `gbrain config set` actually reaches MCP stdio launches, `manageGitignore` distinguishes worktrees from submodules via Git's documented `/modules/` vs `/worktrees/` path segments, and walkers in `extract.ts` + `transcript-discovery.ts` consult a new `pruneDir()` helper at descent time so they don't recurse into vendor directories. + +### What you can now do + +**Upgrade gbrain cleanly on Supabase brains stuck at pre-v0.18, pre-v0.26.5, or pre-v0.34 schemas.** The bootstrap forward-reference class (eleven incidents in two years per the test file's own comment block) loses its largest remaining lineage. Pre-v0.18 brains had `files` without `source_id`/`page_id`, so the `idx_files_source_id` index creation crashed before any migration could run. Pre-v0.26.5 brains had `sources` without the archive lifecycle columns (`archived`, `archived_at`, `archive_expires_at`), but `CREATE TABLE IF NOT EXISTS sources` is a no-op on existing tables so the schema-blob replay couldn't add them — every downstream visibility filter in search/list_pages tripped. Pre-v0.34 brains had `oauth_clients` without the source-scoping columns (`source_id` FK, `federated_read` GIN-indexed array) so the v60+v61+v65 chain failed the same way. All seven columns now have explicit `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` probes in both engines. **And the probes now run on the DDL connection that holds the advisory lock** — pre-fix they ran through the instance pool while the lock sat on a different connection, opening a concurrent-bootstrap race for anyone on Supabase's transaction pooler. Codex caught that during pre-landing review. + +**Trust orphan counts again.** `gbrain orphans` filtered nothing on `deleted_at` — soft-deleted pages (v0.26.5 soft-delete shipped without updating this query) appeared as orphans, inflating counts. Worse, links from soft-deleted source pages still counted as inbound, so a live page could hide from orphan results purely because a deleted page linked to it. The query now filters BOTH sides: `p.deleted_at IS NULL` on the candidate, `src.deleted_at IS NULL` on the inbound-link JOIN. The structural correctness rule for soft-delete (active relationships only) lands the same shape across both engines. + +**`gbrain think` answers over MCP when you've set the key via `gbrain config`.** Before this release, MCP stdio launches (Claude Desktop, Cursor, etc.) returned `(no LLM available — set ANTHROPIC_API_KEY or pass client)` regardless of whether `gbrain config set anthropic_api_key sk-...` had been run, because `runThink` instantiated `new Anthropic()` directly and the Anthropic SDK only reads `process.env.ANTHROPIC_API_KEY`. Stdio launches from desktop apps don't inherit shell env, so the gbrain-config-set key never reached the SDK. `runThink` now routes through `gateway.chat()` — the canonical AI seam per CLAUDE.md that v0.31.12 established for chat/embed/expansion. The gateway reads `anthropic_api_key` from `~/.gbrain/config.json` AND from env, so both paths work. Existing `opts.client` injection is preserved (the test seam stays unchanged); `opts.stubResponse` continues to bypass the LLM call entirely. When neither the key nor a client is available, the graceful "no LLM available" stub still fires with the same `NO_ANTHROPIC_API_KEY` warning — the legacy diagnostic signal lives. + +**Use gbrain inside Conductor worktrees without `.gitignore` management silently breaking.** `manageGitignore` (the storage-tiering helper that updates `.gitignore` to exclude `db_only:` paths) treated every `.git`-as-file as a submodule and skipped management. Both submodules AND worktrees use `.git` as a file (worktrees just have a different file content). The discriminator now matches the gitdir path segment: `/modules/` = submodule, `/worktrees/` = worktree. Worktrees are first-class repos and now get `.gitignore` management. Handles all four combinations of {relative, absolute} × {modules, worktrees} including the absorbed-submodule case from `git submodule absorbgitdirs` that the legacy absolute-vs-relative discriminator would have misclassified. + +**Walkers stop wasting IO on `node_modules` (and frontmatter scans stop emitting false MISSING_OPEN warnings from inside vendor packages).** The canonical `isSyncable` filter had only a dot-prefix exclusion — `.git` and `.obsidian` were blocked but `node_modules` slipped through (no leading dot). A new `pruneDir()` helper centralizes the directory-exclusion logic at single-segment granularity: blocks `node_modules`, dot-prefix dirs, `ops`, and `*.raw` sidecars. Both walkers (`walkMarkdownFiles` in extract.ts, `listTextFiles` in transcript-discovery.ts) consult it at descent time BEFORE recursing, so the IO cost of walking thousands of vendor files is saved. `isSyncable` itself also gains node_modules exclusion — verified blast radius across every existing caller (sync, frontmatter, brain-writer, import) confirms node_modules exclusion was already the desired behavior. `transcript-discovery` also moves from non-recursive to recursive walking with the same pruneDir guard, so transcripts in nested corpus subdirectories (`corpus/2026/...`) become discoverable. + +### Itemized changes + +#### Closed issues +- **#1018** PGLite upgrade wedge: applyForwardReferenceBootstrap missing v60 oauth_clients forward refs +- **#974** Bootstrap gap: applyForwardReferenceBootstrap misses files.source_id + files.page_id +- **#820** v0.13.0 migration files.page_id forward-reference cascade on pre-v0.18 brains +- **#1021** orphans: soft-deleted pages still counted in orphan scan +- **#952** think over MCP returns "no LLM available" even when ANTHROPIC_API_KEY configured via gbrain config +- **#889** Git worktrees misclassified as submodules — breaks .gitignore management on Conductor workflows +- **#923** gbrain frontmatter validate walks node_modules, inflating MISSING_OPEN counts in doctor +- **#202** extract --source fs walker does not respect isSyncable prefix exclusions + +#### Bootstrap structural fix +- Seven new probes in `applyForwardReferenceBootstrap` (both engines): `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. +- Bootstrap now accepts the DDL connection from `initSchema` and runs probes inside the advisory-lock scope (Codex pre-landing review P1). +- New MIGRATIONS-source introspection contract test in `test/schema-bootstrap-coverage.test.ts`: parses `src/core/migrate.ts` for every `ALTER TABLE ... ADD COLUMN`, asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies. Catches the column-only forward-reference class (sources.archived* shape) that the pre-existing CREATE INDEX parser couldn't see. Source-text introspection covers all three migration shapes: top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, and handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`. +- Pre-existing parser bug in `parseBaseTableColumns` fixed: now strips SQL line comments (`-- ...`) and block comments (`/* ... */`) before identifying column names. Pre-fix, any column preceded by a comment line inside the CREATE TABLE body was silently dropped (e.g. `page_kind`, `deleted_at`, `emotional_weight`, `effective_date` — all dropped from coverage). + +#### Walker correctness +- New exported `pruneDir(name: string): boolean` helper in `src/core/sync.ts`. Blocks `node_modules`, dot-prefix directories, `ops`, and `*.raw` sidecars. +- `isSyncable` now applies `pruneDir` per path segment (legacy dot-prefix + `.raw/` + `ops/` checks consolidated). +- `walkMarkdownFiles` (extract.ts) consults `pruneDir` at descent + uses `isSyncable({strategy: 'markdown'})` at file emit. Pre-fix the walker had ad-hoc dot-prefix exclusion and didn't call isSyncable at all. +- `listTextFiles` (transcript-discovery.ts) now recursive with `pruneDir`-guarded descent. Uses its own `.txt`/`.md` predicate rather than the markdown-strategy isSyncable that would have rejected `.txt` and applied markdown-only README/ops exclusions. + +#### Soft-delete correctness +- `findOrphanPages` (both engines) filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side. + +#### MCP think gateway routing +- `runThink` (`src/core/think/index.ts`) drops the direct `new Anthropic()` instantiation. New `tryBuildGatewayClient` probes recipe availability + ANTHROPIC_API_KEY presence (reads BOTH `process.env` and `loadConfig()`'s gbrain config). Returns null on miss, preserving the legacy graceful-degradation path. +- New `chatResultToMessage` adapter converts gateway's `ChatResult` to an Anthropic-Message-shaped object. `mapStopReason` translates provider-neutral stop reasons (`end` / `length` / `tool_calls` / `refusal` / `content_filter` / `other`) to Anthropic's `stop_reason` enum. + +#### Git worktree discriminator +- `manageGitignore` (`src/commands/sync.ts`) reads the `.git` file content and matches the gitdir path segment: `/modules/` → skip (submodule, parent-managed), `/worktrees/` → MANAGE (worktree, first-class repo). Malformed `.git` file (no `gitdir:` prefix, IO error) defaults to MANAGE preserving pre-fix catch{} semantics. + +#### Tests +- `test/helpers/extract-added-columns.ts` (NEW): MIGRATIONS source introspection helper with internal test seam for the regex. +- `test/schema-bootstrap-coverage.test.ts`: extended `REQUIRED_BOOTSTRAP_COVERAGE` with 7 new entries + 4 new test cases (MIGRATIONS contract, sanity check, regex shape variants, planted-bug regression). +- `test/orphans.test.ts`: 3 new cases pinning the both-sides soft-delete filter — soft-deleted candidate excluded, codex C11 link-source case, live-link smoke regression. +- `test/think-gateway-adapter.test.ts` (NEW): 9 cases covering response-shape conversion, stop-reason mapping, model-id normalization (bare + prefixed), unknown-provider fallback, no-API-key fallback, hasAnthropicKey env read. +- `test/storage-sync.test.ts`: 5 new cases for worktree/submodule discrimination (relative + absolute × modules + worktrees, malformed `.git` file). +- `test/sync.test.ts`: 7 new isSyncable + 6 new pruneDir cases including the node_modules CRITICAL regression. + +#### Critical files +- `src/core/{pglite,postgres}-engine.ts` — `applyForwardReferenceBootstrap` (extended; Postgres engine threads DDL connection); `findOrphanPages` (both sides filter). +- `src/core/think/index.ts` — gateway adapter + ThinkLLMClient backward compat. +- `src/core/sync.ts` — new `pruneDir` export; `isSyncable` per-segment pruning. +- `src/commands/{extract,sync}.ts` — walker pruneDir adoption; worktree discriminator. +- `src/core/cycle/transcript-discovery.ts` — recursive walker with pruneDir. +- `test/helpers/extract-added-columns.ts` — NEW. +- `test/schema-bootstrap-coverage.test.ts` — extended. + +#### Follow-ups filed +- `TODOS.md`: runThink full rewrite to drop ThinkLLMClient indirection now that the gateway adapter is in place. Supabase parity test fixture for the bootstrap connection-threading (the bug is fixed; the test fixture proving it under real pooler topology is the residual). + +## To take advantage of v0.35.5.0 + +`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration on a pre-v0.18 / pre-v0.26.5 / pre-v0.34 brain: + +1. **Run the orchestrator manually:** + ```bash + gbrain apply-migrations --yes + ``` +2. **Verify the bootstrap landed:** + ```bash + gbrain doctor --json | grep -E 'schema_version|bootstrap' + ``` +3. **Confirm the new columns exist on Supabase brains** (the wave's primary target): + ```bash + psql "$DATABASE_URL" -c "\d sources" | grep -E 'archived|archive_expires_at' + psql "$DATABASE_URL" -c "\d files" | grep -E 'source_id|page_id' + psql "$DATABASE_URL" -c "\d oauth_clients" | grep -E 'source_id|federated_read' + ``` +4. **If `gbrain think` was returning "no LLM available" over MCP** despite a configured key, no manual action needed — the gateway adapter activates as soon as v0.35.5.0 is installed. +5. **If any step fails**, file an issue with the output of `gbrain doctor` and contents of `~/.gbrain/upgrade-errors.jsonl` at https://github.com/garrytan/gbrain/issues. + ## [0.35.4.0] - 2026-05-16 **Doctor stops crying wolf on clean worker restarts. Entity resolver stops spawning phantom pages. Prefix-expansion gets 58x faster.** diff --git a/CLAUDE.md b/CLAUDE.md index 8e1e02c12..35be02d32 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,9 +43,9 @@ strict behavior when unset. - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -56,7 +56,7 @@ strict behavior when unset. - `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim). - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) - `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags) -- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500. +- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). **v0.35.5.0:** new exported `pruneDir(name: string): boolean` helper is the single source of truth for descent-time directory exclusion across walkers. Blocks `node_modules` (no leading dot, so pre-v0.35.5 walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars. `isSyncable` now applies it per path segment; `walkMarkdownFiles` in `src/commands/extract.ts` and `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing so the IO cost of walking thousands of vendor files is saved. Closes #923 + #202. `manageGitignore` worktree fix in same wave: discriminator now matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) instead of the legacy absolute-vs-relative check that misclassified absorbed submodules and worktrees both. Conductor worktrees are first-class repos and now get `.gitignore` management for storage-tiering. Closes #889. v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local) - `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape. - `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens). @@ -85,6 +85,8 @@ strict behavior when unset. - `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count. - `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow. - `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` (v0.32.6) — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned per Codex; UTF-8-safe truncation; C1 confidence-floor double-enforcement; resolution_kind output drives M7 paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (Codex outside-voice fix — prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — Codex fix to bias from silent skip), M5 trend writes to `eval_contradictions_runs`, M6 source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker — stable cache hit-rate across re-runs). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (P1 batched), `writeContradictionsRun` + `loadContradictionsTrend` (M5), `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` (P2). Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). M1 doctor check surfaces high-severity findings with paste-ready resolution commands. M2 synthesize phase pre-fetches latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. 226 hermetic unit tests + 12 real-Postgres E2E. Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`. Architecture doc: `docs/contradictions.md`. +- `src/core/think/index.ts` (v0.35.5.0 — gateway adapter) — `runThink` no longer instantiates `new Anthropic()` directly. The internal `LLMClient` instance is now built by a small adapter that wraps `gateway.chat()` from `src/core/ai/gateway.ts`, the canonical AI seam v0.31.12 established for chat/embed/expansion. Closes #952: stdio MCP launches (Claude Desktop, Cursor) don't inherit shell env, so the Anthropic SDK's env-only key resolution lost the key any user had set via `gbrain config set anthropic_api_key`. The gateway reads from `~/.gbrain/config.json` AND from env, so both paths work. Test seam preserved: `opts.client?: ThinkLLMClient` injection still works for the 12+ existing tests (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`, etc.); `opts.stubResponse` continues to short-circuit before any LLM call. When neither key nor client is available, the graceful "no LLM available" stub still fires with the same `NO_ANTHROPIC_API_KEY` warning. v0.36.x TODO: drop `ThinkLLMClient` indirection entirely, migrate tests to `__setChatTransportForTests` seam from `src/core/ai/gateway.ts`. +- `src/core/operations.ts` extension (v0.35.5.0 orphans fix) — `findOrphanPages` (both engines) now filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side. Pre-v0.35.5 the query filtered nothing on `deleted_at`, so soft-deleted pages (v0.26.5 soft-delete shipped without updating this query) appeared as orphans AND links from soft-deleted source pages still suppressed live pages from orphan results. Closes #1021. Pinned by `test/orphans.test.ts`'s soft-delete cases. - `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1) — `gbrain eval longmemeval ` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec). - `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". - `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. @@ -571,7 +573,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution), `test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion), `test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage), -`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.), +`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented. **v0.35.5.0:** test now also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`), and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies. Catches the column-only forward-reference class (e.g. `sources.archived` shape from v0.26.5, `oauth_clients.source_id` from v0.34.1) that the pre-existing CREATE INDEX parser couldn't see. Pre-existing parser bug fixed in same wave: `parseBaseTableColumns` now strips SQL line + block comments before identifying column names so commented-out lines no longer hide adjacent columns from coverage.), `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` (v0.26.6 #588 — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each (bootstrap + schema replay + migrations), snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via PGLITE_SCHEMA_SQL or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skip-gracefully without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.), `test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation), `test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin), diff --git a/TODOS.md b/TODOS.md index 255a4b6c3..0a12739df 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,6 +1,13 @@ # TODOS +## dreamy-thompson wave follow-ups (v0.36.x) + +- [ ] **v0.36.x: runThink full rewrite — drop ThinkLLMClient indirection.** v0.36's fix(think) wave landed a gateway-backed adapter at `src/core/think/index.ts:225-251` so `gbrain config set anthropic_api_key` works over MCP stdio (closed #952). The adapter routes through `gateway.chat()` but `runThink` still carries the `ThinkLLMClient` interface as the test seam — it's the last LLM-using path that doesn't use the canonical `__setChatTransportForTests` seam v0.31.12 established for chat/embed. Cleanup: drop `ThinkLLMClient`, drop the `opts.client` injection point, migrate the 12+ existing tests (`test/think-pipeline.serial.test.ts:144,181,222`, `test/think-gateway-adapter.test.ts`, plus 9+ others that stub the interface) to `__setChatTransportForTests`. Pros: codebase consistency, one fewer test-stub pattern, easier to add provider switching for think once it routes through gateway natively. Cons: 12+ test files need migration. Blocked by: v0.36 wave landing on master (so the adapter exists to lean on while migrating tests). Plan reference: D5 + D7 in `~/.claude/plans/ok-i-spun-up-dreamy-thompson.md`. + +- [ ] **v0.36.x: Supabase parity test fixture for `applyForwardReferenceBootstrap`.** v0.36 fixed the underlying bug (bootstrap now uses the DDL connection from `initSchema` so probes run inside the advisory-lock scope) per codex P1 from /ship adversarial review. What remains is the TEST FIXTURE that proves it: the new pre-v18/pre-v34/pre-v60 E2E tests run against local Docker Postgres but not against Supabase-shape pooler topology (transaction pooler + statement_timeout). Real Supabase upgrades have failed multiple times on this exact connection-topology divergence (#699, #820 lineage). Fix: a test fixture that exercises the probe path against deriveDirectUrl + transaction pooler + statement_timeout. Cons: requires Supabase fixture infra OR careful mocking of the connection-selection logic in `db.ts`'s `getDDLConnection` path. + + ## kinshasa-v3 follow-ups (v0.35.4.0) - [ ] **v0.36.x: Fix `supervisor-audit.ts:77` `readSupervisorEvents` to use the dual-week-aware pattern from `stub-guard-audit.ts:readRecentStubGuardEvents`.** The supervisor reader only reads the current ISO-week file, so a 24h sliding window across Monday 00:00 UTC silently loses Sunday's events (they're in last week's file). The new stub-guard reader in v0.35.4.0 fixes this for its own audit log by reading BOTH current and previous week files before timestamp-filtering — the supervisor reader should adopt the same shape. Pin with a unit test that uses a fake-clock fixture set to "Monday 00:01 UTC" with a Sunday 23:55 event in the prior file. Filed during v0.35.4.0 kinshasa-v3 codex outside-voice review. diff --git a/VERSION b/VERSION index 0819fa8d2..7513a6391 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.35.4.0 +0.35.5.0 diff --git a/llms-full.txt b/llms-full.txt index 0d7198fe6..7e8637271 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -151,9 +151,9 @@ strict behavior when unset. - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. - `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -164,7 +164,7 @@ strict behavior when unset. - `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim). - `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`) - `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags) -- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500. +- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). **v0.35.5.0:** new exported `pruneDir(name: string): boolean` helper is the single source of truth for descent-time directory exclusion across walkers. Blocks `node_modules` (no leading dot, so pre-v0.35.5 walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars. `isSyncable` now applies it per path segment; `walkMarkdownFiles` in `src/commands/extract.ts` and `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing so the IO cost of walking thousands of vendor files is saved. Closes #923 + #202. `manageGitignore` worktree fix in same wave: discriminator now matches the gitdir path segment (`/modules/` = submodule, `/worktrees/` = worktree, per Git's documented layout) instead of the legacy absolute-vs-relative check that misclassified absorbed submodules and worktrees both. Conductor worktrees are first-class repos and now get `.gitignore` management for storage-tiering. Closes #889. v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500. - `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local) - `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape. - `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens). @@ -193,6 +193,8 @@ strict behavior when unset. - `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count. - `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow. - `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` (v0.32.6) — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned per Codex; UTF-8-safe truncation; C1 confidence-floor double-enforcement; resolution_kind output drives M7 paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (Codex outside-voice fix — prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — Codex fix to bias from silent skip), M5 trend writes to `eval_contradictions_runs`, M6 source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker — stable cache hit-rate across re-runs). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (P1 batched), `writeContradictionsRun` + `loadContradictionsTrend` (M5), `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` (P2). Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). M1 doctor check surfaces high-severity findings with paste-ready resolution commands. M2 synthesize phase pre-fetches latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. 226 hermetic unit tests + 12 real-Postgres E2E. Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`. Architecture doc: `docs/contradictions.md`. +- `src/core/think/index.ts` (v0.35.5.0 — gateway adapter) — `runThink` no longer instantiates `new Anthropic()` directly. The internal `LLMClient` instance is now built by a small adapter that wraps `gateway.chat()` from `src/core/ai/gateway.ts`, the canonical AI seam v0.31.12 established for chat/embed/expansion. Closes #952: stdio MCP launches (Claude Desktop, Cursor) don't inherit shell env, so the Anthropic SDK's env-only key resolution lost the key any user had set via `gbrain config set anthropic_api_key`. The gateway reads from `~/.gbrain/config.json` AND from env, so both paths work. Test seam preserved: `opts.client?: ThinkLLMClient` injection still works for the 12+ existing tests (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`, etc.); `opts.stubResponse` continues to short-circuit before any LLM call. When neither key nor client is available, the graceful "no LLM available" stub still fires with the same `NO_ANTHROPIC_API_KEY` warning. v0.36.x TODO: drop `ThinkLLMClient` indirection entirely, migrate tests to `__setChatTransportForTests` seam from `src/core/ai/gateway.ts`. +- `src/core/operations.ts` extension (v0.35.5.0 orphans fix) — `findOrphanPages` (both engines) now filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side. Pre-v0.35.5 the query filtered nothing on `deleted_at`, so soft-deleted pages (v0.26.5 soft-delete shipped without updating this query) appeared as orphans AND links from soft-deleted source pages still suppressed live pages from orphan results. Closes #1021. Pinned by `test/orphans.test.ts`'s soft-delete cases. - `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1) — `gbrain eval longmemeval ` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec). - `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)". - `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE. @@ -679,7 +681,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac `test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution), `test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion), `test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage), -`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.), +`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented. **v0.35.5.0:** test now also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`), and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies. Catches the column-only forward-reference class (e.g. `sources.archived` shape from v0.26.5, `oauth_clients.source_id` from v0.34.1) that the pre-existing CREATE INDEX parser couldn't see. Pre-existing parser bug fixed in same wave: `parseBaseTableColumns` now strips SQL line + block comments before identifying column names so commented-out lines no longer hide adjacent columns from coverage.), `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` (v0.26.6 #588 — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each (bootstrap + schema replay + migrations), snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via PGLITE_SCHEMA_SQL or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skip-gracefully without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.), `test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation), `test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin), diff --git a/package.json b/package.json index 4e7cd52c8..8f104bf65 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.35.4.0", + "version": "0.35.5.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 84d8a4662..b2f6c1a6c 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -28,7 +28,7 @@ import { } from '../core/link-extraction.ts'; import { createProgress } from '../core/progress.ts'; import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; -import { pathToSlug } from '../core/sync.ts'; +import { pathToSlug, pruneDir, isSyncable } from '../core/sync.ts'; // Batch size for addLinksBatch / addTimelineEntriesBatch. // Postgres bind-parameter limit is 65535. Links use 4 cols/row → 16K hard ceiling; @@ -63,16 +63,26 @@ interface ExtractResult { // --- Shared walker --- export function walkMarkdownFiles(dir: string): { path: string; relPath: string }[] { + // Descent-time pruning + emit-time isSyncable filter (closes #923, #202). + // Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't + // call isSyncable at all — so it descended into `node_modules/`, emitted + // markdown files from there, AND ignored the canonical exclusion list + // (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor + // subtrees before recursion (saving IO), and isSyncable filters the emit + // set against the canonical markdown-strategy rules. const files: { path: string; relPath: string }[] = []; function walk(d: string) { for (const entry of readdirSync(d)) { - if (entry.startsWith('.')) continue; const full = join(d, entry); try { - if (lstatSync(full).isDirectory()) { + const st = lstatSync(full); + if (st.isDirectory()) { + if (!pruneDir(entry)) continue; walk(full); } else if (entry.endsWith('.md') && !entry.startsWith('_')) { - files.push({ path: full, relPath: relative(dir, full) }); + const rel = relative(dir, full); + if (!isSyncable(rel, { strategy: 'markdown' })) continue; + files.push({ path: full, relPath: rel }); } } catch { /* skip unreadable */ } } diff --git a/src/commands/sync.ts b/src/commands/sync.ts index b297a434d..280f3f7a7 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -1387,17 +1387,35 @@ export function manageGitignore( return; } - // D49: submodule detection. In a submodule, `.git` is a regular file - // (containing `gitdir: ../path/to/parent.git/modules/x`), not a directory. + // Submodule + worktree detection (closes #889 misclassification). + // Both submodules and worktrees use `.git` as a FILE (not a directory), so + // statSync.isFile() doesn't discriminate. Discriminator is the gitdir path + // segment: + // - submodule: gitdir contains `/modules/` (skip — managed by parent) + // - worktree: gitdir contains `/worktrees/` (MANAGE — first-class repo) + // Both contracts are documented Git internal layouts and stable across all 4 + // {relative, absolute} × {modules, worktrees} combinations, including the + // absorbed-submodule case from `git submodule absorbgitdirs`. + // Malformed `.git` file (no `gitdir:` prefix, unreadable) → MANAGE (fail-closed + // toward managing, preserving the pre-#889 catch{} behavior). const dotGit = join(repoPath, '.git'); if (existsSync(dotGit)) { try { if (statSync(dotGit).isFile()) { - console.warn( - `Note: skipping .gitignore management — ${repoPath} is a git submodule. ` + - `Add db_only directories to your parent repo's .gitignore manually.`, - ); - return; + const content = readFileSync(dotGit, 'utf-8'); + const match = content.match(/gitdir:\s*(.+)/); + const gitdir = match ? match[1].trim() : ''; + if (gitdir.includes('/modules/')) { + console.warn( + `Note: skipping .gitignore management — ${repoPath} is a git submodule. ` + + `Add db_only directories to your parent repo's .gitignore manually.`, + ); + return; + } + // Worktree (gitdir contains /worktrees/) OR malformed .git falls through + // to the existing manage path. Worktrees are first-class repos — they + // need .gitignore management too. Malformed → MANAGE preserves the + // pre-#889 fail-closed-toward-managing catch behavior. } } catch { // proceed; can't tell, default to managing diff --git a/src/core/cycle/transcript-discovery.ts b/src/core/cycle/transcript-discovery.ts index 2d8dcc886..0b368ab94 100644 --- a/src/core/cycle/transcript-discovery.ts +++ b/src/core/cycle/transcript-discovery.ts @@ -12,6 +12,7 @@ import { readFileSync, readdirSync, statSync } from 'node:fs'; import { join, basename } from 'node:path'; import { createHash } from 'node:crypto'; +import { pruneDir } from '../sync.ts'; export interface DiscoveredTranscript { /** Absolute path to the transcript file. */ @@ -119,22 +120,38 @@ function matchesAnyExclude(text: string, patterns: RegExp[]): boolean { } function listTextFiles(dir: string): string[] { - let entries: string[]; - try { - entries = readdirSync(dir); - } catch { - return []; - } + // Recursive walk with descent-time pruning (closes codex C12/C13 spec gap). + // Accepts BOTH .txt and .md per transcript-discovery's domain rules — does + // NOT use isSyncable({strategy:'markdown'}) because that predicate rejects + // .txt and applies markdown-only README/ops exclusions transcripts don't share. + // + // pruneDir at descent time skips node_modules / .git / .obsidian / .raw / + // .cache / ops / etc. before recursion — saves the IO cost of walking + // vendor subtrees. const out: string[] = []; - for (const name of entries) { - if (!name.endsWith('.txt') && !name.endsWith('.md')) continue; - const full = join(dir, name); + function walk(d: string) { + let entries: string[]; try { - if (statSync(full).isFile()) out.push(full); + entries = readdirSync(d); } catch { - // skip unreadable entries + return; + } + for (const name of entries) { + const full = join(d, name); + try { + const st = statSync(full); + if (st.isDirectory()) { + if (!pruneDir(name)) continue; + walk(full); + } else if (st.isFile() && (name.endsWith('.txt') || name.endsWith('.md'))) { + out.push(full); + } + } catch { + // skip unreadable entries + } } } + walk(dir); return out.sort(); } diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 03ab90b41..6b7a759bd 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -288,7 +288,27 @@ export class PGLiteEngine implements BrainEngine { EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema='public' AND table_name='ingest_log') AS ingest_log_exists, EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_schema='public' AND table_name='ingest_log' AND column_name='source_id') AS ingest_log_source_id_exists + WHERE table_schema='public' AND table_name='ingest_log' AND column_name='source_id') AS ingest_log_source_id_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema='public' AND table_name='files') AS files_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='files' AND column_name='source_id') AS files_source_id_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='files' AND column_name='page_id') AS files_page_id_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema='public' AND table_name='oauth_clients') AS oauth_clients_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='oauth_clients' AND column_name='source_id') AS oauth_clients_source_id_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='oauth_clients' AND column_name='federated_read') AS oauth_clients_federated_read_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema='public' AND table_name='sources') AS sources_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='sources' AND column_name='archived') AS sources_archived_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='sources' AND column_name='archived_at') AS sources_archived_at_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='sources' AND column_name='archive_expires_at') AS sources_archive_expires_at_exists `); const probe = rows[0] as { pages_exists: boolean; @@ -309,6 +329,16 @@ export class PGLiteEngine implements BrainEngine { subagent_provider_id_exists: boolean; ingest_log_exists: boolean; ingest_log_source_id_exists: boolean; + files_exists: boolean; + files_source_id_exists: boolean; + files_page_id_exists: boolean; + oauth_clients_exists: boolean; + oauth_clients_source_id_exists: boolean; + oauth_clients_federated_read_exists: boolean; + sources_exists: boolean; + sources_archived_exists: boolean; + sources_archived_at_exists: boolean; + sources_archive_expires_at_exists: boolean; }; const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists; @@ -332,27 +362,55 @@ export class PGLiteEngine implements BrainEngine { // references source_id. Old brains have ingest_log without source_id; // bootstrap adds the column before SCHEMA_SQL replay creates the index. const needsIngestLogSourceId = probe.ingest_log_exists && !probe.ingest_log_source_id_exists; + // v0.18 (v18): files.source_id + files.page_id added; idx_files_source_id + // and idx_files_page_id in PGLITE_SCHEMA_SQL crash without them. + const needsFilesBootstrap = probe.files_exists + && (!probe.files_source_id_exists || !probe.files_page_id_exists); + // v0.34.1 (v60+v61+v65): oauth_clients.source_id + federated_read added; + // FK to sources(id) + GIN index idx_oauth_clients_federated_read in + // PGLITE_SCHEMA_SQL crash without them. + const needsOauthClientsBootstrap = probe.oauth_clients_exists + && (!probe.oauth_clients_source_id_exists || !probe.oauth_clients_federated_read_exists); + // v0.26.5 (v34): sources.archived + archived_at + archive_expires_at added + // for soft-delete lifecycle. Not directly referenced by indexes BUT + // PGLITE_SCHEMA_SQL's `CREATE TABLE IF NOT EXISTS sources` is a no-op on + // pre-existing sources tables (won't add columns), so visibility filters + // referencing these columns trip on old brains. The bootstrap closes the + // gap before any visibility-filter SQL runs. + const needsSourcesArchive = probe.sources_exists + && (!probe.sources_archived_exists + || !probe.sources_archived_at_exists + || !probe.sources_archive_expires_at_exists); // Fresh installs (no tables yet) and modern brains both no-op. if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt && !needsChunksEmbeddingImage && !needsMcpLogBootstrap && !needsSubagentProviderId - && !needsPagesRecency && !needsIngestLogSourceId) return; + && !needsPagesRecency && !needsIngestLogSourceId + && !needsFilesBootstrap && !needsOauthClientsBootstrap + && !needsSourcesArchive) return; console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap'); if (needsPagesBootstrap) { // Mirror schema-embedded.ts shape for `sources` so the subsequent // PGLITE_SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op. + // Archive columns (v34) are folded in here so a pre-v18 brain doesn't + // need needsSourcesArchive to also fire — bootstrap creates a complete + // v34-shape sources in one go. needsSourcesArchive then only fires on + // the pre-v34 case (sources exists, archive cols don't). await this.db.exec(` CREATE TABLE IF NOT EXISTS sources ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - local_path TEXT, - last_commit TEXT, - last_sync_at TIMESTAMPTZ, - config JSONB NOT NULL DEFAULT '{}'::jsonb, - created_at TIMESTAMPTZ NOT NULL DEFAULT now() + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + local_path TEXT, + last_commit TEXT, + last_sync_at TIMESTAMPTZ, + config JSONB NOT NULL DEFAULT '{}'::jsonb, + archived BOOLEAN NOT NULL DEFAULT FALSE, + archived_at TIMESTAMPTZ, + archive_expires_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); INSERT INTO sources (id, name, config) VALUES ('default', 'default', '{"federated": true}'::jsonb) @@ -465,6 +523,50 @@ export class PGLiteEngine implements BrainEngine { ALTER TABLE ingest_log ADD COLUMN IF NOT EXISTS source_id TEXT NOT NULL DEFAULT 'default'; `); } + + if (needsFilesBootstrap) { + // v18 (files_provenance_columns) adds source_id + page_id to files plus + // idx_files_source_id and idx_files_page_id in PGLITE_SCHEMA_SQL. Pre-v18 + // brains crash on the CREATE INDEX. Bootstrap adds both columns; v18 + // runs later via runMigrations and is idempotent. + await this.db.exec(` + ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id TEXT + NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE; + ALTER TABLE files ADD COLUMN IF NOT EXISTS page_id INTEGER + REFERENCES pages(id) ON DELETE SET NULL; + `); + } + + if (needsOauthClientsBootstrap) { + // v60+v61+v65 (oauth_clients_source_id_fk, oauth_clients_federated_read_column, + // oauth_clients_federated_read_gin_index) add source_id + federated_read + // and the GIN index idx_oauth_clients_federated_read. PGLITE_SCHEMA_SQL's + // FK + index references crash on pre-v60 brains. Bootstrap mirrors the + // v60+v61 column shape; v60-v65 run later via runMigrations and are + // idempotent (and handle backfill + RESTRICT-flip). + await this.db.exec(` + ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS source_id TEXT + DEFAULT 'default' REFERENCES sources(id) ON DELETE SET NULL; + ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS federated_read TEXT[] + NOT NULL DEFAULT '{}'; + `); + } + + if (needsSourcesArchive) { + // v34 (destructive_guard_columns) promotes archive lifecycle from JSONB + // config to real columns on sources. PGLITE_SCHEMA_SQL's + // `CREATE TABLE IF NOT EXISTS sources` is a no-op against an existing + // pre-v34 sources table, so the column-add never lands until the v34 + // migration runs. v34's UPDATE statements + downstream visibility filters + // (search/query/list_pages) need the columns to exist on the table + // schema. Bootstrap adds the three columns; v34 runs later via + // runMigrations and is idempotent (and handles JSONB → column backfill). + await this.db.exec(` + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN NOT NULL DEFAULT FALSE; + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived_at TIMESTAMPTZ; + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archive_expires_at TIMESTAMPTZ; + `); + } } async withReservedConnection(fn: (conn: ReservedConnection) => Promise): Promise { @@ -1882,15 +1984,25 @@ export class PGLiteEngine implements BrainEngine { } async findOrphanPages(): Promise> { + // Soft-delete filter on BOTH sides: + // - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates + // - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound + // Without the link-source filter, a live page can hide from orphan results purely + // because a soft-deleted page links to it. v0.26.5 invariant; codex C11. const { rows } = await this.db.query( `SELECT p.slug, COALESCE(p.title, p.slug) AS title, p.frontmatter->>'domain' AS domain FROM pages p - WHERE NOT EXISTS ( - SELECT 1 FROM links l WHERE l.to_page_id = p.id - ) + WHERE p.deleted_at IS NULL + AND NOT EXISTS ( + SELECT 1 + FROM links l + JOIN pages src ON src.id = l.from_page_id + WHERE l.to_page_id = p.id + AND src.deleted_at IS NULL + ) ORDER BY p.slug` ); return rows as Array<{ slug: string; title: string; domain: string | null }>; diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index c7538849e..b8c3c7b49 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -232,7 +232,12 @@ export class PostgresEngine implements BrainEngine { // Pre-schema bootstrap: add forward-referenced state the embedded schema // blob requires but that older brains don't have yet (issues #366/#375/ // #378/#396 + #266/#357). Idempotent on fresh installs and modern brains. - await this.applyForwardReferenceBootstrap(); + // Threads the DDL connection (same one holding the advisory lock above) + // so bootstrap probes run on the locked connection — without this, the + // probes ran through `this.sql` (the pooler/instance pool) outside the + // lock, opening a concurrent-bootstrap race for Supabase users on the + // transaction pooler. Codex P1 finding from v0.36 dreamy-thompson wave. + await this.applyForwardReferenceBootstrap(conn); await conn.unsafe(sqlText); @@ -294,8 +299,13 @@ export class PostgresEngine implements BrainEngine { * `test/schema-bootstrap-coverage.test.ts` (PGLite side) and * `test/e2e/postgres-bootstrap.test.ts` (Postgres side). */ - private async applyForwardReferenceBootstrap(): Promise { - const conn = this.sql; + private async applyForwardReferenceBootstrap(injectedConn?: postgres.Sql): Promise { + // Use the caller-provided connection (DDL pool, holding the advisory lock + // from initSchema) when available — falls back to this.sql for backward + // compatibility with any unit-test path that still calls bootstrap directly. + // Production path always passes the DDL conn so bootstrap probes run inside + // the same lock scope as SCHEMA_SQL replay. + const conn = injectedConn ?? this.sql; // Single round-trip probe for every forward-reference target. // current_schema() resolves to whatever search_path the connection uses, @@ -319,6 +329,16 @@ export class PostgresEngine implements BrainEngine { subagent_provider_id_exists: boolean; ingest_log_exists: boolean; ingest_log_source_id_exists: boolean; + files_exists: boolean; + files_source_id_exists: boolean; + files_page_id_exists: boolean; + oauth_clients_exists: boolean; + oauth_clients_source_id_exists: boolean; + oauth_clients_federated_read_exists: boolean; + sources_exists: boolean; + sources_archived_exists: boolean; + sources_archived_at_exists: boolean; + sources_archive_expires_at_exists: boolean; }[]>` SELECT EXISTS (SELECT 1 FROM information_schema.tables @@ -356,7 +376,27 @@ export class PostgresEngine implements BrainEngine { EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = 'ingest_log') AS ingest_log_exists, EXISTS (SELECT 1 FROM information_schema.columns - WHERE table_schema = current_schema() AND table_name = 'ingest_log' AND column_name = 'source_id') AS ingest_log_source_id_exists + WHERE table_schema = current_schema() AND table_name = 'ingest_log' AND column_name = 'source_id') AS ingest_log_source_id_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema = current_schema() AND table_name = 'files') AS files_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'files' AND column_name = 'source_id') AS files_source_id_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'files' AND column_name = 'page_id') AS files_page_id_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema = current_schema() AND table_name = 'oauth_clients') AS oauth_clients_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'oauth_clients' AND column_name = 'source_id') AS oauth_clients_source_id_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'oauth_clients' AND column_name = 'federated_read') AS oauth_clients_federated_read_exists, + EXISTS (SELECT 1 FROM information_schema.tables + WHERE table_schema = current_schema() AND table_name = 'sources') AS sources_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'archived') AS sources_archived_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'archived_at') AS sources_archived_at_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'archive_expires_at') AS sources_archive_expires_at_exists `; const probe = probeRows[0]!; @@ -386,26 +426,52 @@ export class PostgresEngine implements BrainEngine { // source_id. Old brains have ingest_log without source_id; bootstrap adds // the column before SCHEMA_SQL replay creates the index. const needsIngestLogSourceId = probe.ingest_log_exists && !probe.ingest_log_source_id_exists; + // v0.18 (v18): files.source_id + files.page_id added; idx_files_source_id + // and idx_files_page_id in SCHEMA_SQL crash without them. + const needsFilesBootstrap = probe.files_exists + && (!probe.files_source_id_exists || !probe.files_page_id_exists); + // v0.34.1 (v60+v61+v65): oauth_clients.source_id + federated_read added; + // FK to sources(id) + GIN index idx_oauth_clients_federated_read in + // SCHEMA_SQL crash without them. + const needsOauthClientsBootstrap = probe.oauth_clients_exists + && (!probe.oauth_clients_source_id_exists || !probe.oauth_clients_federated_read_exists); + // v0.26.5 (v34): sources.archived + archived_at + archive_expires_at added + // for soft-delete lifecycle. SCHEMA_SQL's `CREATE TABLE IF NOT EXISTS sources` + // is a no-op on pre-existing sources tables (won't add columns), so the + // visibility filters in search/list_pages trip on old brains. Bootstrap + // closes the gap before any visibility-filter SQL runs. + const needsSourcesArchive = probe.sources_exists + && (!probe.sources_archived_exists + || !probe.sources_archived_at_exists + || !probe.sources_archive_expires_at_exists); if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt && !needsMcpLogBootstrap && !needsSubagentProviderId && !needsChunksEmbeddingImage && !needsPagesRecency - && !needsIngestLogSourceId) return; + && !needsIngestLogSourceId && !needsFilesBootstrap + && !needsOauthClientsBootstrap && !needsSourcesArchive) return; console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap'); if (needsPagesBootstrap) { // Mirror schema-embedded.ts's `sources` shape so the subsequent // SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op. + // Archive columns (v34) are folded in here so a pre-v18 brain doesn't + // need needsSourcesArchive to also fire — bootstrap creates a complete + // v34-shape sources in one go. needsSourcesArchive then only fires on + // the pre-v34 case (sources exists, archive cols don't). await conn.unsafe(` CREATE TABLE IF NOT EXISTS sources ( - id TEXT PRIMARY KEY, - name TEXT NOT NULL UNIQUE, - local_path TEXT, - last_commit TEXT, - last_sync_at TIMESTAMPTZ, - config JSONB NOT NULL DEFAULT '{}'::jsonb, - created_at TIMESTAMPTZ NOT NULL DEFAULT now() + id TEXT PRIMARY KEY, + name TEXT NOT NULL UNIQUE, + local_path TEXT, + last_commit TEXT, + last_sync_at TIMESTAMPTZ, + config JSONB NOT NULL DEFAULT '{}'::jsonb, + archived BOOLEAN NOT NULL DEFAULT FALSE, + archived_at TIMESTAMPTZ, + archive_expires_at TIMESTAMPTZ, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); INSERT INTO sources (id, name, config) VALUES ('default', 'default', '{"federated": true}'::jsonb) @@ -518,6 +584,50 @@ export class PostgresEngine implements BrainEngine { ALTER TABLE ingest_log ADD COLUMN IF NOT EXISTS source_id TEXT NOT NULL DEFAULT 'default'; `); } + + if (needsFilesBootstrap) { + // v18 (files_provenance_columns) adds source_id + page_id to files plus + // idx_files_source_id and idx_files_page_id in SCHEMA_SQL. Pre-v18 brains + // crash on the CREATE INDEX. Bootstrap adds both columns; v18 runs later + // via runMigrations and is idempotent. + await conn.unsafe(` + ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id TEXT + NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE; + ALTER TABLE files ADD COLUMN IF NOT EXISTS page_id INTEGER + REFERENCES pages(id) ON DELETE SET NULL; + `); + } + + if (needsOauthClientsBootstrap) { + // v60+v61+v65 (oauth_clients_source_id_fk, oauth_clients_federated_read_column, + // oauth_clients_federated_read_gin_index) add source_id + federated_read + // and the GIN index idx_oauth_clients_federated_read. SCHEMA_SQL's + // FK + index references crash on pre-v60 brains. Bootstrap mirrors the + // v60+v61 column shape; v60-v65 run later via runMigrations and are + // idempotent (and handle backfill + the v64 RESTRICT-flip). + await conn.unsafe(` + ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS source_id TEXT + DEFAULT 'default' REFERENCES sources(id) ON DELETE SET NULL; + ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS federated_read TEXT[] + NOT NULL DEFAULT '{}'; + `); + } + + if (needsSourcesArchive) { + // v34 (destructive_guard_columns) promotes archive lifecycle from JSONB + // config to real columns on sources. SCHEMA_SQL's `CREATE TABLE IF NOT EXISTS + // sources` is a no-op against an existing pre-v34 sources table, so the + // column-add never lands until the v34 migration runs. v34's UPDATE + // statements + downstream visibility filters (search/query/list_pages) + // need the columns to exist on the table schema. Bootstrap adds the + // three columns; v34 runs later via runMigrations and is idempotent + // (and handles JSONB → column backfill). + await conn.unsafe(` + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN NOT NULL DEFAULT FALSE; + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived_at TIMESTAMPTZ; + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archive_expires_at TIMESTAMPTZ; + `); + } } async transaction(fn: (engine: BrainEngine) => Promise): Promise { @@ -1901,15 +2011,25 @@ export class PostgresEngine implements BrainEngine { async findOrphanPages(): Promise> { const sql = this.sql; + // Soft-delete filter on BOTH sides: + // - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates + // - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound + // Without the link-source filter, a live page can hide from orphan results purely + // because a soft-deleted page links to it. v0.26.5 invariant; codex C11. const rows = await sql` SELECT p.slug, COALESCE(p.title, p.slug) AS title, p.frontmatter->>'domain' AS domain FROM pages p - WHERE NOT EXISTS ( - SELECT 1 FROM links l WHERE l.to_page_id = p.id - ) + WHERE p.deleted_at IS NULL + AND NOT EXISTS ( + SELECT 1 + FROM links l + JOIN pages src ON src.id = l.from_page_id + WHERE l.to_page_id = p.id + AND src.deleted_at IS NULL + ) ORDER BY p.slug `; return rows as unknown as Array<{ slug: string; title: string; domain: string | null }>; diff --git a/src/core/sync.ts b/src/core/sync.ts index 89778243c..a5abecc61 100644 --- a/src/core/sync.ts +++ b/src/core/sync.ts @@ -208,6 +208,47 @@ function matchesAnyGlob(path: string, patterns?: string[]): boolean { return patterns.some((pattern) => globToRegex(pattern).test(normalized)); } +/** + * Directory names that walkers must NEVER descend into. Used at descent + * time (before recursion) to prune entire subtrees — saves the IO cost of + * walking thousands of vendor / generated / hidden files only to filter + * them at file-emit time. Used by every walker in gbrain (sync, extract, + * transcript-discovery, etc.). + * + * Pattern: dirname matching at single path-segment granularity. Walkers + * call `pruneDir(entry.name)` on each subdirectory before recursing. + * + * `node_modules` lacks a leading dot so the dot-prefix exclusion in + * isSyncable below doesn't catch it; explicit entry here closes the + * latent walker bug (#923, #202). + */ +const PRUNE_DIR_NAMES = new Set([ + 'node_modules', + '.raw', + 'ops', +]); + +/** + * Should this directory be descended into? Returns `false` for vendor / hidden / + * generated dirs that walkers should skip BEFORE recursing. Catches + * `node_modules` (latent bug — no leading dot), dot-prefix dirs (`.git`, + * `.obsidian`, `.raw`, `.cache`, etc. via the leading-dot heuristic), and the + * explicit `PRUNE_DIR_NAMES` set above. + * + * `name` is a single path segment (basename of the directory entry), NOT a + * full path. Walkers consult this on each subdirectory entry during recursion. + */ +export function pruneDir(name: string): boolean { + if (!name) return true; + if (name.startsWith('.')) return false; + if (PRUNE_DIR_NAMES.has(name)) return false; + // `.raw` is the literal directory name; `*.raw` is the gbrain sidecar + // convention (e.g. `people/pedro.raw/` holds raw source for pedro.md). + // Both forms should be skipped at descent time. + if (name.endsWith('.raw')) return false; + return true; +} + /** * Filter a file path to determine if it should be synced to GBrain. * Strategy-aware: 'markdown' (default) = .md/.mdx only, 'code' = code files only, 'auto' = both. @@ -217,20 +258,17 @@ export function isSyncable(path: string, opts: SyncableOptions = {}): boolean { if (!isAllowedByStrategy(path, strategy)) return false; - // Skip hidden directories - if (path.split('/').some(p => p.startsWith('.'))) return false; - - // Skip .raw/ sidecar directories - if (path.includes('.raw/')) return false; + // Skip every path segment that pruneDir would block walkers from descending + // into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars, + // `node_modules/` (latent bug fix), and `ops/` at any depth. + const segments = path.split('/'); + if (segments.some(p => !pruneDir(p))) return false; // Skip meta files that aren't pages const skipFiles = ['schema.md', 'index.md', 'log.md', 'README.md']; - const basename = path.split('/').pop() || ''; + const basename = segments[segments.length - 1] || ''; if (skipFiles.includes(basename)) return false; - // Skip ops/ directory - if (path.startsWith('ops/')) return false; - if (opts.include && opts.include.length > 0 && !matchesAnyGlob(path, opts.include)) return false; if (opts.exclude && opts.exclude.length > 0 && matchesAnyGlob(path, opts.exclude)) return false; diff --git a/src/core/think/index.ts b/src/core/think/index.ts index f083263c6..3f767e054 100644 --- a/src/core/think/index.ts +++ b/src/core/think/index.ts @@ -17,13 +17,17 @@ * for those flags per Codex P1 #7. */ -import Anthropic from '@anthropic-ai/sdk'; +import type Anthropic from '@anthropic-ai/sdk'; import type { BrainEngine, SynthesisEvidenceInput } from '../engine.ts'; import { runGather, renderPagesBlock, takesHitToTakeForPrompt } from './gather.ts'; import { renderTakesBlock } from './sanitize.ts'; import { buildThinkSystemPrompt, buildThinkUserMessage } from './prompt.ts'; import { resolveCitations, type ParsedCitation } from './cite-render.ts'; import { resolveModel } from '../model-config.ts'; +import { chat as gatewayChat, type ChatResult } from '../ai/gateway.ts'; +import { resolveRecipe } from '../ai/model-resolver.ts'; +import { AIConfigError } from '../ai/errors.ts'; +import { loadConfig } from '../config.ts'; /** Anthropic Messages client interface — same shape used by subagent.ts so test stubs can be shared. */ export interface ThinkLLMClient { @@ -222,7 +226,20 @@ export async function runThink( if (opts.stubResponse) { response = opts.stubResponse; } else { - if (!opts.client && !process.env.ANTHROPIC_API_KEY) { + // Build a ThinkLLMClient. Three sources, in priority order: + // 1. opts.client (test injection — preserved as test seam) + // 2. Gateway adapter (routes through gateway.chat() — picks up + // anthropic_api_key from gbrain config OR env, gateway rate-leases, + // retry, prompt caching, the canonical seam per CLAUDE.md) + // 3. Graceful fallback ("no LLM available" stub) — when gateway is + // unconfigured AND no env var is set, return without throwing. + // + // Pre-v0.36, this code path constructed `new Anthropic()` directly. + // That bypassed gateway config (gbrain config set anthropic_api_key) + // because the Anthropic SDK only reads process.env.ANTHROPIC_API_KEY. + // Closes #952 (think over MCP returns "no LLM available"). + const client = opts.client ?? await tryBuildGatewayClient(modelUsed); + if (!client) { warnings.push('NO_ANTHROPIC_API_KEY'); // Degrade gracefully: return the gather without synthesis. Better than throwing. return { @@ -244,11 +261,6 @@ export async function runThink( }, }; } - // Anthropic SDK exposes the create method via .messages — match the structural signature. - const realClient = new Anthropic(); - const client: ThinkLLMClient = opts.client ?? { - create: (params, opts2) => realClient.messages.create(params, opts2), - }; const result = await client.create({ model: modelUsed, max_tokens: DEFAULT_MAX_OUTPUT_TOKENS, @@ -347,3 +359,185 @@ export async function persistSynthesis( const persisted = await persistCitations(engine, page.id, result.citations); return { slug, evidenceInserted: persisted.inserted, warnings: persisted.warnings }; } + +// ───────────────────────────────────────────────────────────────── +// Gateway adapter for #952 (think over MCP returns "no LLM available"). +// ───────────────────────────────────────────────────────────────── +// Pre-v0.36, runThink instantiated `new Anthropic()` directly and read +// ANTHROPIC_API_KEY from process.env. Claude Desktop's stdio MCP launch +// doesn't inherit shell env, so `gbrain config set anthropic_api_key sk-...` +// (which writes to ~/.gbrain/config.json) never reached the SDK and every +// MCP think call degraded to "no LLM available." +// +// The adapter routes through gateway.chat() — the canonical seam per +// CLAUDE.md. Gateway reads the API key from gbrain config OR env, picks +// up prompt caching, rate-leases, retry, and the test seam +// (__setChatTransportForTests) that v0.31.12 already established. +// +// Per plan-eng-review D10 (cross-model tension with codex C7+C8+C9+C10), +// the adapter implements four fixes: +// 1. Drop the new Anthropic() direct path entirely — always route through gateway +// 2. Real availability check via try/catch around resolveRecipe + assertion +// (NOT the false-positive `getChatModel()` truthy check) +// 3. Model-id resolution: handle both bare (`claude-opus-4-7`) and +// provider-prefixed (`anthropic:claude-opus-4-7`) shapes +// 4. Response-shape conversion: ChatResult → Anthropic.Message +// +// `opts.client` injection path is preserved (test seam — see ThinkLLMClient). +// `opts.stubResponse` path is preserved (pure-test escape). +// ───────────────────────────────────────────────────────────────── + +/** + * Try to build a gateway-backed ThinkLLMClient for the given model. + * Returns null when the gateway cannot resolve a usable chat provider for + * this model (missing API key for the resolved provider, unknown provider, + * touchpoint not supported, etc.). Caller falls through to the graceful + * "no LLM available" stub on null. + */ +async function tryBuildGatewayClient(modelUsed: string): Promise { + // Normalize: ensure provider:model shape. resolveModel returns bare + // anthropic ids (e.g. `claude-opus-4-7`); gateway.chat needs `anthropic:...`. + const modelStr = modelUsed.includes(':') ? modelUsed : `anthropic:${modelUsed}`; + + // Availability probe: resolveRecipe throws on unknown provider; assertTouchpoint + // throws if the resolved recipe doesn't support chat. Both are AIConfigError. + let providerId: string; + try { + const { parsed } = resolveRecipe(modelStr); + providerId = parsed.providerId; + } catch (e) { + if (e instanceof AIConfigError) return null; + throw e; + } + + // API-key availability probe. The gateway lazily checks keys inside + // instantiateChat at first .chat() call and throws AIConfigError on miss. + // Pre-checking here preserves the legacy "NO_ANTHROPIC_API_KEY" warning + // signal AND avoids paying for a wasted gateway call when the user clearly + // has no key configured. Reads BOTH the gbrain config file (`anthropic_api_key` + // set via `gbrain config set`) AND the process env, matching gateway's + // own loadConfig precedence. + if (providerId === 'anthropic' && !hasAnthropicKey()) return null; + + return { + create: async (params): Promise => { + // Build ChatOpts from Anthropic.MessageCreateParamsNonStreaming. + const messages = params.messages.map(m => ({ + role: m.role, + content: typeof m.content === 'string' + ? m.content + : (Array.isArray(m.content) ? m.content.map(b => 'text' in b ? b.text : '').join('') : ''), + })); + const system = typeof params.system === 'string' + ? params.system + : (Array.isArray(params.system) ? params.system.map(b => 'text' in b ? b.text : '').join('') : undefined); + + let result: ChatResult; + try { + result = await gatewayChat({ + model: modelStr, + system, + messages, + maxTokens: params.max_tokens, + }); + } catch (e) { + // AIConfigError at chat time = missing API key for resolved provider. + // Surface as a sentinel "no LLM available"-shaped Message so the + // existing JSON-parse path produces the graceful degradation answer. + if (e instanceof AIConfigError) { + return buildGracefulMessage(modelStr) as unknown as Anthropic.Message; + } + throw e; + } + return chatResultToMessage(result, modelStr) as unknown as Anthropic.Message; + }, + }; +} + +/** + * Convert gateway's `ChatResult` into an Anthropic-Message-shaped object. + * The caller (`runThink`) parses `result.content[0].text` as JSON; the + * other fields (usage, stop_reason) are returned with best-effort mapping + * for downstream telemetry compat. + */ +function chatResultToMessage(result: ChatResult, modelStr: string): { + id: string; + type: 'message'; + role: 'assistant'; + model: string; + content: Array<{ type: 'text'; text: string }>; + usage: { input_tokens: number; output_tokens: number }; + stop_reason: 'end_turn' | 'max_tokens' | 'tool_use' | 'stop_sequence'; +} { + return { + id: '', + type: 'message', + role: 'assistant', + model: modelStr, + content: [{ type: 'text', text: result.text }], + usage: { + input_tokens: result.usage.input_tokens, + output_tokens: result.usage.output_tokens, + }, + stop_reason: mapStopReason(result.stopReason), + }; +} + +function hasAnthropicKey(): boolean { + if (process.env.ANTHROPIC_API_KEY) return true; + try { + const cfg = loadConfig(); + if (cfg?.anthropic_api_key) return true; + } catch { + // loadConfig may throw on first-run installs; treat as no key available. + } + return false; +} + +function mapStopReason(s: ChatResult['stopReason']): 'end_turn' | 'max_tokens' | 'tool_use' | 'stop_sequence' { + switch (s) { + case 'end': return 'end_turn'; + case 'length': return 'max_tokens'; + case 'tool_calls': return 'tool_use'; + // 'refusal', 'content_filter', 'other' → end_turn (no Anthropic equivalent) + default: return 'end_turn'; + } +} + +/** + * Sentinel Message returned when gateway.chat throws AIConfigError (typically + * missing API key for the resolved provider). The caller's JSON parser will + * fail on this text, fall through to `LLM_OUTPUT_NOT_JSON`, and surface the + * sentinel as the answer — matches the legacy graceful-degradation shape. + */ +function buildGracefulMessage(modelStr: string): { + id: string; + type: 'message'; + role: 'assistant'; + model: string; + content: Array<{ type: 'text'; text: string }>; + usage: { input_tokens: number; output_tokens: number }; + stop_reason: 'end_turn'; +} { + return { + id: '', + type: 'message', + role: 'assistant', + model: modelStr, + content: [{ type: 'text', text: '(no LLM available — set anthropic_api_key via gbrain config or ANTHROPIC_API_KEY env)' }], + usage: { input_tokens: 0, output_tokens: 0 }, + stop_reason: 'end_turn', + }; +} + +// Test-only exports for the adapter helpers. The functions live at module +// scope (not inside runThink) so they can be unit-tested directly. Naming +// follows the `__` prefix convention already established by +// `__setChatTransportForTests` in gateway.ts. +export const __thinkAdapter = { + tryBuildGatewayClient, + chatResultToMessage, + mapStopReason, + buildGracefulMessage, + hasAnthropicKey, +}; diff --git a/test/helpers/extract-added-columns.ts b/test/helpers/extract-added-columns.ts new file mode 100644 index 000000000..091c98aae --- /dev/null +++ b/test/helpers/extract-added-columns.ts @@ -0,0 +1,97 @@ +/** + * MIGRATIONS-source introspection for the bootstrap CI guard. + * + * The v0.28.5 SQL-parser at the bottom of `test/schema-bootstrap-coverage.test.ts` + * walks every `CREATE INDEX` in `PGLITE_SCHEMA_SQL` to enforce that indexed + * columns either live in the table's CREATE TABLE body OR are added by + * `applyForwardReferenceBootstrap`. That structural check kills the + * column-with-index forward-reference class. + * + * It does NOT kill the column-ONLY forward-reference class. v0.26.5 (v34) + * added `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` + * for the soft-delete lifecycle. Those columns aren't indexed, but + * `CREATE TABLE IF NOT EXISTS sources` is a no-op on a pre-v34 brain, so the + * archive columns never land on the schema when the schema blob replays — + * downstream visibility filters in search / list_pages trip immediately. + * + * This helper closes the gap by walking the source text of + * `src/core/migrate.ts` and extracting every `ALTER TABLE ... ADD COLUMN` + * the file contains, regardless of whether it lives in a migration's `sql` + * field, `sqlFor.{postgres,pglite}` field, or inside a `handler` function + * body that calls `engine.runMigration(N, \`...\`)`. The test in + * `schema-bootstrap-coverage.test.ts` then asserts every such (table, column) + * pair is also covered by the bootstrap. + * + * Why source-file introspection (vs walking the MIGRATIONS object array): + * v34's `destructive_guard_columns` migration uses a `handler` function that + * embeds SQL inside `engine.runMigration(34, \`ALTER TABLE sources ADD COLUMN ...\`)`. + * The handler body isn't reachable via Migration.sql / Migration.sqlFor. Reading + * the source text catches ALL three shapes uniformly: top-level sql, sqlFor + * overrides, and handler-embedded `engine.runMigration` calls. + * + * Why regex-on-our-own-source is safe (vs the DDL-parser-of-prod-schema trap + * codex flagged in plan review): the input is migrate.ts — our own code, with + * consistent shape. ALTER TABLE ADD COLUMN is unambiguous. Future contributors + * who add columns follow the same shape (or fail this test and learn). + */ + +import { readFileSync } from 'fs'; +import { resolve } from 'path'; + +export interface AddedColumnRef { + table: string; + column: string; +} + +/** + * Extract every `ALTER TABLE ADD COLUMN ` reference from a SQL + * string. Handles: + * - `ALTER TABLE [IF EXISTS] [ONLY]
ADD COLUMN [IF NOT EXISTS] ` + * - Quoted identifiers (`"col"`, `\`col\``) + * - Multi-statement strings with mixed ALTER + CREATE + UPDATE + */ +function extractAlterAddColumnsFromSql(sql: string): Array<{ table: string; column: string }> { + const result: Array<{ table: string; column: string }> = []; + // Identifier shape: optional quote, word chars, optional matching quote. + // Handles bare `pages`, double-quoted `"pages"`, and backtick `\`pages\``. + const re = /ALTER\s+TABLE\s+(?:IF\s+EXISTS\s+)?(?:ONLY\s+)?["`]?(\w+)["`]?\s+ADD\s+COLUMN\s+(?:IF\s+NOT\s+EXISTS\s+)?["`]?(\w+)["`]?/gi; + let m: RegExpExecArray | null; + while ((m = re.exec(sql)) !== null) { + result.push({ table: m[1].toLowerCase(), column: m[2].toLowerCase() }); + } + return result; +} + +/** + * Read `src/core/migrate.ts` and emit every (table, column) pair that ANY + * `ALTER TABLE ... ADD COLUMN` reference in the file targets. Deduped: + * later occurrences of the same (table, column) are dropped. + * + * Source-file introspection covers all three migration shapes: + * - top-level `sql:` field (e.g. v22 ingest_log_source_id) + * - `sqlFor.postgres` / `sqlFor.pglite` overrides + * - handler-body `engine.runMigration(N, \`ALTER TABLE ... ADD COLUMN ...\`)` + * (e.g. v34 destructive_guard_columns) + * + * The function intentionally does NOT track which migration each column + * came from. That mapping requires parsing the surrounding migration object + * structure, which adds fragility for marginal value (the failure message + * names the table.column; the contributor can grep for it in migrate.ts). + */ +export function extractAddedColumnsFromMigrations(): AddedColumnRef[] { + const migratePath = resolve(process.cwd(), 'src/core/migrate.ts'); + const source = readFileSync(migratePath, 'utf-8'); + + const seen = new Set(); + const result: AddedColumnRef[] = []; + for (const ref of extractAlterAddColumnsFromSql(source)) { + const key = `${ref.table}.${ref.column}`; + if (seen.has(key)) continue; + seen.add(key); + result.push(ref); + } + return result; +} + +// Test-internal exports so the parser itself can be unit-tested. +export const __internal = { extractAlterAddColumnsFromSql }; diff --git a/test/orphans.test.ts b/test/orphans.test.ts index 7989dad78..7d56bce01 100644 --- a/test/orphans.test.ts +++ b/test/orphans.test.ts @@ -289,4 +289,84 @@ describe('findOrphans (engine-injected)', () => { expect(result.total_orphans).toBe(0); expect(result.total_pages).toBe(0); }); + + // ──────────────────────────────────────────────────────────────── + // Soft-delete filtering on BOTH sides (v0.26.5 invariant; codex C11) + // ──────────────────────────────────────────────────────────────── + + test('REGRESSION: soft-deleted page with no inbound is NOT in orphan results', async () => { + // Candidate-filter regression. Pre-fix, findOrphanPages returned every + // page without inbound links — including soft-deleted ones. Now the + // outer query filters p.deleted_at IS NULL. + await engine.putPage('people/alice', { + type: 'person', + title: 'Alice', + compiled_truth: 'Alice has no inbound links and is soft-deleted.', + timeline: '', + }); + // Soft-delete alice directly via the DB handle (no engine method exposed). + await (engine as any).db.query( + `UPDATE pages SET deleted_at = now() WHERE slug = 'people/alice'` + ); + + const rows = await queryOrphanPages(engine); + const slugs = rows.map(r => r.slug); + expect(slugs).not.toContain('people/alice'); + }); + + test('REGRESSION: live page with ONLY inbound link from soft-deleted source IS orphan (codex C11)', async () => { + // Link-source-filter regression. Pre-fix, a live page that had ONE + // inbound link from a soft-deleted source was hidden from orphan + // results because the EXISTS check didn't filter the source side. + // Now the inner JOIN filters src.deleted_at IS NULL too. + await engine.putPage('people/alice', { + type: 'person', + title: 'Alice', + compiled_truth: 'Alice was soft-deleted but used to link to Bob.', + timeline: '', + }); + await engine.putPage('people/bob', { + type: 'person', + title: 'Bob', + compiled_truth: 'Bob has no live inbound links.', + timeline: '', + }); + await engine.addLink('people/alice', 'people/bob', 'mentioned', 'references', 'markdown'); + + // Soft-delete alice. Bob's ONLY inbound link is now from a deleted page. + await (engine as any).db.query( + `UPDATE pages SET deleted_at = now() WHERE slug = 'people/alice'` + ); + + const rows = await queryOrphanPages(engine); + const slugs = rows.map(r => r.slug).sort(); + // alice is soft-deleted → not in results (candidate filter). + // bob has no LIVE inbound link → IS in results (link-source filter — codex C11). + expect(slugs).not.toContain('people/alice'); + expect(slugs).toContain('people/bob'); + }); + + test('live page with inbound link from LIVE source is NOT orphan (regression for unchanged behavior)', async () => { + // Sanity check: the soft-delete filter must NOT break the basic + // "live link counts" case. + await engine.putPage('people/alice', { + type: 'person', + title: 'Alice', + compiled_truth: 'Alice is live and links to Bob.', + timeline: '', + }); + await engine.putPage('people/bob', { + type: 'person', + title: 'Bob', + compiled_truth: 'Bob has a live inbound from Alice.', + timeline: '', + }); + await engine.addLink('people/alice', 'people/bob', 'mentioned', 'references', 'markdown'); + + const rows = await queryOrphanPages(engine); + const slugs = rows.map(r => r.slug); + // alice is orphan (no inbound), bob is NOT (alice links to him). + expect(slugs).toContain('people/alice'); + expect(slugs).not.toContain('people/bob'); + }); }); diff --git a/test/schema-bootstrap-coverage.test.ts b/test/schema-bootstrap-coverage.test.ts index ec28a3977..5e9bb0a79 100644 --- a/test/schema-bootstrap-coverage.test.ts +++ b/test/schema-bootstrap-coverage.test.ts @@ -108,6 +108,27 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [ // created_at DESC)`. Old brains have ingest_log without source_id; bootstrap // adds the column before SCHEMA_SQL replay creates the index. { kind: 'column', table: 'ingest_log', column: 'source_id' }, + // v0.18 (v18) — forward-referenced by `CREATE INDEX idx_files_source_id ON + // files(source_id)` and `CREATE INDEX idx_files_page_id ON files(page_id)`. + // Pre-v18 brains have files without these columns; bootstrap adds them + // before SCHEMA_SQL replay creates the indexes. + { kind: 'column', table: 'files', column: 'source_id' }, + { kind: 'column', table: 'files', column: 'page_id' }, + // v0.34.1 (v60+v61+v65) — forward-referenced by the FK + // `oauth_clients.source_id REFERENCES sources(id)` and the GIN index + // `idx_oauth_clients_federated_read ON oauth_clients USING GIN (federated_read)`. + // Pre-v60 brains have oauth_clients without these columns; bootstrap adds + // them before SCHEMA_SQL replay creates the FK + index. + { kind: 'column', table: 'oauth_clients', column: 'source_id' }, + { kind: 'column', table: 'oauth_clients', column: 'federated_read' }, + // v0.26.5 (v34) — promotes archive lifecycle from JSONB config to real + // columns on sources. CREATE TABLE IF NOT EXISTS is a no-op on existing + // sources tables, so the visibility filters in search/list_pages that + // reference these columns trip on pre-v34 brains. Bootstrap adds them + // before any visibility-filter SQL runs. + { kind: 'column', table: 'sources', column: 'archived' }, + { kind: 'column', table: 'sources', column: 'archived_at' }, + { kind: 'column', table: 'sources', column: 'archive_expires_at' }, ]; test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => { @@ -168,8 +189,26 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in ALTER TABLE pages DROP COLUMN IF EXISTS import_filename; ALTER TABLE pages DROP COLUMN IF EXISTS salience_touched_at; ALTER TABLE pages DROP COLUMN IF EXISTS emotional_weight; + + DROP INDEX IF EXISTS idx_ingest_log_source_type_created; + ALTER TABLE ingest_log DROP COLUMN IF EXISTS source_id; + + DROP INDEX IF EXISTS idx_files_source_id; + DROP INDEX IF EXISTS idx_files_page_id; + ALTER TABLE files DROP COLUMN IF EXISTS source_id; + ALTER TABLE files DROP COLUMN IF EXISTS page_id; + + DROP INDEX IF EXISTS idx_oauth_clients_federated_read; + ALTER TABLE oauth_clients DROP COLUMN IF EXISTS source_id; + ALTER TABLE oauth_clients DROP COLUMN IF EXISTS federated_read; `); + // Note: we don't strip sources.archived* here because they're inline in the + // sources CREATE TABLE definition (no separate ALTER TABLE), and the + // earlier `DROP TABLE IF EXISTS sources CASCADE` already nuked them. + // The bootstrap's needsPagesBootstrap branch recreates sources without the + // archive columns; the new needsSourcesArchive probe adds them. + // Run bootstrap in isolation (NOT initSchema). This is what we're testing. await (engine as any).applyForwardReferenceBootstrap(); @@ -311,7 +350,15 @@ function parseBaseTableColumns(sql: string): Map> { parts.push(body.slice(start)); for (const partRaw of parts) { - const part = partRaw.trim(); + // Strip SQL line comments (`-- ...` to end of line) and block + // comments (`/* ... */`) before identifying the column name. + // Without this, a column definition preceded by a comment inside + // the CREATE TABLE body is silently dropped (the comment is the + // "first identifier" and the parser bails out). + const stripped = partRaw + .replace(/--[^\n]*/g, '') + .replace(/\/\*[\s\S]*?\*\//g, ''); + const part = stripped.trim(); if (!part) continue; // Skip constraint lines. if (/^(CONSTRAINT|PRIMARY|UNIQUE|CHECK|FOREIGN|EXCLUDE)\b/i.test(part)) continue; @@ -537,3 +584,184 @@ test('every CREATE INDEX column in PGLITE_SCHEMA_SQL is covered by CREATE TABLE ); } }, 30000); + +// ───────────────────────────────────────────────────────────────── +// v0.36+ — MIGRATIONS introspection: catch the column-only forward-ref class. +// ───────────────────────────────────────────────────────────────── +// The CREATE INDEX parser above kills the column-with-index forward-ref class. +// v0.26.5 (v34) introduced a column-ONLY class: `sources.archived` + +// `sources.archived_at` + `sources.archive_expires_at` aren't indexed but +// `CREATE TABLE IF NOT EXISTS sources` is a no-op on pre-v34 brains. The +// schema-blob replay never adds the archive columns, so downstream visibility +// filters trip immediately. +// +// This test walks every `ALTER TABLE ... ADD COLUMN` in the MIGRATIONS array +// (our own structured code, not arbitrary Postgres DDL) and asserts every +// (table, column) pair is also added by `applyForwardReferenceBootstrap`. +// Future contributors who add a migration with ALTER TABLE ADD COLUMN AND +// forget to extend the bootstrap will see this test fail at PR time with a +// paste-ready `Add probe for
.` message. +// +// Why regex-on-our-own-SQL is safe vs regex-on-prod-Postgres-DDL: every +// migration's SQL string is authored by us with consistent shape. The +// ALTER TABLE ADD COLUMN pattern is stable across all 60+ existing +// migrations. We control the input, not Postgres. +// +// Exemption mechanism: some migrations add columns that are intentionally +// not in the schema blob (one-off transition columns later dropped, etc.). +// Those go in the COLUMN_EXEMPTIONS set below with a brief rationale. +// ───────────────────────────────────────────────────────────────── + +const COLUMN_EXEMPTIONS = new Set([ + // Schema-blob-not-yet-refreshed: each of these columns is added by a + // migration but NOT (yet) referenced by `PGLITE_SCHEMA_SQL` (neither in a + // CREATE TABLE body nor in any CREATE INDEX). Bootstrap doesn't need to + // add them because there's no forward reference for the schema blob's + // replay to trip on. The migration handles every upgrade path correctly: + // - fresh install: schema blob replays, then migration adds the column. + // - pre-existing brain missing the column: migration adds it via ALTER. + // - pre-existing brain already on this column: ALTER ... IF NOT EXISTS no-ops. + // If a future migration adds a CREATE INDEX that references one of these + // columns, the existing v0.28.5 CREATE-INDEX parser will catch it and + // force a bootstrap probe (and the exemption should be removed). + // + // Refreshing PGLITE_SCHEMA_SQL is a separate concern handled by + // `bun run build:schema` from src/schema.sql; not gated by this test. + 'minion_jobs.quiet_hours', + 'minion_jobs.stagger_key', + 'sources.chunker_version', + 'access_tokens.permissions', + 'takes.resolved_quality', + 'pages.emotional_weight_recomputed_at', + 'facts.notability', + 'facts.row_num', + 'facts.source_markdown_slug', + 'pages.chunker_version', + 'pages.source_path', + 'content_chunks.edges_backfilled_at', + 'query_cache.knobs_hash', +]); + +test('every ALTER TABLE ADD COLUMN in MIGRATIONS is covered by applyForwardReferenceBootstrap (column-only class)', async () => { + const { extractAddedColumnsFromMigrations } = await import('./helpers/extract-added-columns.ts'); + const { readFileSync } = await import('fs'); + const { resolve: resolvePath } = await import('path'); + const { PGLITE_SCHEMA_SQL } = await import('../src/core/pglite-schema.ts'); + + const enginePath = resolvePath(process.cwd(), 'src/core/pglite-engine.ts'); + const engineSrc = readFileSync(enginePath, 'utf-8'); + const bootstrapAdds = parseAlterAddColumns(engineSrc); + + // Bootstrap's own CREATE TABLE statements (e.g. needsPagesBootstrap inlines + // `archived BOOLEAN ...` inside the CREATE TABLE sources block). Those + // count as covered without a separate ALTER TABLE ADD COLUMN. + const bootstrapCreateTableCols = parseBaseTableColumns(engineSrc); + + // PGLITE_SCHEMA_SQL's CREATE TABLE definitions. The schema blob defines + // every modern table inline; columns added by migrations are typically + // ALSO updated in the schema blob so fresh installs get them natively. + // The bootstrap is only needed when: (a) the table existed before the + // migration ran (so CREATE TABLE IF NOT EXISTS is a no-op on old brains) + // AND (b) the column has a forward-reference index OR a downstream filter + // that breaks on old brains. Schema-blob coverage handles the fresh case. + const schemaCreateTableCols = parseBaseTableColumns(PGLITE_SCHEMA_SQL); + + const migrationAdds = extractAddedColumnsFromMigrations(); + + const covered = (table: string, column: string): boolean => { + if (COLUMN_EXEMPTIONS.has(`${table}.${column}`)) return true; + if (bootstrapAdds.some(a => a.table === table && a.column === column)) return true; + const bootstrapCols = bootstrapCreateTableCols.get(table); + if (bootstrapCols && bootstrapCols.has(column)) return true; + const schemaCols = schemaCreateTableCols.get(table); + if (schemaCols && schemaCols.has(column)) return true; + return false; + }; + + const uncovered: typeof migrationAdds = []; + for (const ref of migrationAdds) { + if (!covered(ref.table, ref.column)) { + uncovered.push(ref); + } + } + + if (uncovered.length > 0) { + const list = uncovered + .map(u => ` ${u.table}.${u.column}`) + .join('\n'); + throw new Error( + `MIGRATIONS file (src/core/migrate.ts) adds ${uncovered.length} (table, column) pair(s) that ` + + `applyForwardReferenceBootstrap does NOT cover:\n${list}\n\n` + + `Fix one of:\n` + + ` 1. Add a probe + ALTER TABLE ADD COLUMN in applyForwardReferenceBootstrap ` + + `(src/core/pglite-engine.ts AND src/core/postgres-engine.ts), OR\n` + + ` 2. If the column is intentionally not in the schema blob ` + + `(transitional / handler-only / later-dropped), add the (table, column) ` + + `to COLUMN_EXEMPTIONS in test/schema-bootstrap-coverage.test.ts with a ` + + `brief rationale comment.`, + ); + } +}); + +test('extractAddedColumnsFromMigrations sanity-checks against known migration column additions', async () => { + // Lightweight sanity test that the helper extracts the columns we expect + // for a few well-known v34 / v60 / v61 migrations. Catches regex + // regressions in the helper itself. + const { extractAddedColumnsFromMigrations } = await import('./helpers/extract-added-columns.ts'); + const refs = extractAddedColumnsFromMigrations(); + const has = (table: string, column: string) => + refs.some(r => r.table === table && r.column === column); + // v34 sources.archived* (the codex C1 case) + expect(has('sources', 'archived')).toBe(true); + expect(has('sources', 'archived_at')).toBe(true); + expect(has('sources', 'archive_expires_at')).toBe(true); + // v60+v61 oauth_clients.* + expect(has('oauth_clients', 'source_id')).toBe(true); + expect(has('oauth_clients', 'federated_read')).toBe(true); + // v18 files.* + expect(has('files', 'source_id')).toBe(true); + expect(has('files', 'page_id')).toBe(true); +}); + +test('extractAlterAddColumnsFromSql handles representative migration SQL shapes', async () => { + const { __internal } = await import('./helpers/extract-added-columns.ts'); + const fn = __internal.extractAlterAddColumnsFromSql; + + // Standard shape (with IF NOT EXISTS) + expect(fn('ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN')).toEqual([ + { table: 'sources', column: 'archived' }, + ]); + // No IF NOT EXISTS (older migrations) + expect(fn('ALTER TABLE pages ADD COLUMN deleted_at TIMESTAMPTZ;')).toEqual([ + { table: 'pages', column: 'deleted_at' }, + ]); + // Multi-statement, mixed + expect(fn(` + CREATE INDEX foo ON bar(x); + ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS source_id TEXT REFERENCES sources(id); + ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS federated_read TEXT[] NOT NULL DEFAULT '{}'; + UPDATE oauth_clients SET source_id = 'default'; + `)).toEqual([ + { table: 'oauth_clients', column: 'source_id' }, + { table: 'oauth_clients', column: 'federated_read' }, + ]); + // Quoted identifiers + expect(fn('ALTER TABLE "pages" ADD COLUMN "effective_date" TIMESTAMPTZ')).toEqual([ + { table: 'pages', column: 'effective_date' }, + ]); + // ALTER TABLE IF EXISTS / ONLY variants + expect(fn('ALTER TABLE IF EXISTS ONLY content_chunks ADD COLUMN language TEXT')).toEqual([ + { table: 'content_chunks', column: 'language' }, + ]); +}); + +test('planted-bug: simulated unprovided column produces a clear failure message', async () => { + // Negative case — regression guard. If the contract test silently passes + // on uncovered columns, the gate is fake. This test plants a fake column + // in a fake SQL string and verifies the helper extracts it (proving the + // gate would catch it in the real contract test). + const { __internal } = await import('./helpers/extract-added-columns.ts'); + const fn = __internal.extractAlterAddColumnsFromSql; + const planted = fn('ALTER TABLE pages ADD COLUMN IF NOT EXISTS planted_test_col TEXT'); + expect(planted).toEqual([{ table: 'pages', column: 'planted_test_col' }]); +}); diff --git a/test/storage-sync.test.ts b/test/storage-sync.test.ts index d383efc36..c386717b5 100644 --- a/test/storage-sync.test.ts +++ b/test/storage-sync.test.ts @@ -141,4 +141,50 @@ describe('manageGitignore', () => { manageGitignore(tmp); expect(warnings.some((w) => /Could not (read|update)/.test(w))).toBe(true); }); + + // ──────────────────────────────────────────────────────────────── + // Worktree vs submodule discrimination (closes #889) + // ──────────────────────────────────────────────────────────────── + + test('REGRESSION: submodule with relative gitdir/modules/ → skip (D49 contract)', () => { + writeStorageConfig(); + writeFileSync(join(tmp, '.git'), 'gitdir: ../.git/modules/sub\n'); + manageGitignore(tmp); + expect(existsSync(join(tmp, '.gitignore'))).toBe(false); + expect(warnings.some((w) => /submodule/.test(w))).toBe(true); + }); + + test('absorbed submodule with absolute gitdir/modules/ → skip (closes edge case)', () => { + writeStorageConfig(); + // After `git submodule absorbgitdirs`, the gitdir path becomes absolute. + writeFileSync(join(tmp, '.git'), 'gitdir: /home/user/parent/.git/modules/sub\n'); + manageGitignore(tmp); + expect(existsSync(join(tmp, '.gitignore'))).toBe(false); + expect(warnings.some((w) => /submodule/.test(w))).toBe(true); + }); + + test('CRITICAL: worktree with absolute gitdir/worktrees/ → MANAGE (closes #889)', () => { + writeStorageConfig(); + writeFileSync(join(tmp, '.git'), 'gitdir: /home/user/repo/.git/worktrees/feature-branch\n'); + manageGitignore(tmp); + expect(existsSync(join(tmp, '.gitignore'))).toBe(true); + expect(warnings.filter((w) => /submodule/.test(w))).toEqual([]); + }); + + test('worktree with relative gitdir/worktrees/ → MANAGE', () => { + writeStorageConfig(); + writeFileSync(join(tmp, '.git'), 'gitdir: ../.git/worktrees/feature-branch\n'); + manageGitignore(tmp); + expect(existsSync(join(tmp, '.gitignore'))).toBe(true); + expect(warnings.filter((w) => /submodule/.test(w))).toEqual([]); + }); + + test('malformed .git file (no gitdir: prefix) → MANAGE (preserves catch behavior)', () => { + writeStorageConfig(); + writeFileSync(join(tmp, '.git'), 'garbage content\n'); + manageGitignore(tmp); + // No gitdir prefix → not a submodule → MANAGE. + expect(existsSync(join(tmp, '.gitignore'))).toBe(true); + expect(warnings.filter((w) => /submodule/.test(w))).toEqual([]); + }); }); diff --git a/test/sync.test.ts b/test/sync.test.ts index a7d0d67cc..14be8a0a7 100644 --- a/test/sync.test.ts +++ b/test/sync.test.ts @@ -1,5 +1,5 @@ import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test'; -import { buildSyncManifest, isSyncable, pathToSlug } from '../src/core/sync.ts'; +import { buildSyncManifest, isSyncable, pathToSlug, pruneDir } from '../src/core/sync.ts'; import { buildGitInvocation } from '../src/commands/sync.ts'; import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; import { join } from 'path'; @@ -99,6 +99,56 @@ describe('isSyncable', () => { expect(isSyncable('ops/deploy-log.md')).toBe(false); expect(isSyncable('ops/config.md')).toBe(false); }); + + // ──────────────────────────────────────────────────────────────── + // v0.36 walker drift fix (closes #923, #202): node_modules exclusion + // ──────────────────────────────────────────────────────────────── + + test('CRITICAL latent-bug regression: rejects node_modules paths at any depth', () => { + // Pre-v0.36, isSyncable had no node_modules check. Any markdown file + // under a non-dot `node_modules` directory slipped through. This is + // the canonical latent-bug fix gated by IRON RULE per the wave plan. + expect(isSyncable('node_modules/some-pkg/README.md')).toBe(false); + expect(isSyncable('node_modules/some-pkg/CHANGELOG.md')).toBe(false); + expect(isSyncable('node_modules/some-pkg/docs/api.md')).toBe(false); + expect(isSyncable('apps/web/node_modules/dep/notes.md')).toBe(false); + }); +}); + +describe('pruneDir', () => { + test('blocks node_modules (no leading dot, the latent-bug case)', () => { + expect(pruneDir('node_modules')).toBe(false); + }); + + test('blocks dot-prefix dirs (.git, .obsidian, .raw, .cache, etc.)', () => { + expect(pruneDir('.git')).toBe(false); + expect(pruneDir('.obsidian')).toBe(false); + expect(pruneDir('.raw')).toBe(false); + expect(pruneDir('.cache')).toBe(false); + expect(pruneDir('.vscode')).toBe(false); + }); + + test('blocks ops (gbrain operational dir)', () => { + expect(pruneDir('ops')).toBe(false); + }); + + test('blocks *.raw sidecar dirs (gbrain convention)', () => { + expect(pruneDir('.raw')).toBe(false); + expect(pruneDir('pedro.raw')).toBe(false); + expect(pruneDir('article.raw')).toBe(false); + }); + + test('allows normal content dirs', () => { + expect(pruneDir('wiki')).toBe(true); + expect(pruneDir('people')).toBe(true); + expect(pruneDir('meetings')).toBe(true); + expect(pruneDir('corpus')).toBe(true); + expect(pruneDir('2026')).toBe(true); + }); + + test('empty string returns true (defensive default)', () => { + expect(pruneDir('')).toBe(true); + }); }); describe('pathToSlug', () => { diff --git a/test/think-gateway-adapter.test.ts b/test/think-gateway-adapter.test.ts new file mode 100644 index 000000000..c97ff1f9c --- /dev/null +++ b/test/think-gateway-adapter.test.ts @@ -0,0 +1,106 @@ +/** + * Gateway adapter tests for runThink (#952 fix). + * + * Pre-v0.36, runThink instantiated `new Anthropic()` directly. Closing #952 + * routed it through gateway.chat() so MCP stdio launches pick up + * `anthropic_api_key` from gbrain config instead of process.env. + * + * The adapter shape was determined by plan-eng-review D10 (cross-model + * tension D10 with codex C7+C8+C9+C10): + * - drop new Anthropic() entirely + * - real availability check (NOT a false-positive `getChatModel()` truthy) + * - model-id normalization (bare → provider-prefixed) + * - response-shape conversion (ChatResult → Anthropic.Message) + * + * These tests pin the four spec points. Hermetic — no real LLM call. + */ + +import { describe, test, expect } from 'bun:test'; +import { __thinkAdapter } from '../src/core/think/index.ts'; +import { withEnv } from './helpers/with-env.ts'; + +describe('think gateway adapter — response shape conversion', () => { + test('chatResultToMessage maps ChatResult.text to Anthropic.Message content[0].text', () => { + const out = __thinkAdapter.chatResultToMessage( + { + text: '{"answer":"hi","citations":[],"gaps":[]}', + blocks: [], + stopReason: 'end', + usage: { input_tokens: 5, output_tokens: 2, cache_read_tokens: 0, cache_creation_tokens: 0 }, + model: 'anthropic:claude-opus-4-7', + providerId: 'anthropic', + }, + 'anthropic:claude-opus-4-7', + ); + expect(out.content[0].type).toBe('text'); + expect(out.content[0].text).toBe('{"answer":"hi","citations":[],"gaps":[]}'); + expect(out.usage.input_tokens).toBe(5); + expect(out.usage.output_tokens).toBe(2); + expect(out.stop_reason).toBe('end_turn'); + expect(out.model).toBe('anthropic:claude-opus-4-7'); + }); + + test('mapStopReason covers the full provider-neutral stop-reason set', () => { + expect(__thinkAdapter.mapStopReason('end')).toBe('end_turn'); + expect(__thinkAdapter.mapStopReason('length')).toBe('max_tokens'); + expect(__thinkAdapter.mapStopReason('tool_calls')).toBe('tool_use'); + // 'refusal', 'content_filter', 'other' → end_turn (no Anthropic equivalent). + expect(__thinkAdapter.mapStopReason('refusal')).toBe('end_turn'); + expect(__thinkAdapter.mapStopReason('content_filter')).toBe('end_turn'); + expect(__thinkAdapter.mapStopReason('other')).toBe('end_turn'); + }); +}); + +describe('think gateway adapter — model-id normalization', () => { + test('tryBuildGatewayClient accepts bare anthropic model ids and prefixes anthropic:', async () => { + // Bare model: `claude-opus-4-7` → must resolve through `anthropic:` recipe. + // resolveRecipe will throw AIConfigError for unknown providers, so a + // successful build proves the prefix landed. + await withEnv({ ANTHROPIC_API_KEY: 'sk-test-fake' }, async () => { + const client = await __thinkAdapter.tryBuildGatewayClient('claude-opus-4-7'); + expect(client).not.toBeNull(); + }); + }); + + test('tryBuildGatewayClient accepts already-prefixed provider:model strings', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test-fake' }, async () => { + const client = await __thinkAdapter.tryBuildGatewayClient('anthropic:claude-sonnet-4-6'); + expect(client).not.toBeNull(); + }); + }); + + test('tryBuildGatewayClient returns null on unknown provider (AIConfigError → graceful fallback)', async () => { + const client = await __thinkAdapter.tryBuildGatewayClient('nonexistent-provider:foo-1'); + expect(client).toBeNull(); + }); + + test('tryBuildGatewayClient returns null when ANTHROPIC_API_KEY is absent (preserves legacy NO_ANTHROPIC_API_KEY signal)', async () => { + await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => { + const client = await __thinkAdapter.tryBuildGatewayClient('claude-opus-4-7'); + expect(client).toBeNull(); + }); + }); + + test('hasAnthropicKey reads process.env', async () => { + await withEnv({ ANTHROPIC_API_KEY: 'sk-test-key' }, async () => { + expect(__thinkAdapter.hasAnthropicKey()).toBe(true); + }); + await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => { + expect(__thinkAdapter.hasAnthropicKey()).toBe(false); + }); + }); +}); + +describe('think gateway adapter — graceful fallback shape', () => { + test('buildGracefulMessage produces a parseable Anthropic.Message-shaped object', () => { + const m = __thinkAdapter.buildGracefulMessage('anthropic:claude-opus-4-7'); + expect(m.type).toBe('message'); + expect(m.role).toBe('assistant'); + expect(m.content[0].type).toBe('text'); + expect(m.content[0].text).toContain('no LLM available'); + expect(m.content[0].text).toContain('gbrain config'); + expect(m.usage.input_tokens).toBe(0); + expect(m.usage.output_tokens).toBe(0); + expect(m.stop_reason).toBe('end_turn'); + }); +});