mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.35.5.0 fix wave: bootstrap + orphans + think MCP + worktree + walker (#1111)
* fix(bootstrap): extend probes for files/oauth_clients/sources.archived* + add MIGRATIONS introspection guard Adds 7 new forward-reference probes to applyForwardReferenceBootstrap on both engines, closes the column-only forward-ref class via a new MIGRATIONS-source introspection contract test. New probes: - files.source_id + files.page_id (v18 forward refs) - oauth_clients.source_id + oauth_clients.federated_read (v60+v61+v65) - sources.archived + archived_at + archive_expires_at (v34 promoted from JSONB) The sources.archived* columns are the codex-flagged class: they're added inline in v34's CREATE TABLE definition but `CREATE TABLE IF NOT EXISTS sources` is a no-op on pre-v34 brains, so downstream visibility filters (search/list_pages) trip on old brains. needsPagesBootstrap now folds archive columns into its CREATE TABLE so pre-v0.18 brains get a v34-shape sources in one go; needsSourcesArchive then only fires on the pre-v34 case (sources exists, archive cols don't). Closes the structural bug class via test/helpers/extract-added-columns.ts: reads src/core/migrate.ts as text and extracts every ALTER TABLE ADD COLUMN. The new contract test asserts every (table, column) pair is covered by EITHER the bootstrap's ALTER TABLE statements, the bootstrap's CREATE TABLE definitions, OR the schema blob's CREATE TABLE bodies. The column-only class (no index, no FK; just an inline CREATE TABLE column the schema blob can't add to existing tables) is now caught at PR time. Source-text introspection catches all three migration shapes uniformly: - top-level `sql:` field - `sqlFor.postgres` / `sqlFor.pglite` overrides - handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)` (v34 shape) Pre-existing parseBaseTableColumns parser bug fixed: now strips `--` line comments and `/* ... */` blocks before identifying column names. Without this, a column preceded by a comment was silently dropped. Catches pages.page_kind and others that were silently uncovered. 13 columns added by migrations but not in PGLITE_SCHEMA_SQL are exempted with a unified rationale: they have no schema-blob forward reference; migration handles all upgrade paths cleanly. Refreshing the schema blob is a separate concern. Issues closed: #1018 (v60 oauth_clients), #974 (files.source_id/page_id), #820 (v0.13.0 migration files.page_id cascade); pre-empts the sources.archived class before any pre-v34 brain trips on it. Tests: - 9 cases in test/schema-bootstrap-coverage.test.ts (5 existing + 4 new) - helper-level unit tests cover SQL shape variants (IF NOT EXISTS, quoted identifiers, ALTER TABLE IF EXISTS ONLY, multi-statement) - planted-bug regression verifies the gate actually catches new uncovered columns Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(orphans): filter soft-deleted pages on both candidate and link-source sides Closes #1021. The v0.26.5 soft-delete invariant requires that findOrphanPages exclude both: 1. Candidate pages that are themselves soft-deleted 2. Inbound links from soft-deleted source pages Pre-fix, findOrphanPages had no deleted_at filter at all. Soft-deleted pages with no inbound links were counted as orphans (inflating counts). Pre-codex-tension-D11, only the candidate-side filter was planned. Codex C11 caught the second case: a live page that has ONE inbound link from a soft-deleted source page was hidden from orphan results — the link still existed in the links table, the EXISTS subquery saw it, the page looked "linked." Now the inner JOIN on pages enforces src.deleted_at IS NULL. Three regression tests pin the contract: - soft-deleted page with no inbound → NOT orphan - live page with ONLY inbound link from soft-deleted source → IS orphan - live page with live inbound → NOT orphan (smoke check that the new filters don't break unchanged behavior) Engine parity: same SQL shape on both Postgres and PGLite engines. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(think): route runThink through gateway.chat adapter (closes #952) Pre-fix, runThink instantiated `new Anthropic()` directly and read ANTHROPIC_API_KEY from process.env. Claude Desktop's stdio MCP launch doesn't inherit shell env, so `gbrain config set anthropic_api_key sk-...` (writes to ~/.gbrain/config.json) never reached the SDK and every MCP think call degraded to "no LLM available." The adapter routes through gateway.chat() — the canonical seam per CLAUDE.md. Gateway reads the API key from gbrain config OR env, picks up prompt caching, rate-leases, retry, and the test seam (__setChatTransportForTests) that v0.31.12 established. Per plan-eng-review D10 (cross-model tension with codex C7+C8+C9+C10), four spec points landed: 1. Drop `new Anthropic()` direct path entirely. Every non-stub LLM call from runThink routes through gateway. 2. Real availability check (NOT a false-positive `getChatModel()` truthy). `tryBuildGatewayClient` probes both the recipe (resolveRecipe throws AIConfigError on unknown providers) AND the API key (reads process.env + loadConfig at the gbrain config layer for parity with gateway's own auth resolution). Returns null on miss; runThink takes the graceful "no LLM available" early-return preserving the legacy NO_ANTHROPIC_API_KEY warning signal. 3. Model-id normalization. resolveModel returns bare anthropic ids (claude-opus-4-7); gateway.chat needs provider:model. Adapter auto-prefixes anthropic: when the id is bare. Provider:model strings pass through unchanged. 4. Response-shape conversion. ChatResult → Anthropic.Message via chatResultToMessage. mapStopReason translates gateway's provider-neutral stop reasons (end / length / tool_calls / refusal / content_filter / other) to Anthropic's stop_reason ('end_turn' / 'max_tokens' / 'tool_use'); refusal/content_filter/other fall through to end_turn (no Anthropic equivalent). Usage tokens pass through. `opts.client` injection preserved (test seam — see ThinkLLMClient). `opts.stubResponse` preserved (pure-test escape). Tests: - test/think-gateway-adapter.test.ts (9 cases): response shape, stop reason mapping, model-id normalization (bare + prefixed), provider unknown returns null, ANTHROPIC_API_KEY absent returns null (regression for legacy graceful degradation), hasAnthropicKey reads process.env correctly. Uses withEnv per the test-isolation contract. - test/think-pipeline.serial.test.ts (17 existing cases): unchanged; the graceful-degradation case at line 213 still produces the NO_ANTHROPIC_API_KEY warning because tryBuildGatewayClient returns null when no key is configured, taking the legacy early-return path. Closes #952. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(sync): distinguish git worktree from submodule via path-segment match (closes #889) Pre-fix, `manageGitignore` treated every `.git`-as-file as a submodule and skipped gitignore management. Both submodules AND worktrees use `.git` as a file (not a directory), so the legacy `statSync.isFile()` check couldn't discriminate. Worktrees got misclassified as submodules and their .gitignore wasn't managed. Per plan-eng-review D4 (chose path-segment match over absolute-vs- relative path heuristic): the gitdir path contains: - `/modules/<name>` for submodules (skip — managed by parent repo) - `/worktrees/<name>` for worktrees (MANAGE — first-class repo) Both are documented Git internal layouts, stable across all 4 {relative, absolute} × {modules, worktrees} combinations including the absorbed-submodule edge case from `git submodule absorbgitdirs` (where the submodule's gitdir flips to an absolute path). Malformed `.git` file (no `gitdir:` prefix, IO error) → MANAGE, preserving the pre-#889 catch{} fail-closed-toward-managing semantics. Tests (5 new + 1 regression renamed): - REGRESSION: submodule relative gitdir/modules/ → skip (D49 contract) - absorbed submodule absolute gitdir/modules/ → skip (edge case) - CRITICAL: worktree absolute gitdir/worktrees/ → MANAGE (closes #889) - worktree relative gitdir/worktrees/ → MANAGE - malformed .git file → MANAGE (preserves catch behavior) - regular .git directory → MANAGE (existing smoke) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(walkers): pruneDir helper + descent-time exclusion + transcript predicate (closes #923, #202) Per plan-eng-review D12 (cross-model tension with codex C12+C13), three structural changes: 1. Extract `pruneDir(name)` helper in src/core/sync.ts. Returns false for directory names walkers must NEVER descend into: `node_modules` (latent bug — no leading dot), dot-prefix dirs (`.git`, `.obsidian`, `.raw`, `.cache`, etc.), `ops`, and `*.raw` sidecar dirs (gbrain convention — `people/pedro.raw/` holds raw source for pedro.md). Walkers consult it at descent time BEFORE recursion, saving the IO cost of walking entire vendor / hidden / sidecar subtrees only to filter them at file-emit time. 2. `isSyncable` itself gains the same exclusion set (via pruneDir on each path segment). Closes the latent bug where node_modules markdown files slipped through: `node_modules/some-pkg/README.md` returned true pre-fix because the legacy dot-prefix check only blocked `.node_modules` (with a leading dot), not the actual `node_modules`. CRITICAL regression test in test/sync.test.ts pins the contract per IRON RULE. 3. Two walkers rewritten to use pruneDir at descent + per-walker file predicate at emit: - `walkMarkdownFiles` (src/commands/extract.ts): pruneDir + isSyncable ({strategy:'markdown'}). Pre-fix this walker had ONLY an ad-hoc dot-prefix exclusion and didn't call isSyncable at all — descended into node_modules, emitted markdown files from there, ignored README/ ops/.raw filters. - `listTextFiles` (src/core/cycle/transcript-discovery.ts): pruneDir + own .txt/.md predicate. DOES NOT use isSyncable({strategy:'markdown'}) because transcripts accept .txt and don't share markdown sync's README/ops exclusions (codex C12). Also made RECURSIVE — pre-fix it walked only the top dir, so transcripts in `corpus/2026/` were invisible (codex C14 — descent-time pruning is the right shape but the test would have passed vacuously on a non-recursive walker). Verified blast radius before adding node_modules: every existing isSyncable caller (sync.ts:558-561 sync filter, frontmatter.ts:264 validate, brain-writer.ts:305 reverse-write, import.ts:454 import filter) wants node_modules excluded — this is a latent-bug fix, not a behavior change for any legitimate caller. Tests: - 7 new isSyncable cases including the node_modules CRITICAL regression - 6 new pruneDir cases (node_modules, dot-prefix, ops, *.raw, content dirs that should pass, empty-string default) - Existing extract.test.ts + extract-fs.test.ts unchanged and passing Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(todos): file v0.36.x follow-ups for runThink rewrite + Supabase bootstrap parity Two follow-up TODOs filed during the v0.36 dreamy-thompson wave: 1. runThink full rewrite (D5+D7 from plan-eng-review): drop the ThinkLLMClient indirection now that v0.36 routes through gateway.chat. 12+ tests need migration to __setChatTransportForTests. Blocked by this wave landing. 2. Supabase parity test for applyForwardReferenceBootstrap (codex C6 residual): real Docker Postgres E2E catches schema correctness but not Supabase pooler/direct-pool routing. The probe uses this.sql but PostgresEngine.initSchema chooses a DDL connection; the divergence has caused multiple historical wedges (#699, #820 lineage). Both entries include full context per the CLAUDE.md TODOS-format spec (what, why, pros, cons, blocked-by, plan reference). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(bootstrap): thread DDL connection through applyForwardReferenceBootstrap Codex adversarial review during /ship caught a P1: initSchema selected a DDL connection, took pg_advisory_lock(42) on it, but applyForwardReferenceBootstrap used `this.sql` (the instance pool) inside. Bootstrap probes ran outside the lock scope on a different connection. Failure mode: two concurrent gbrain instances could BOTH enter the bootstrap block on Supabase transaction-pooler setups because the advisory lock was held on a different connection than the one running ALTER TABLE. The pooler's statement_timeout could also kill the probes mid-flight without affecting the lock-holder, leaving an inconsistent schema state. Fix: applyForwardReferenceBootstrap now accepts an optional connection parameter. initSchema passes the DDL conn (the one holding the lock). this.sql remains the fallback for any unit-test path that calls bootstrap directly. PGLite engine doesn't need this change — single connection, no pooler. This was pre-existing (every prior probe used this.sql), but the v0.36 wave is explicitly about fixing the Supabase upgrade-wedge class. Codex's position was correct: don't ship the wave with the underlying connection mismatch still there. The Supabase parity TEST FIXTURE follow-up remains on TODOS.md (test infra needed to PROVE the fix works under real pooler topology), but the bug itself is closed. 15/15 bootstrap tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.35.5.0) Six-correctness-fix wave: bootstrap forward-ref class (4 issues + 1 pre-empt), orphans soft-delete leak (both sides), runThink → gateway.chat adapter, git worktree vs submodule discriminator, walker pruneDir + descent-time exclusion, plus a Codex-P1 catch during /ship that threaded the DDL connection through applyForwardReferenceBootstrap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CLAUDE.md for v0.35.5.0 backend correctness wave Fold v0.35.5.0 file-level annotations into CLAUDE.md: - postgres-engine.ts + pglite-engine.ts: 7 new applyForwardReferenceBootstrap probes (files.source_id/page_id, oauth_clients.source_id/federated_read, sources.archived/archived_at/archive_expires_at) + DDL connection threading - test/schema-bootstrap-coverage.test.ts: new MIGRATIONS-source introspection guard + parseBaseTableColumns comment-stripping fix - src/core/sync.ts: new pruneDir helper + manageGitignore worktree discriminator - src/core/think/index.ts (new entry): runThink gateway adapter for MCP stdio key resolution - src/core/operations.ts (new entry): findOrphanPages soft-delete filter Regenerate llms-full.txt via bun run build:llms. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
0c6fcab555
commit
4446e9f9d2
@@ -2,6 +2,99 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.35.5.0] - 2026-05-16
|
||||
|
||||
**Upgrade goes through on stuck Supabase brains. Orphan counts get honest. `gbrain think` over MCP actually answers. Worktrees stop being misclassified. node_modules stops leaking.**
|
||||
|
||||
A correctness wave. Five user-visible bugs closed; one structural CI guard added so the bootstrap class can't bite the same shape again. The shape of the change: `applyForwardReferenceBootstrap` now probes for seven previously-missing columns (files.source_id/page_id, oauth_clients.source_id/federated_read, sources.archived/archived_at/archive_expires_at) AND runs on the same DDL connection that holds the advisory lock, `findOrphanPages` filters soft-deleted pages on BOTH the candidate side AND the link-source side, `runThink` routes through `gateway.chat()` so the API key configured via `gbrain config set` actually reaches MCP stdio launches, `manageGitignore` distinguishes worktrees from submodules via Git's documented `/modules/` vs `/worktrees/` path segments, and walkers in `extract.ts` + `transcript-discovery.ts` consult a new `pruneDir()` helper at descent time so they don't recurse into vendor directories.
|
||||
|
||||
### What you can now do
|
||||
|
||||
**Upgrade gbrain cleanly on Supabase brains stuck at pre-v0.18, pre-v0.26.5, or pre-v0.34 schemas.** The bootstrap forward-reference class (eleven incidents in two years per the test file's own comment block) loses its largest remaining lineage. Pre-v0.18 brains had `files` without `source_id`/`page_id`, so the `idx_files_source_id` index creation crashed before any migration could run. Pre-v0.26.5 brains had `sources` without the archive lifecycle columns (`archived`, `archived_at`, `archive_expires_at`), but `CREATE TABLE IF NOT EXISTS sources` is a no-op on existing tables so the schema-blob replay couldn't add them — every downstream visibility filter in search/list_pages tripped. Pre-v0.34 brains had `oauth_clients` without the source-scoping columns (`source_id` FK, `federated_read` GIN-indexed array) so the v60+v61+v65 chain failed the same way. All seven columns now have explicit `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` probes in both engines. **And the probes now run on the DDL connection that holds the advisory lock** — pre-fix they ran through the instance pool while the lock sat on a different connection, opening a concurrent-bootstrap race for anyone on Supabase's transaction pooler. Codex caught that during pre-landing review.
|
||||
|
||||
**Trust orphan counts again.** `gbrain orphans` filtered nothing on `deleted_at` — soft-deleted pages (v0.26.5 soft-delete shipped without updating this query) appeared as orphans, inflating counts. Worse, links from soft-deleted source pages still counted as inbound, so a live page could hide from orphan results purely because a deleted page linked to it. The query now filters BOTH sides: `p.deleted_at IS NULL` on the candidate, `src.deleted_at IS NULL` on the inbound-link JOIN. The structural correctness rule for soft-delete (active relationships only) lands the same shape across both engines.
|
||||
|
||||
**`gbrain think` answers over MCP when you've set the key via `gbrain config`.** Before this release, MCP stdio launches (Claude Desktop, Cursor, etc.) returned `(no LLM available — set ANTHROPIC_API_KEY or pass client)` regardless of whether `gbrain config set anthropic_api_key sk-...` had been run, because `runThink` instantiated `new Anthropic()` directly and the Anthropic SDK only reads `process.env.ANTHROPIC_API_KEY`. Stdio launches from desktop apps don't inherit shell env, so the gbrain-config-set key never reached the SDK. `runThink` now routes through `gateway.chat()` — the canonical AI seam per CLAUDE.md that v0.31.12 established for chat/embed/expansion. The gateway reads `anthropic_api_key` from `~/.gbrain/config.json` AND from env, so both paths work. Existing `opts.client` injection is preserved (the test seam stays unchanged); `opts.stubResponse` continues to bypass the LLM call entirely. When neither the key nor a client is available, the graceful "no LLM available" stub still fires with the same `NO_ANTHROPIC_API_KEY` warning — the legacy diagnostic signal lives.
|
||||
|
||||
**Use gbrain inside Conductor worktrees without `.gitignore` management silently breaking.** `manageGitignore` (the storage-tiering helper that updates `.gitignore` to exclude `db_only:` paths) treated every `.git`-as-file as a submodule and skipped management. Both submodules AND worktrees use `.git` as a file (worktrees just have a different file content). The discriminator now matches the gitdir path segment: `/modules/<name>` = submodule, `/worktrees/<name>` = worktree. Worktrees are first-class repos and now get `.gitignore` management. Handles all four combinations of {relative, absolute} × {modules, worktrees} including the absorbed-submodule case from `git submodule absorbgitdirs` that the legacy absolute-vs-relative discriminator would have misclassified.
|
||||
|
||||
**Walkers stop wasting IO on `node_modules` (and frontmatter scans stop emitting false MISSING_OPEN warnings from inside vendor packages).** The canonical `isSyncable` filter had only a dot-prefix exclusion — `.git` and `.obsidian` were blocked but `node_modules` slipped through (no leading dot). A new `pruneDir()` helper centralizes the directory-exclusion logic at single-segment granularity: blocks `node_modules`, dot-prefix dirs, `ops`, and `*.raw` sidecars. Both walkers (`walkMarkdownFiles` in extract.ts, `listTextFiles` in transcript-discovery.ts) consult it at descent time BEFORE recursing, so the IO cost of walking thousands of vendor files is saved. `isSyncable` itself also gains node_modules exclusion — verified blast radius across every existing caller (sync, frontmatter, brain-writer, import) confirms node_modules exclusion was already the desired behavior. `transcript-discovery` also moves from non-recursive to recursive walking with the same pruneDir guard, so transcripts in nested corpus subdirectories (`corpus/2026/...`) become discoverable.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Closed issues
|
||||
- **#1018** PGLite upgrade wedge: applyForwardReferenceBootstrap missing v60 oauth_clients forward refs
|
||||
- **#974** Bootstrap gap: applyForwardReferenceBootstrap misses files.source_id + files.page_id
|
||||
- **#820** v0.13.0 migration files.page_id forward-reference cascade on pre-v0.18 brains
|
||||
- **#1021** orphans: soft-deleted pages still counted in orphan scan
|
||||
- **#952** think over MCP returns "no LLM available" even when ANTHROPIC_API_KEY configured via gbrain config
|
||||
- **#889** Git worktrees misclassified as submodules — breaks .gitignore management on Conductor workflows
|
||||
- **#923** gbrain frontmatter validate walks node_modules, inflating MISSING_OPEN counts in doctor
|
||||
- **#202** extract --source fs walker does not respect isSyncable prefix exclusions
|
||||
|
||||
#### Bootstrap structural fix
|
||||
- Seven new probes in `applyForwardReferenceBootstrap` (both engines): `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`.
|
||||
- Bootstrap now accepts the DDL connection from `initSchema` and runs probes inside the advisory-lock scope (Codex pre-landing review P1).
|
||||
- New MIGRATIONS-source introspection contract test in `test/schema-bootstrap-coverage.test.ts`: parses `src/core/migrate.ts` for every `ALTER TABLE ... ADD COLUMN`, asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies. Catches the column-only forward-reference class (sources.archived* shape) that the pre-existing CREATE INDEX parser couldn't see. Source-text introspection covers all three migration shapes: top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, and handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`.
|
||||
- Pre-existing parser bug in `parseBaseTableColumns` fixed: now strips SQL line comments (`-- ...`) and block comments (`/* ... */`) before identifying column names. Pre-fix, any column preceded by a comment line inside the CREATE TABLE body was silently dropped (e.g. `page_kind`, `deleted_at`, `emotional_weight`, `effective_date` — all dropped from coverage).
|
||||
|
||||
#### Walker correctness
|
||||
- New exported `pruneDir(name: string): boolean` helper in `src/core/sync.ts`. Blocks `node_modules`, dot-prefix directories, `ops`, and `*.raw` sidecars.
|
||||
- `isSyncable` now applies `pruneDir` per path segment (legacy dot-prefix + `.raw/` + `ops/` checks consolidated).
|
||||
- `walkMarkdownFiles` (extract.ts) consults `pruneDir` at descent + uses `isSyncable({strategy: 'markdown'})` at file emit. Pre-fix the walker had ad-hoc dot-prefix exclusion and didn't call isSyncable at all.
|
||||
- `listTextFiles` (transcript-discovery.ts) now recursive with `pruneDir`-guarded descent. Uses its own `.txt`/`.md` predicate rather than the markdown-strategy isSyncable that would have rejected `.txt` and applied markdown-only README/ops exclusions.
|
||||
|
||||
#### Soft-delete correctness
|
||||
- `findOrphanPages` (both engines) filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side.
|
||||
|
||||
#### MCP think gateway routing
|
||||
- `runThink` (`src/core/think/index.ts`) drops the direct `new Anthropic()` instantiation. New `tryBuildGatewayClient` probes recipe availability + ANTHROPIC_API_KEY presence (reads BOTH `process.env` and `loadConfig()`'s gbrain config). Returns null on miss, preserving the legacy graceful-degradation path.
|
||||
- New `chatResultToMessage` adapter converts gateway's `ChatResult` to an Anthropic-Message-shaped object. `mapStopReason` translates provider-neutral stop reasons (`end` / `length` / `tool_calls` / `refusal` / `content_filter` / `other`) to Anthropic's `stop_reason` enum.
|
||||
|
||||
#### Git worktree discriminator
|
||||
- `manageGitignore` (`src/commands/sync.ts`) reads the `.git` file content and matches the gitdir path segment: `/modules/<name>` → skip (submodule, parent-managed), `/worktrees/<name>` → MANAGE (worktree, first-class repo). Malformed `.git` file (no `gitdir:` prefix, IO error) defaults to MANAGE preserving pre-fix catch{} semantics.
|
||||
|
||||
#### Tests
|
||||
- `test/helpers/extract-added-columns.ts` (NEW): MIGRATIONS source introspection helper with internal test seam for the regex.
|
||||
- `test/schema-bootstrap-coverage.test.ts`: extended `REQUIRED_BOOTSTRAP_COVERAGE` with 7 new entries + 4 new test cases (MIGRATIONS contract, sanity check, regex shape variants, planted-bug regression).
|
||||
- `test/orphans.test.ts`: 3 new cases pinning the both-sides soft-delete filter — soft-deleted candidate excluded, codex C11 link-source case, live-link smoke regression.
|
||||
- `test/think-gateway-adapter.test.ts` (NEW): 9 cases covering response-shape conversion, stop-reason mapping, model-id normalization (bare + prefixed), unknown-provider fallback, no-API-key fallback, hasAnthropicKey env read.
|
||||
- `test/storage-sync.test.ts`: 5 new cases for worktree/submodule discrimination (relative + absolute × modules + worktrees, malformed `.git` file).
|
||||
- `test/sync.test.ts`: 7 new isSyncable + 6 new pruneDir cases including the node_modules CRITICAL regression.
|
||||
|
||||
#### Critical files
|
||||
- `src/core/{pglite,postgres}-engine.ts` — `applyForwardReferenceBootstrap` (extended; Postgres engine threads DDL connection); `findOrphanPages` (both sides filter).
|
||||
- `src/core/think/index.ts` — gateway adapter + ThinkLLMClient backward compat.
|
||||
- `src/core/sync.ts` — new `pruneDir` export; `isSyncable` per-segment pruning.
|
||||
- `src/commands/{extract,sync}.ts` — walker pruneDir adoption; worktree discriminator.
|
||||
- `src/core/cycle/transcript-discovery.ts` — recursive walker with pruneDir.
|
||||
- `test/helpers/extract-added-columns.ts` — NEW.
|
||||
- `test/schema-bootstrap-coverage.test.ts` — extended.
|
||||
|
||||
#### Follow-ups filed
|
||||
- `TODOS.md`: runThink full rewrite to drop ThinkLLMClient indirection now that the gateway adapter is in place. Supabase parity test fixture for the bootstrap connection-threading (the bug is fixed; the test fixture proving it under real pooler topology is the residual).
|
||||
|
||||
## To take advantage of v0.35.5.0
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't, or if `gbrain doctor` warns about a partial migration on a pre-v0.18 / pre-v0.26.5 / pre-v0.34 brain:
|
||||
|
||||
1. **Run the orchestrator manually:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
2. **Verify the bootstrap landed:**
|
||||
```bash
|
||||
gbrain doctor --json | grep -E 'schema_version|bootstrap'
|
||||
```
|
||||
3. **Confirm the new columns exist on Supabase brains** (the wave's primary target):
|
||||
```bash
|
||||
psql "$DATABASE_URL" -c "\d sources" | grep -E 'archived|archive_expires_at'
|
||||
psql "$DATABASE_URL" -c "\d files" | grep -E 'source_id|page_id'
|
||||
psql "$DATABASE_URL" -c "\d oauth_clients" | grep -E 'source_id|federated_read'
|
||||
```
|
||||
4. **If `gbrain think` was returning "no LLM available" over MCP** despite a configured key, no manual action needed — the gateway adapter activates as soon as v0.35.5.0 is installed.
|
||||
5. **If any step fails**, file an issue with the output of `gbrain doctor` and contents of `~/.gbrain/upgrade-errors.jsonl` at https://github.com/garrytan/gbrain/issues.
|
||||
|
||||
## [0.35.4.0] - 2026-05-16
|
||||
|
||||
**Doctor stops crying wolf on clean worker restarts. Entity resolver stops spawning phantom pages. Prefix-expansion gets 58x faster.**
|
||||
|
||||
@@ -43,9 +43,9 @@ strict behavior when unset.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise<Array<{slug, source_id}>>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity.
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity.
|
||||
- `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。!?`), `CJK_CLAUSE_DELIMITERS` (`;:,、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`.
|
||||
- `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger.
|
||||
- `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers.
|
||||
@@ -56,7 +56,7 @@ strict behavior when unset.
|
||||
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). **v0.35.5.0:** new exported `pruneDir(name: string): boolean` helper is the single source of truth for descent-time directory exclusion across walkers. Blocks `node_modules` (no leading dot, so pre-v0.35.5 walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars. `isSyncable` now applies it per path segment; `walkMarkdownFiles` in `src/commands/extract.ts` and `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing so the IO cost of walking thousands of vendor files is saved. Closes #923 + #202. `manageGitignore` worktree fix in same wave: discriminator now matches the gitdir path segment (`/modules/<name>` = submodule, `/worktrees/<name>` = worktree, per Git's documented layout) instead of the legacy absolute-vs-relative check that misclassified absorbed submodules and worktrees both. Conductor worktrees are first-class repos and now get `.gitignore` management for storage-tiering. Closes #889. v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
|
||||
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
|
||||
@@ -85,6 +85,8 @@ strict behavior when unset.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` (v0.32.6) — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned per Codex; UTF-8-safe truncation; C1 confidence-floor double-enforcement; resolution_kind output drives M7 paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (Codex outside-voice fix — prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — Codex fix to bias from silent skip), M5 trend writes to `eval_contradictions_runs`, M6 source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker — stable cache hit-rate across re-runs). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (P1 batched), `writeContradictionsRun` + `loadContradictionsTrend` (M5), `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` (P2). Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). M1 doctor check surfaces high-severity findings with paste-ready resolution commands. M2 synthesize phase pre-fetches latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. 226 hermetic unit tests + 12 real-Postgres E2E. Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`. Architecture doc: `docs/contradictions.md`.
|
||||
- `src/core/think/index.ts` (v0.35.5.0 — gateway adapter) — `runThink` no longer instantiates `new Anthropic()` directly. The internal `LLMClient` instance is now built by a small adapter that wraps `gateway.chat()` from `src/core/ai/gateway.ts`, the canonical AI seam v0.31.12 established for chat/embed/expansion. Closes #952: stdio MCP launches (Claude Desktop, Cursor) don't inherit shell env, so the Anthropic SDK's env-only key resolution lost the key any user had set via `gbrain config set anthropic_api_key`. The gateway reads from `~/.gbrain/config.json` AND from env, so both paths work. Test seam preserved: `opts.client?: ThinkLLMClient` injection still works for the 12+ existing tests (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`, etc.); `opts.stubResponse` continues to short-circuit before any LLM call. When neither key nor client is available, the graceful "no LLM available" stub still fires with the same `NO_ANTHROPIC_API_KEY` warning. v0.36.x TODO: drop `ThinkLLMClient` indirection entirely, migrate tests to `__setChatTransportForTests` seam from `src/core/ai/gateway.ts`.
|
||||
- `src/core/operations.ts` extension (v0.35.5.0 orphans fix) — `findOrphanPages` (both engines) now filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side. Pre-v0.35.5 the query filtered nothing on `deleted_at`, so soft-deleted pages (v0.26.5 soft-delete shipped without updating this query) appeared as orphans AND links from soft-deleted source pages still suppressed live pages from orphan results. Closes #1021. Pinned by `test/orphans.test.ts`'s soft-delete cases.
|
||||
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1) — `gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `<chat_session id="..." date="...">` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec).
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
@@ -571,7 +573,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion),
|
||||
`test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage),
|
||||
`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.),
|
||||
`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented. **v0.35.5.0:** test now also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`), and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies. Catches the column-only forward-reference class (e.g. `sources.archived` shape from v0.26.5, `oauth_clients.source_id` from v0.34.1) that the pre-existing CREATE INDEX parser couldn't see. Pre-existing parser bug fixed in same wave: `parseBaseTableColumns` now strips SQL line + block comments before identifying column names so commented-out lines no longer hide adjacent columns from coverage.),
|
||||
`test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` (v0.26.6 #588 — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each (bootstrap + schema replay + migrations), snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via PGLITE_SCHEMA_SQL or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skip-gracefully without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.),
|
||||
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
|
||||
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
|
||||
|
||||
@@ -1,6 +1,13 @@
|
||||
# TODOS
|
||||
|
||||
|
||||
## dreamy-thompson wave follow-ups (v0.36.x)
|
||||
|
||||
- [ ] **v0.36.x: runThink full rewrite — drop ThinkLLMClient indirection.** v0.36's fix(think) wave landed a gateway-backed adapter at `src/core/think/index.ts:225-251` so `gbrain config set anthropic_api_key` works over MCP stdio (closed #952). The adapter routes through `gateway.chat()` but `runThink` still carries the `ThinkLLMClient` interface as the test seam — it's the last LLM-using path that doesn't use the canonical `__setChatTransportForTests` seam v0.31.12 established for chat/embed. Cleanup: drop `ThinkLLMClient`, drop the `opts.client` injection point, migrate the 12+ existing tests (`test/think-pipeline.serial.test.ts:144,181,222`, `test/think-gateway-adapter.test.ts`, plus 9+ others that stub the interface) to `__setChatTransportForTests`. Pros: codebase consistency, one fewer test-stub pattern, easier to add provider switching for think once it routes through gateway natively. Cons: 12+ test files need migration. Blocked by: v0.36 wave landing on master (so the adapter exists to lean on while migrating tests). Plan reference: D5 + D7 in `~/.claude/plans/ok-i-spun-up-dreamy-thompson.md`.
|
||||
|
||||
- [ ] **v0.36.x: Supabase parity test fixture for `applyForwardReferenceBootstrap`.** v0.36 fixed the underlying bug (bootstrap now uses the DDL connection from `initSchema` so probes run inside the advisory-lock scope) per codex P1 from /ship adversarial review. What remains is the TEST FIXTURE that proves it: the new pre-v18/pre-v34/pre-v60 E2E tests run against local Docker Postgres but not against Supabase-shape pooler topology (transaction pooler + statement_timeout). Real Supabase upgrades have failed multiple times on this exact connection-topology divergence (#699, #820 lineage). Fix: a test fixture that exercises the probe path against deriveDirectUrl + transaction pooler + statement_timeout. Cons: requires Supabase fixture infra OR careful mocking of the connection-selection logic in `db.ts`'s `getDDLConnection` path.
|
||||
|
||||
|
||||
## kinshasa-v3 follow-ups (v0.35.4.0)
|
||||
|
||||
- [ ] **v0.36.x: Fix `supervisor-audit.ts:77` `readSupervisorEvents` to use the dual-week-aware pattern from `stub-guard-audit.ts:readRecentStubGuardEvents`.** The supervisor reader only reads the current ISO-week file, so a 24h sliding window across Monday 00:00 UTC silently loses Sunday's events (they're in last week's file). The new stub-guard reader in v0.35.4.0 fixes this for its own audit log by reading BOTH current and previous week files before timestamp-filtering — the supervisor reader should adopt the same shape. Pin with a unit test that uses a fake-clock fixture set to "Monday 00:01 UTC" with a Sunday 23:55 event in the prior file. Filed during v0.35.4.0 kinshasa-v3 codex outside-voice review.
|
||||
|
||||
+6
-4
@@ -151,9 +151,9 @@ strict behavior when unset.
|
||||
- `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `<prefix>/*` matches recursive children; bare `<prefix>` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents/<id>/...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path.
|
||||
- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise<Array<{slug, source_id}>>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`)
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains.
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820.
|
||||
- `src/core/pglite-schema.ts` — PGLite-specific DDL (pgvector, pg_trgm, triggers)
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity.
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity.
|
||||
- `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。!?`), `CJK_CLAUSE_DELIMITERS` (`;:,、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`.
|
||||
- `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger.
|
||||
- `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers.
|
||||
@@ -164,7 +164,7 @@ strict behavior when unset.
|
||||
- `src/core/db.ts` — Connection management, schema initialization. v0.22.1 (#363, contributed by @orendi84): `resolveSessionTimeouts()` returns `statement_timeout` + `idle_in_transaction_session_timeout` (defaults: 5min each, env-overridable via `GBRAIN_STATEMENT_TIMEOUT` / `GBRAIN_IDLE_TX_TIMEOUT` / `GBRAIN_CLIENT_CHECK_INTERVAL`). Both `connect()` (module singleton) and `PostgresEngine.connect()` (worker pool) consume the result via postgres.js's `connection` option, sending GUCs as startup parameters that survive PgBouncer transaction mode (unlike the prior `setSessionDefaults` post-pool SET, kept as a back-compat no-op shim).
|
||||
- `src/commands/migrate-engine.ts` — Bidirectional engine migration (`gbrain migrate --to supabase/pglite`)
|
||||
- `src/core/import-file.ts` — importFromFile + importFromContent (chunk + embed + tags)
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
|
||||
- `src/core/sync.ts` — Pure sync functions (manifest parsing, filtering, slug conversion). **v0.35.5.0:** new exported `pruneDir(name: string): boolean` helper is the single source of truth for descent-time directory exclusion across walkers. Blocks `node_modules` (no leading dot, so pre-v0.35.5 walkers slipped through and inflated MISSING_OPEN counts via vendor packages), dot-prefix dirs, `ops/`, and `*.raw` sidecars. `isSyncable` now applies it per path segment; `walkMarkdownFiles` in `src/commands/extract.ts` and `listTextFiles` in `src/core/cycle/transcript-discovery.ts` consult it BEFORE recursing so the IO cost of walking thousands of vendor files is saved. Closes #923 + #202. `manageGitignore` worktree fix in same wave: discriminator now matches the gitdir path segment (`/modules/<name>` = submodule, `/worktrees/<name>` = worktree, per Git's documented layout) instead of the legacy absolute-vs-relative check that misclassified absorbed submodules and worktrees both. Conductor worktrees are first-class repos and now get `.gitignore` management for storage-tiering. Closes #889. v0.22.12 (#500, foundation by @wintermute via #501): `classifyErrorCode(errorMsg)` regex-based classifier with 12 codes (`SLUG_MISMATCH`, `YAML_PARSE`, `YAML_DUPLICATE_KEY`, `MISSING_OPEN`, `MISSING_CLOSE`, `NESTED_QUOTES`, `EMPTY_FRONTMATTER`, `NULL_BYTES`, `INVALID_UTF8`, `STATEMENT_TIMEOUT`, `FILE_TOO_LARGE`, `SYMLINK_NOT_ALLOWED`) plus `UNKNOWN` fallback. `summarizeFailuresByCode(failures)` returns sorted `[{code, count}]`. `code?` optional field on `SyncFailure`; backfilled at ack time on pre-v0.22.12 entries. `acknowledgeSyncFailures()` returns `AcknowledgeResult { count, summary }`. Three regexes (`MISSING_OPEN`, `MISSING_CLOSE`, `EMPTY_FRONTMATTER`) broadened to match actual `markdown.ts:159-244` validator message strings, not just the literal code-name prefix. `FILE_TOO_LARGE` covers all three production size sites in `import-file.ts:199, 352, 401`; `SYMLINK_NOT_ALLOWED` covers the rejection at `:347`. Closes the silent-skip pattern that motivated #500.
|
||||
- `src/core/storage.ts` — Pluggable storage interface (S3, Supabase Storage, local)
|
||||
- `src/core/storage-config.ts` (v0.22.11) — Storage tiering: `loadStorageConfig` reads `gbrain.yml`, normalizes deprecated keys (`git_tracked` / `supabase_only`) to canonical (`db_tracked` / `db_only`) with once-per-process deprecation warning, and runs `normalizeAndValidateStorageConfig` (auto-fixes missing trailing `/`, throws `StorageConfigError` on tier overlap). Path-segment matcher: `media/x/` does NOT match `media/xerox/foo`. Replaces gray-matter (broken on delimiter-less YAML) with a dedicated parser for the `gbrain.yml` shape.
|
||||
- `src/core/disk-walk.ts` (v0.22.11) — `walkBrainRepo(repoPath)` returns `Map<slug, {size, mtimeMs}>` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens).
|
||||
@@ -193,6 +193,8 @@ strict behavior when unset.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` (v0.32.6) — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned per Codex; UTF-8-safe truncation; C1 confidence-floor double-enforcement; resolution_kind output drives M7 paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (Codex outside-voice fix — prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — Codex fix to bias from silent skip), M5 trend writes to `eval_contradictions_runs`, M6 source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker — stable cache hit-rate across re-runs). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (P1 batched), `writeContradictionsRun` + `loadContradictionsTrend` (M5), `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` (P2). Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). M1 doctor check surfaces high-severity findings with paste-ready resolution commands. M2 synthesize phase pre-fetches latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. 226 hermetic unit tests + 12 real-Postgres E2E. Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`. Architecture doc: `docs/contradictions.md`.
|
||||
- `src/core/think/index.ts` (v0.35.5.0 — gateway adapter) — `runThink` no longer instantiates `new Anthropic()` directly. The internal `LLMClient` instance is now built by a small adapter that wraps `gateway.chat()` from `src/core/ai/gateway.ts`, the canonical AI seam v0.31.12 established for chat/embed/expansion. Closes #952: stdio MCP launches (Claude Desktop, Cursor) don't inherit shell env, so the Anthropic SDK's env-only key resolution lost the key any user had set via `gbrain config set anthropic_api_key`. The gateway reads from `~/.gbrain/config.json` AND from env, so both paths work. Test seam preserved: `opts.client?: ThinkLLMClient` injection still works for the 12+ existing tests (`test/think-pipeline.serial.test.ts`, `test/think-gateway-adapter.test.ts`, etc.); `opts.stubResponse` continues to short-circuit before any LLM call. When neither key nor client is available, the graceful "no LLM available" stub still fires with the same `NO_ANTHROPIC_API_KEY` warning. v0.36.x TODO: drop `ThinkLLMClient` indirection entirely, migrate tests to `__setChatTransportForTests` seam from `src/core/ai/gateway.ts`.
|
||||
- `src/core/operations.ts` extension (v0.35.5.0 orphans fix) — `findOrphanPages` (both engines) now filters `p.deleted_at IS NULL` on the candidate side AND adds `JOIN pages src ON src.id = l.from_page_id WHERE src.deleted_at IS NULL` to the EXISTS subquery on the link-source side. Pre-v0.35.5 the query filtered nothing on `deleted_at`, so soft-deleted pages (v0.26.5 soft-delete shipped without updating this query) appeared as orphans AND links from soft-deleted source pages still suppressed live pages from orphan results. Closes #1021. Pinned by `test/orphans.test.ts`'s soft-delete cases.
|
||||
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1) — `gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `<chat_session id="..." date="...">` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec).
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
@@ -679,7 +681,7 @@ parity), `test/cli.test.ts` (CLI structure), `test/config.test.ts` (config redac
|
||||
`test/file-migration.test.ts` (file migration), `test/file-resolver.test.ts` (file resolution),
|
||||
`test/import-resume.test.ts` (import checkpoints), `test/migrate.test.ts` (migration; v8/v9 helper-btree-index SQL structural assertions + 1000-row wall-clock fixtures that guard the O(n²)→O(n log n) fix + v0.13.1 assertions on v12/v13 SQL shape, `sqlFor` + `transaction:false` runner semantics, the `max_stalled DEFAULT 1` regression guard, and v0.22.6.1 v24 `sqlFor.pglite: ''` no-op assertion),
|
||||
`test/bootstrap.test.ts` (v0.22.6.1 — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on simulated pre-v0.18 brain, fresh-install regression guard, pre-v0.13 `links` shape coverage),
|
||||
`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented.),
|
||||
`test/schema-bootstrap-coverage.test.ts` (v0.22.6.1 CI guard — `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in PGLITE_SCHEMA_SQL; the test fails loudly if `applyForwardReferenceBootstrap` skips one. When you add a column-with-index to the embedded schema blob, you extend both arrays or this guard fails. The pattern that broke gbrain ten times in two years is now structurally prevented. **v0.35.5.0:** test now also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`), and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies. Catches the column-only forward-reference class (e.g. `sources.archived` shape from v0.26.5, `oauth_clients.source_id` from v0.34.1) that the pre-existing CREATE INDEX parser couldn't see. Pre-existing parser bug fixed in same wave: `parseBaseTableColumns` now strips SQL line + block comments before identifying column names so commented-out lines no longer hide adjacent columns from coverage.),
|
||||
`test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` (v0.26.6 #588 — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each (bootstrap + schema replay + migrations), snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via PGLITE_SCHEMA_SQL or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skip-gracefully without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.),
|
||||
`test/setup-branching.test.ts` (setup flow), `test/slug-validation.test.ts` (slug validation),
|
||||
`test/storage.test.ts` (storage backends), `test/supabase-admin.test.ts` (Supabase admin),
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.35.4.0",
|
||||
"version": "0.35.5.0",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
+14
-4
@@ -28,7 +28,7 @@ import {
|
||||
} from '../core/link-extraction.ts';
|
||||
import { createProgress } from '../core/progress.ts';
|
||||
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
|
||||
import { pathToSlug } from '../core/sync.ts';
|
||||
import { pathToSlug, pruneDir, isSyncable } from '../core/sync.ts';
|
||||
|
||||
// Batch size for addLinksBatch / addTimelineEntriesBatch.
|
||||
// Postgres bind-parameter limit is 65535. Links use 4 cols/row → 16K hard ceiling;
|
||||
@@ -63,16 +63,26 @@ interface ExtractResult {
|
||||
// --- Shared walker ---
|
||||
|
||||
export function walkMarkdownFiles(dir: string): { path: string; relPath: string }[] {
|
||||
// Descent-time pruning + emit-time isSyncable filter (closes #923, #202).
|
||||
// Pre-fix, this walker had only an ad-hoc dot-prefix exclusion and didn't
|
||||
// call isSyncable at all — so it descended into `node_modules/`, emitted
|
||||
// markdown files from there, AND ignored the canonical exclusion list
|
||||
// (`.raw/`, `ops/`, README.md, etc.). Now: pruneDir skips entire vendor
|
||||
// subtrees before recursion (saving IO), and isSyncable filters the emit
|
||||
// set against the canonical markdown-strategy rules.
|
||||
const files: { path: string; relPath: string }[] = [];
|
||||
function walk(d: string) {
|
||||
for (const entry of readdirSync(d)) {
|
||||
if (entry.startsWith('.')) continue;
|
||||
const full = join(d, entry);
|
||||
try {
|
||||
if (lstatSync(full).isDirectory()) {
|
||||
const st = lstatSync(full);
|
||||
if (st.isDirectory()) {
|
||||
if (!pruneDir(entry)) continue;
|
||||
walk(full);
|
||||
} else if (entry.endsWith('.md') && !entry.startsWith('_')) {
|
||||
files.push({ path: full, relPath: relative(dir, full) });
|
||||
const rel = relative(dir, full);
|
||||
if (!isSyncable(rel, { strategy: 'markdown' })) continue;
|
||||
files.push({ path: full, relPath: rel });
|
||||
}
|
||||
} catch { /* skip unreadable */ }
|
||||
}
|
||||
|
||||
+25
-7
@@ -1387,17 +1387,35 @@ export function manageGitignore(
|
||||
return;
|
||||
}
|
||||
|
||||
// D49: submodule detection. In a submodule, `.git` is a regular file
|
||||
// (containing `gitdir: ../path/to/parent.git/modules/x`), not a directory.
|
||||
// Submodule + worktree detection (closes #889 misclassification).
|
||||
// Both submodules and worktrees use `.git` as a FILE (not a directory), so
|
||||
// statSync.isFile() doesn't discriminate. Discriminator is the gitdir path
|
||||
// segment:
|
||||
// - submodule: gitdir contains `/modules/<name>` (skip — managed by parent)
|
||||
// - worktree: gitdir contains `/worktrees/<name>` (MANAGE — first-class repo)
|
||||
// Both contracts are documented Git internal layouts and stable across all 4
|
||||
// {relative, absolute} × {modules, worktrees} combinations, including the
|
||||
// absorbed-submodule case from `git submodule absorbgitdirs`.
|
||||
// Malformed `.git` file (no `gitdir:` prefix, unreadable) → MANAGE (fail-closed
|
||||
// toward managing, preserving the pre-#889 catch{} behavior).
|
||||
const dotGit = join(repoPath, '.git');
|
||||
if (existsSync(dotGit)) {
|
||||
try {
|
||||
if (statSync(dotGit).isFile()) {
|
||||
console.warn(
|
||||
`Note: skipping .gitignore management — ${repoPath} is a git submodule. ` +
|
||||
`Add db_only directories to your parent repo's .gitignore manually.`,
|
||||
);
|
||||
return;
|
||||
const content = readFileSync(dotGit, 'utf-8');
|
||||
const match = content.match(/gitdir:\s*(.+)/);
|
||||
const gitdir = match ? match[1].trim() : '';
|
||||
if (gitdir.includes('/modules/')) {
|
||||
console.warn(
|
||||
`Note: skipping .gitignore management — ${repoPath} is a git submodule. ` +
|
||||
`Add db_only directories to your parent repo's .gitignore manually.`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
// Worktree (gitdir contains /worktrees/) OR malformed .git falls through
|
||||
// to the existing manage path. Worktrees are first-class repos — they
|
||||
// need .gitignore management too. Malformed → MANAGE preserves the
|
||||
// pre-#889 fail-closed-toward-managing catch behavior.
|
||||
}
|
||||
} catch {
|
||||
// proceed; can't tell, default to managing
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
import { readFileSync, readdirSync, statSync } from 'node:fs';
|
||||
import { join, basename } from 'node:path';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { pruneDir } from '../sync.ts';
|
||||
|
||||
export interface DiscoveredTranscript {
|
||||
/** Absolute path to the transcript file. */
|
||||
@@ -119,22 +120,38 @@ function matchesAnyExclude(text: string, patterns: RegExp[]): boolean {
|
||||
}
|
||||
|
||||
function listTextFiles(dir: string): string[] {
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(dir);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
// Recursive walk with descent-time pruning (closes codex C12/C13 spec gap).
|
||||
// Accepts BOTH .txt and .md per transcript-discovery's domain rules — does
|
||||
// NOT use isSyncable({strategy:'markdown'}) because that predicate rejects
|
||||
// .txt and applies markdown-only README/ops exclusions transcripts don't share.
|
||||
//
|
||||
// pruneDir at descent time skips node_modules / .git / .obsidian / .raw /
|
||||
// .cache / ops / etc. before recursion — saves the IO cost of walking
|
||||
// vendor subtrees.
|
||||
const out: string[] = [];
|
||||
for (const name of entries) {
|
||||
if (!name.endsWith('.txt') && !name.endsWith('.md')) continue;
|
||||
const full = join(dir, name);
|
||||
function walk(d: string) {
|
||||
let entries: string[];
|
||||
try {
|
||||
if (statSync(full).isFile()) out.push(full);
|
||||
entries = readdirSync(d);
|
||||
} catch {
|
||||
// skip unreadable entries
|
||||
return;
|
||||
}
|
||||
for (const name of entries) {
|
||||
const full = join(d, name);
|
||||
try {
|
||||
const st = statSync(full);
|
||||
if (st.isDirectory()) {
|
||||
if (!pruneDir(name)) continue;
|
||||
walk(full);
|
||||
} else if (st.isFile() && (name.endsWith('.txt') || name.endsWith('.md'))) {
|
||||
out.push(full);
|
||||
}
|
||||
} catch {
|
||||
// skip unreadable entries
|
||||
}
|
||||
}
|
||||
}
|
||||
walk(dir);
|
||||
return out.sort();
|
||||
}
|
||||
|
||||
|
||||
+124
-12
@@ -288,7 +288,27 @@ export class PGLiteEngine implements BrainEngine {
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema='public' AND table_name='ingest_log') AS ingest_log_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='ingest_log' AND column_name='source_id') AS ingest_log_source_id_exists
|
||||
WHERE table_schema='public' AND table_name='ingest_log' AND column_name='source_id') AS ingest_log_source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema='public' AND table_name='files') AS files_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='files' AND column_name='source_id') AS files_source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='files' AND column_name='page_id') AS files_page_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema='public' AND table_name='oauth_clients') AS oauth_clients_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='oauth_clients' AND column_name='source_id') AS oauth_clients_source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='oauth_clients' AND column_name='federated_read') AS oauth_clients_federated_read_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema='public' AND table_name='sources') AS sources_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='sources' AND column_name='archived') AS sources_archived_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='sources' AND column_name='archived_at') AS sources_archived_at_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema='public' AND table_name='sources' AND column_name='archive_expires_at') AS sources_archive_expires_at_exists
|
||||
`);
|
||||
const probe = rows[0] as {
|
||||
pages_exists: boolean;
|
||||
@@ -309,6 +329,16 @@ export class PGLiteEngine implements BrainEngine {
|
||||
subagent_provider_id_exists: boolean;
|
||||
ingest_log_exists: boolean;
|
||||
ingest_log_source_id_exists: boolean;
|
||||
files_exists: boolean;
|
||||
files_source_id_exists: boolean;
|
||||
files_page_id_exists: boolean;
|
||||
oauth_clients_exists: boolean;
|
||||
oauth_clients_source_id_exists: boolean;
|
||||
oauth_clients_federated_read_exists: boolean;
|
||||
sources_exists: boolean;
|
||||
sources_archived_exists: boolean;
|
||||
sources_archived_at_exists: boolean;
|
||||
sources_archive_expires_at_exists: boolean;
|
||||
};
|
||||
|
||||
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
|
||||
@@ -332,27 +362,55 @@ export class PGLiteEngine implements BrainEngine {
|
||||
// references source_id. Old brains have ingest_log without source_id;
|
||||
// bootstrap adds the column before SCHEMA_SQL replay creates the index.
|
||||
const needsIngestLogSourceId = probe.ingest_log_exists && !probe.ingest_log_source_id_exists;
|
||||
// v0.18 (v18): files.source_id + files.page_id added; idx_files_source_id
|
||||
// and idx_files_page_id in PGLITE_SCHEMA_SQL crash without them.
|
||||
const needsFilesBootstrap = probe.files_exists
|
||||
&& (!probe.files_source_id_exists || !probe.files_page_id_exists);
|
||||
// v0.34.1 (v60+v61+v65): oauth_clients.source_id + federated_read added;
|
||||
// FK to sources(id) + GIN index idx_oauth_clients_federated_read in
|
||||
// PGLITE_SCHEMA_SQL crash without them.
|
||||
const needsOauthClientsBootstrap = probe.oauth_clients_exists
|
||||
&& (!probe.oauth_clients_source_id_exists || !probe.oauth_clients_federated_read_exists);
|
||||
// v0.26.5 (v34): sources.archived + archived_at + archive_expires_at added
|
||||
// for soft-delete lifecycle. Not directly referenced by indexes BUT
|
||||
// PGLITE_SCHEMA_SQL's `CREATE TABLE IF NOT EXISTS sources` is a no-op on
|
||||
// pre-existing sources tables (won't add columns), so visibility filters
|
||||
// referencing these columns trip on old brains. The bootstrap closes the
|
||||
// gap before any visibility-filter SQL runs.
|
||||
const needsSourcesArchive = probe.sources_exists
|
||||
&& (!probe.sources_archived_exists
|
||||
|| !probe.sources_archived_at_exists
|
||||
|| !probe.sources_archive_expires_at_exists);
|
||||
|
||||
// Fresh installs (no tables yet) and modern brains both no-op.
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap
|
||||
&& !needsPagesDeletedAt && !needsChunksEmbeddingImage
|
||||
&& !needsMcpLogBootstrap && !needsSubagentProviderId
|
||||
&& !needsPagesRecency && !needsIngestLogSourceId) return;
|
||||
&& !needsPagesRecency && !needsIngestLogSourceId
|
||||
&& !needsFilesBootstrap && !needsOauthClientsBootstrap
|
||||
&& !needsSourcesArchive) return;
|
||||
|
||||
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
|
||||
|
||||
if (needsPagesBootstrap) {
|
||||
// Mirror schema-embedded.ts shape for `sources` so the subsequent
|
||||
// PGLITE_SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op.
|
||||
// Archive columns (v34) are folded in here so a pre-v18 brain doesn't
|
||||
// need needsSourcesArchive to also fire — bootstrap creates a complete
|
||||
// v34-shape sources in one go. needsSourcesArchive then only fires on
|
||||
// the pre-v34 case (sources exists, archive cols don't).
|
||||
await this.db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
archived BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
archived_at TIMESTAMPTZ,
|
||||
archive_expires_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
INSERT INTO sources (id, name, config)
|
||||
VALUES ('default', 'default', '{"federated": true}'::jsonb)
|
||||
@@ -465,6 +523,50 @@ export class PGLiteEngine implements BrainEngine {
|
||||
ALTER TABLE ingest_log ADD COLUMN IF NOT EXISTS source_id TEXT NOT NULL DEFAULT 'default';
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsFilesBootstrap) {
|
||||
// v18 (files_provenance_columns) adds source_id + page_id to files plus
|
||||
// idx_files_source_id and idx_files_page_id in PGLITE_SCHEMA_SQL. Pre-v18
|
||||
// brains crash on the CREATE INDEX. Bootstrap adds both columns; v18
|
||||
// runs later via runMigrations and is idempotent.
|
||||
await this.db.exec(`
|
||||
ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id TEXT
|
||||
NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE;
|
||||
ALTER TABLE files ADD COLUMN IF NOT EXISTS page_id INTEGER
|
||||
REFERENCES pages(id) ON DELETE SET NULL;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsOauthClientsBootstrap) {
|
||||
// v60+v61+v65 (oauth_clients_source_id_fk, oauth_clients_federated_read_column,
|
||||
// oauth_clients_federated_read_gin_index) add source_id + federated_read
|
||||
// and the GIN index idx_oauth_clients_federated_read. PGLITE_SCHEMA_SQL's
|
||||
// FK + index references crash on pre-v60 brains. Bootstrap mirrors the
|
||||
// v60+v61 column shape; v60-v65 run later via runMigrations and are
|
||||
// idempotent (and handle backfill + RESTRICT-flip).
|
||||
await this.db.exec(`
|
||||
ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS source_id TEXT
|
||||
DEFAULT 'default' REFERENCES sources(id) ON DELETE SET NULL;
|
||||
ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS federated_read TEXT[]
|
||||
NOT NULL DEFAULT '{}';
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsSourcesArchive) {
|
||||
// v34 (destructive_guard_columns) promotes archive lifecycle from JSONB
|
||||
// config to real columns on sources. PGLITE_SCHEMA_SQL's
|
||||
// `CREATE TABLE IF NOT EXISTS sources` is a no-op against an existing
|
||||
// pre-v34 sources table, so the column-add never lands until the v34
|
||||
// migration runs. v34's UPDATE statements + downstream visibility filters
|
||||
// (search/query/list_pages) need the columns to exist on the table
|
||||
// schema. Bootstrap adds the three columns; v34 runs later via
|
||||
// runMigrations and is idempotent (and handles JSONB → column backfill).
|
||||
await this.db.exec(`
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN NOT NULL DEFAULT FALSE;
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived_at TIMESTAMPTZ;
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archive_expires_at TIMESTAMPTZ;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
|
||||
@@ -1882,15 +1984,25 @@ export class PGLiteEngine implements BrainEngine {
|
||||
}
|
||||
|
||||
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||
// Soft-delete filter on BOTH sides:
|
||||
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
||||
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
||||
// Without the link-source filter, a live page can hide from orphan results purely
|
||||
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT
|
||||
p.slug,
|
||||
COALESCE(p.title, p.slug) AS title,
|
||||
p.frontmatter->>'domain' AS domain
|
||||
FROM pages p
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM links l WHERE l.to_page_id = p.id
|
||||
)
|
||||
WHERE p.deleted_at IS NULL
|
||||
AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM links l
|
||||
JOIN pages src ON src.id = l.from_page_id
|
||||
WHERE l.to_page_id = p.id
|
||||
AND src.deleted_at IS NULL
|
||||
)
|
||||
ORDER BY p.slug`
|
||||
);
|
||||
return rows as Array<{ slug: string; title: string; domain: string | null }>;
|
||||
|
||||
+135
-15
@@ -232,7 +232,12 @@ export class PostgresEngine implements BrainEngine {
|
||||
// Pre-schema bootstrap: add forward-referenced state the embedded schema
|
||||
// blob requires but that older brains don't have yet (issues #366/#375/
|
||||
// #378/#396 + #266/#357). Idempotent on fresh installs and modern brains.
|
||||
await this.applyForwardReferenceBootstrap();
|
||||
// Threads the DDL connection (same one holding the advisory lock above)
|
||||
// so bootstrap probes run on the locked connection — without this, the
|
||||
// probes ran through `this.sql` (the pooler/instance pool) outside the
|
||||
// lock, opening a concurrent-bootstrap race for Supabase users on the
|
||||
// transaction pooler. Codex P1 finding from v0.36 dreamy-thompson wave.
|
||||
await this.applyForwardReferenceBootstrap(conn);
|
||||
|
||||
await conn.unsafe(sqlText);
|
||||
|
||||
@@ -294,8 +299,13 @@ export class PostgresEngine implements BrainEngine {
|
||||
* `test/schema-bootstrap-coverage.test.ts` (PGLite side) and
|
||||
* `test/e2e/postgres-bootstrap.test.ts` (Postgres side).
|
||||
*/
|
||||
private async applyForwardReferenceBootstrap(): Promise<void> {
|
||||
const conn = this.sql;
|
||||
private async applyForwardReferenceBootstrap(injectedConn?: postgres.Sql): Promise<void> {
|
||||
// Use the caller-provided connection (DDL pool, holding the advisory lock
|
||||
// from initSchema) when available — falls back to this.sql for backward
|
||||
// compatibility with any unit-test path that still calls bootstrap directly.
|
||||
// Production path always passes the DDL conn so bootstrap probes run inside
|
||||
// the same lock scope as SCHEMA_SQL replay.
|
||||
const conn = injectedConn ?? this.sql;
|
||||
|
||||
// Single round-trip probe for every forward-reference target.
|
||||
// current_schema() resolves to whatever search_path the connection uses,
|
||||
@@ -319,6 +329,16 @@ export class PostgresEngine implements BrainEngine {
|
||||
subagent_provider_id_exists: boolean;
|
||||
ingest_log_exists: boolean;
|
||||
ingest_log_source_id_exists: boolean;
|
||||
files_exists: boolean;
|
||||
files_source_id_exists: boolean;
|
||||
files_page_id_exists: boolean;
|
||||
oauth_clients_exists: boolean;
|
||||
oauth_clients_source_id_exists: boolean;
|
||||
oauth_clients_federated_read_exists: boolean;
|
||||
sources_exists: boolean;
|
||||
sources_archived_exists: boolean;
|
||||
sources_archived_at_exists: boolean;
|
||||
sources_archive_expires_at_exists: boolean;
|
||||
}[]>`
|
||||
SELECT
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
@@ -356,7 +376,27 @@ export class PostgresEngine implements BrainEngine {
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = current_schema() AND table_name = 'ingest_log') AS ingest_log_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'ingest_log' AND column_name = 'source_id') AS ingest_log_source_id_exists
|
||||
WHERE table_schema = current_schema() AND table_name = 'ingest_log' AND column_name = 'source_id') AS ingest_log_source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = current_schema() AND table_name = 'files') AS files_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'files' AND column_name = 'source_id') AS files_source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'files' AND column_name = 'page_id') AS files_page_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = current_schema() AND table_name = 'oauth_clients') AS oauth_clients_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'oauth_clients' AND column_name = 'source_id') AS oauth_clients_source_id_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'oauth_clients' AND column_name = 'federated_read') AS oauth_clients_federated_read_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.tables
|
||||
WHERE table_schema = current_schema() AND table_name = 'sources') AS sources_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'archived') AS sources_archived_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'archived_at') AS sources_archived_at_exists,
|
||||
EXISTS (SELECT 1 FROM information_schema.columns
|
||||
WHERE table_schema = current_schema() AND table_name = 'sources' AND column_name = 'archive_expires_at') AS sources_archive_expires_at_exists
|
||||
`;
|
||||
const probe = probeRows[0]!;
|
||||
|
||||
@@ -386,26 +426,52 @@ export class PostgresEngine implements BrainEngine {
|
||||
// source_id. Old brains have ingest_log without source_id; bootstrap adds
|
||||
// the column before SCHEMA_SQL replay creates the index.
|
||||
const needsIngestLogSourceId = probe.ingest_log_exists && !probe.ingest_log_source_id_exists;
|
||||
// v0.18 (v18): files.source_id + files.page_id added; idx_files_source_id
|
||||
// and idx_files_page_id in SCHEMA_SQL crash without them.
|
||||
const needsFilesBootstrap = probe.files_exists
|
||||
&& (!probe.files_source_id_exists || !probe.files_page_id_exists);
|
||||
// v0.34.1 (v60+v61+v65): oauth_clients.source_id + federated_read added;
|
||||
// FK to sources(id) + GIN index idx_oauth_clients_federated_read in
|
||||
// SCHEMA_SQL crash without them.
|
||||
const needsOauthClientsBootstrap = probe.oauth_clients_exists
|
||||
&& (!probe.oauth_clients_source_id_exists || !probe.oauth_clients_federated_read_exists);
|
||||
// v0.26.5 (v34): sources.archived + archived_at + archive_expires_at added
|
||||
// for soft-delete lifecycle. SCHEMA_SQL's `CREATE TABLE IF NOT EXISTS sources`
|
||||
// is a no-op on pre-existing sources tables (won't add columns), so the
|
||||
// visibility filters in search/list_pages trip on old brains. Bootstrap
|
||||
// closes the gap before any visibility-filter SQL runs.
|
||||
const needsSourcesArchive = probe.sources_exists
|
||||
&& (!probe.sources_archived_exists
|
||||
|| !probe.sources_archived_at_exists
|
||||
|| !probe.sources_archive_expires_at_exists);
|
||||
|
||||
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap
|
||||
&& !needsPagesDeletedAt && !needsMcpLogBootstrap && !needsSubagentProviderId
|
||||
&& !needsChunksEmbeddingImage && !needsPagesRecency
|
||||
&& !needsIngestLogSourceId) return;
|
||||
&& !needsIngestLogSourceId && !needsFilesBootstrap
|
||||
&& !needsOauthClientsBootstrap && !needsSourcesArchive) return;
|
||||
|
||||
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
|
||||
|
||||
if (needsPagesBootstrap) {
|
||||
// Mirror schema-embedded.ts's `sources` shape so the subsequent
|
||||
// SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op.
|
||||
// Archive columns (v34) are folded in here so a pre-v18 brain doesn't
|
||||
// need needsSourcesArchive to also fire — bootstrap creates a complete
|
||||
// v34-shape sources in one go. needsSourcesArchive then only fires on
|
||||
// the pre-v34 case (sources exists, archive cols don't).
|
||||
await conn.unsafe(`
|
||||
CREATE TABLE IF NOT EXISTS sources (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
local_path TEXT,
|
||||
last_commit TEXT,
|
||||
last_sync_at TIMESTAMPTZ,
|
||||
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
archived BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
archived_at TIMESTAMPTZ,
|
||||
archive_expires_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
INSERT INTO sources (id, name, config)
|
||||
VALUES ('default', 'default', '{"federated": true}'::jsonb)
|
||||
@@ -518,6 +584,50 @@ export class PostgresEngine implements BrainEngine {
|
||||
ALTER TABLE ingest_log ADD COLUMN IF NOT EXISTS source_id TEXT NOT NULL DEFAULT 'default';
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsFilesBootstrap) {
|
||||
// v18 (files_provenance_columns) adds source_id + page_id to files plus
|
||||
// idx_files_source_id and idx_files_page_id in SCHEMA_SQL. Pre-v18 brains
|
||||
// crash on the CREATE INDEX. Bootstrap adds both columns; v18 runs later
|
||||
// via runMigrations and is idempotent.
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id TEXT
|
||||
NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE;
|
||||
ALTER TABLE files ADD COLUMN IF NOT EXISTS page_id INTEGER
|
||||
REFERENCES pages(id) ON DELETE SET NULL;
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsOauthClientsBootstrap) {
|
||||
// v60+v61+v65 (oauth_clients_source_id_fk, oauth_clients_federated_read_column,
|
||||
// oauth_clients_federated_read_gin_index) add source_id + federated_read
|
||||
// and the GIN index idx_oauth_clients_federated_read. SCHEMA_SQL's
|
||||
// FK + index references crash on pre-v60 brains. Bootstrap mirrors the
|
||||
// v60+v61 column shape; v60-v65 run later via runMigrations and are
|
||||
// idempotent (and handle backfill + the v64 RESTRICT-flip).
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS source_id TEXT
|
||||
DEFAULT 'default' REFERENCES sources(id) ON DELETE SET NULL;
|
||||
ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS federated_read TEXT[]
|
||||
NOT NULL DEFAULT '{}';
|
||||
`);
|
||||
}
|
||||
|
||||
if (needsSourcesArchive) {
|
||||
// v34 (destructive_guard_columns) promotes archive lifecycle from JSONB
|
||||
// config to real columns on sources. SCHEMA_SQL's `CREATE TABLE IF NOT EXISTS
|
||||
// sources` is a no-op against an existing pre-v34 sources table, so the
|
||||
// column-add never lands until the v34 migration runs. v34's UPDATE
|
||||
// statements + downstream visibility filters (search/query/list_pages)
|
||||
// need the columns to exist on the table schema. Bootstrap adds the
|
||||
// three columns; v34 runs later via runMigrations and is idempotent
|
||||
// (and handles JSONB → column backfill).
|
||||
await conn.unsafe(`
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN NOT NULL DEFAULT FALSE;
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived_at TIMESTAMPTZ;
|
||||
ALTER TABLE sources ADD COLUMN IF NOT EXISTS archive_expires_at TIMESTAMPTZ;
|
||||
`);
|
||||
}
|
||||
}
|
||||
|
||||
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
||||
@@ -1901,15 +2011,25 @@ export class PostgresEngine implements BrainEngine {
|
||||
|
||||
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
||||
const sql = this.sql;
|
||||
// Soft-delete filter on BOTH sides:
|
||||
// - candidate: p.deleted_at IS NULL — soft-deleted pages aren't orphan candidates
|
||||
// - link source: src.deleted_at IS NULL — links FROM soft-deleted pages don't count as inbound
|
||||
// Without the link-source filter, a live page can hide from orphan results purely
|
||||
// because a soft-deleted page links to it. v0.26.5 invariant; codex C11.
|
||||
const rows = await sql`
|
||||
SELECT
|
||||
p.slug,
|
||||
COALESCE(p.title, p.slug) AS title,
|
||||
p.frontmatter->>'domain' AS domain
|
||||
FROM pages p
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM links l WHERE l.to_page_id = p.id
|
||||
)
|
||||
WHERE p.deleted_at IS NULL
|
||||
AND NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM links l
|
||||
JOIN pages src ON src.id = l.from_page_id
|
||||
WHERE l.to_page_id = p.id
|
||||
AND src.deleted_at IS NULL
|
||||
)
|
||||
ORDER BY p.slug
|
||||
`;
|
||||
return rows as unknown as Array<{ slug: string; title: string; domain: string | null }>;
|
||||
|
||||
+47
-9
@@ -208,6 +208,47 @@ function matchesAnyGlob(path: string, patterns?: string[]): boolean {
|
||||
return patterns.some((pattern) => globToRegex(pattern).test(normalized));
|
||||
}
|
||||
|
||||
/**
|
||||
* Directory names that walkers must NEVER descend into. Used at descent
|
||||
* time (before recursion) to prune entire subtrees — saves the IO cost of
|
||||
* walking thousands of vendor / generated / hidden files only to filter
|
||||
* them at file-emit time. Used by every walker in gbrain (sync, extract,
|
||||
* transcript-discovery, etc.).
|
||||
*
|
||||
* Pattern: dirname matching at single path-segment granularity. Walkers
|
||||
* call `pruneDir(entry.name)` on each subdirectory before recursing.
|
||||
*
|
||||
* `node_modules` lacks a leading dot so the dot-prefix exclusion in
|
||||
* isSyncable below doesn't catch it; explicit entry here closes the
|
||||
* latent walker bug (#923, #202).
|
||||
*/
|
||||
const PRUNE_DIR_NAMES = new Set<string>([
|
||||
'node_modules',
|
||||
'.raw',
|
||||
'ops',
|
||||
]);
|
||||
|
||||
/**
|
||||
* Should this directory be descended into? Returns `false` for vendor / hidden /
|
||||
* generated dirs that walkers should skip BEFORE recursing. Catches
|
||||
* `node_modules` (latent bug — no leading dot), dot-prefix dirs (`.git`,
|
||||
* `.obsidian`, `.raw`, `.cache`, etc. via the leading-dot heuristic), and the
|
||||
* explicit `PRUNE_DIR_NAMES` set above.
|
||||
*
|
||||
* `name` is a single path segment (basename of the directory entry), NOT a
|
||||
* full path. Walkers consult this on each subdirectory entry during recursion.
|
||||
*/
|
||||
export function pruneDir(name: string): boolean {
|
||||
if (!name) return true;
|
||||
if (name.startsWith('.')) return false;
|
||||
if (PRUNE_DIR_NAMES.has(name)) return false;
|
||||
// `.raw` is the literal directory name; `*.raw` is the gbrain sidecar
|
||||
// convention (e.g. `people/pedro.raw/` holds raw source for pedro.md).
|
||||
// Both forms should be skipped at descent time.
|
||||
if (name.endsWith('.raw')) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Filter a file path to determine if it should be synced to GBrain.
|
||||
* Strategy-aware: 'markdown' (default) = .md/.mdx only, 'code' = code files only, 'auto' = both.
|
||||
@@ -217,20 +258,17 @@ export function isSyncable(path: string, opts: SyncableOptions = {}): boolean {
|
||||
|
||||
if (!isAllowedByStrategy(path, strategy)) return false;
|
||||
|
||||
// Skip hidden directories
|
||||
if (path.split('/').some(p => p.startsWith('.'))) return false;
|
||||
|
||||
// Skip .raw/ sidecar directories
|
||||
if (path.includes('.raw/')) return false;
|
||||
// Skip every path segment that pruneDir would block walkers from descending
|
||||
// into. Catches hidden dirs (`.git`, `.obsidian`), `.raw/` sidecars,
|
||||
// `node_modules/` (latent bug fix), and `ops/` at any depth.
|
||||
const segments = path.split('/');
|
||||
if (segments.some(p => !pruneDir(p))) return false;
|
||||
|
||||
// Skip meta files that aren't pages
|
||||
const skipFiles = ['schema.md', 'index.md', 'log.md', 'README.md'];
|
||||
const basename = path.split('/').pop() || '';
|
||||
const basename = segments[segments.length - 1] || '';
|
||||
if (skipFiles.includes(basename)) return false;
|
||||
|
||||
// Skip ops/ directory
|
||||
if (path.startsWith('ops/')) return false;
|
||||
|
||||
if (opts.include && opts.include.length > 0 && !matchesAnyGlob(path, opts.include)) return false;
|
||||
if (opts.exclude && opts.exclude.length > 0 && matchesAnyGlob(path, opts.exclude)) return false;
|
||||
|
||||
|
||||
+201
-7
@@ -17,13 +17,17 @@
|
||||
* for those flags per Codex P1 #7.
|
||||
*/
|
||||
|
||||
import Anthropic from '@anthropic-ai/sdk';
|
||||
import type Anthropic from '@anthropic-ai/sdk';
|
||||
import type { BrainEngine, SynthesisEvidenceInput } from '../engine.ts';
|
||||
import { runGather, renderPagesBlock, takesHitToTakeForPrompt } from './gather.ts';
|
||||
import { renderTakesBlock } from './sanitize.ts';
|
||||
import { buildThinkSystemPrompt, buildThinkUserMessage } from './prompt.ts';
|
||||
import { resolveCitations, type ParsedCitation } from './cite-render.ts';
|
||||
import { resolveModel } from '../model-config.ts';
|
||||
import { chat as gatewayChat, type ChatResult } from '../ai/gateway.ts';
|
||||
import { resolveRecipe } from '../ai/model-resolver.ts';
|
||||
import { AIConfigError } from '../ai/errors.ts';
|
||||
import { loadConfig } from '../config.ts';
|
||||
|
||||
/** Anthropic Messages client interface — same shape used by subagent.ts so test stubs can be shared. */
|
||||
export interface ThinkLLMClient {
|
||||
@@ -222,7 +226,20 @@ export async function runThink(
|
||||
if (opts.stubResponse) {
|
||||
response = opts.stubResponse;
|
||||
} else {
|
||||
if (!opts.client && !process.env.ANTHROPIC_API_KEY) {
|
||||
// Build a ThinkLLMClient. Three sources, in priority order:
|
||||
// 1. opts.client (test injection — preserved as test seam)
|
||||
// 2. Gateway adapter (routes through gateway.chat() — picks up
|
||||
// anthropic_api_key from gbrain config OR env, gateway rate-leases,
|
||||
// retry, prompt caching, the canonical seam per CLAUDE.md)
|
||||
// 3. Graceful fallback ("no LLM available" stub) — when gateway is
|
||||
// unconfigured AND no env var is set, return without throwing.
|
||||
//
|
||||
// Pre-v0.36, this code path constructed `new Anthropic()` directly.
|
||||
// That bypassed gateway config (gbrain config set anthropic_api_key)
|
||||
// because the Anthropic SDK only reads process.env.ANTHROPIC_API_KEY.
|
||||
// Closes #952 (think over MCP returns "no LLM available").
|
||||
const client = opts.client ?? await tryBuildGatewayClient(modelUsed);
|
||||
if (!client) {
|
||||
warnings.push('NO_ANTHROPIC_API_KEY');
|
||||
// Degrade gracefully: return the gather without synthesis. Better than throwing.
|
||||
return {
|
||||
@@ -244,11 +261,6 @@ export async function runThink(
|
||||
},
|
||||
};
|
||||
}
|
||||
// Anthropic SDK exposes the create method via .messages — match the structural signature.
|
||||
const realClient = new Anthropic();
|
||||
const client: ThinkLLMClient = opts.client ?? {
|
||||
create: (params, opts2) => realClient.messages.create(params, opts2),
|
||||
};
|
||||
const result = await client.create({
|
||||
model: modelUsed,
|
||||
max_tokens: DEFAULT_MAX_OUTPUT_TOKENS,
|
||||
@@ -347,3 +359,185 @@ export async function persistSynthesis(
|
||||
const persisted = await persistCitations(engine, page.id, result.citations);
|
||||
return { slug, evidenceInserted: persisted.inserted, warnings: persisted.warnings };
|
||||
}
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Gateway adapter for #952 (think over MCP returns "no LLM available").
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Pre-v0.36, runThink instantiated `new Anthropic()` directly and read
|
||||
// ANTHROPIC_API_KEY from process.env. Claude Desktop's stdio MCP launch
|
||||
// doesn't inherit shell env, so `gbrain config set anthropic_api_key sk-...`
|
||||
// (which writes to ~/.gbrain/config.json) never reached the SDK and every
|
||||
// MCP think call degraded to "no LLM available."
|
||||
//
|
||||
// The adapter routes through gateway.chat() — the canonical seam per
|
||||
// CLAUDE.md. Gateway reads the API key from gbrain config OR env, picks
|
||||
// up prompt caching, rate-leases, retry, and the test seam
|
||||
// (__setChatTransportForTests) that v0.31.12 already established.
|
||||
//
|
||||
// Per plan-eng-review D10 (cross-model tension with codex C7+C8+C9+C10),
|
||||
// the adapter implements four fixes:
|
||||
// 1. Drop the new Anthropic() direct path entirely — always route through gateway
|
||||
// 2. Real availability check via try/catch around resolveRecipe + assertion
|
||||
// (NOT the false-positive `getChatModel()` truthy check)
|
||||
// 3. Model-id resolution: handle both bare (`claude-opus-4-7`) and
|
||||
// provider-prefixed (`anthropic:claude-opus-4-7`) shapes
|
||||
// 4. Response-shape conversion: ChatResult → Anthropic.Message
|
||||
//
|
||||
// `opts.client` injection path is preserved (test seam — see ThinkLLMClient).
|
||||
// `opts.stubResponse` path is preserved (pure-test escape).
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Try to build a gateway-backed ThinkLLMClient for the given model.
|
||||
* Returns null when the gateway cannot resolve a usable chat provider for
|
||||
* this model (missing API key for the resolved provider, unknown provider,
|
||||
* touchpoint not supported, etc.). Caller falls through to the graceful
|
||||
* "no LLM available" stub on null.
|
||||
*/
|
||||
async function tryBuildGatewayClient(modelUsed: string): Promise<ThinkLLMClient | null> {
|
||||
// Normalize: ensure provider:model shape. resolveModel returns bare
|
||||
// anthropic ids (e.g. `claude-opus-4-7`); gateway.chat needs `anthropic:...`.
|
||||
const modelStr = modelUsed.includes(':') ? modelUsed : `anthropic:${modelUsed}`;
|
||||
|
||||
// Availability probe: resolveRecipe throws on unknown provider; assertTouchpoint
|
||||
// throws if the resolved recipe doesn't support chat. Both are AIConfigError.
|
||||
let providerId: string;
|
||||
try {
|
||||
const { parsed } = resolveRecipe(modelStr);
|
||||
providerId = parsed.providerId;
|
||||
} catch (e) {
|
||||
if (e instanceof AIConfigError) return null;
|
||||
throw e;
|
||||
}
|
||||
|
||||
// API-key availability probe. The gateway lazily checks keys inside
|
||||
// instantiateChat at first .chat() call and throws AIConfigError on miss.
|
||||
// Pre-checking here preserves the legacy "NO_ANTHROPIC_API_KEY" warning
|
||||
// signal AND avoids paying for a wasted gateway call when the user clearly
|
||||
// has no key configured. Reads BOTH the gbrain config file (`anthropic_api_key`
|
||||
// set via `gbrain config set`) AND the process env, matching gateway's
|
||||
// own loadConfig precedence.
|
||||
if (providerId === 'anthropic' && !hasAnthropicKey()) return null;
|
||||
|
||||
return {
|
||||
create: async (params): Promise<Anthropic.Message> => {
|
||||
// Build ChatOpts from Anthropic.MessageCreateParamsNonStreaming.
|
||||
const messages = params.messages.map(m => ({
|
||||
role: m.role,
|
||||
content: typeof m.content === 'string'
|
||||
? m.content
|
||||
: (Array.isArray(m.content) ? m.content.map(b => 'text' in b ? b.text : '').join('') : ''),
|
||||
}));
|
||||
const system = typeof params.system === 'string'
|
||||
? params.system
|
||||
: (Array.isArray(params.system) ? params.system.map(b => 'text' in b ? b.text : '').join('') : undefined);
|
||||
|
||||
let result: ChatResult;
|
||||
try {
|
||||
result = await gatewayChat({
|
||||
model: modelStr,
|
||||
system,
|
||||
messages,
|
||||
maxTokens: params.max_tokens,
|
||||
});
|
||||
} catch (e) {
|
||||
// AIConfigError at chat time = missing API key for resolved provider.
|
||||
// Surface as a sentinel "no LLM available"-shaped Message so the
|
||||
// existing JSON-parse path produces the graceful degradation answer.
|
||||
if (e instanceof AIConfigError) {
|
||||
return buildGracefulMessage(modelStr) as unknown as Anthropic.Message;
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
return chatResultToMessage(result, modelStr) as unknown as Anthropic.Message;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert gateway's `ChatResult` into an Anthropic-Message-shaped object.
|
||||
* The caller (`runThink`) parses `result.content[0].text` as JSON; the
|
||||
* other fields (usage, stop_reason) are returned with best-effort mapping
|
||||
* for downstream telemetry compat.
|
||||
*/
|
||||
function chatResultToMessage(result: ChatResult, modelStr: string): {
|
||||
id: string;
|
||||
type: 'message';
|
||||
role: 'assistant';
|
||||
model: string;
|
||||
content: Array<{ type: 'text'; text: string }>;
|
||||
usage: { input_tokens: number; output_tokens: number };
|
||||
stop_reason: 'end_turn' | 'max_tokens' | 'tool_use' | 'stop_sequence';
|
||||
} {
|
||||
return {
|
||||
id: '',
|
||||
type: 'message',
|
||||
role: 'assistant',
|
||||
model: modelStr,
|
||||
content: [{ type: 'text', text: result.text }],
|
||||
usage: {
|
||||
input_tokens: result.usage.input_tokens,
|
||||
output_tokens: result.usage.output_tokens,
|
||||
},
|
||||
stop_reason: mapStopReason(result.stopReason),
|
||||
};
|
||||
}
|
||||
|
||||
function hasAnthropicKey(): boolean {
|
||||
if (process.env.ANTHROPIC_API_KEY) return true;
|
||||
try {
|
||||
const cfg = loadConfig();
|
||||
if (cfg?.anthropic_api_key) return true;
|
||||
} catch {
|
||||
// loadConfig may throw on first-run installs; treat as no key available.
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function mapStopReason(s: ChatResult['stopReason']): 'end_turn' | 'max_tokens' | 'tool_use' | 'stop_sequence' {
|
||||
switch (s) {
|
||||
case 'end': return 'end_turn';
|
||||
case 'length': return 'max_tokens';
|
||||
case 'tool_calls': return 'tool_use';
|
||||
// 'refusal', 'content_filter', 'other' → end_turn (no Anthropic equivalent)
|
||||
default: return 'end_turn';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sentinel Message returned when gateway.chat throws AIConfigError (typically
|
||||
* missing API key for the resolved provider). The caller's JSON parser will
|
||||
* fail on this text, fall through to `LLM_OUTPUT_NOT_JSON`, and surface the
|
||||
* sentinel as the answer — matches the legacy graceful-degradation shape.
|
||||
*/
|
||||
function buildGracefulMessage(modelStr: string): {
|
||||
id: string;
|
||||
type: 'message';
|
||||
role: 'assistant';
|
||||
model: string;
|
||||
content: Array<{ type: 'text'; text: string }>;
|
||||
usage: { input_tokens: number; output_tokens: number };
|
||||
stop_reason: 'end_turn';
|
||||
} {
|
||||
return {
|
||||
id: '',
|
||||
type: 'message',
|
||||
role: 'assistant',
|
||||
model: modelStr,
|
||||
content: [{ type: 'text', text: '(no LLM available — set anthropic_api_key via gbrain config or ANTHROPIC_API_KEY env)' }],
|
||||
usage: { input_tokens: 0, output_tokens: 0 },
|
||||
stop_reason: 'end_turn',
|
||||
};
|
||||
}
|
||||
|
||||
// Test-only exports for the adapter helpers. The functions live at module
|
||||
// scope (not inside runThink) so they can be unit-tested directly. Naming
|
||||
// follows the `__` prefix convention already established by
|
||||
// `__setChatTransportForTests` in gateway.ts.
|
||||
export const __thinkAdapter = {
|
||||
tryBuildGatewayClient,
|
||||
chatResultToMessage,
|
||||
mapStopReason,
|
||||
buildGracefulMessage,
|
||||
hasAnthropicKey,
|
||||
};
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* MIGRATIONS-source introspection for the bootstrap CI guard.
|
||||
*
|
||||
* The v0.28.5 SQL-parser at the bottom of `test/schema-bootstrap-coverage.test.ts`
|
||||
* walks every `CREATE INDEX` in `PGLITE_SCHEMA_SQL` to enforce that indexed
|
||||
* columns either live in the table's CREATE TABLE body OR are added by
|
||||
* `applyForwardReferenceBootstrap`. That structural check kills the
|
||||
* column-with-index forward-reference class.
|
||||
*
|
||||
* It does NOT kill the column-ONLY forward-reference class. v0.26.5 (v34)
|
||||
* added `sources.archived` + `sources.archived_at` + `sources.archive_expires_at`
|
||||
* for the soft-delete lifecycle. Those columns aren't indexed, but
|
||||
* `CREATE TABLE IF NOT EXISTS sources` is a no-op on a pre-v34 brain, so the
|
||||
* archive columns never land on the schema when the schema blob replays —
|
||||
* downstream visibility filters in search / list_pages trip immediately.
|
||||
*
|
||||
* This helper closes the gap by walking the source text of
|
||||
* `src/core/migrate.ts` and extracting every `ALTER TABLE ... ADD COLUMN`
|
||||
* the file contains, regardless of whether it lives in a migration's `sql`
|
||||
* field, `sqlFor.{postgres,pglite}` field, or inside a `handler` function
|
||||
* body that calls `engine.runMigration(N, \`...\`)`. The test in
|
||||
* `schema-bootstrap-coverage.test.ts` then asserts every such (table, column)
|
||||
* pair is also covered by the bootstrap.
|
||||
*
|
||||
* Why source-file introspection (vs walking the MIGRATIONS object array):
|
||||
* v34's `destructive_guard_columns` migration uses a `handler` function that
|
||||
* embeds SQL inside `engine.runMigration(34, \`ALTER TABLE sources ADD COLUMN ...\`)`.
|
||||
* The handler body isn't reachable via Migration.sql / Migration.sqlFor. Reading
|
||||
* the source text catches ALL three shapes uniformly: top-level sql, sqlFor
|
||||
* overrides, and handler-embedded `engine.runMigration` calls.
|
||||
*
|
||||
* Why regex-on-our-own-source is safe (vs the DDL-parser-of-prod-schema trap
|
||||
* codex flagged in plan review): the input is migrate.ts — our own code, with
|
||||
* consistent shape. ALTER TABLE ADD COLUMN is unambiguous. Future contributors
|
||||
* who add columns follow the same shape (or fail this test and learn).
|
||||
*/
|
||||
|
||||
import { readFileSync } from 'fs';
|
||||
import { resolve } from 'path';
|
||||
|
||||
export interface AddedColumnRef {
|
||||
table: string;
|
||||
column: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract every `ALTER TABLE <table> ADD COLUMN <col>` reference from a SQL
|
||||
* string. Handles:
|
||||
* - `ALTER TABLE [IF EXISTS] [ONLY] <table> ADD COLUMN [IF NOT EXISTS] <col>`
|
||||
* - Quoted identifiers (`"col"`, `\`col\``)
|
||||
* - Multi-statement strings with mixed ALTER + CREATE + UPDATE
|
||||
*/
|
||||
function extractAlterAddColumnsFromSql(sql: string): Array<{ table: string; column: string }> {
|
||||
const result: Array<{ table: string; column: string }> = [];
|
||||
// Identifier shape: optional quote, word chars, optional matching quote.
|
||||
// Handles bare `pages`, double-quoted `"pages"`, and backtick `\`pages\``.
|
||||
const re = /ALTER\s+TABLE\s+(?:IF\s+EXISTS\s+)?(?:ONLY\s+)?["`]?(\w+)["`]?\s+ADD\s+COLUMN\s+(?:IF\s+NOT\s+EXISTS\s+)?["`]?(\w+)["`]?/gi;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = re.exec(sql)) !== null) {
|
||||
result.push({ table: m[1].toLowerCase(), column: m[2].toLowerCase() });
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* Read `src/core/migrate.ts` and emit every (table, column) pair that ANY
|
||||
* `ALTER TABLE ... ADD COLUMN` reference in the file targets. Deduped:
|
||||
* later occurrences of the same (table, column) are dropped.
|
||||
*
|
||||
* Source-file introspection covers all three migration shapes:
|
||||
* - top-level `sql:` field (e.g. v22 ingest_log_source_id)
|
||||
* - `sqlFor.postgres` / `sqlFor.pglite` overrides
|
||||
* - handler-body `engine.runMigration(N, \`ALTER TABLE ... ADD COLUMN ...\`)`
|
||||
* (e.g. v34 destructive_guard_columns)
|
||||
*
|
||||
* The function intentionally does NOT track which migration each column
|
||||
* came from. That mapping requires parsing the surrounding migration object
|
||||
* structure, which adds fragility for marginal value (the failure message
|
||||
* names the table.column; the contributor can grep for it in migrate.ts).
|
||||
*/
|
||||
export function extractAddedColumnsFromMigrations(): AddedColumnRef[] {
|
||||
const migratePath = resolve(process.cwd(), 'src/core/migrate.ts');
|
||||
const source = readFileSync(migratePath, 'utf-8');
|
||||
|
||||
const seen = new Set<string>();
|
||||
const result: AddedColumnRef[] = [];
|
||||
for (const ref of extractAlterAddColumnsFromSql(source)) {
|
||||
const key = `${ref.table}.${ref.column}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
result.push(ref);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
// Test-internal exports so the parser itself can be unit-tested.
|
||||
export const __internal = { extractAlterAddColumnsFromSql };
|
||||
@@ -289,4 +289,84 @@ describe('findOrphans (engine-injected)', () => {
|
||||
expect(result.total_orphans).toBe(0);
|
||||
expect(result.total_pages).toBe(0);
|
||||
});
|
||||
|
||||
// ────────────────────────────────────────────────────────────────
|
||||
// Soft-delete filtering on BOTH sides (v0.26.5 invariant; codex C11)
|
||||
// ────────────────────────────────────────────────────────────────
|
||||
|
||||
test('REGRESSION: soft-deleted page with no inbound is NOT in orphan results', async () => {
|
||||
// Candidate-filter regression. Pre-fix, findOrphanPages returned every
|
||||
// page without inbound links — including soft-deleted ones. Now the
|
||||
// outer query filters p.deleted_at IS NULL.
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice has no inbound links and is soft-deleted.',
|
||||
timeline: '',
|
||||
});
|
||||
// Soft-delete alice directly via the DB handle (no engine method exposed).
|
||||
await (engine as any).db.query(
|
||||
`UPDATE pages SET deleted_at = now() WHERE slug = 'people/alice'`
|
||||
);
|
||||
|
||||
const rows = await queryOrphanPages(engine);
|
||||
const slugs = rows.map(r => r.slug);
|
||||
expect(slugs).not.toContain('people/alice');
|
||||
});
|
||||
|
||||
test('REGRESSION: live page with ONLY inbound link from soft-deleted source IS orphan (codex C11)', async () => {
|
||||
// Link-source-filter regression. Pre-fix, a live page that had ONE
|
||||
// inbound link from a soft-deleted source was hidden from orphan
|
||||
// results because the EXISTS check didn't filter the source side.
|
||||
// Now the inner JOIN filters src.deleted_at IS NULL too.
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice was soft-deleted but used to link to Bob.',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('people/bob', {
|
||||
type: 'person',
|
||||
title: 'Bob',
|
||||
compiled_truth: 'Bob has no live inbound links.',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.addLink('people/alice', 'people/bob', 'mentioned', 'references', 'markdown');
|
||||
|
||||
// Soft-delete alice. Bob's ONLY inbound link is now from a deleted page.
|
||||
await (engine as any).db.query(
|
||||
`UPDATE pages SET deleted_at = now() WHERE slug = 'people/alice'`
|
||||
);
|
||||
|
||||
const rows = await queryOrphanPages(engine);
|
||||
const slugs = rows.map(r => r.slug).sort();
|
||||
// alice is soft-deleted → not in results (candidate filter).
|
||||
// bob has no LIVE inbound link → IS in results (link-source filter — codex C11).
|
||||
expect(slugs).not.toContain('people/alice');
|
||||
expect(slugs).toContain('people/bob');
|
||||
});
|
||||
|
||||
test('live page with inbound link from LIVE source is NOT orphan (regression for unchanged behavior)', async () => {
|
||||
// Sanity check: the soft-delete filter must NOT break the basic
|
||||
// "live link counts" case.
|
||||
await engine.putPage('people/alice', {
|
||||
type: 'person',
|
||||
title: 'Alice',
|
||||
compiled_truth: 'Alice is live and links to Bob.',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.putPage('people/bob', {
|
||||
type: 'person',
|
||||
title: 'Bob',
|
||||
compiled_truth: 'Bob has a live inbound from Alice.',
|
||||
timeline: '',
|
||||
});
|
||||
await engine.addLink('people/alice', 'people/bob', 'mentioned', 'references', 'markdown');
|
||||
|
||||
const rows = await queryOrphanPages(engine);
|
||||
const slugs = rows.map(r => r.slug);
|
||||
// alice is orphan (no inbound), bob is NOT (alice links to him).
|
||||
expect(slugs).toContain('people/alice');
|
||||
expect(slugs).not.toContain('people/bob');
|
||||
});
|
||||
});
|
||||
|
||||
@@ -108,6 +108,27 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [
|
||||
// created_at DESC)`. Old brains have ingest_log without source_id; bootstrap
|
||||
// adds the column before SCHEMA_SQL replay creates the index.
|
||||
{ kind: 'column', table: 'ingest_log', column: 'source_id' },
|
||||
// v0.18 (v18) — forward-referenced by `CREATE INDEX idx_files_source_id ON
|
||||
// files(source_id)` and `CREATE INDEX idx_files_page_id ON files(page_id)`.
|
||||
// Pre-v18 brains have files without these columns; bootstrap adds them
|
||||
// before SCHEMA_SQL replay creates the indexes.
|
||||
{ kind: 'column', table: 'files', column: 'source_id' },
|
||||
{ kind: 'column', table: 'files', column: 'page_id' },
|
||||
// v0.34.1 (v60+v61+v65) — forward-referenced by the FK
|
||||
// `oauth_clients.source_id REFERENCES sources(id)` and the GIN index
|
||||
// `idx_oauth_clients_federated_read ON oauth_clients USING GIN (federated_read)`.
|
||||
// Pre-v60 brains have oauth_clients without these columns; bootstrap adds
|
||||
// them before SCHEMA_SQL replay creates the FK + index.
|
||||
{ kind: 'column', table: 'oauth_clients', column: 'source_id' },
|
||||
{ kind: 'column', table: 'oauth_clients', column: 'federated_read' },
|
||||
// v0.26.5 (v34) — promotes archive lifecycle from JSONB config to real
|
||||
// columns on sources. CREATE TABLE IF NOT EXISTS is a no-op on existing
|
||||
// sources tables, so the visibility filters in search/list_pages that
|
||||
// reference these columns trip on pre-v34 brains. Bootstrap adds them
|
||||
// before any visibility-filter SQL runs.
|
||||
{ kind: 'column', table: 'sources', column: 'archived' },
|
||||
{ kind: 'column', table: 'sources', column: 'archived_at' },
|
||||
{ kind: 'column', table: 'sources', column: 'archive_expires_at' },
|
||||
];
|
||||
|
||||
test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => {
|
||||
@@ -168,8 +189,26 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS import_filename;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS salience_touched_at;
|
||||
ALTER TABLE pages DROP COLUMN IF EXISTS emotional_weight;
|
||||
|
||||
DROP INDEX IF EXISTS idx_ingest_log_source_type_created;
|
||||
ALTER TABLE ingest_log DROP COLUMN IF EXISTS source_id;
|
||||
|
||||
DROP INDEX IF EXISTS idx_files_source_id;
|
||||
DROP INDEX IF EXISTS idx_files_page_id;
|
||||
ALTER TABLE files DROP COLUMN IF EXISTS source_id;
|
||||
ALTER TABLE files DROP COLUMN IF EXISTS page_id;
|
||||
|
||||
DROP INDEX IF EXISTS idx_oauth_clients_federated_read;
|
||||
ALTER TABLE oauth_clients DROP COLUMN IF EXISTS source_id;
|
||||
ALTER TABLE oauth_clients DROP COLUMN IF EXISTS federated_read;
|
||||
`);
|
||||
|
||||
// Note: we don't strip sources.archived* here because they're inline in the
|
||||
// sources CREATE TABLE definition (no separate ALTER TABLE), and the
|
||||
// earlier `DROP TABLE IF EXISTS sources CASCADE` already nuked them.
|
||||
// The bootstrap's needsPagesBootstrap branch recreates sources without the
|
||||
// archive columns; the new needsSourcesArchive probe adds them.
|
||||
|
||||
// Run bootstrap in isolation (NOT initSchema). This is what we're testing.
|
||||
await (engine as any).applyForwardReferenceBootstrap();
|
||||
|
||||
@@ -311,7 +350,15 @@ function parseBaseTableColumns(sql: string): Map<string, Set<string>> {
|
||||
parts.push(body.slice(start));
|
||||
|
||||
for (const partRaw of parts) {
|
||||
const part = partRaw.trim();
|
||||
// Strip SQL line comments (`-- ...` to end of line) and block
|
||||
// comments (`/* ... */`) before identifying the column name.
|
||||
// Without this, a column definition preceded by a comment inside
|
||||
// the CREATE TABLE body is silently dropped (the comment is the
|
||||
// "first identifier" and the parser bails out).
|
||||
const stripped = partRaw
|
||||
.replace(/--[^\n]*/g, '')
|
||||
.replace(/\/\*[\s\S]*?\*\//g, '');
|
||||
const part = stripped.trim();
|
||||
if (!part) continue;
|
||||
// Skip constraint lines.
|
||||
if (/^(CONSTRAINT|PRIMARY|UNIQUE|CHECK|FOREIGN|EXCLUDE)\b/i.test(part)) continue;
|
||||
@@ -537,3 +584,184 @@ test('every CREATE INDEX column in PGLITE_SCHEMA_SQL is covered by CREATE TABLE
|
||||
);
|
||||
}
|
||||
}, 30000);
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// v0.36+ — MIGRATIONS introspection: catch the column-only forward-ref class.
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// The CREATE INDEX parser above kills the column-with-index forward-ref class.
|
||||
// v0.26.5 (v34) introduced a column-ONLY class: `sources.archived` +
|
||||
// `sources.archived_at` + `sources.archive_expires_at` aren't indexed but
|
||||
// `CREATE TABLE IF NOT EXISTS sources` is a no-op on pre-v34 brains. The
|
||||
// schema-blob replay never adds the archive columns, so downstream visibility
|
||||
// filters trip immediately.
|
||||
//
|
||||
// This test walks every `ALTER TABLE ... ADD COLUMN` in the MIGRATIONS array
|
||||
// (our own structured code, not arbitrary Postgres DDL) and asserts every
|
||||
// (table, column) pair is also added by `applyForwardReferenceBootstrap`.
|
||||
// Future contributors who add a migration with ALTER TABLE ADD COLUMN AND
|
||||
// forget to extend the bootstrap will see this test fail at PR time with a
|
||||
// paste-ready `Add probe for <table>.<column>` message.
|
||||
//
|
||||
// Why regex-on-our-own-SQL is safe vs regex-on-prod-Postgres-DDL: every
|
||||
// migration's SQL string is authored by us with consistent shape. The
|
||||
// ALTER TABLE ADD COLUMN pattern is stable across all 60+ existing
|
||||
// migrations. We control the input, not Postgres.
|
||||
//
|
||||
// Exemption mechanism: some migrations add columns that are intentionally
|
||||
// not in the schema blob (one-off transition columns later dropped, etc.).
|
||||
// Those go in the COLUMN_EXEMPTIONS set below with a brief rationale.
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
const COLUMN_EXEMPTIONS = new Set<string>([
|
||||
// Schema-blob-not-yet-refreshed: each of these columns is added by a
|
||||
// migration but NOT (yet) referenced by `PGLITE_SCHEMA_SQL` (neither in a
|
||||
// CREATE TABLE body nor in any CREATE INDEX). Bootstrap doesn't need to
|
||||
// add them because there's no forward reference for the schema blob's
|
||||
// replay to trip on. The migration handles every upgrade path correctly:
|
||||
// - fresh install: schema blob replays, then migration adds the column.
|
||||
// - pre-existing brain missing the column: migration adds it via ALTER.
|
||||
// - pre-existing brain already on this column: ALTER ... IF NOT EXISTS no-ops.
|
||||
// If a future migration adds a CREATE INDEX that references one of these
|
||||
// columns, the existing v0.28.5 CREATE-INDEX parser will catch it and
|
||||
// force a bootstrap probe (and the exemption should be removed).
|
||||
//
|
||||
// Refreshing PGLITE_SCHEMA_SQL is a separate concern handled by
|
||||
// `bun run build:schema` from src/schema.sql; not gated by this test.
|
||||
'minion_jobs.quiet_hours',
|
||||
'minion_jobs.stagger_key',
|
||||
'sources.chunker_version',
|
||||
'access_tokens.permissions',
|
||||
'takes.resolved_quality',
|
||||
'pages.emotional_weight_recomputed_at',
|
||||
'facts.notability',
|
||||
'facts.row_num',
|
||||
'facts.source_markdown_slug',
|
||||
'pages.chunker_version',
|
||||
'pages.source_path',
|
||||
'content_chunks.edges_backfilled_at',
|
||||
'query_cache.knobs_hash',
|
||||
]);
|
||||
|
||||
test('every ALTER TABLE ADD COLUMN in MIGRATIONS is covered by applyForwardReferenceBootstrap (column-only class)', async () => {
|
||||
const { extractAddedColumnsFromMigrations } = await import('./helpers/extract-added-columns.ts');
|
||||
const { readFileSync } = await import('fs');
|
||||
const { resolve: resolvePath } = await import('path');
|
||||
const { PGLITE_SCHEMA_SQL } = await import('../src/core/pglite-schema.ts');
|
||||
|
||||
const enginePath = resolvePath(process.cwd(), 'src/core/pglite-engine.ts');
|
||||
const engineSrc = readFileSync(enginePath, 'utf-8');
|
||||
const bootstrapAdds = parseAlterAddColumns(engineSrc);
|
||||
|
||||
// Bootstrap's own CREATE TABLE statements (e.g. needsPagesBootstrap inlines
|
||||
// `archived BOOLEAN ...` inside the CREATE TABLE sources block). Those
|
||||
// count as covered without a separate ALTER TABLE ADD COLUMN.
|
||||
const bootstrapCreateTableCols = parseBaseTableColumns(engineSrc);
|
||||
|
||||
// PGLITE_SCHEMA_SQL's CREATE TABLE definitions. The schema blob defines
|
||||
// every modern table inline; columns added by migrations are typically
|
||||
// ALSO updated in the schema blob so fresh installs get them natively.
|
||||
// The bootstrap is only needed when: (a) the table existed before the
|
||||
// migration ran (so CREATE TABLE IF NOT EXISTS is a no-op on old brains)
|
||||
// AND (b) the column has a forward-reference index OR a downstream filter
|
||||
// that breaks on old brains. Schema-blob coverage handles the fresh case.
|
||||
const schemaCreateTableCols = parseBaseTableColumns(PGLITE_SCHEMA_SQL);
|
||||
|
||||
const migrationAdds = extractAddedColumnsFromMigrations();
|
||||
|
||||
const covered = (table: string, column: string): boolean => {
|
||||
if (COLUMN_EXEMPTIONS.has(`${table}.${column}`)) return true;
|
||||
if (bootstrapAdds.some(a => a.table === table && a.column === column)) return true;
|
||||
const bootstrapCols = bootstrapCreateTableCols.get(table);
|
||||
if (bootstrapCols && bootstrapCols.has(column)) return true;
|
||||
const schemaCols = schemaCreateTableCols.get(table);
|
||||
if (schemaCols && schemaCols.has(column)) return true;
|
||||
return false;
|
||||
};
|
||||
|
||||
const uncovered: typeof migrationAdds = [];
|
||||
for (const ref of migrationAdds) {
|
||||
if (!covered(ref.table, ref.column)) {
|
||||
uncovered.push(ref);
|
||||
}
|
||||
}
|
||||
|
||||
if (uncovered.length > 0) {
|
||||
const list = uncovered
|
||||
.map(u => ` ${u.table}.${u.column}`)
|
||||
.join('\n');
|
||||
throw new Error(
|
||||
`MIGRATIONS file (src/core/migrate.ts) adds ${uncovered.length} (table, column) pair(s) that ` +
|
||||
`applyForwardReferenceBootstrap does NOT cover:\n${list}\n\n` +
|
||||
`Fix one of:\n` +
|
||||
` 1. Add a probe + ALTER TABLE ADD COLUMN in applyForwardReferenceBootstrap ` +
|
||||
`(src/core/pglite-engine.ts AND src/core/postgres-engine.ts), OR\n` +
|
||||
` 2. If the column is intentionally not in the schema blob ` +
|
||||
`(transitional / handler-only / later-dropped), add the (table, column) ` +
|
||||
`to COLUMN_EXEMPTIONS in test/schema-bootstrap-coverage.test.ts with a ` +
|
||||
`brief rationale comment.`,
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('extractAddedColumnsFromMigrations sanity-checks against known migration column additions', async () => {
|
||||
// Lightweight sanity test that the helper extracts the columns we expect
|
||||
// for a few well-known v34 / v60 / v61 migrations. Catches regex
|
||||
// regressions in the helper itself.
|
||||
const { extractAddedColumnsFromMigrations } = await import('./helpers/extract-added-columns.ts');
|
||||
const refs = extractAddedColumnsFromMigrations();
|
||||
const has = (table: string, column: string) =>
|
||||
refs.some(r => r.table === table && r.column === column);
|
||||
// v34 sources.archived* (the codex C1 case)
|
||||
expect(has('sources', 'archived')).toBe(true);
|
||||
expect(has('sources', 'archived_at')).toBe(true);
|
||||
expect(has('sources', 'archive_expires_at')).toBe(true);
|
||||
// v60+v61 oauth_clients.*
|
||||
expect(has('oauth_clients', 'source_id')).toBe(true);
|
||||
expect(has('oauth_clients', 'federated_read')).toBe(true);
|
||||
// v18 files.*
|
||||
expect(has('files', 'source_id')).toBe(true);
|
||||
expect(has('files', 'page_id')).toBe(true);
|
||||
});
|
||||
|
||||
test('extractAlterAddColumnsFromSql handles representative migration SQL shapes', async () => {
|
||||
const { __internal } = await import('./helpers/extract-added-columns.ts');
|
||||
const fn = __internal.extractAlterAddColumnsFromSql;
|
||||
|
||||
// Standard shape (with IF NOT EXISTS)
|
||||
expect(fn('ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN')).toEqual([
|
||||
{ table: 'sources', column: 'archived' },
|
||||
]);
|
||||
// No IF NOT EXISTS (older migrations)
|
||||
expect(fn('ALTER TABLE pages ADD COLUMN deleted_at TIMESTAMPTZ;')).toEqual([
|
||||
{ table: 'pages', column: 'deleted_at' },
|
||||
]);
|
||||
// Multi-statement, mixed
|
||||
expect(fn(`
|
||||
CREATE INDEX foo ON bar(x);
|
||||
ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS source_id TEXT REFERENCES sources(id);
|
||||
ALTER TABLE oauth_clients ADD COLUMN IF NOT EXISTS federated_read TEXT[] NOT NULL DEFAULT '{}';
|
||||
UPDATE oauth_clients SET source_id = 'default';
|
||||
`)).toEqual([
|
||||
{ table: 'oauth_clients', column: 'source_id' },
|
||||
{ table: 'oauth_clients', column: 'federated_read' },
|
||||
]);
|
||||
// Quoted identifiers
|
||||
expect(fn('ALTER TABLE "pages" ADD COLUMN "effective_date" TIMESTAMPTZ')).toEqual([
|
||||
{ table: 'pages', column: 'effective_date' },
|
||||
]);
|
||||
// ALTER TABLE IF EXISTS / ONLY variants
|
||||
expect(fn('ALTER TABLE IF EXISTS ONLY content_chunks ADD COLUMN language TEXT')).toEqual([
|
||||
{ table: 'content_chunks', column: 'language' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('planted-bug: simulated unprovided column produces a clear failure message', async () => {
|
||||
// Negative case — regression guard. If the contract test silently passes
|
||||
// on uncovered columns, the gate is fake. This test plants a fake column
|
||||
// in a fake SQL string and verifies the helper extracts it (proving the
|
||||
// gate would catch it in the real contract test).
|
||||
const { __internal } = await import('./helpers/extract-added-columns.ts');
|
||||
const fn = __internal.extractAlterAddColumnsFromSql;
|
||||
const planted = fn('ALTER TABLE pages ADD COLUMN IF NOT EXISTS planted_test_col TEXT');
|
||||
expect(planted).toEqual([{ table: 'pages', column: 'planted_test_col' }]);
|
||||
});
|
||||
|
||||
@@ -141,4 +141,50 @@ describe('manageGitignore', () => {
|
||||
manageGitignore(tmp);
|
||||
expect(warnings.some((w) => /Could not (read|update)/.test(w))).toBe(true);
|
||||
});
|
||||
|
||||
// ────────────────────────────────────────────────────────────────
|
||||
// Worktree vs submodule discrimination (closes #889)
|
||||
// ────────────────────────────────────────────────────────────────
|
||||
|
||||
test('REGRESSION: submodule with relative gitdir/modules/ → skip (D49 contract)', () => {
|
||||
writeStorageConfig();
|
||||
writeFileSync(join(tmp, '.git'), 'gitdir: ../.git/modules/sub\n');
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
|
||||
expect(warnings.some((w) => /submodule/.test(w))).toBe(true);
|
||||
});
|
||||
|
||||
test('absorbed submodule with absolute gitdir/modules/ → skip (closes edge case)', () => {
|
||||
writeStorageConfig();
|
||||
// After `git submodule absorbgitdirs`, the gitdir path becomes absolute.
|
||||
writeFileSync(join(tmp, '.git'), 'gitdir: /home/user/parent/.git/modules/sub\n');
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(false);
|
||||
expect(warnings.some((w) => /submodule/.test(w))).toBe(true);
|
||||
});
|
||||
|
||||
test('CRITICAL: worktree with absolute gitdir/worktrees/ → MANAGE (closes #889)', () => {
|
||||
writeStorageConfig();
|
||||
writeFileSync(join(tmp, '.git'), 'gitdir: /home/user/repo/.git/worktrees/feature-branch\n');
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(true);
|
||||
expect(warnings.filter((w) => /submodule/.test(w))).toEqual([]);
|
||||
});
|
||||
|
||||
test('worktree with relative gitdir/worktrees/ → MANAGE', () => {
|
||||
writeStorageConfig();
|
||||
writeFileSync(join(tmp, '.git'), 'gitdir: ../.git/worktrees/feature-branch\n');
|
||||
manageGitignore(tmp);
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(true);
|
||||
expect(warnings.filter((w) => /submodule/.test(w))).toEqual([]);
|
||||
});
|
||||
|
||||
test('malformed .git file (no gitdir: prefix) → MANAGE (preserves catch behavior)', () => {
|
||||
writeStorageConfig();
|
||||
writeFileSync(join(tmp, '.git'), 'garbage content\n');
|
||||
manageGitignore(tmp);
|
||||
// No gitdir prefix → not a submodule → MANAGE.
|
||||
expect(existsSync(join(tmp, '.gitignore'))).toBe(true);
|
||||
expect(warnings.filter((w) => /submodule/.test(w))).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
+51
-1
@@ -1,5 +1,5 @@
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
|
||||
import { buildSyncManifest, isSyncable, pathToSlug } from '../src/core/sync.ts';
|
||||
import { buildSyncManifest, isSyncable, pathToSlug, pruneDir } from '../src/core/sync.ts';
|
||||
import { buildGitInvocation } from '../src/commands/sync.ts';
|
||||
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
@@ -99,6 +99,56 @@ describe('isSyncable', () => {
|
||||
expect(isSyncable('ops/deploy-log.md')).toBe(false);
|
||||
expect(isSyncable('ops/config.md')).toBe(false);
|
||||
});
|
||||
|
||||
// ────────────────────────────────────────────────────────────────
|
||||
// v0.36 walker drift fix (closes #923, #202): node_modules exclusion
|
||||
// ────────────────────────────────────────────────────────────────
|
||||
|
||||
test('CRITICAL latent-bug regression: rejects node_modules paths at any depth', () => {
|
||||
// Pre-v0.36, isSyncable had no node_modules check. Any markdown file
|
||||
// under a non-dot `node_modules` directory slipped through. This is
|
||||
// the canonical latent-bug fix gated by IRON RULE per the wave plan.
|
||||
expect(isSyncable('node_modules/some-pkg/README.md')).toBe(false);
|
||||
expect(isSyncable('node_modules/some-pkg/CHANGELOG.md')).toBe(false);
|
||||
expect(isSyncable('node_modules/some-pkg/docs/api.md')).toBe(false);
|
||||
expect(isSyncable('apps/web/node_modules/dep/notes.md')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('pruneDir', () => {
|
||||
test('blocks node_modules (no leading dot, the latent-bug case)', () => {
|
||||
expect(pruneDir('node_modules')).toBe(false);
|
||||
});
|
||||
|
||||
test('blocks dot-prefix dirs (.git, .obsidian, .raw, .cache, etc.)', () => {
|
||||
expect(pruneDir('.git')).toBe(false);
|
||||
expect(pruneDir('.obsidian')).toBe(false);
|
||||
expect(pruneDir('.raw')).toBe(false);
|
||||
expect(pruneDir('.cache')).toBe(false);
|
||||
expect(pruneDir('.vscode')).toBe(false);
|
||||
});
|
||||
|
||||
test('blocks ops (gbrain operational dir)', () => {
|
||||
expect(pruneDir('ops')).toBe(false);
|
||||
});
|
||||
|
||||
test('blocks *.raw sidecar dirs (gbrain convention)', () => {
|
||||
expect(pruneDir('.raw')).toBe(false);
|
||||
expect(pruneDir('pedro.raw')).toBe(false);
|
||||
expect(pruneDir('article.raw')).toBe(false);
|
||||
});
|
||||
|
||||
test('allows normal content dirs', () => {
|
||||
expect(pruneDir('wiki')).toBe(true);
|
||||
expect(pruneDir('people')).toBe(true);
|
||||
expect(pruneDir('meetings')).toBe(true);
|
||||
expect(pruneDir('corpus')).toBe(true);
|
||||
expect(pruneDir('2026')).toBe(true);
|
||||
});
|
||||
|
||||
test('empty string returns true (defensive default)', () => {
|
||||
expect(pruneDir('')).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe('pathToSlug', () => {
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
/**
|
||||
* Gateway adapter tests for runThink (#952 fix).
|
||||
*
|
||||
* Pre-v0.36, runThink instantiated `new Anthropic()` directly. Closing #952
|
||||
* routed it through gateway.chat() so MCP stdio launches pick up
|
||||
* `anthropic_api_key` from gbrain config instead of process.env.
|
||||
*
|
||||
* The adapter shape was determined by plan-eng-review D10 (cross-model
|
||||
* tension D10 with codex C7+C8+C9+C10):
|
||||
* - drop new Anthropic() entirely
|
||||
* - real availability check (NOT a false-positive `getChatModel()` truthy)
|
||||
* - model-id normalization (bare → provider-prefixed)
|
||||
* - response-shape conversion (ChatResult → Anthropic.Message)
|
||||
*
|
||||
* These tests pin the four spec points. Hermetic — no real LLM call.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { __thinkAdapter } from '../src/core/think/index.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
describe('think gateway adapter — response shape conversion', () => {
|
||||
test('chatResultToMessage maps ChatResult.text to Anthropic.Message content[0].text', () => {
|
||||
const out = __thinkAdapter.chatResultToMessage(
|
||||
{
|
||||
text: '{"answer":"hi","citations":[],"gaps":[]}',
|
||||
blocks: [],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 5, output_tokens: 2, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-opus-4-7',
|
||||
providerId: 'anthropic',
|
||||
},
|
||||
'anthropic:claude-opus-4-7',
|
||||
);
|
||||
expect(out.content[0].type).toBe('text');
|
||||
expect(out.content[0].text).toBe('{"answer":"hi","citations":[],"gaps":[]}');
|
||||
expect(out.usage.input_tokens).toBe(5);
|
||||
expect(out.usage.output_tokens).toBe(2);
|
||||
expect(out.stop_reason).toBe('end_turn');
|
||||
expect(out.model).toBe('anthropic:claude-opus-4-7');
|
||||
});
|
||||
|
||||
test('mapStopReason covers the full provider-neutral stop-reason set', () => {
|
||||
expect(__thinkAdapter.mapStopReason('end')).toBe('end_turn');
|
||||
expect(__thinkAdapter.mapStopReason('length')).toBe('max_tokens');
|
||||
expect(__thinkAdapter.mapStopReason('tool_calls')).toBe('tool_use');
|
||||
// 'refusal', 'content_filter', 'other' → end_turn (no Anthropic equivalent).
|
||||
expect(__thinkAdapter.mapStopReason('refusal')).toBe('end_turn');
|
||||
expect(__thinkAdapter.mapStopReason('content_filter')).toBe('end_turn');
|
||||
expect(__thinkAdapter.mapStopReason('other')).toBe('end_turn');
|
||||
});
|
||||
});
|
||||
|
||||
describe('think gateway adapter — model-id normalization', () => {
|
||||
test('tryBuildGatewayClient accepts bare anthropic model ids and prefixes anthropic:', async () => {
|
||||
// Bare model: `claude-opus-4-7` → must resolve through `anthropic:` recipe.
|
||||
// resolveRecipe will throw AIConfigError for unknown providers, so a
|
||||
// successful build proves the prefix landed.
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test-fake' }, async () => {
|
||||
const client = await __thinkAdapter.tryBuildGatewayClient('claude-opus-4-7');
|
||||
expect(client).not.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
test('tryBuildGatewayClient accepts already-prefixed provider:model strings', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test-fake' }, async () => {
|
||||
const client = await __thinkAdapter.tryBuildGatewayClient('anthropic:claude-sonnet-4-6');
|
||||
expect(client).not.toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
test('tryBuildGatewayClient returns null on unknown provider (AIConfigError → graceful fallback)', async () => {
|
||||
const client = await __thinkAdapter.tryBuildGatewayClient('nonexistent-provider:foo-1');
|
||||
expect(client).toBeNull();
|
||||
});
|
||||
|
||||
test('tryBuildGatewayClient returns null when ANTHROPIC_API_KEY is absent (preserves legacy NO_ANTHROPIC_API_KEY signal)', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => {
|
||||
const client = await __thinkAdapter.tryBuildGatewayClient('claude-opus-4-7');
|
||||
expect(client).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
test('hasAnthropicKey reads process.env', async () => {
|
||||
await withEnv({ ANTHROPIC_API_KEY: 'sk-test-key' }, async () => {
|
||||
expect(__thinkAdapter.hasAnthropicKey()).toBe(true);
|
||||
});
|
||||
await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => {
|
||||
expect(__thinkAdapter.hasAnthropicKey()).toBe(false);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('think gateway adapter — graceful fallback shape', () => {
|
||||
test('buildGracefulMessage produces a parseable Anthropic.Message-shaped object', () => {
|
||||
const m = __thinkAdapter.buildGracefulMessage('anthropic:claude-opus-4-7');
|
||||
expect(m.type).toBe('message');
|
||||
expect(m.role).toBe('assistant');
|
||||
expect(m.content[0].type).toBe('text');
|
||||
expect(m.content[0].text).toContain('no LLM available');
|
||||
expect(m.content[0].text).toContain('gbrain config');
|
||||
expect(m.usage.input_tokens).toBe(0);
|
||||
expect(m.usage.output_tokens).toBe(0);
|
||||
expect(m.stop_reason).toBe('end_turn');
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user