mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
+1








7c27fa129b
* fix(oauth): default omitted authorize scope to client's full grant When a client omits `scope` on /authorize, the authorize() grant computed `(params.scopes || []).filter(...)` → the empty set. That empty grant was written to oauth_codes and propagated into the access AND refresh tokens, so every request failed `insufficient_scope` even though the client was registered with e.g. `read write`. Because refresh inherits the stored grant, it never self-healed — reconnecting just minted another empty-scoped token. Some MCP connectors (observed with Claude Desktop) omit `scope` on /authorize, so they hit this on every connection. Fix: when no scope is requested, default to the client's full registered scope (RFC 6749 §3.3 permits a server default). This mirrors exchangeClientCredentials, which already does `requestedScope ? ... : allowedScopes`. The result is still clamped to the allowed set, so an explicit over-broad request cannot escalate. Adds test/oauth-authorize-scope-default.test.ts covering: omitted/empty → inherits full grant; explicit subset honored; clamp preserved (over-broad and disallowed-only requests cannot escalate or trigger inheritance). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(sync): skip Python venv/ in the code walker collectSyncableFiles (first-sync walker) and the incremental PRUNE_DIR_NAMES set skipped node_modules but not Python venv/. On a Python repo the walker descended into venv/ (thousands of files); the resulting slug collisions crashed putPage's INSERT ... ON CONFLICT ... RETURNING with "undefined is not an object (evaluating 'row.deleted_at')". Add `venv` alongside node_modules in both the import.ts inline skip and PRUNE_DIR_NAMES. venv is the Python equivalent of node_modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(gateway): carry asymmetric input_type across the AI SDK to the wire body (#1400) dimsProviderOptions() threads input_type ('query' | 'document') into providerOptions.openaiCompatible for asymmetric models (ZE zembed-1, Voyage v3+), but the AI SDK's openai-compatible adapter validates providerOptions against a fixed schema and silently drops the field before building the HTTP body. Every embedQuery() was therefore encoded document-side: the ZE shim's hard default fired ('document'), Voyage and local openai-compat servers got no input_type at all, and asymmetric retrieval silently collapsed toward surface-token overlap — while the providerOptions-level contract test stayed green. Fix: an AsyncLocalStorage (same pattern as __budgetStore) populated in embedSubBatch() only when providerOptions actually threads an input_type, read at body-rewrite time by the fetch shims: - zeroEntropyCompatFetch: recovers the threaded value; document default preserved for ingest paths. - voyageCompatFetch: opt-in like the dims.ts Voyage branch — inject only when threaded; the field stays off the wire otherwise. - NEW openAICompatAsymmetricFetch: fallthrough default for every other openai-compatible recipe (llama-server, litellm, ollama, ...) — the canonical local/proxy paths for asymmetric models. Strict pass-through when nothing was threaded, so symmetric deployments see zero wire change; recipes with their own compat fetch (azure) keep it via the compat.fetch ?? precedence. KNOBS_HASH_VERSION bumped 10→11: cached query_cache rows were keyed on document-side query vectors; pre-fix rows must not be served to post-fix lookups (same convention as the v=3 embedding-provider bump). One-time global cold-miss on upgrade; refills within cache.ttl_seconds. Tests: test/embed-input-type-wire.test.ts runs the REAL SDK transport with a mocked global fetch and asserts on the outbound body — the only layer where this regression is observable. Covers ZE hosted, llama-server, litellm, ollama (query + document sides) and pins the pass-through for non-asymmetric models and Voyage's opt-in shape. 4 of the original 7 assertions fail on master, proving the pin. One structural pin in test/ai/zeroentropy-compat-fetch.test.ts updated to the new line shape (same semantic); KEY_FILES.md gateway.ts entry updated to the new truth. Supersedes #1400 (closed unmerged) — same ALS mechanism, extended to Voyage + all openai-compatible recipes. Credit to @billy-armstrong for the original diagnosis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(sync): honor .gitignore in code walk; prune vendor/dist/build collectSyncableFiles (the full-sync / dry-run enumerator) reimplemented its own directory skip list inline (node_modules || ops), bypassing the canonical pruneDir gate and ignoring .gitignore entirely. On a Laravel/PHP repo this descended into vendor/ (~50k Composer files), storage/, and public/build/, trying to import 52k dependency/build files and flooding the index with library internals (a 35-min sync that never finished, killed by the watchdog at 3%). - collectSyncableFiles now enumerates via `git ls-files --cached --others --exclude-standard` when dir is a git work tree, so the walk honors .gitignore (tracked + untracked-not-ignored). Falls back to the FS walk for non-git dirs. EroLab: 52164 -> 1028 files. - The FS fallback now prunes through the canonical pruneDir() instead of a drifted inline list, so the two skip lists can't diverge again. - PRUNE_DIR_NAMES gains vendor/dist/build (dependency + build-output trees). Addresses #1483 (.gbrainignore), #1159 (--respect-gitignore), and the maintainer's #1942 vendor/dist/build prune. Walker regression suites (sync-walker-symlink, brain-writer-walk-prune, sync, sync-walker-submodule) green: 90 pass. * fix(config): ignore DATABASE_URL auto-loaded from cwd .env (#427) Bun merges .env files from the process cwd into process.env before any user code runs. loadConfig() prefers env DATABASE_URL over ~/.gbrain/config.json, so any gbrain invocation from inside a web-app checkout silently retargets the brain at that app's database — reads go to the wrong DB and apply-migrations can write gbrain's schema into a production app database (#427). effectiveEnvDatabaseUrl() re-parses the .env files Bun auto-loads from cwd and treats a DATABASE_URL whose value matches one of them as file-origin: ignored, with a one-time stderr notice. GBRAIN_DATABASE_URL and genuinely exported DATABASE_URLs are honored unchanged, so the operator escape hatch and the e2e suite's env-provided URL keep working. Applied at loadConfig, getDbUrlSource (doctor parity), init --non-interactive, and migrate --to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cli): arm the disconnect hard-deadline at teardown entry, not before the op body The 10s force-exit timer in the shared-op dispatch was armed BEFORE the try block, so any op whose handler ran past 10s wall-clock was killed mid-flight with process.exit(0) and zero stdout. On a slow Postgres pooler (6-10s per fresh connection) a healthy `gbrain search` was force-exited every time — an empty 'success' indistinguishable from no results. The v0.42.20.0 exitCode honor can't help: a mid-op kill fires before any error path sets exitCode. Move the arming into the finally (teardown entry), matching the fall-through owner-disconnect site later in main(): the timer still bounds a hung drain/disconnect (the C13 contract) but can no longer kill a slow-but-progressing op. Verified on a transaction-pooler Supabase brain: search went from 0 bytes/exit 0 at 10s to real results at ~21s. * fix(import): stamp source_id on extracted call-graph edges importCodeFile built CodeEdgeInput rows without source_id, so every edge landed NULL. getCallersOf/getCalleesOf filter `AND source_id = <scoped>` whenever a worktree pin or --source is in play — NULL never matches, so scoped call-graph queries silently returned 0 rows on multi-source brains even though the edges existed (2,122 edges, 26 targeting the probed symbol, count 0 returned). One-line fix: carry the sourceId already in scope into the edge input. Existing NULL rows backfill with: UPDATE code_edges_symbol e SET source_id = p.source_id FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE c.id = e.from_chunk_id AND e.source_id IS NULL; (same for code_edges_chunk). Verified: code-callers returns 21 callers where it returned 0. * docs(migrations): NULL embeddings BEFORE the column-type alter The Postgres recipe ordered ALTER COLUMN TYPE vector(N) before the UPDATE that clears stale embeddings. pgvector refuses to cast existing vectors across dimensions ('expected 1024 dimensions, not 1536'), so the recipe as written aborts the transaction on any brain that has embeddings — which is every brain doing this migration. Swap the steps: NULLs cast fine. * fix: honor legacy token source grants in oauth * fix(cli): bound read-scope op handlers at 180s wallclock (pre-landing review) With the hard-deadline timer correctly scoped to teardown, a genuinely wedged read handler (hung pooler connection mid-query) would hang the CLI forever — the #1633 zombie class the old pre-try timer accidentally bounded at 10s. Reads now get a generous withTimeout (180s default, far above any healthy slow-pooler run; --timeout=Ns overrides; exit 124 with the teardown finally still draining + disconnecting). Writes/admin stay unbounded: a long import/embed must never be killed by a default. * fix(import): stamp unscoped edges 'default', matching the pages-table default Review catch: 'sourceId ?? null' fixed the scoped path but left the unscoped one (reindex --code without --source, importCodeFile callers without opts.sourceId) stranding edges at NULL while their pages land under the schema default (pages.source_id DEFAULT 'default') — so getCallersOf(sym, { sourceId: 'default' }) missed them. Same bug, other door. Fallback is now 'default'. * fix(core): runtime dim-migration recipe NULLs embeddings before the alter Review catch: the doc fix corrected docs/embedding-migrations.md, but embeddingMismatchMessage still PRINTED the broken order — ALTER before UPDATE ... SET embedding = NULL — and linked to the now-contradicting doc. pgvector refuses to cast existing vectors across dimensions, so the printed recipe aborted on any brain that has embeddings. Swap the steps and say why inline. * feat(migrate): v116 — backfill NULL edge source_id + index from_symbol_qualified 1. Backfill: edges written before the stamping fix sit at source_id=NULL and stay invisible to scoped call-graph queries until repaired. Derive each edge's source from its own from_chunk's page (pages.source_id is NOT NULL DEFAULT 'default'). Same SQL verified live on a 2,122-edge production brain. 2. Indexes: getCalleesOf filters both edge tables on from_symbol_qualified, which had no index — every callee lookup was a seq scan, amplified per-BFS-node by the recursive code walk. With NULL edges repaired, scoped walks actually expand, so the latent cost becomes real. Mirrored into src/schema.sql; schema-embedded.ts regenerated. * docs(migrations): align the rationale list with the corrected recipe order The 'Why we don't do this automatically' list still said alter-then-wipe; reorder to wipe-then-alter and replace the fragile 'step 3' numeric cross-reference with a name-based one. * test: regression coverage for edge source_id stamping, timer placement, recipe order - import-code-edges-source-id: scoped import stamps edges + scoped getCallersOf/getCalleesOf match (verified failing pre-fix), plus the unscoped-import case asserting 'default' stamping. - cli-force-exit-teardown-arming: structural pin — the hard-deadline timer arms inside the finally (teardown entry), never before the op body; daemon guard, unref, clearTimeout intact. - embedding-dim-check: recipe order pinned — UPDATE precedes ALTER so the printed SQL can't drift from docs/embedding-migrations.md again. * fix(cli): hard-exit after teardown on wallclock timeout; bound makeContext too Adversarial review, two findings on the new timeout path: 1. On timeout the finally drained, disconnected, then CLEARED the hard-deadline timer — removing the only backstop while the abandoned handler (withTimeout races, it does not cancel) can hold ref'd sockets/SDK timers that keep Bun's loop alive: 'timed out' printed, process immortal — the zombie class this branch exists to kill, resurrected through its own fix. The finally now exits explicitly after teardown completes on the timeout path. 2. makeContext does DB I/O (resolveSourceId) for EVERY op and sat outside any bound — a pooler wedge at context build hung reads, writes, and admin alike. It now shares the same wallclock bound. * fix(import): normalize edge source once — closes the '' door and the unscoped chunk fan-out Adversarial review: txOpts used truthiness while the edge stamp used nullish — sourceId:'' put pages under 'default' but stamped edges '', FK-violating against sources(id) and silently dropping the file's whole call graph in the best-effort catch. The unscoped getChunks could also fan out to same-slug chunks from another source. One normalized edgeSourceId (sourceId || 'default') now drives both the chunk lookup and the stamp. * fix(engine): default edge source_id to 'default' at the insert layer (both engines) Adversarial review: addCodeEdges still wrote e.source_id ?? null, so any future caller that forgets the field reintroduces invisible NULL edges the day after the v116 backfill runs. A NULL source_id is invisible to every scoped call-graph query; default to the schema-default source the way the pages table does. Applied to both engines (parity). * fix(core): facts alter recipe NULLs embeddings before cross-dimension alters Adversarial review: buildFactsAlterRecipe shipped the same defect class this branch fixes for content_chunks 350 lines up — a cross-dimension ALTER ... USING cast that pgvector refuses while rows hold old-width vectors. Dimension changes now wipe first (the facts pipeline re-embeds on next write); same-dim type swaps (halfvec <-> vector) keep the lossless cast and PRESERVE data. Both behaviors pinned by tests. * v0.42.39.0 chore: version bump + CHANGELOG + TODOS Marks the v0.42.20.0 'decouple the op-dispatch force-exit timer' follow-up complete — this branch ships exactly that decoupling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(postgres-engine): atomic JSONB merge in updateSourceConfig — eliminate lost-update race ## Problem `updateSourceConfig` used a read-then-write pattern: read the current `config` row, normalize it in JavaScript, then write the merged result back with `SET config = <normalized> || <patch>`. Under concurrent callers (two background autopilot/cycle paths patching different keys simultaneously), both callers can read the same stale row. The later `SET config = ...` then clobbers the earlier patch, silently dropping whatever keys the first caller wrote. Reproduced at 21/25 lost-update events under real Postgres with parallel callers. ## Fix Fold the normalization and merge into a single atomic `UPDATE … SET config = CASE … END || patch` statement. Because the `SET` expression evaluates against the row-locked latest version of `config`, there is no snapshot window between the read and the write. Concurrent callers now converge correctly (50/50 clean in reproduction test). The `CASE` also normalizes historical bad JSONB shapes inline: - `object` — used as-is - `string` — double-encoded config; inner text parsed with the SQL `IS JSON` guard (Postgres 16+) so unparseable strings fall back to `{}` instead of raising `invalid input syntax for type json` - `array` — array of patch objects aggregated into a flat object via `jsonb_object_agg` - anything else — falls back to `{}` `pglite-engine.updateSourceConfig` already used an atomic `||` merge; this change brings postgres-engine to parity. ## Test Added two assertions to `test/list-all-sources.test.ts`: 1. JSONB string holding non-JSON text normalizes to `{}` (no cast throw) 2. JSONB string holding double-encoded valid JSON is parsed then merged * fix(doctor): five correctness fixes — stale locks, content sanity, graph coverage, exit code, gateway guard ## 1. Stale lock break hints cover gbrain-cycle: keys The doctor stale-lock report only recognized `gbrain-sync:` lock prefixes; everything else fell back to `gbrain sync --break-lock`, which is wrong for dream/autopilot cycle locks. A `gbrain-cycle:<source>` or `gbrain-cycle` lock now suggests `gbrain dream --break-lock [--source <name>]`, and unknown lock shapes fall back to `gbrain doctor` instead of a misleading sync command. ## 2. content_sanity_audit_recent counts reject and quarantine as hard failures v0.42 renamed the hard disposition path: rejected pages emit a `reject` event and quarantined junk pages emit `quarantine`; `hard_block` is now only the pre-v0.42 legacy alias. The status check only counted `hard_block`, so fresh `reject` / `quarantine` events from the new path cleared as `ok` whenever fewer than 10 events existed. The check now sums all three for the hard count, and `soft_block + flag` for the soft count. ## 3. graph_coverage excludes test fixture entity pages from the denominator Brains seeded with code sources (e.g. a sync of the gbrain repo itself) could accumulate test fixture pages typed as `entity` / `person`. Including these in the entity-count denominator diluted coverage and produced spurious warnings ("Entity link coverage 0%, timeline 0%") on knowledge-only brains with no real entity pages. The check now queries a per-entity stats CTE that excludes `tools/gbrain/test/*` slugs and the `templates/new-person` stub, with an additional guard for the all-fixture case (`eligibleEntityCount = 0`). ## 4. process.exitCode instead of process.exit at doctor main exit point `process.exit(hasFail ? 1 : 0)` was a hard kill that prevented cleanup handlers (Bun unload events, open DB connections) from running. Using `process.exitCode = hasFail ? 1 : 0` defers the actual termination until the end of the event loop, allowing cleanup to complete. ## 5. checkSubagentCapability exported for test seams + gateway loop guard The function was private, making it untestable in isolation. It is now exported. Additionally, users running gbrain with a non-Anthropic chat model via `agent.use_gateway_loop=true` no longer receive a spurious warning that `ANTHROPIC_API_KEY` is missing — subagents route via the gateway loop in that configuration and do not need the key directly. ## Tests Doctor test suite: 77 pass, 0 fail (no regressions). * fix(engine): deleteFactsForPage excludeSourcePrefixes (#1928) + reconnect() parity (#2034) Engine-layer API for two cycle/availability fixes that share these files: - deleteFactsForPage gains optional excludeSourcePrefixes so the fence reconcile can protect non-fence facts (e.g. cli: conversation facts). - reconnect(ctx?) is now a first-class BrainEngine method on both engines (PostgresEngine already had it; PGLite gains config capture + reconnect) so callers stop using disconnect()+bare connect(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(cycle): stop extract_facts from wiping conversation facts (#1928) The fence reconcile delete-then-reinsert wiped cli:-origin facts (no fence to recreate them); a failed-sync full walk turned it brain-wide (1829 rows, 0 reinserted, status ok). Now: exclude cli: rows from the wipe, do NOT inherit the failed-sync->full-walk fallback for this destructive phase, and warn on net-negative reconcile. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(autopilot,supervisor): reconnect() instead of disconnect()+bare connect() (#2034) The autopilot health-probe recovery called connect() with no args after disconnect(), losing the startup config (database_url undefined -> FATAL restart-loop on every DB blip) and opening a null-pool window. Both call sites now use engine.reconnect(), which restores the captured config. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(write-through): mirror to the assigned source's local_path, never the global repo (#2018) put_page write-through resolved the disk target from the global sync.repo_path, so a default-source page (local_path NULL) got written into an unrelated federated source's working tree. Now it uses the assigned source's own local_path; NULL local_path skips (no leak); the global path is used only as a sole-source fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite-lock): heartbeat + steal-grace so live holders are never stolen (#2058) A live holder's lock was force-removed after 5min age alone, letting a second process share the single-writer data dir -> WAL corruption. The lock now heartbeats while held; a holder is reaped only when its PID is dead OR its heartbeat went stale past the steal grace. Pairs PID liveness with heartbeat age to also defeat PID reuse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(migrate,doctor): self-heal idx_timeline_dedup drift (#2038) A migration renumbered during a merge (v102) could be recorded-as-applied without its DDL running, leaving the 3-column index so every timeline write failed the 4-column ON CONFLICT. runMigrations now always runs a shape-keyed drift repair (dedupe-then-rebuild) even when no migration is pending, and doctor surfaces the drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(timeline): un-silence the swallowed batch catch; pin Date-batch round-trip (#2057) The meetings extractor's bare catch {} hid a brain-wide timeline-write failure (0 entries, no error). It now counts + surfaces batch errors. Adds a Date-bearing batch regression test proving the #1861 jsonb_to_recordset refactor already fixed the original ::text[] cast failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v0.42.41.0) Triage fix wave: 6 authored critical fixes (#1928 facts wipe, #2018 write-through leak, #2034 reconnect loop, #2058 WAL lock, #2038 timeline migration drift, #2057 timeline silent-empty) + community PRs #2064 #2052 #2020 #2033 #2074 #2075 #2009 #2072 #2073. TODOS: deferred #1994 #1963 #2050. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: address adversarial review findings (#1928, #2058, #2038, #2057) Codex as-built review of the authored fixes surfaced 4 real issues: - #2058: add a pid+acquired_at ownership token. A stale holder reaped + replaced past the grace must NOT let its resumed heartbeat refresh, nor releaseLock remove, the NEW owner's lock (re-opened the concurrent-writer hole). Heartbeat and release now verify the on-disk lock is still ours. + regression test. - #1928: the destructive-full-walk guard keyed off phases.includes('sync'), which wrongly suppressed a legitimate full reconcile when sync was SKIPPED (no engine / no brainDir). Key off a syncAttempted flag set only when sync actually ran. - #2038: dedupe keeps MIN(id) not MIN(ctid) — deterministic and consistent with the existing v-migration lower-id rule. - #2057: the extract CLI caller now surfaces batch_errors (stderr + exit 1) instead of printing a clean success over failed inserts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(key-files): sync reference to v0.42.41.0 triage-wave behavior Update KEY_FILES.md to current-state truth for the shipped fixes (no release-history clauses, per the reference-doc discipline): - write-through.ts (#2018): resolves the disk target from the assigned source's own local_path; sole-source falls back to sync.repo_path, multi-source skips with source_has_no_local_path rather than leak. - engine.ts (#2034): reconnect() is now a REQUIRED lifecycle method on both engines; config-restoring, never disconnect()+bare connect(). - migrate.ts (#2073): document v116 edge source_id backfill + callee index, and the always-run (version-counter-blind) timeline dedup self-heal. - new entry for timeline-dedup-repair.ts (#2038) + the timeline_dedup_index doctor check. - new entry for pglite-lock.ts (#2058): heartbeat + steal-grace (GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS) so a live holder is never stolen. - extract-facts.ts (#1928): cli:-fact protection, no failed-sync full-walk inheritance, net_fact_deletion warn floor. bun run build:llms re-run (KEY_FILES is link-only so bundles unchanged); freshness + current-state guards green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(write-through): preserve nested multi-source layout; narrow #2018 leak guard The first #2018 fix skipped any no-local_path source on a multi-source brain, which broke the legitimate nested layout (a source without its own tree nests under the host repo at .sources/<id>/ — pinned by put-page-write-through.test). Narrow the guard: a no-local_path source nests under sync.repo_path as before; only SKIP when sync.repo_path is literally another source's own local_path (the actual leak — writing there pollutes that sibling's repo). Caught by the sharded suite. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: satisfy test-isolation guard for the new lock/reconnect tests CI `verify` flagged 3 intra-process isolation violations in the tests added this wave (the parallel runner shares one process per shard): - pglite-lock.test.ts: the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS mutation now goes through withEnv() instead of a raw process.env write (R1). - pglite-reconnect: renamed to *.serial.test.ts — it creates per-test engines to exercise the connect/reconnect lifecycle, which doesn't fit the shared beforeAll-engine model (R3/R4). verify is now 30/30; both files green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(pglite): reconnect() is a no-op for in-memory engines (#2034) CI serial-tests + test(5) caught two in-branch regressions from the #2034 PGLite reconnect(): - worker/queue claim-error recovery + their renewLock e2e test assume PGLite reconnect is absent/no-op (queue.ts documents it). Making it a real disconnect+reopen wiped an in-memory engine's state mid-job. reconnect() now no-ops for in-memory (no database_path) — file-backed still re-opens the dir (state persists on disk). Restores the documented worker assumption. - connection-resilience 'Supervisor still has the 3-strikes-then-reconnect path' pinned the removed unsafe-cast text; updated to assert the direct this.engine.reconnect() call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: quarantine embed-input-type-wire to serial lane (CI test(5) leak) #2033's embed-input-type-wire.test.ts configures a 1280-dim embedding gateway; the active dimension survived into engine-find-trajectory when CI's 10-way hash-disjoint sharding co-located them (this branch's added files reshuffled the assignment), failing 7 trajectory tests with 'expected 1280 dimensions, not 1536'. resetGateway() in afterEach clears the gateway but the dimension still leaked. It mutates global gateway/embedding state, so it belongs in the serial lane (own bun process, true isolation) by the repo's own definition. Root-caused by reproducing the exact failing pair locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Austin Arnett <austin@sdsconsultinggroup.org> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Dave MacDonald <djmacdonald@ucdavis.edu> Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com> Co-authored-by: Alex P. <12667893+aphaiboon@users.noreply.github.com> Co-authored-by: Garry Tan <bo.m.liu@gmail.com> Co-authored-by: jbarol <barol.j@gmail.com> Co-authored-by: maxpetrusenkoagent <max.petrusenko.agent@gmail.com> Co-authored-by: PAI <pai@scaffolde.ai>
708 lines
29 KiB
TypeScript
708 lines
29 KiB
TypeScript
/**
|
|
* Detect existing-brain embedding-dimension mismatch (v0.28.5 — A4).
|
|
*
|
|
* `gbrain init --embedding-dimensions N` on an existing brain whose
|
|
* `content_chunks.embedding` column is a different `vector(M)` would
|
|
* silently create a config/column drift: the config gets templated to N
|
|
* but the column stays at M. The first sync write blows up with
|
|
* "expected M, got N" — the silent-corruption pattern v0.28.5 is shipped
|
|
* to kill.
|
|
*
|
|
* Loud-failure path: `gbrain init` AND `gbrain doctor` both consult this
|
|
* helper. On mismatch they emit the same inline ALTER recipe (see
|
|
* `embeddingMismatchMessage`) plus a pointer to `docs/embedding-migrations.md`.
|
|
*/
|
|
|
|
import type { BrainEngine } from './engine.ts';
|
|
import { PGVECTOR_HNSW_VECTOR_MAX_DIMS } from './vector-index.ts';
|
|
import { gbrainPath } from './config.ts';
|
|
import { resolveRecipe } from './ai/model-resolver.ts';
|
|
import type { Recipe } from './ai/types.ts';
|
|
import { AIConfigError } from './ai/errors.ts';
|
|
import {
|
|
supportsVoyageOutputDimension,
|
|
isValidVoyageOutputDim,
|
|
VOYAGE_VALID_OUTPUT_DIMS,
|
|
supportsZeroEntropyDimension,
|
|
isValidZeroEntropyDim,
|
|
ZEROENTROPY_VALID_DIMS,
|
|
isOpenAITextEmbedding3Model,
|
|
isValidOpenAITextEmbedding3Dim,
|
|
maxOpenAITextEmbedding3Dim,
|
|
} from './ai/dims.ts';
|
|
|
|
/**
|
|
* pgvector supports vector(N) columns up to 16000 dimensions. HNSW indexing
|
|
* is capped at PGVECTOR_HNSW_VECTOR_MAX_DIMS (2000); above that, exact scan
|
|
* still works but searches are slower.
|
|
*
|
|
* The preflight resolver below uses this as the hard upper bound so anything
|
|
* pgvector itself would reject (e.g. an accidental `embedding_dimensions: 99999`)
|
|
* fails at init time rather than at first embed.
|
|
*/
|
|
export const PGVECTOR_COLUMN_MAX_DIMS = 16000;
|
|
|
|
/**
|
|
* v0.37 (D9): runtime guard for the deferred-setup mode.
|
|
*
|
|
* Init's `--no-embedding` opt-in writes `embedding_disabled: true` to
|
|
* config.json. Every embed callsite (CLI: `gbrain embed`, `gbrain import`;
|
|
* library: `runEmbedCore`) consults this guard so the user gets a clear
|
|
* "configure embedding first" message rather than an opaque gateway error
|
|
* at first vector write.
|
|
*
|
|
* Returns void on the happy path. Throws `EmbeddingDisabledError` when the
|
|
* config has `embedding_disabled: true`. The error type lets callers in
|
|
* CLI mode print a paste-ready hint + exit 1, and library callers (Minion
|
|
* handlers) bubble it back as a structured job failure.
|
|
*/
|
|
export class EmbeddingDisabledError extends Error {
|
|
constructor(message: string) {
|
|
super(message);
|
|
this.name = 'EmbeddingDisabledError';
|
|
}
|
|
}
|
|
|
|
export function assertEmbeddingEnabled(cfg: { embedding_disabled?: boolean } | null): void {
|
|
if (cfg?.embedding_disabled) {
|
|
throw new EmbeddingDisabledError(
|
|
'This brain was initialized with `--no-embedding` (deferred setup).\n' +
|
|
'Configure an embedding provider before running embed / import:\n' +
|
|
' gbrain config set embedding_model <provider>:<model>\n' +
|
|
' gbrain config set embedding_dimensions <N>\n' +
|
|
' gbrain init --force --embedding-model <provider>:<model> # re-init to size schema\n',
|
|
);
|
|
}
|
|
}
|
|
|
|
export interface ColumnDimResult {
|
|
/** Whether the `content_chunks.embedding` column exists. False on a fresh brain. */
|
|
exists: boolean;
|
|
/** Parsed `vector(N)` dimension if known. null when the column doesn't exist or the type isn't vector. */
|
|
dims: number | null;
|
|
}
|
|
|
|
/**
|
|
* Read the actual dimension of `content_chunks.embedding` from the engine.
|
|
*
|
|
* Uses information_schema + a vector-specific catalog query. Returns
|
|
* { exists: false, dims: null } on a fresh brain that doesn't have the
|
|
* column yet. Returns { exists: true, dims: null } on a brain whose
|
|
* column type isn't `vector` (shouldn't happen but defensive).
|
|
*/
|
|
export async function readContentChunksEmbeddingDim(engine: BrainEngine): Promise<ColumnDimResult> {
|
|
// Probe column existence first to avoid noisy errors on fresh brains.
|
|
const existsRows = await engine.executeRaw<{ exists: boolean }>(
|
|
`SELECT EXISTS (
|
|
SELECT 1 FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = 'content_chunks'
|
|
AND column_name = 'embedding'
|
|
) AS exists`,
|
|
);
|
|
const exists = !!existsRows?.[0]?.exists;
|
|
if (!exists) return { exists: false, dims: null };
|
|
|
|
// pgvector stores dim in pg_type.typmod when atttypmod is set; format_type
|
|
// returns the human-readable `vector(N)`. We parse N out of that.
|
|
const formatRows = await engine.executeRaw<{ formatted: string | null }>(
|
|
`SELECT format_type(a.atttypid, a.atttypmod) AS formatted
|
|
FROM pg_attribute a
|
|
JOIN pg_class c ON c.oid = a.attrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.relname = 'content_chunks'
|
|
AND a.attname = 'embedding'
|
|
AND NOT a.attisdropped`,
|
|
);
|
|
const formatted = formatRows?.[0]?.formatted ?? null;
|
|
if (!formatted) return { exists: true, dims: null };
|
|
|
|
const m = formatted.match(/vector\((\d+)\)/i);
|
|
return { exists: true, dims: m ? parseInt(m[1], 10) : null };
|
|
}
|
|
|
|
/**
|
|
* Build the human-readable recipe printed when an existing brain's column
|
|
* dim doesn't match the requested dim.
|
|
*
|
|
* v0.37 fix wave (Lane D.1): branches on engine kind because the recipes
|
|
* are fundamentally different:
|
|
*
|
|
* - **PGLite** has no native pgvector extension (the WASM build can't
|
|
* `ALTER COLUMN TYPE vector(N)`), so the only path is wipe-and-reinit
|
|
* via `gbrain init --pglite --embedding-model X --embedding-dimensions N`.
|
|
* The recipe derives the active database path so users don't paste a
|
|
* stale literal that ignores `GBRAIN_HOME` / `--path` / their config.
|
|
* - **Postgres** keeps the existing four-step SQL recipe.
|
|
*
|
|
* The old recipe pointed at `gbrain config set embedding_model X` which
|
|
* is a no-op for the embed pipeline (the embed gateway reads file plane,
|
|
* not DB plane). After Lane C.2 that command refuses; the recipe now
|
|
* points at the actual fix path.
|
|
*/
|
|
export interface EmbeddingMismatchOpts {
|
|
currentDims: number;
|
|
requestedDims: number;
|
|
requestedModel?: string;
|
|
source?: 'init' | 'doctor' | 'embed';
|
|
/**
|
|
* PGLite vs Postgres branching. Required so the recipe matches the
|
|
* brain's actual engine. Pre-v0.37 default was 'postgres' (the SQL
|
|
* recipe), which produced the wrong recipe for the default install
|
|
* on PGLite.
|
|
*/
|
|
engineKind: 'pglite' | 'postgres';
|
|
/**
|
|
* Active PGLite database path. Used only for the PGLite branch; if
|
|
* omitted, falls back to the default `gbrainPath('brain.pglite')`.
|
|
* Resolving at the call site is preferred because the caller knows
|
|
* about `--path` flags and `GBRAIN_HOME` overrides.
|
|
*/
|
|
databasePath?: string;
|
|
}
|
|
|
|
export function embeddingMismatchMessage(opts: EmbeddingMismatchOpts): string {
|
|
const { currentDims, requestedDims, requestedModel, source, engineKind, databasePath } = opts;
|
|
const header = source === 'doctor'
|
|
? `Embedding dimension mismatch detected.`
|
|
: `Refusing to silently re-template existing brain.`;
|
|
|
|
if (engineKind === 'pglite') {
|
|
const activePath = databasePath ?? gbrainPath('brain.pglite');
|
|
const modelArg = requestedModel ? ` --embedding-model ${requestedModel}` : '';
|
|
const lines = [
|
|
header,
|
|
``,
|
|
` Existing column: vector(${currentDims})`,
|
|
` Requested: vector(${requestedDims})${requestedModel ? ` (${requestedModel})` : ''}`,
|
|
``,
|
|
`Switching dims is destructive: it drops every embedding in your brain.`,
|
|
`PGLite cannot ALTER vector column types (pgvector ships as embedded WASM,`,
|
|
`not a native extension). Wipe-and-reinit is the only path.`,
|
|
``,
|
|
`Recommended (one command):`,
|
|
``,
|
|
` gbrain reinit-pglite${modelArg} --embedding-dimensions ${requestedDims}`,
|
|
``,
|
|
`Or by hand:`,
|
|
``,
|
|
` mv ${activePath} ${activePath}.bak`,
|
|
` gbrain init --pglite${modelArg} --embedding-dimensions ${requestedDims}`,
|
|
` gbrain sync # re-imports your brain repo from disk`,
|
|
` gbrain embed --stale`,
|
|
``,
|
|
`Full guide: docs/embedding-migrations.md`,
|
|
];
|
|
return lines.join('\n');
|
|
}
|
|
|
|
// Postgres branch — preserve the existing SQL recipe.
|
|
const supportsHnsw = requestedDims <= PGVECTOR_HNSW_VECTOR_MAX_DIMS;
|
|
const reindexLine = supportsHnsw
|
|
? `CREATE INDEX IF NOT EXISTS idx_chunks_embedding\n ON content_chunks USING hnsw (embedding vector_cosine_ops);`
|
|
: `-- Skip reindex. dims=${requestedDims} exceeds pgvector's HNSW cap of ${PGVECTOR_HNSW_VECTOR_MAX_DIMS};\n-- searchVector falls back to exact scan.`;
|
|
|
|
const modelArg = requestedModel ? ` --embedding-model ${requestedModel}` : '';
|
|
const lines = [
|
|
header,
|
|
``,
|
|
` Existing column: vector(${currentDims})`,
|
|
` Requested: vector(${requestedDims})${requestedModel ? ` (${requestedModel})` : ''}`,
|
|
``,
|
|
`Switching dims is destructive: it drops every embedding in your brain and`,
|
|
`requires a full re-embed (potentially hours and $1-100 in API calls).`,
|
|
``,
|
|
`Recipe (run against your Postgres brain):`,
|
|
``,
|
|
` BEGIN;`,
|
|
` DROP INDEX IF EXISTS idx_chunks_embedding;`,
|
|
` -- NULL embeddings BEFORE the alter: pgvector refuses to cast existing`,
|
|
` -- vectors across dimensions and aborts the transaction. NULLs cast fine.`,
|
|
` UPDATE content_chunks SET embedding = NULL, embedded_at = NULL;`,
|
|
` ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(${requestedDims});`,
|
|
` ${reindexLine.split('\n').join('\n ')}`,
|
|
` COMMIT;`,
|
|
``,
|
|
`Then re-init config (file plane is canonical post-v0.37):`,
|
|
` gbrain init --supabase${modelArg} --embedding-dimensions ${requestedDims}`,
|
|
` gbrain embed --stale`,
|
|
``,
|
|
`Full guide: docs/embedding-migrations.md`,
|
|
];
|
|
return lines.join('\n');
|
|
}
|
|
|
|
// ============================================================================
|
|
// v0.37.x — preflight schema-dim resolution (D11 + D12)
|
|
//
|
|
// Resolves the dim that the PGLite schema substitution will use BEFORE
|
|
// `engine.initSchema()` runs, so init can't create a column whose width
|
|
// disagrees with the gateway-resolved provider. Pure functions, no I/O —
|
|
// init calls them, exits early on error, never writes anything to disk in
|
|
// the failure path. The post-init invariant assertion stays as a regression
|
|
// guardrail; after this resolver lands it can never fire.
|
|
// ============================================================================
|
|
|
|
/** Tagged-union result of preflight resolution. */
|
|
export type ResolveSchemaDimResult =
|
|
| { ok: true; dim: number; model: string; provider: string; recipeDefault: number }
|
|
| { ok: false; error: string };
|
|
|
|
/** Inputs for the embedding-tier preflight resolver. */
|
|
export interface ResolveSchemaEmbeddingDimOpts {
|
|
/** `provider:model` string (e.g. `openai:text-embedding-3-large`). Required. */
|
|
embedding_model: string;
|
|
/** Explicit override (Matryoshka step, custom dim). Optional. */
|
|
embedding_dimensions?: number;
|
|
}
|
|
|
|
/**
|
|
* Resolve the dim that will land in `content_chunks.embedding`'s vector(N)
|
|
* column. Caller is `init.ts:initPGLite` before any DB write happens.
|
|
*
|
|
* Validations:
|
|
* 1. `embedding_model` parses as `provider:model`.
|
|
* 2. Provider is a known recipe.
|
|
* 3. Recipe declares an `embedding` touchpoint.
|
|
* 4. Resolved dim is a positive integer.
|
|
* 5. Resolved dim ≤ PGVECTOR_COLUMN_MAX_DIMS (16000).
|
|
* 6. If user passed `embedding_dimensions`, it either matches
|
|
* `recipe.touchpoints.embedding.default_dims` OR is in the recipe's
|
|
* `dims_options` list (Matryoshka providers). Otherwise reject — the
|
|
* user picked a model that doesn't support custom dims.
|
|
*/
|
|
export function resolveSchemaEmbeddingDim(opts: ResolveSchemaEmbeddingDimOpts): ResolveSchemaDimResult {
|
|
try {
|
|
const { recipe, parsed } = resolveRecipe(opts.embedding_model);
|
|
const tp = recipe.touchpoints.embedding;
|
|
if (!tp) {
|
|
return {
|
|
ok: false,
|
|
error:
|
|
`Provider "${recipe.id}" does not offer embedding models. ` +
|
|
`Pick a recipe with an embedding touchpoint (gbrain providers list).`,
|
|
};
|
|
}
|
|
return validateDimAgainstTouchpoint(parsed.modelId, recipe, tp.default_dims, tp.dims_options, opts.embedding_dimensions);
|
|
} catch (err) {
|
|
return { ok: false, error: err instanceof AIConfigError ? err.message : String(err) };
|
|
}
|
|
}
|
|
|
|
/** Inputs for the multimodal-tier preflight resolver (D12). */
|
|
export interface ResolveSchemaMultimodalDimOpts {
|
|
/** `provider:model` string for the multimodal endpoint. Required. */
|
|
embedding_multimodal_model: string;
|
|
/** Explicit override. Optional. */
|
|
embedding_multimodal_dimensions?: number;
|
|
}
|
|
|
|
/**
|
|
* Resolve the dim that will land in `content_chunks.embedding_multimodal`'s
|
|
* vector(N) column. Mirrors `resolveSchemaEmbeddingDim` but also checks the
|
|
* recipe-level `supports_multimodal` flag and the per-model
|
|
* `multimodal_models` allow-list (some recipes like Voyage mix text-only
|
|
* and multimodal models in one embedding touchpoint).
|
|
*/
|
|
export function resolveSchemaMultimodalDim(opts: ResolveSchemaMultimodalDimOpts): ResolveSchemaDimResult {
|
|
try {
|
|
const { recipe, parsed } = resolveRecipe(opts.embedding_multimodal_model);
|
|
const tp = recipe.touchpoints.embedding;
|
|
if (!tp) {
|
|
return {
|
|
ok: false,
|
|
error:
|
|
`Provider "${recipe.id}" does not offer embedding models. ` +
|
|
`Pick a recipe with an embedding touchpoint that supports multimodal input.`,
|
|
};
|
|
}
|
|
if (!tp.supports_multimodal) {
|
|
return {
|
|
ok: false,
|
|
error:
|
|
`Provider "${recipe.id}" does not support multimodal embeddings. ` +
|
|
`Configured recipes that do: voyage (voyage-multimodal-3). ` +
|
|
`Run \`gbrain providers list\` to see touchpoint coverage.`,
|
|
};
|
|
}
|
|
if (tp.multimodal_models && !tp.multimodal_models.includes(parsed.modelId)) {
|
|
return {
|
|
ok: false,
|
|
error:
|
|
`Model "${parsed.modelId}" is not in provider "${recipe.id}"'s multimodal allow-list ` +
|
|
`(allowed: ${tp.multimodal_models.join(', ')}). ` +
|
|
`Pick a multimodal-capable model from this provider.`,
|
|
};
|
|
}
|
|
return validateDimAgainstTouchpoint(parsed.modelId, recipe, tp.default_dims, tp.dims_options, opts.embedding_multimodal_dimensions);
|
|
} catch (err) {
|
|
return { ok: false, error: err instanceof AIConfigError ? err.message : String(err) };
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Shared validation of a requested dim against a recipe touchpoint's
|
|
* declared dims, including provider-specific Matryoshka allow-lists.
|
|
*
|
|
* Recipes (`src/core/ai/recipes/*.ts`) declare `default_dims` per touchpoint
|
|
* but do NOT generally encode Matryoshka steps as `dims_options`. The
|
|
* per-provider valid-dim allow-lists live in `src/core/ai/dims.ts`:
|
|
* - `VOYAGE_VALID_OUTPUT_DIMS` (256/512/1024/2048) for flexible Voyage models
|
|
* - `ZEROENTROPY_VALID_DIMS` (2560/1280/640/320/160/80/40) for ZE zembed-1
|
|
* - OpenAI text-embedding-3-* accepts ANY positive integer up to the
|
|
* model's native size (1536 small / 3072 large)
|
|
*
|
|
* Validation order:
|
|
* 1. recipe-declared `dims_options` (highest precedence — recipe author
|
|
* knows their backend)
|
|
* 2. provider-specific dim.ts allow-lists (for known Matryoshka providers)
|
|
* 3. fall through to "this model only emits default_dims" rejection
|
|
*/
|
|
function validateDimAgainstTouchpoint(
|
|
modelId: string,
|
|
recipe: Recipe,
|
|
defaultDims: number,
|
|
dimsOptions: number[] | undefined,
|
|
requestedDims: number | undefined,
|
|
): ResolveSchemaDimResult {
|
|
const dim = requestedDims ?? defaultDims;
|
|
|
|
if (!Number.isInteger(dim) || dim <= 0) {
|
|
return {
|
|
ok: false,
|
|
error: `Embedding dimensions must be a positive integer; got ${JSON.stringify(dim)}.`,
|
|
};
|
|
}
|
|
if (dim > PGVECTOR_COLUMN_MAX_DIMS) {
|
|
return {
|
|
ok: false,
|
|
error:
|
|
`Embedding dimensions ${dim} exceed pgvector's column cap of ${PGVECTOR_COLUMN_MAX_DIMS}. ` +
|
|
`Pick a model that returns ≤${PGVECTOR_COLUMN_MAX_DIMS} dims.`,
|
|
};
|
|
}
|
|
|
|
if (requestedDims !== undefined && requestedDims !== defaultDims) {
|
|
// User asked for a non-default dim. Walk the precedence chain.
|
|
const customDimOk = isCustomDimValidForProvider(recipe, modelId, requestedDims, dimsOptions);
|
|
if (!customDimOk.valid) {
|
|
return { ok: false, error: customDimOk.error };
|
|
}
|
|
}
|
|
|
|
return {
|
|
ok: true,
|
|
dim,
|
|
model: `${recipe.id}:${modelId}`,
|
|
provider: recipe.id,
|
|
recipeDefault: defaultDims,
|
|
};
|
|
}
|
|
|
|
interface CustomDimCheck {
|
|
valid: boolean;
|
|
error: string;
|
|
}
|
|
|
|
function isCustomDimValidForProvider(
|
|
recipe: Recipe,
|
|
modelId: string,
|
|
requestedDims: number,
|
|
dimsOptions: number[] | undefined,
|
|
): CustomDimCheck {
|
|
// Tier 1: recipe-declared dims_options.
|
|
if (dimsOptions && dimsOptions.length > 0) {
|
|
if (dimsOptions.includes(requestedDims)) return { valid: true, error: '' };
|
|
return {
|
|
valid: false,
|
|
error:
|
|
`Provider "${recipe.id}" model "${modelId}" rejects custom dimensions ${requestedDims} ` +
|
|
`(allowed: ${dimsOptions.join(', ')}).`,
|
|
};
|
|
}
|
|
|
|
// Tier 2: provider-specific Matryoshka allow-lists.
|
|
if (recipe.id === 'voyage' && supportsVoyageOutputDimension(modelId)) {
|
|
if (isValidVoyageOutputDim(requestedDims)) return { valid: true, error: '' };
|
|
return {
|
|
valid: false,
|
|
error:
|
|
`Voyage model "${modelId}" rejects custom dimensions ${requestedDims} ` +
|
|
`(allowed: ${VOYAGE_VALID_OUTPUT_DIMS.join(', ')}).`,
|
|
};
|
|
}
|
|
if (recipe.id === 'zeroentropyai' && supportsZeroEntropyDimension(modelId)) {
|
|
if (isValidZeroEntropyDim(requestedDims)) return { valid: true, error: '' };
|
|
return {
|
|
valid: false,
|
|
error:
|
|
`ZeroEntropy model "${modelId}" does not support custom dimensions ${requestedDims} ` +
|
|
`(allowed: ${ZEROENTROPY_VALID_DIMS.join(', ')}).`,
|
|
};
|
|
}
|
|
if (recipe.id === 'openai' && isOpenAITextEmbedding3Model(modelId)) {
|
|
if (isValidOpenAITextEmbedding3Dim(modelId, requestedDims)) return { valid: true, error: '' };
|
|
const maxDim = maxOpenAITextEmbedding3Dim(modelId);
|
|
return {
|
|
valid: false,
|
|
error:
|
|
`OpenAI ${modelId} accepts dimensions 1..${maxDim}, got ${requestedDims}.`,
|
|
};
|
|
}
|
|
|
|
// Tier 3: provider not known to support custom dims at all.
|
|
return {
|
|
valid: false,
|
|
error:
|
|
`Provider "${recipe.id}" model "${modelId}" does not support custom dimensions ${requestedDims} ` +
|
|
`(this model only emits its default vector size). ` +
|
|
`Either drop --embedding-dimensions or pick a Matryoshka-aware model.`,
|
|
};
|
|
}
|
|
|
|
// ───────────────────────────────────────────────────────────────────────
|
|
// v0.41.15.0 (T5 + T6) — facts.embedding column drift detection.
|
|
//
|
|
// Migration v40 reads `config.embedding_dimensions` at MIGRATION time and
|
|
// creates `facts.embedding` as `halfvec(N)` (or `vector(N)` on pgvector
|
|
// < 0.7). If the user later changes embedding provider without re-running
|
|
// migrations, the column type stays at the old N and the first insert
|
|
// dies with an opaque pgvector error. Two surfaces close the gap:
|
|
//
|
|
// 1. `readFactsEmbeddingDim(engine)` — column-type probe used by the
|
|
// `gbrain doctor` `embedding_dim_mismatch` check to surface drift.
|
|
// 2. `assertFactsEmbeddingDimMatchesConfig(engine)` — preflight thrown
|
|
// at the top of every fact-writing path (extract-conversation-facts
|
|
// startup, the cycle extract_facts phase, facts:absorb op). Result
|
|
// cached per process so the SELECT runs once per startup.
|
|
//
|
|
// Both helpers handle the `vector(N)` AND `halfvec(N)` shapes because
|
|
// migration v40 falls back to `vector` on pgvector < 0.7 (codex #19).
|
|
// ───────────────────────────────────────────────────────────────────────
|
|
|
|
/**
|
|
* Discriminated result of `readFactsEmbeddingDim`. Carries the column
|
|
* type (vector vs halfvec) alongside the dim so callers can render
|
|
* paste-ready ALTER recipes that target the right type + opclass.
|
|
*/
|
|
export interface FactsColumnDimResult {
|
|
/** Whether the `facts.embedding` column exists (false on pre-v40 brains). */
|
|
exists: boolean;
|
|
/** Parsed dim from format_type, or null when the column doesn't exist. */
|
|
dims: number | null;
|
|
/** Column type — `halfvec` (pgvector >=0.7) or `vector` (older). */
|
|
columnType: 'halfvec' | 'vector' | null;
|
|
}
|
|
|
|
/**
|
|
* Read the actual width + type of `facts.embedding`. Mirrors
|
|
* `readContentChunksEmbeddingDim` but for the facts table; covers
|
|
* BOTH `vector(N)` and `halfvec(N)` shapes per codex #19.
|
|
*
|
|
* Returns `{exists: false, dims: null, columnType: null}` on pre-v40
|
|
* brains (facts table absent) and a fully-populated result otherwise.
|
|
*/
|
|
export async function readFactsEmbeddingDim(engine: BrainEngine): Promise<FactsColumnDimResult> {
|
|
// Probe the embedding column directly. The facts table itself may
|
|
// exist on a partial-v40 brain but without the embedding column on
|
|
// very-old upgrade chains; both null branches yield exists:false.
|
|
const existsRows = await engine.executeRaw<{ exists: boolean }>(
|
|
`SELECT EXISTS (
|
|
SELECT 1 FROM information_schema.columns
|
|
WHERE table_schema = 'public'
|
|
AND table_name = 'facts'
|
|
AND column_name = 'embedding'
|
|
) AS exists`,
|
|
);
|
|
const exists = !!existsRows?.[0]?.exists;
|
|
if (!exists) return { exists: false, dims: null, columnType: null };
|
|
|
|
const formatRows = await engine.executeRaw<{ formatted: string | null }>(
|
|
`SELECT format_type(a.atttypid, a.atttypmod) AS formatted
|
|
FROM pg_attribute a
|
|
JOIN pg_class c ON c.oid = a.attrelid
|
|
JOIN pg_namespace n ON n.oid = c.relnamespace
|
|
WHERE n.nspname = 'public'
|
|
AND c.relname = 'facts'
|
|
AND a.attname = 'embedding'
|
|
AND NOT a.attisdropped`,
|
|
);
|
|
const formatted = formatRows?.[0]?.formatted ?? null;
|
|
if (!formatted) return { exists: true, dims: null, columnType: null };
|
|
|
|
// Order matters: try `halfvec(N)` BEFORE `vector(N)` because the
|
|
// half-vector regex would otherwise be shadowed by the generic
|
|
// `vector` match (halfvec is a separate pgvector type that also
|
|
// contains "vec" as a substring).
|
|
const halfMatch = formatted.match(/halfvec\((\d+)\)/i);
|
|
if (halfMatch) {
|
|
return { exists: true, dims: parseInt(halfMatch[1], 10), columnType: 'halfvec' };
|
|
}
|
|
const vecMatch = formatted.match(/vector\((\d+)\)/i);
|
|
if (vecMatch) {
|
|
return { exists: true, dims: parseInt(vecMatch[1], 10), columnType: 'vector' };
|
|
}
|
|
return { exists: true, dims: null, columnType: null };
|
|
}
|
|
|
|
/** Tagged error thrown by `assertFactsEmbeddingDimMatchesConfig` on drift. */
|
|
export class FactsEmbeddingDimMismatchError extends Error {
|
|
readonly tag = 'FACTS_EMBEDDING_DIM_MISMATCH' as const;
|
|
constructor(
|
|
message: string,
|
|
public readonly columnDims: number,
|
|
public readonly configuredDims: number,
|
|
public readonly columnType: 'halfvec' | 'vector',
|
|
) {
|
|
super(message);
|
|
this.name = 'FactsEmbeddingDimMismatchError';
|
|
}
|
|
}
|
|
|
|
/**
|
|
* v0.41.15.0 (D15): build the paste-ready ALTER recipe for facts dim
|
|
* drift (codex #18). Postgres-only — facts.embedding ALTER on PGLite
|
|
* is not supported by the embedded pgvector WASM. The recipe is the
|
|
* full DROP INDEX + ALTER USING + CREATE INDEX flow, NOT a bare
|
|
* `ALTER TYPE ... REINDEX` (which doesn't actually rewrite the index
|
|
* after a type change).
|
|
*/
|
|
export function buildFactsAlterRecipe(
|
|
columnDims: number,
|
|
configuredDims: number,
|
|
columnType: 'halfvec' | 'vector',
|
|
): string {
|
|
const opclass = columnType === 'halfvec' ? 'halfvec_cosine_ops' : 'vector_cosine_ops';
|
|
const targetType = columnType === 'halfvec' ? `halfvec(${configuredDims})` : `vector(${configuredDims})`;
|
|
const dimsChanged = columnDims !== configuredDims;
|
|
return [
|
|
`-- ALTER ${columnType}(${columnDims}) → ${columnType}(${configuredDims}) on indexed column.`,
|
|
`-- HOLD a maintenance window: this rewrites every row's embedding.`,
|
|
`-- Coordinate with any active extract-conversation-facts backfill.`,
|
|
`DROP INDEX IF EXISTS idx_facts_embedding_hnsw;`,
|
|
// Same-dim type swaps (halfvec <-> vector) cast row data losslessly and
|
|
// MUST keep it. Cross-dimension changes are different: pgvector refuses
|
|
// to cast existing vectors across dimensions ("expected N dimensions,
|
|
// not M") and aborts the transaction — and the old-space vectors are
|
|
// unusable at the new width anyway. NULL them first; the facts pipeline
|
|
// re-embeds on the next write.
|
|
...(dimsChanged
|
|
? [
|
|
`-- Dimension change: NULL embeddings BEFORE the alter — pgvector`,
|
|
`-- refuses cross-dimension casts and aborts the transaction.`,
|
|
`UPDATE facts SET embedding = NULL;`,
|
|
]
|
|
: []),
|
|
`ALTER TABLE facts ALTER COLUMN embedding TYPE ${targetType}`,
|
|
` USING embedding::${targetType};`,
|
|
`CREATE INDEX idx_facts_embedding_hnsw`,
|
|
` ON facts USING hnsw (embedding ${opclass})`,
|
|
` WHERE embedding IS NOT NULL AND expired_at IS NULL;`,
|
|
].join('\n');
|
|
}
|
|
|
|
/**
|
|
* Per-process cache for `assertFactsEmbeddingDimMatchesConfig`. The
|
|
* probe is a cheap SELECT but runs at the top of every fact-writing
|
|
* call site; caching keeps the cost off the hot path. The cache
|
|
* stores the engine's `kind + a synthetic instance marker` so a fresh
|
|
* engine connection in the same process re-probes. Test seam below
|
|
* clears the cache between cases.
|
|
*/
|
|
const _factsDimCheckCache = new WeakMap<BrainEngine, { ok: true } | { err: FactsEmbeddingDimMismatchError }>();
|
|
|
|
/** Test seam: clear the per-process facts-dim cache. */
|
|
export function _resetFactsDimCheckCacheForTest(): void {
|
|
// WeakMap has no clear() — but tests can pass fresh engine instances
|
|
// to get fresh probes. This noop helper documents the intent.
|
|
}
|
|
|
|
/**
|
|
* Preflight check: throws FactsEmbeddingDimMismatchError when the
|
|
* configured embedding dimensions don't match the facts.embedding
|
|
* column width. Called at the top of every fact-writing path so users
|
|
* see a clear paste-ready ALTER hint BEFORE the first insert (which
|
|
* would otherwise fail with the opaque pgvector "expected vector(N),
|
|
* got vector(M)" error).
|
|
*
|
|
* Caches the result per engine instance for the process lifetime —
|
|
* one SELECT at startup, zero per-page cost. Successful probes return
|
|
* void; mismatches throw the tagged class.
|
|
*
|
|
* Skipped on:
|
|
* - PGLite engines (the facts table on PGLite uses the same
|
|
* embedded pgvector that migrated content_chunks; if dim drift
|
|
* exists, the `--no-embedding` runtime guard already covers it).
|
|
* - Brains without the facts.embedding column (pre-v40 install
|
|
* chains; the migration that creates the column hasn't run, so
|
|
* no possible drift exists).
|
|
* - Brains with no `embedding_dimensions` config (fresh installs;
|
|
* gateway defaults take over and align with migration defaults).
|
|
*/
|
|
export async function assertFactsEmbeddingDimMatchesConfig(engine: BrainEngine): Promise<void> {
|
|
const cached = _factsDimCheckCache.get(engine);
|
|
if (cached) {
|
|
if ('err' in cached) throw cached.err;
|
|
return;
|
|
}
|
|
|
|
// PGLite + non-Postgres engines: skip. (PGLite ships a single
|
|
// pgvector version; the column and config are wired together at
|
|
// initSchema time, so the bug class doesn't apply.)
|
|
if (engine.kind !== 'postgres') {
|
|
_factsDimCheckCache.set(engine, { ok: true });
|
|
return;
|
|
}
|
|
|
|
const col = await readFactsEmbeddingDim(engine);
|
|
if (!col.exists || col.dims === null || col.columnType === null) {
|
|
// No facts.embedding column → migration v40 hasn't run yet → no
|
|
// possible drift. Cache as ok; the migration runner will pick up
|
|
// the right dims from config when it lands.
|
|
_factsDimCheckCache.set(engine, { ok: true });
|
|
return;
|
|
}
|
|
|
|
// Read the configured dims directly from the gateway. This matches
|
|
// what gateway.embed() will produce — single source of truth.
|
|
let configuredDims: number;
|
|
try {
|
|
// Lazy-import to avoid the gateway pulling in at module-load
|
|
// time (matters for tests that mock the gateway).
|
|
const { getEmbeddingDimensions } = await import('./ai/gateway.ts');
|
|
configuredDims = getEmbeddingDimensions();
|
|
} catch {
|
|
// Gateway not configured (rare; usually means the brain hasn't
|
|
// been initialized yet). Skip the check — the fact-writing path
|
|
// will fail with a clearer "gateway not configured" error.
|
|
_factsDimCheckCache.set(engine, { ok: true });
|
|
return;
|
|
}
|
|
|
|
if (col.dims === configuredDims) {
|
|
_factsDimCheckCache.set(engine, { ok: true });
|
|
return;
|
|
}
|
|
|
|
const recipe = buildFactsAlterRecipe(col.dims, configuredDims, col.columnType);
|
|
const message = [
|
|
`facts.embedding is ${col.columnType}(${col.dims}) but configured embedding_dimensions is ${configuredDims}.`,
|
|
`Refusing to attempt fact inserts that would fail with an opaque pgvector error.`,
|
|
``,
|
|
`Paste-ready fix (review carefully — this rewrites the facts table):`,
|
|
``,
|
|
recipe,
|
|
``,
|
|
`Or run \`gbrain doctor --json\` for the full diagnostic + fix surface.`,
|
|
].join('\n');
|
|
const err = new FactsEmbeddingDimMismatchError(
|
|
message,
|
|
col.dims,
|
|
configuredDims,
|
|
col.columnType,
|
|
);
|
|
_factsDimCheckCache.set(engine, { err });
|
|
throw err;
|
|
}
|