From 7c27fa129bf8e7e2b031e7c7c8ac144b3ecd9b12 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 12 Jun 2026 06:05:34 -0700 Subject: [PATCH] =?UTF-8?q?v0.42.41.0=20fix:=20triage=20wave=20=E2=80=94?= =?UTF-8?q?=206=20data-loss/availability=20fixes=20+=209=20community=20PRs?= =?UTF-8?q?=20(#2128)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(oauth): default omitted authorize scope to client's full grant When a client omits `scope` on /authorize, the authorize() grant computed `(params.scopes || []).filter(...)` → the empty set. That empty grant was written to oauth_codes and propagated into the access AND refresh tokens, so every request failed `insufficient_scope` even though the client was registered with e.g. `read write`. Because refresh inherits the stored grant, it never self-healed — reconnecting just minted another empty-scoped token. Some MCP connectors (observed with Claude Desktop) omit `scope` on /authorize, so they hit this on every connection. Fix: when no scope is requested, default to the client's full registered scope (RFC 6749 §3.3 permits a server default). This mirrors exchangeClientCredentials, which already does `requestedScope ? ... : allowedScopes`. The result is still clamped to the allowed set, so an explicit over-broad request cannot escalate. Adds test/oauth-authorize-scope-default.test.ts covering: omitted/empty → inherits full grant; explicit subset honored; clamp preserved (over-broad and disallowed-only requests cannot escalate or trigger inheritance). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(sync): skip Python venv/ in the code walker collectSyncableFiles (first-sync walker) and the incremental PRUNE_DIR_NAMES set skipped node_modules but not Python venv/. On a Python repo the walker descended into venv/ (thousands of files); the resulting slug collisions crashed putPage's INSERT ... ON CONFLICT ... RETURNING with "undefined is not an object (evaluating 'row.deleted_at')". Add `venv` alongside node_modules in both the import.ts inline skip and PRUNE_DIR_NAMES. venv is the Python equivalent of node_modules. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(gateway): carry asymmetric input_type across the AI SDK to the wire body (#1400) dimsProviderOptions() threads input_type ('query' | 'document') into providerOptions.openaiCompatible for asymmetric models (ZE zembed-1, Voyage v3+), but the AI SDK's openai-compatible adapter validates providerOptions against a fixed schema and silently drops the field before building the HTTP body. Every embedQuery() was therefore encoded document-side: the ZE shim's hard default fired ('document'), Voyage and local openai-compat servers got no input_type at all, and asymmetric retrieval silently collapsed toward surface-token overlap — while the providerOptions-level contract test stayed green. Fix: an AsyncLocalStorage (same pattern as __budgetStore) populated in embedSubBatch() only when providerOptions actually threads an input_type, read at body-rewrite time by the fetch shims: - zeroEntropyCompatFetch: recovers the threaded value; document default preserved for ingest paths. - voyageCompatFetch: opt-in like the dims.ts Voyage branch — inject only when threaded; the field stays off the wire otherwise. - NEW openAICompatAsymmetricFetch: fallthrough default for every other openai-compatible recipe (llama-server, litellm, ollama, ...) — the canonical local/proxy paths for asymmetric models. Strict pass-through when nothing was threaded, so symmetric deployments see zero wire change; recipes with their own compat fetch (azure) keep it via the compat.fetch ?? precedence. KNOBS_HASH_VERSION bumped 10→11: cached query_cache rows were keyed on document-side query vectors; pre-fix rows must not be served to post-fix lookups (same convention as the v=3 embedding-provider bump). One-time global cold-miss on upgrade; refills within cache.ttl_seconds. Tests: test/embed-input-type-wire.test.ts runs the REAL SDK transport with a mocked global fetch and asserts on the outbound body — the only layer where this regression is observable. Covers ZE hosted, llama-server, litellm, ollama (query + document sides) and pins the pass-through for non-asymmetric models and Voyage's opt-in shape. 4 of the original 7 assertions fail on master, proving the pin. One structural pin in test/ai/zeroentropy-compat-fetch.test.ts updated to the new line shape (same semantic); KEY_FILES.md gateway.ts entry updated to the new truth. Supersedes #1400 (closed unmerged) — same ALS mechanism, extended to Voyage + all openai-compatible recipes. Credit to @billy-armstrong for the original diagnosis. Co-Authored-By: Claude Fable 5 * fix(sync): honor .gitignore in code walk; prune vendor/dist/build collectSyncableFiles (the full-sync / dry-run enumerator) reimplemented its own directory skip list inline (node_modules || ops), bypassing the canonical pruneDir gate and ignoring .gitignore entirely. On a Laravel/PHP repo this descended into vendor/ (~50k Composer files), storage/, and public/build/, trying to import 52k dependency/build files and flooding the index with library internals (a 35-min sync that never finished, killed by the watchdog at 3%). - collectSyncableFiles now enumerates via `git ls-files --cached --others --exclude-standard` when dir is a git work tree, so the walk honors .gitignore (tracked + untracked-not-ignored). Falls back to the FS walk for non-git dirs. EroLab: 52164 -> 1028 files. - The FS fallback now prunes through the canonical pruneDir() instead of a drifted inline list, so the two skip lists can't diverge again. - PRUNE_DIR_NAMES gains vendor/dist/build (dependency + build-output trees). Addresses #1483 (.gbrainignore), #1159 (--respect-gitignore), and the maintainer's #1942 vendor/dist/build prune. Walker regression suites (sync-walker-symlink, brain-writer-walk-prune, sync, sync-walker-submodule) green: 90 pass. * fix(config): ignore DATABASE_URL auto-loaded from cwd .env (#427) Bun merges .env files from the process cwd into process.env before any user code runs. loadConfig() prefers env DATABASE_URL over ~/.gbrain/config.json, so any gbrain invocation from inside a web-app checkout silently retargets the brain at that app's database — reads go to the wrong DB and apply-migrations can write gbrain's schema into a production app database (#427). effectiveEnvDatabaseUrl() re-parses the .env files Bun auto-loads from cwd and treats a DATABASE_URL whose value matches one of them as file-origin: ignored, with a one-time stderr notice. GBRAIN_DATABASE_URL and genuinely exported DATABASE_URLs are honored unchanged, so the operator escape hatch and the e2e suite's env-provided URL keep working. Applied at loadConfig, getDbUrlSource (doctor parity), init --non-interactive, and migrate --to. Co-Authored-By: Claude Fable 5 * fix(cli): arm the disconnect hard-deadline at teardown entry, not before the op body The 10s force-exit timer in the shared-op dispatch was armed BEFORE the try block, so any op whose handler ran past 10s wall-clock was killed mid-flight with process.exit(0) and zero stdout. On a slow Postgres pooler (6-10s per fresh connection) a healthy `gbrain search` was force-exited every time — an empty 'success' indistinguishable from no results. The v0.42.20.0 exitCode honor can't help: a mid-op kill fires before any error path sets exitCode. Move the arming into the finally (teardown entry), matching the fall-through owner-disconnect site later in main(): the timer still bounds a hung drain/disconnect (the C13 contract) but can no longer kill a slow-but-progressing op. Verified on a transaction-pooler Supabase brain: search went from 0 bytes/exit 0 at 10s to real results at ~21s. * fix(import): stamp source_id on extracted call-graph edges importCodeFile built CodeEdgeInput rows without source_id, so every edge landed NULL. getCallersOf/getCalleesOf filter `AND source_id = ` whenever a worktree pin or --source is in play — NULL never matches, so scoped call-graph queries silently returned 0 rows on multi-source brains even though the edges existed (2,122 edges, 26 targeting the probed symbol, count 0 returned). One-line fix: carry the sourceId already in scope into the edge input. Existing NULL rows backfill with: UPDATE code_edges_symbol e SET source_id = p.source_id FROM content_chunks c JOIN pages p ON p.id = c.page_id WHERE c.id = e.from_chunk_id AND e.source_id IS NULL; (same for code_edges_chunk). Verified: code-callers returns 21 callers where it returned 0. * docs(migrations): NULL embeddings BEFORE the column-type alter The Postgres recipe ordered ALTER COLUMN TYPE vector(N) before the UPDATE that clears stale embeddings. pgvector refuses to cast existing vectors across dimensions ('expected 1024 dimensions, not 1536'), so the recipe as written aborts the transaction on any brain that has embeddings — which is every brain doing this migration. Swap the steps: NULLs cast fine. * fix: honor legacy token source grants in oauth * fix(cli): bound read-scope op handlers at 180s wallclock (pre-landing review) With the hard-deadline timer correctly scoped to teardown, a genuinely wedged read handler (hung pooler connection mid-query) would hang the CLI forever — the #1633 zombie class the old pre-try timer accidentally bounded at 10s. Reads now get a generous withTimeout (180s default, far above any healthy slow-pooler run; --timeout=Ns overrides; exit 124 with the teardown finally still draining + disconnecting). Writes/admin stay unbounded: a long import/embed must never be killed by a default. * fix(import): stamp unscoped edges 'default', matching the pages-table default Review catch: 'sourceId ?? null' fixed the scoped path but left the unscoped one (reindex --code without --source, importCodeFile callers without opts.sourceId) stranding edges at NULL while their pages land under the schema default (pages.source_id DEFAULT 'default') — so getCallersOf(sym, { sourceId: 'default' }) missed them. Same bug, other door. Fallback is now 'default'. * fix(core): runtime dim-migration recipe NULLs embeddings before the alter Review catch: the doc fix corrected docs/embedding-migrations.md, but embeddingMismatchMessage still PRINTED the broken order — ALTER before UPDATE ... SET embedding = NULL — and linked to the now-contradicting doc. pgvector refuses to cast existing vectors across dimensions, so the printed recipe aborted on any brain that has embeddings. Swap the steps and say why inline. * feat(migrate): v116 — backfill NULL edge source_id + index from_symbol_qualified 1. Backfill: edges written before the stamping fix sit at source_id=NULL and stay invisible to scoped call-graph queries until repaired. Derive each edge's source from its own from_chunk's page (pages.source_id is NOT NULL DEFAULT 'default'). Same SQL verified live on a 2,122-edge production brain. 2. Indexes: getCalleesOf filters both edge tables on from_symbol_qualified, which had no index — every callee lookup was a seq scan, amplified per-BFS-node by the recursive code walk. With NULL edges repaired, scoped walks actually expand, so the latent cost becomes real. Mirrored into src/schema.sql; schema-embedded.ts regenerated. * docs(migrations): align the rationale list with the corrected recipe order The 'Why we don't do this automatically' list still said alter-then-wipe; reorder to wipe-then-alter and replace the fragile 'step 3' numeric cross-reference with a name-based one. * test: regression coverage for edge source_id stamping, timer placement, recipe order - import-code-edges-source-id: scoped import stamps edges + scoped getCallersOf/getCalleesOf match (verified failing pre-fix), plus the unscoped-import case asserting 'default' stamping. - cli-force-exit-teardown-arming: structural pin — the hard-deadline timer arms inside the finally (teardown entry), never before the op body; daemon guard, unref, clearTimeout intact. - embedding-dim-check: recipe order pinned — UPDATE precedes ALTER so the printed SQL can't drift from docs/embedding-migrations.md again. * fix(cli): hard-exit after teardown on wallclock timeout; bound makeContext too Adversarial review, two findings on the new timeout path: 1. On timeout the finally drained, disconnected, then CLEARED the hard-deadline timer — removing the only backstop while the abandoned handler (withTimeout races, it does not cancel) can hold ref'd sockets/SDK timers that keep Bun's loop alive: 'timed out' printed, process immortal — the zombie class this branch exists to kill, resurrected through its own fix. The finally now exits explicitly after teardown completes on the timeout path. 2. makeContext does DB I/O (resolveSourceId) for EVERY op and sat outside any bound — a pooler wedge at context build hung reads, writes, and admin alike. It now shares the same wallclock bound. * fix(import): normalize edge source once — closes the '' door and the unscoped chunk fan-out Adversarial review: txOpts used truthiness while the edge stamp used nullish — sourceId:'' put pages under 'default' but stamped edges '', FK-violating against sources(id) and silently dropping the file's whole call graph in the best-effort catch. The unscoped getChunks could also fan out to same-slug chunks from another source. One normalized edgeSourceId (sourceId || 'default') now drives both the chunk lookup and the stamp. * fix(engine): default edge source_id to 'default' at the insert layer (both engines) Adversarial review: addCodeEdges still wrote e.source_id ?? null, so any future caller that forgets the field reintroduces invisible NULL edges the day after the v116 backfill runs. A NULL source_id is invisible to every scoped call-graph query; default to the schema-default source the way the pages table does. Applied to both engines (parity). * fix(core): facts alter recipe NULLs embeddings before cross-dimension alters Adversarial review: buildFactsAlterRecipe shipped the same defect class this branch fixes for content_chunks 350 lines up — a cross-dimension ALTER ... USING cast that pgvector refuses while rows hold old-width vectors. Dimension changes now wipe first (the facts pipeline re-embeds on next write); same-dim type swaps (halfvec <-> vector) keep the lossless cast and PRESERVE data. Both behaviors pinned by tests. * v0.42.39.0 chore: version bump + CHANGELOG + TODOS Marks the v0.42.20.0 'decouple the op-dispatch force-exit timer' follow-up complete — this branch ships exactly that decoupling. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(postgres-engine): atomic JSONB merge in updateSourceConfig — eliminate lost-update race ## Problem `updateSourceConfig` used a read-then-write pattern: read the current `config` row, normalize it in JavaScript, then write the merged result back with `SET config = || `. Under concurrent callers (two background autopilot/cycle paths patching different keys simultaneously), both callers can read the same stale row. The later `SET config = ...` then clobbers the earlier patch, silently dropping whatever keys the first caller wrote. Reproduced at 21/25 lost-update events under real Postgres with parallel callers. ## Fix Fold the normalization and merge into a single atomic `UPDATE … SET config = CASE … END || patch` statement. Because the `SET` expression evaluates against the row-locked latest version of `config`, there is no snapshot window between the read and the write. Concurrent callers now converge correctly (50/50 clean in reproduction test). The `CASE` also normalizes historical bad JSONB shapes inline: - `object` — used as-is - `string` — double-encoded config; inner text parsed with the SQL `IS JSON` guard (Postgres 16+) so unparseable strings fall back to `{}` instead of raising `invalid input syntax for type json` - `array` — array of patch objects aggregated into a flat object via `jsonb_object_agg` - anything else — falls back to `{}` `pglite-engine.updateSourceConfig` already used an atomic `||` merge; this change brings postgres-engine to parity. ## Test Added two assertions to `test/list-all-sources.test.ts`: 1. JSONB string holding non-JSON text normalizes to `{}` (no cast throw) 2. JSONB string holding double-encoded valid JSON is parsed then merged * fix(doctor): five correctness fixes — stale locks, content sanity, graph coverage, exit code, gateway guard ## 1. Stale lock break hints cover gbrain-cycle: keys The doctor stale-lock report only recognized `gbrain-sync:` lock prefixes; everything else fell back to `gbrain sync --break-lock`, which is wrong for dream/autopilot cycle locks. A `gbrain-cycle:` or `gbrain-cycle` lock now suggests `gbrain dream --break-lock [--source ]`, and unknown lock shapes fall back to `gbrain doctor` instead of a misleading sync command. ## 2. content_sanity_audit_recent counts reject and quarantine as hard failures v0.42 renamed the hard disposition path: rejected pages emit a `reject` event and quarantined junk pages emit `quarantine`; `hard_block` is now only the pre-v0.42 legacy alias. The status check only counted `hard_block`, so fresh `reject` / `quarantine` events from the new path cleared as `ok` whenever fewer than 10 events existed. The check now sums all three for the hard count, and `soft_block + flag` for the soft count. ## 3. graph_coverage excludes test fixture entity pages from the denominator Brains seeded with code sources (e.g. a sync of the gbrain repo itself) could accumulate test fixture pages typed as `entity` / `person`. Including these in the entity-count denominator diluted coverage and produced spurious warnings ("Entity link coverage 0%, timeline 0%") on knowledge-only brains with no real entity pages. The check now queries a per-entity stats CTE that excludes `tools/gbrain/test/*` slugs and the `templates/new-person` stub, with an additional guard for the all-fixture case (`eligibleEntityCount = 0`). ## 4. process.exitCode instead of process.exit at doctor main exit point `process.exit(hasFail ? 1 : 0)` was a hard kill that prevented cleanup handlers (Bun unload events, open DB connections) from running. Using `process.exitCode = hasFail ? 1 : 0` defers the actual termination until the end of the event loop, allowing cleanup to complete. ## 5. checkSubagentCapability exported for test seams + gateway loop guard The function was private, making it untestable in isolation. It is now exported. Additionally, users running gbrain with a non-Anthropic chat model via `agent.use_gateway_loop=true` no longer receive a spurious warning that `ANTHROPIC_API_KEY` is missing — subagents route via the gateway loop in that configuration and do not need the key directly. ## Tests Doctor test suite: 77 pass, 0 fail (no regressions). * fix(engine): deleteFactsForPage excludeSourcePrefixes (#1928) + reconnect() parity (#2034) Engine-layer API for two cycle/availability fixes that share these files: - deleteFactsForPage gains optional excludeSourcePrefixes so the fence reconcile can protect non-fence facts (e.g. cli: conversation facts). - reconnect(ctx?) is now a first-class BrainEngine method on both engines (PostgresEngine already had it; PGLite gains config capture + reconnect) so callers stop using disconnect()+bare connect(). Co-Authored-By: Claude Fable 5 * fix(cycle): stop extract_facts from wiping conversation facts (#1928) The fence reconcile delete-then-reinsert wiped cli:-origin facts (no fence to recreate them); a failed-sync full walk turned it brain-wide (1829 rows, 0 reinserted, status ok). Now: exclude cli: rows from the wipe, do NOT inherit the failed-sync->full-walk fallback for this destructive phase, and warn on net-negative reconcile. Co-Authored-By: Claude Fable 5 * fix(autopilot,supervisor): reconnect() instead of disconnect()+bare connect() (#2034) The autopilot health-probe recovery called connect() with no args after disconnect(), losing the startup config (database_url undefined -> FATAL restart-loop on every DB blip) and opening a null-pool window. Both call sites now use engine.reconnect(), which restores the captured config. Co-Authored-By: Claude Fable 5 * fix(write-through): mirror to the assigned source's local_path, never the global repo (#2018) put_page write-through resolved the disk target from the global sync.repo_path, so a default-source page (local_path NULL) got written into an unrelated federated source's working tree. Now it uses the assigned source's own local_path; NULL local_path skips (no leak); the global path is used only as a sole-source fallback. Co-Authored-By: Claude Fable 5 * fix(pglite-lock): heartbeat + steal-grace so live holders are never stolen (#2058) A live holder's lock was force-removed after 5min age alone, letting a second process share the single-writer data dir -> WAL corruption. The lock now heartbeats while held; a holder is reaped only when its PID is dead OR its heartbeat went stale past the steal grace. Pairs PID liveness with heartbeat age to also defeat PID reuse. Co-Authored-By: Claude Fable 5 * fix(migrate,doctor): self-heal idx_timeline_dedup drift (#2038) A migration renumbered during a merge (v102) could be recorded-as-applied without its DDL running, leaving the 3-column index so every timeline write failed the 4-column ON CONFLICT. runMigrations now always runs a shape-keyed drift repair (dedupe-then-rebuild) even when no migration is pending, and doctor surfaces the drift. Co-Authored-By: Claude Fable 5 * fix(timeline): un-silence the swallowed batch catch; pin Date-batch round-trip (#2057) The meetings extractor's bare catch {} hid a brain-wide timeline-write failure (0 entries, no error). It now counts + surfaces batch errors. Adds a Date-bearing batch regression test proving the #1861 jsonb_to_recordset refactor already fixed the original ::text[] cast failure. Co-Authored-By: Claude Fable 5 * chore: bump version and changelog (v0.42.41.0) Triage fix wave: 6 authored critical fixes (#1928 facts wipe, #2018 write-through leak, #2034 reconnect loop, #2058 WAL lock, #2038 timeline migration drift, #2057 timeline silent-empty) + community PRs #2064 #2052 #2020 #2033 #2074 #2075 #2009 #2072 #2073. TODOS: deferred #1994 #1963 #2050. Co-Authored-By: Claude Fable 5 * fix: address adversarial review findings (#1928, #2058, #2038, #2057) Codex as-built review of the authored fixes surfaced 4 real issues: - #2058: add a pid+acquired_at ownership token. A stale holder reaped + replaced past the grace must NOT let its resumed heartbeat refresh, nor releaseLock remove, the NEW owner's lock (re-opened the concurrent-writer hole). Heartbeat and release now verify the on-disk lock is still ours. + regression test. - #1928: the destructive-full-walk guard keyed off phases.includes('sync'), which wrongly suppressed a legitimate full reconcile when sync was SKIPPED (no engine / no brainDir). Key off a syncAttempted flag set only when sync actually ran. - #2038: dedupe keeps MIN(id) not MIN(ctid) — deterministic and consistent with the existing v-migration lower-id rule. - #2057: the extract CLI caller now surfaces batch_errors (stderr + exit 1) instead of printing a clean success over failed inserts. Co-Authored-By: Claude Fable 5 * docs(key-files): sync reference to v0.42.41.0 triage-wave behavior Update KEY_FILES.md to current-state truth for the shipped fixes (no release-history clauses, per the reference-doc discipline): - write-through.ts (#2018): resolves the disk target from the assigned source's own local_path; sole-source falls back to sync.repo_path, multi-source skips with source_has_no_local_path rather than leak. - engine.ts (#2034): reconnect() is now a REQUIRED lifecycle method on both engines; config-restoring, never disconnect()+bare connect(). - migrate.ts (#2073): document v116 edge source_id backfill + callee index, and the always-run (version-counter-blind) timeline dedup self-heal. - new entry for timeline-dedup-repair.ts (#2038) + the timeline_dedup_index doctor check. - new entry for pglite-lock.ts (#2058): heartbeat + steal-grace (GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS) so a live holder is never stolen. - extract-facts.ts (#1928): cli:-fact protection, no failed-sync full-walk inheritance, net_fact_deletion warn floor. bun run build:llms re-run (KEY_FILES is link-only so bundles unchanged); freshness + current-state guards green. Co-Authored-By: Claude Fable 5 * fix(write-through): preserve nested multi-source layout; narrow #2018 leak guard The first #2018 fix skipped any no-local_path source on a multi-source brain, which broke the legitimate nested layout (a source without its own tree nests under the host repo at .sources// — pinned by put-page-write-through.test). Narrow the guard: a no-local_path source nests under sync.repo_path as before; only SKIP when sync.repo_path is literally another source's own local_path (the actual leak — writing there pollutes that sibling's repo). Caught by the sharded suite. Co-Authored-By: Claude Fable 5 * test: satisfy test-isolation guard for the new lock/reconnect tests CI `verify` flagged 3 intra-process isolation violations in the tests added this wave (the parallel runner shares one process per shard): - pglite-lock.test.ts: the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS mutation now goes through withEnv() instead of a raw process.env write (R1). - pglite-reconnect: renamed to *.serial.test.ts — it creates per-test engines to exercise the connect/reconnect lifecycle, which doesn't fit the shared beforeAll-engine model (R3/R4). verify is now 30/30; both files green. Co-Authored-By: Claude Fable 5 * fix(pglite): reconnect() is a no-op for in-memory engines (#2034) CI serial-tests + test(5) caught two in-branch regressions from the #2034 PGLite reconnect(): - worker/queue claim-error recovery + their renewLock e2e test assume PGLite reconnect is absent/no-op (queue.ts documents it). Making it a real disconnect+reopen wiped an in-memory engine's state mid-job. reconnect() now no-ops for in-memory (no database_path) — file-backed still re-opens the dir (state persists on disk). Restores the documented worker assumption. - connection-resilience 'Supervisor still has the 3-strikes-then-reconnect path' pinned the removed unsafe-cast text; updated to assert the direct this.engine.reconnect() call. Co-Authored-By: Claude Fable 5 * test: quarantine embed-input-type-wire to serial lane (CI test(5) leak) #2033's embed-input-type-wire.test.ts configures a 1280-dim embedding gateway; the active dimension survived into engine-find-trajectory when CI's 10-way hash-disjoint sharding co-located them (this branch's added files reshuffled the assignment), failing 7 trajectory tests with 'expected 1280 dimensions, not 1536'. resetGateway() in afterEach clears the gateway but the dimension still leaked. It mutates global gateway/embedding state, so it belongs in the serial lane (own bun process, true isolation) by the repo's own definition. Root-caused by reproducing the exact failing pair locally. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Austin Arnett Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: Dave MacDonald Co-authored-by: pabloglzg <186649799+pabloglzg@users.noreply.github.com> Co-authored-by: Alex P. <12667893+aphaiboon@users.noreply.github.com> Co-authored-by: Garry Tan Co-authored-by: jbarol Co-authored-by: maxpetrusenkoagent Co-authored-by: PAI --- CHANGELOG.md | 33 +++ TODOS.md | 43 +++- VERSION | 2 +- docs/architecture/KEY_FILES.md | 12 +- docs/embedding-migrations.md | 17 +- package.json | 2 +- src/cli.ts | 125 ++++++++-- src/commands/autopilot.ts | 9 +- src/commands/doctor.ts | 105 ++++++++- src/commands/extract.ts | 11 + src/commands/import.ts | 69 +++++- src/commands/init.ts | 8 +- src/commands/migrate-engine.ts | 5 +- src/core/ai/gateway.ts | 109 ++++++++- src/core/config.ts | 101 +++++++- src/core/cycle.ts | 37 ++- src/core/cycle/extract-facts.ts | 13 +- src/core/doctor-categories.ts | 1 + src/core/embedding-dim-check.ts | 18 +- src/core/engine.ts | 24 +- src/core/extract-timeline-from-meetings.ts | 26 ++- src/core/import-file.ts | 21 +- src/core/legacy-token-scope.ts | 22 ++ src/core/migrate.ts | 66 ++++++ src/core/minions/supervisor.ts | 19 +- src/core/oauth-provider.ts | 73 ++++-- src/core/pglite-engine.ts | 49 +++- src/core/pglite-lock.ts | 111 ++++++++- src/core/postgres-engine.ts | 71 +++++- src/core/schema-embedded.ts | 58 +++-- src/core/search/mode.ts | 11 +- src/core/sync.ts | 13 ++ src/core/timeline-dedup-repair.ts | 120 ++++++++++ src/core/write-through.ts | 59 ++++- src/mcp/http-transport.ts | 26 +-- src/schema.sql | 9 + test/ai/zeroentropy-compat-fetch.test.ts | 12 +- test/cli-force-exit-teardown-arming.test.ts | 56 +++++ test/config-env-hijack.test.ts | 144 ++++++++++++ test/connection-resilience.test.ts | 6 +- test/cross-modal-phase1.test.ts | 8 +- test/embed-input-type-wire.serial.test.ts | 247 ++++++++++++++++++++ test/embedding-dim-check-facts.test.ts | 21 ++ test/embedding-dim-check.test.ts | 20 ++ test/facts-reconcile-protect-cli.test.ts | 78 +++++++ test/import-code-edges-source-id.test.ts | 112 +++++++++ test/list-all-sources.test.ts | 47 ++++ test/oauth-authorize-scope-default.test.ts | 89 +++++++ test/oauth.test.ts | 28 +++ test/pglite-lock.test.ts | 94 ++++++++ test/pglite-reconnect.serial.test.ts | 45 ++++ test/search-alias-resolved-boost.test.ts | 4 +- test/search-mode.test.ts | 10 +- test/search/knobs-hash-reranker.test.ts | 7 +- test/timeline-date-batch.test.ts | 76 ++++++ test/timeline-dedup-repair.test.ts | 119 ++++++++++ test/write-through.test.ts | 52 +++++ 57 files changed, 2571 insertions(+), 202 deletions(-) create mode 100644 src/core/legacy-token-scope.ts create mode 100644 src/core/timeline-dedup-repair.ts create mode 100644 test/cli-force-exit-teardown-arming.test.ts create mode 100644 test/config-env-hijack.test.ts create mode 100644 test/embed-input-type-wire.serial.test.ts create mode 100644 test/facts-reconcile-protect-cli.test.ts create mode 100644 test/import-code-edges-source-id.test.ts create mode 100644 test/oauth-authorize-scope-default.test.ts create mode 100644 test/pglite-reconnect.serial.test.ts create mode 100644 test/timeline-date-batch.test.ts create mode 100644 test/timeline-dedup-repair.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 6aeaf6617..a620e693d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,39 @@ All notable changes to GBrain will be documented in this file. +## [0.42.41.0] - 2026-06-11 + +**A correctness-and-reliability wave: your conversation facts survive a cycle, write-through stops polluting other repos, autopilot rides out a DB blip instead of crash-looping, and concurrent PGLite processes stop corrupting each other.** A triage of open reports surfaced six bugs with no fix yet plus a batch of community PRs; this ships them together, each with a regression test. + +The headline is data durability. The `extract_facts` cycle phase reconciles a page's facts from its `## Facts` fence by deleting then reinserting — but conversation facts (written by `extract-conversation-facts`) live on pages that have no fence, so a cycle could delete them and reinsert nothing. A failed sync made it worse by escalating the phase to a full-brain walk. Both are fixed: the reconcile now protects non-fence (`cli:`-origin) facts, the destructive phase no longer inherits the failed-sync full-walk, and a reconcile that removes far more than it adds now reports `warn` instead of a silent `ok`. + +Three more silent failures, each found in production and now self-healing or loud: +- **Timeline writes that silently stopped.** A migration renumbered during a merge could be recorded as applied without its index change ever running, so every timeline insert failed its `ON CONFLICT`. `gbrain` now repairs the index shape on every migrate pass (dedupe-then-rebuild, even when nothing is pending), `gbrain doctor` reports the drift, and the meetings extractor no longer swallows batch errors. +- **Autopilot crash-loop on transient DB errors.** The health-probe recovery called `connect()` without its config, so every reconnect threw and the process exited on any blip. It now uses `reconnect()`, which restores the captured config; `reconnect()` is a first-class method on both engines. +- **WAL corruption from concurrent PGLite processes.** A live, working process holding the data-dir lock could have it stolen after five minutes. The lock now heartbeats while held and is only reclaimed from a dead or genuinely stalled holder. + +### Added +- **`timeline_dedup_index` doctor check + always-run repair (#2038).** Detects and heals a stale `idx_timeline_dedup` shape; `gbrain apply-migrations --force-schema` triggers the repair on demand. Reported by @jbarol. +- **`BrainEngine.reconnect()` on both engines (#2034).** Config-restoring reconnect, replacing the `disconnect()`+bare-`connect()` pattern. + +### Fixed +- **Conversation facts survive a fence reconcile (#1928).** The cycle's per-page wipe now excludes non-fence (`cli:`) facts, the destructive phase no longer full-walks on a failed sync, and net-negative reconciles surface as `warn`. +- **`put_page` write-through no longer leaks into an unrelated source's repo (#2018).** It mirrors to the assigned source's own `local_path`; a source without one is skipped rather than written into the global repo path. +- **Autopilot survives transient DB errors instead of crash-looping (#2034).** +- **Concurrent PGLite processes no longer corrupt the WAL (#2058).** Heartbeat + steal-grace replaces the age-only stale-lock check. +- **Timeline migration drift self-heals; the extractor stops swallowing errors (#2038, #2057).** A Date-typed batch date round-trips correctly (verified by test). +- **A cwd `.env` `DATABASE_URL` no longer silently retargets the brain (#2064, closes #427).** Reported by @bomliu. +- **`gbrain sync --strategy code` honors `.gitignore` and skips `vendor`/`dist`/`build`/`venv` (#2052, #2020).** Reported by @aphaiboon and @dMac716. +- **Asymmetric embedding `input_type` reaches the wire across every openai-compatible recipe (#2033, supersedes #1400).** Query vectors are no longer document-typed. By @pabloglzg; original diagnosis by @billy-armstrong. +- **`updateSourceConfig` JSONB merge is atomic (#2074).** Eliminates a concurrent-writer lost-update race. Reported by @pai-scaffolde. +- **`gbrain doctor` correctness pass (#2075):** stale-lock hints, content sanity, graph coverage, exit code, gateway guard. Reported by @pai-scaffolde. +- **`gbrain search` returns results instead of exiting 0 empty on slow poolers; scoped `code-callers`/`code-callees` find their edges (#2073).** Migration v116 backfills NULL edge `source_id` and indexes `from_symbol_qualified`. Reported by @jbarol. +- **OAuth scope handling (#2009, #2072):** an omitted authorize scope now defaults to the client's registered grant (clamped to it, so no widening) instead of an empty grant that never self-heals; legacy token source grants are honored through a single shared scope parser. By @austinrarnett and @maxpetrusenkoagent. + +### To take advantage of v0.42.41.0 + +`gbrain upgrade`. The timeline-index repair and migration v116 run automatically on the next migrate pass — if `gbrain doctor` flagged a timeline or call-graph problem before, re-run it after upgrade. No config changes required; the facts, write-through, autopilot, and lock fixes apply on restart. + ## [0.42.40.0] - 2026-06-09 **`gbrain extract --stale` no longer aborts partway through a brain that contains emoji or other non-BMP characters.** On a large brain, link/timeline extraction could die with `invalid input syntax for type json` and commit nothing — and because the staleness bookmark only advances on a clean finish, every retry re-hit the same point and extraction stayed wedged. The cause: the link-context excerpt was sliced by raw UTF-16 index, so a window boundary landing inside an emoji's surrogate pair left an unpaired surrogate half in the text, which Postgres rejects when the batch is serialized to JSONB — taking down the whole batch, not just the one row. (PGLite is more permissive here, so this primarily bit the managed-Postgres engine.) diff --git a/TODOS.md b/TODOS.md index ba96fc81a..cf8baa784 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,31 @@ # TODOS +## gbrain triage wave follow-ups (filed v0.42.41.0) + +Deferred from the v0.42.41.0 fix wave (eng-reviewed as separate scope, not hotfixes). +See plan + GSTACK REVIEW REPORT at +`~/.claude/plans/system-instruction-you-are-working-zany-thacker.md`. + +- [ ] **P1 — supervisor: retry-with-backoff instead of hard stop on transient DB outages (#1994).** + `max_crashes_exceeded` gives up permanently; a transient pooler blip that trips the + counter wedges the supervisor until manual restart. **Why:** the #2034 reconnect fix + makes the engine recover, but the supervisor still hard-stops. **Where:** + `src/core/minions/supervisor.ts` crash-count loop — add exponential backoff with a + much higher (or no) permanent-give-up threshold for recoverable errors. +- [ ] **P2 — PGLite `reindex-frontmatter` / backfill statement_timeout boost (#1963).** + Community RCA: `SET LOCAL statement_timeout` is gated on `engine.kind === 'postgres'`, + so PGLite inherits the 30s session default and trips on non-trivial batches; the CLI + then swallows the error and exits 0. **Where:** `src/core/backfill-effective-date.ts` + (boost on PGLite too, or per-row updates) + the cli.ts catch that hides it. +- [ ] **P2 — autopilot drain-worker concurrency self-deadlock (#2050).** Drain-worker + runs at concurrency=1, so any cycle phase that spawns a subagent (patterns, synthesize) + deadlocks waiting on a worker slot it can't get. **Where:** autopilot drain-worker + dispatch — raise concurrency or exempt subagent-spawning phases. +- [ ] **P3 — name-keyed migration ledger (#2038 structural follow-up).** The always-run + index drift probe heals the one known case; the general fix is keying applied-migration + tracking by stable name rather than version integer so a renumber can't strand a + migration as recorded-but-not-executed. **Where:** `src/core/migrate.ts` ledger. + ## gbrain#1981 Retrieval Reflex follow-ups (v0.43+) Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extractor @@ -1800,11 +1826,6 @@ Three items deferred: self-heals via stale-reclaim). The common sync SUCCESS path already drains via handleCliOnly's finally. Convert for graceful drain on sync error exits. -- [ ] **(v0.42.20.0 follow-up) Decouple the op-dispatch force-exit timer** so it - wraps `engine.disconnect()` only (it's armed before the handler today, doubling - as a blanket handler watchdog) and fix its misleading "engine.disconnect() did - not return…" message that fires even when the handler (not disconnect) was slow. - - [ ] **(v0.42.20.0 follow-up) Gateway idle-timeout (vs absolute) for streaming chat.** `withDefaultTimeout` uses an absolute `AbortSignal.timeout`; a streaming generation actively producing tokens past the chat default (300s) would abort. @@ -3574,6 +3595,18 @@ keeping both skills' triggers intact for chaining. ## Completed +### ~~(v0.42.20.0 follow-up) Decouple the op-dispatch force-exit timer~~ +**Completed:** v0.42.39.0 (2026-06-10) + +The timer now arms at teardown entry (inside the op-dispatch finally, before +drain + disconnect) so it bounds ONLY disconnect — no longer doubling as a +blanket handler watchdog that killed slow-but-healthy ops at 10s with exit 0 +and empty stdout. Its "engine.disconnect() did not return…" message is now +accurate by construction (it can only fire during teardown). Read-scope +handlers + context build got their own explicit wallclock bound (180s default, +`--timeout=Ns`, exit 124, hard-exit after teardown) in the same wave. Pinned by +`test/cli-force-exit-teardown-arming.test.ts`. + ### ~~Checks 5 + 6 for check-resolvable~~ **Completed:** v0.19.0 (2026-04-22) diff --git a/VERSION b/VERSION index cc968e114..06c56e14a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.40.0 \ No newline at end of file +0.42.41.0 diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 198b2759a..3491e3d5c 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -9,7 +9,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this (`scripts/check-key-files-current-state.sh`). - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Exports upload validators `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` is a REQUIRED field flagging untrusted callers; `OperationContext.allowedSlugPrefixes` is the trusted-workspace allow-list set by the dream cycle; `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link skipped only when `remote=true && !trustedWorkspace`. Every `Operation` carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`; `sync_brain`, `file_upload`, `file_list`, `file_url` are `admin + localOnly` (rejected over HTTP). Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) use FAIL-CLOSED semantics: `ctx.remote === false` for trusted-only sites, `ctx.remote !== false` for "untrust unless explicit-false" — anything not strictly `false` is treated as remote (closes the HTTP MCP shell-job RCE where a read+write OAuth token could submit `shell` jobs). `sourceScopeOpts(ctx)` encodes the source-scoped read precedence ladder — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId`/`ctx.auth.sourceId`) over nothing; every read-side op handler routes through it so a source-bound OAuth client can't see neighboring sources via `search`/`query`/`list_pages`/`get_page`/`find_experts`/query's image path. `put_page`'s inline disk write-through is the shared `writePageThrough` helper (`src/core/write-through.ts`), ATOMIC via temp-sibling + rename so a crash or concurrent `gbrain sync` can't read a half-written `.md`; same helper backs `gbrain brainstorm/lsd --save`. Link provenance surface (#1941): `add_link` (`gbrain link`/`link-add`) + `remove_link` (`gbrain unlink`/`link-rm`) expose `link_source`/`link_type`; `add_link` rejects the reconciliation-managed built-ins via `MANAGED_LINK_SOURCES` (`markdown`/`frontmatter`/`mentions`/`wikilink-resolved`) and defaults omitted provenance to `'manual'` (the engine's own default stays `'markdown'` for internal callers); `list_link_sources` (`gbrain link-sources`, read) lists provenances via `sourceScopeOpts`. CLI aliases register through `cliHints.aliases` (collision-guarded in `src/cli.ts`). -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput`/`TimelineBatchInput` for the bulk-insert API (`addLinksBatch`/`addTimelineEntriesBatch`). `readonly kind: 'postgres' | 'pglite'` discriminator lets `src/core/migrate.ts` and others branch without `instanceof` + dynamic imports. Methods: `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[],$2::text[],$3::real[])` composite-keyed on `(slug, source_id)`), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` has `sort?: 'updated_desc'|'updated_asc'|'created_desc'|'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines. `listAllPageRefs(): Promise>` ordered by `(source_id, slug)` — cheap cross-source enumeration replacing the `getAllSlugs()→getPage(slug)` N+1 (which silently defaulted to `source_id='default'`); parity across postgres-engine.ts + pglite-engine.ts; Pinned by `test/e2e/multi-source-bug-class.test.ts`. `SearchOpts`+`PageFilters` add `sourceIds?: string[]` (federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when set, preserve scalar `sourceId` fast path when unset); `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId`/`opts.sourceIds`. `traverseGraph` opts has `frontierCap?: number` (per-iteration recursive-CTE cap, approx per-BFS-layer); return type `Promise` for MCP wire stability; export `TraverseGraphOpts`; Postgres uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, PGLite mirrors with positional params; Pinned by `test/regressions/v0_36_frontier_cap.test.ts`. Phantom-redirect methods: `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (content_hash refresh required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug`+`source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence; parity at `test/phantom-redirect-engine-parity.test.ts`. `getAdjacencyBoosts(pageIds): Promise>` powers the per-query graph-signals stage — one SQL query returning inbound-link counts among top-K plus a cross-source count (links from differing `source_id`); `COALESCE(p.source_id,'default')` null safety, `HAVING >= 1`, cross-source CASE-WHEN excludes the target's own source; parity SQL across both engines; `SearchResult` gains optional `base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta` + internal staging fields; Pinned by `test/e2e/graph-signals-engine.test.ts`. Two REQUIRED methods: `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup); `sourceId` REQUIRED on both at the type level (asymmetric with single-row `deletePage` which keeps optional/'default'); both short-circuit on empty input and throw when `> DELETE_BATCH_SIZE`. Embedding-signature stale-detection quartet: `countStaleChunks(opts?)` gains optional `signature?: string` widening the stale predicate from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (NULL signature is GRANDFATHERED, never counted; omit `signature` for the legacy NULL-only count); `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` = `SUM(LENGTH(chunk_text))` over stale chunks (same predicate + embed_skip filter + optional sourceId scope), used by `gbrain sync --all` cost preview via `estimateCostFromChars`; `setPageEmbeddingSignature(slug, {sourceId?, signature})` stamps `pages.embedding_signature` after a page's chunks (re)embed, idempotent no-op when page absent; `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` NULLs `embedding`+`embedded_at` on every chunk whose page signature is set AND differs, returning the count, called BEFORE `listStaleChunks` so signature-drift pages flow through the NULL-embedding keyset cursor unchanged (NULL never invalidated). Widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. Free-text alias layer: `resolveAliases(aliasNorms, opts?): Promise>>` (READ; maps each normalized alias to declaring `(slug, source_id)` pairs, source-scoped) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE; replaces the full alias set, delete-then-insert, empty clears, idempotent on the unique triple), called by the `importFromContent` ingest projection and the `reindex --aliases` backfill; parity across both engines, Pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines injects the shared `buildBestPerPagePoolCte` per-page max-pool so a page surfaces on its strongest chunk. `executeRawDirect(sql, params?, opts?)` is the lock-hot-path sibling of `executeRaw`: same single-statement contract, but routes to the direct session-mode pool when dual-pool is active (Postgres/Supabase port 5432) so a long-held lock heartbeat survives the transaction pooler's per-transaction connection recycling; PGLite delegates straight to `executeRaw` (no pooler). Both engines implement it; the Minion lock path (`claim`/`renewLock`) is the consumer. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput`/`TimelineBatchInput` for the bulk-insert API (`addLinksBatch`/`addTimelineEntriesBatch`). `readonly kind: 'postgres' | 'pglite'` discriminator lets `src/core/migrate.ts` and others branch without `instanceof` + dynamic imports. Methods: `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[],$2::text[],$3::real[])` composite-keyed on `(slug, source_id)`), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` has `sort?: 'updated_desc'|'updated_asc'|'created_desc'|'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines. `listAllPageRefs(): Promise>` ordered by `(source_id, slug)` — cheap cross-source enumeration replacing the `getAllSlugs()→getPage(slug)` N+1 (which silently defaulted to `source_id='default'`); parity across postgres-engine.ts + pglite-engine.ts; Pinned by `test/e2e/multi-source-bug-class.test.ts`. `SearchOpts`+`PageFilters` add `sourceIds?: string[]` (federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when set, preserve scalar `sourceId` fast path when unset); `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId`/`opts.sourceIds`. `traverseGraph` opts has `frontierCap?: number` (per-iteration recursive-CTE cap, approx per-BFS-layer); return type `Promise` for MCP wire stability; export `TraverseGraphOpts`; Postgres uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, PGLite mirrors with positional params; Pinned by `test/regressions/v0_36_frontier_cap.test.ts`. Phantom-redirect methods: `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (content_hash refresh required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug`+`source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence; parity at `test/phantom-redirect-engine-parity.test.ts`. `getAdjacencyBoosts(pageIds): Promise>` powers the per-query graph-signals stage — one SQL query returning inbound-link counts among top-K plus a cross-source count (links from differing `source_id`); `COALESCE(p.source_id,'default')` null safety, `HAVING >= 1`, cross-source CASE-WHEN excludes the target's own source; parity SQL across both engines; `SearchResult` gains optional `base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta` + internal staging fields; Pinned by `test/e2e/graph-signals-engine.test.ts`. Two REQUIRED methods: `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup); `sourceId` REQUIRED on both at the type level (asymmetric with single-row `deletePage` which keeps optional/'default'); both short-circuit on empty input and throw when `> DELETE_BATCH_SIZE`. Embedding-signature stale-detection quartet: `countStaleChunks(opts?)` gains optional `signature?: string` widening the stale predicate from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (NULL signature is GRANDFATHERED, never counted; omit `signature` for the legacy NULL-only count); `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` = `SUM(LENGTH(chunk_text))` over stale chunks (same predicate + embed_skip filter + optional sourceId scope), used by `gbrain sync --all` cost preview via `estimateCostFromChars`; `setPageEmbeddingSignature(slug, {sourceId?, signature})` stamps `pages.embedding_signature` after a page's chunks (re)embed, idempotent no-op when page absent; `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` NULLs `embedding`+`embedded_at` on every chunk whose page signature is set AND differs, returning the count, called BEFORE `listStaleChunks` so signature-drift pages flow through the NULL-embedding keyset cursor unchanged (NULL never invalidated). Widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. Free-text alias layer: `resolveAliases(aliasNorms, opts?): Promise>>` (READ; maps each normalized alias to declaring `(slug, source_id)` pairs, source-scoped) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE; replaces the full alias set, delete-then-insert, empty clears, idempotent on the unique triple), called by the `importFromContent` ingest projection and the `reindex --aliases` backfill; parity across both engines, Pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines injects the shared `buildBestPerPagePoolCte` per-page max-pool so a page surfaces on its strongest chunk. `executeRawDirect(sql, params?, opts?)` is the lock-hot-path sibling of `executeRaw`: same single-statement contract, but routes to the direct session-mode pool when dual-pool is active (Postgres/Supabase port 5432) so a long-held lock heartbeat survives the transaction pooler's per-transaction connection recycling; PGLite delegates straight to `executeRaw` (no pooler). Both engines implement it; the Minion lock path (`claim`/`renewLock`) is the consumer. `reconnect(ctx?: {error?})` is a REQUIRED lifecycle method on BOTH engines: it recovers a dropped connection using the config captured at the last `connect()`, so callers (autopilot health probe, `batchRetry`) never `disconnect()` + bare `connect()` (which loses the config and throws `database_url undefined` forever, and opens a null-connection window). PostgresEngine rebuilds its pool with a `_reconnecting` reentrancy guard and emits pool-recovery audit; PGLite is single-writer in-process so it just restores the saved data dir for parity. - `src/core/engine-constants.ts` — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/background-work.ts` (#1762/#1745/#1775) — process background-work registry: the single owner of "drain every fire-and-forget DB-write sink before the CLI disconnects." `registerBackgroundWorkDrainer({name, order, drain(timeoutMs), abort?})` + `drainAllBackgroundWorkForCliExit({timeoutMs})` over a `Map` (idempotent registration by name; `__registerDrainerForTest` returns an unregister handle). Drains in explicit `(order, name)` order — facts FIRST (order 0) so its abort-path DB `logIngest` runs against the freshest live engine — and AWAITS `abort()` only when `drain()` reports `unfinished>0`. Best-effort per drainer: one sink's failure never blocks the others or the disconnect. FOUR sinks register at module import (rule-of-four): `facts/queue.ts` (order 0; `abort`=`shutdown()` cancels a hung facts:absorb Haiku via internalAbort), `last-retrieved.ts` (order 1), `search/hybrid.ts` (order 2; `awaitPendingSearchCacheWrites` bounded via `Promise.race`), `eval-capture.ts` (order 3; `captureEvalCandidate` self-tracks its promise via `awaitPendingEvalCaptures`). Both CLI-exit paths (`src/cli.ts` op-dispatch finally + `handleCliOnly` finally) call it before `engine.disconnect()` — closing the PGLite busy-loop where `db.close()` raced an in-flight job and pinned the single-writer lock (#1762). CLI-EXIT-ONLY: the facts `shutdown()` abort is permanent process state, never call in a long-lived `gbrain serve`. Companion changes: `src/core/ai/gateway.ts` `withDefaultTimeout(caller, ms)` bounds every outbound AI call (chat 300s, embed+multimodal 60s; env `GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS`; composed with caller signals via `AbortSignal.any`) and the op-dispatch + handleCliOnly force-exit timers `process.exit(process.exitCode ?? 0)` so a hung disconnect can't mask an errored op as success; `src/core/postgres-engine.ts` `reconnect()` module-mode branch re-establishes via idempotent `db.connect()` + `connectionManager.setReadPool` refresh instead of `db.disconnect()` (no null window for concurrent ops; fail-loud on real connect failure — #1745); `src/core/search/hybrid.ts` `embedQueryBounded` + a shared `QueryEmbedDeadline` (6s, floored 2s per embed via `MIN_QUERY_EMBED_BUDGET_MS`; env `GBRAIN_QUERY_EMBED_TIMEOUT_MS`) bounds the cache-lookup AND inner query embeds so a stalled provider falls back to keyword instead of hanging past the 10s force-exit (#1775). Incorporates + hardens PR #1763 (@ElliotDrel). Pinned by `test/core/background-work.test.ts`, `test/search/query-embed-deadline.test.ts`, `test/eval-capture-drain.test.ts`, `test/e2e/postgres-reconnect-singleton.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts`, `test/fix-wave-structural.test.ts`. - `src/core/search/graph-signals.ts` — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring at full score, demote the rest). All three inherit the floor-ratio gate preventing weak pages from being boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width`. `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (incl. the IRON-RULE floor-gate regression). @@ -25,6 +25,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/doctor.ts` extension — `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded. Pinned by cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`). - `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all BrainEngine methods. `listLinkSources({sourceId?, sourceIds?})` returns distinct `link_source` provenances + counts (`ORDER BY count DESC, link_source ASC NULLS LAST`; scalar + federated scoped; parity with postgres-engine.ts) powering `gbrain link-sources`. `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the whole batch as one JSONB document via `jsonb_to_recordset(($1::jsonb)->'rows')` (bound through `executeRawJsonb` with a `{ rows }` wrapper; rows built by the shared `src/core/batch-rows.ts` helpers, NUL-stripped), and are `batchRetry`-wrapped. `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error (macOS 26.3 WASM bug #223, points at `gbrain doctor`); the lock is released on failure so the next process can retry cleanly. `searchKeyword`/`searchKeywordChunks` multiply `ts_rank` by the source-factor CASE at chunk grain; `searchVector` is a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`, inner LIMIT scales with offset to preserve pagination. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for forward-referenced state the embedded blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target, plus `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`) and adds only what's missing; threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope; no-op on fresh installs and modern brains (closes the upgrade-wedge bug class #239/#243/#266/#357/#366/#374/#375/#378/#395/#396/#1018/#974/#820). `getBrainScore` returns 100/100 with full breakdown (35/25/15/15/10) when `pageCount === 0` (vacuous truth — empty brain has no coverage problem); Pinned by `test/brain-score-breakdown.test.ts` empty-brain assertion + `test/doctor-report-remote.serial.test.ts`. `disconnect()` uses snapshot+early-null (snapshot `_db`/`_lock`, null instance fields BEFORE any `await` so a concurrent `connect()` can't see a partial mid-close state) wrapped in try/finally guaranteeing lock-release even if `db.close()` throws; KEEPS close-then-release order (release-then-close was rejected: it would widen the window where a sibling process connects to a still-closing brain); Pinned by `test/pglite-engine-disconnect.serial.test.ts`. Exports `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` routing the catch-block hint by failure shape (`bunfs` matches literal `$$bunfs` OR `ENOENT[\s\S]*pglite\.data` co-occurrence, surfaces a paste-ready `bun upgrade` + Node fallback; `macos-26-3` keeps the #223 link; `unknown` falls through); Pinned by `test/pglite-init-classifier.test.ts`. Implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding, caller-chunking primitive throwing when input exceeds `DELETE_BATCH_SIZE`, `deletePages` returns `RETURNING slug` rows so callers filter `pagesAffected` to confirmed deletes. Implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, widened `countStaleChunks({sourceId?, signature?})` (the `signature` opt widens via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)`, NULL grandfathered); parity SQL with postgres-engine.ts. PGLite-specific DDL (pgvector, pg_trgm, triggers). +- `src/core/pglite-lock.ts` (#2058) — advisory data-dir lock so two processes never open the same embedded PGLite (single-connection WASM Postgres) at once. Atomic `mkdir` of `.gbrain-lock/` + a `lock` file carrying `{pid, acquired_at, refreshed_at}`. A held lock HEARTBEATS its `refreshed_at` every 30s (`.unref()`ed timer) so a long-running but LIVE holder (an `embed` job can run for minutes) is never mistaken for stale. A waiting acquirer reaps the holder only when it is dead (PID gone) OR has stopped refreshing past the steal grace (`GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS`, default 600s) — pairing PID liveness with heartbeat freshness defeats BOTH the WAL-corruption bug (stealing a live writer) and the PID-reuse false positive (a recycled PID reading as alive). Each holder carries an ownership token (`:`); the heartbeat and `releaseLock` verify the on-disk lock is STILL theirs before touching it, so a stalled-then-resumed holder that was already reaped + replaced can't clobber the new owner. In-memory engines take no lock (no file, no concurrent access). Pinned by `test/pglite-lock.test.ts`. - `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the batch as one JSONB document — `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages ...` bound through `executeRawJsonb({ rows })` — which encodes arbitrary free text safely (the old `unnest(${arr}::text[])` array-literal path crashed Postgres with "malformed array literal" on calendar/Zoom context, gbrain#1861) and sidesteps the 65535-parameter cap; takes declares native recordset column types (`page_id int, weight real, active boolean, …`) so no per-element casts; all three are `batchRetry`-wrapped. `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection. `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by `batchRetry` on a retryable connection error). Concurrent callers share one in-flight `_reconnectPromise` (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic `db.connect()` token on the connect leg. `reconnect(ctx?)` accepts the triggering error and records a pool-recovery audit event (`reap_detected`/`reconnect_other`/`reconnect_succeeded`/`reconnect_failed`) for the `pool_reap_health` doctor check. `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same probe set as PGLite (extended for column-only forward-reference cases: `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`/`archived_at`/`archive_expires_at`); the entire probe path runs on the DDL connection threaded from `initSchema` (closing a concurrent-bootstrap race for Supabase pooler users); closes #1018/#974/#820. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls `db.disconnect()` when it owns the singleton (`_ownsModuleSingleton`, set from the `db.connect()` creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` + `test/postgres-engine-singleton-ownership.test.ts`. `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`. - `src/core/cjk.ts` — Single source of truth for CJK detection. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables). Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()`. Separate surface from `sync-failures.jsonl` — that file carries bookmark-gating semantics that info events shouldn't trigger. @@ -101,7 +102,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. `BATCH_SIZE=100` (per-recipe pre-split + recursive halving + adaptive shrink-on-miss live in the gateway; the outer paginator is for progress-callback granularity, not batch protection). `estimateEmbeddingCostUsd(tokens)` prices against the currently-configured model's rate via `currentEmbeddingPricePerMTok()` (resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate 0.13 only when the gateway is unconfigured or the model is unknown to the pricing table). `EMBEDDING_COST_PER_1K_TOKENS` retained for back-compat with direct importers/tests. `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from current and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so cost gate and embed decision can't drift. `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. - `src/core/ai/dims.ts` — per-provider `providerOptions` resolver for embed-time dimension passthrough; the single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, `isValidVoyageOutputDim(dims)`. Voyage path uses the SDK-supported `dimensions` field (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built (the AI SDK's openai-compatible adapter doesn't recognize the wire-key, so sending it from here would be silently dropped and Voyage would return its default 1024-dim). Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary (most common trigger: `embedding_model: voyage:voyage-4-large` without `embedding_dimensions`, falling back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). - `src/core/ai/types.ts` — provider/recipe types. `EmbeddingTouchpoint` has optional `chars_per_token` (default 4, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling), both consulted only when `max_batch_tokens` is also set; Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64). Pre-split budget = `max_batch_tokens × safety_factor / chars_per_token`. `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes mixing text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`); when omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` (OpenAI text + Voyage images without flipping the primary pipeline). `Recipe.default_headers?: Record` (static) and `Recipe.resolveDefaultHeaders?(env)` (env-templated) seam for per-recipe headers riding alongside auth on every openai-compat touchpoint; mutually exclusive (declaring both throws `AIConfigError` at gateway-configure time); keys conflicting with the resolved auth header (`Authorization`, the resolver's custom header) rejected at `applyResolveAuth` call time so defaults can't shadow auth. Used by OpenRouter for the `HTTP-Referer` + `X-OpenRouter-Title` + `X-Title` attribution triple. -- `src/core/ai/gateway.ts` — unified seam for every AI call. `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path embeds via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default; the hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}` and the gateway resolves the matching recipe via `instantiateEmbedding()`. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down. `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL `/embeddings → /models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response `{results: [{embedding}], usage: {total_bytes, total_tokens}}` → `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates. Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name). Wired in `instantiateEmbedding()` via the `recipe.id === 'zeroentropyai'` branch. `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'`. `_rerankTransport` test seam mirrors `_embedTransport`. `embedQuery(text)` threads `inputType: 'query'` through `dimsProviderOptions()` (4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch; `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model`; `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries; the Voyage `/multimodalembeddings` path is unchanged (gateway selects by recipe `implementation` tag). Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions`. Pinned by `test/openai-compat-multimodal.test.ts`. Module-scoped `_embedTransport` defaults to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive `embed()` with a stubbed transport. `splitByTokenBudget` and `isTokenLimitError` exported `@internal` (pure functions reused by the test file). Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model`); after the recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared (closes the Voyage-text-only-into-multimodal-endpoint footgun before any HTTP call). `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel`. Exported `VoyageResponseTooLargeError` tagged class: `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length at `:595`, Layer 2 per-embedding base64 at `:619`) throw it; the inbound response-rewriter's try/catch (which swallows parse failures so misshaped responses fall through to the SDK parser) checks `instanceof VoyageResponseTooLargeError` and rethrows so the cap is actually effective (regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line). AI SDK v6 toolLoop compat (`gbrain skillopt` rollouts AND production background `subagent` jobs both route through `chat()` / `toolLoop`): in `chat()`, tool defs wrap the raw JSON Schema with the SDK's `jsonSchema()` helper (`inputSchema: jsonSchema(t.inputSchema)`) — v6's `asSchema()` treats a bare `{jsonSchema: ...}` object as a thunk and throws "schema is not a function"; new exported pure `toModelMessages(messages: ChatMessage[]): unknown[]` converts gbrain's provider-neutral `ChatMessage[]` into v6 `ModelMessage[]` — tool results (pushed by `toolLoop` as `role:'user'` with bare-value tool-result blocks) become a dedicated `role:'tool'` message with structured `output:{type:'json'|'text'|'error-text', value}` parts; `null` output preserved as `{type:'json', value:null}` (not dropped); text/tool-call blocks pass through with v6 field names (`toolCallId`/`toolName`/`input`); applied at the `generateText` call (`messages: toModelMessages(opts.messages)`). The converter is the load-bearing fix for the production subagent path, not just skillopt. Pinned by `test/gateway-model-messages.test.ts`. Companion fix in `src/core/skillopt/rollout.ts`: the inline `paramsToSchema` dropped `items` on array params; it now uses the shared `paramDefToSchema` from `src/mcp/tool-defs.ts` (single source of truth, recursive on items/enum/default). +- `src/core/ai/gateway.ts` — unified seam for every AI call. `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path embeds via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default; the hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}` and the gateway resolves the matching recipe via `instantiateEmbedding()`. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down. `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL `/embeddings → /models/embed`, injects `input_type` (default `'document'`; the threaded `'query'|'document'` crosses the SDK boundary via the module-level `__embedInputTypeStore` AsyncLocalStorage populated in `embedSubBatch()`, because the AI SDK's openai-compatible adapter strips `input_type` from `providerOptions` before building the wire body — #1400; `voyageCompatFetch` injects it opt-in the same way, and `openAICompatAsymmetricFetch` is the fallthrough shim for every other openai-compat recipe — llama-server/litellm/ollama — a strict pass-through when nothing was threaded) and explicit `encoding_format: 'float'`, and rewrites the response `{results: [{embedding}], usage: {total_bytes, total_tokens}}` → `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates. Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name). Wired in `instantiateEmbedding()` via the `recipe.id === 'zeroentropyai'` branch. `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'`. `_rerankTransport` test seam mirrors `_embedTransport`. `embedQuery(text)` threads `inputType: 'query'` through `dimsProviderOptions()` (4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch; `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model`; `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries; the Voyage `/multimodalembeddings` path is unchanged (gateway selects by recipe `implementation` tag). Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions`. Pinned by `test/openai-compat-multimodal.test.ts`. Module-scoped `_embedTransport` defaults to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive `embed()` with a stubbed transport. `splitByTokenBudget` and `isTokenLimitError` exported `@internal` (pure functions reused by the test file). Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model`); after the recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared (closes the Voyage-text-only-into-multimodal-endpoint footgun before any HTTP call). `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel`. Exported `VoyageResponseTooLargeError` tagged class: `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length at `:595`, Layer 2 per-embedding base64 at `:619`) throw it; the inbound response-rewriter's try/catch (which swallows parse failures so misshaped responses fall through to the SDK parser) checks `instanceof VoyageResponseTooLargeError` and rethrows so the cap is actually effective (regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line). AI SDK v6 toolLoop compat (`gbrain skillopt` rollouts AND production background `subagent` jobs both route through `chat()` / `toolLoop`): in `chat()`, tool defs wrap the raw JSON Schema with the SDK's `jsonSchema()` helper (`inputSchema: jsonSchema(t.inputSchema)`) — v6's `asSchema()` treats a bare `{jsonSchema: ...}` object as a thunk and throws "schema is not a function"; new exported pure `toModelMessages(messages: ChatMessage[]): unknown[]` converts gbrain's provider-neutral `ChatMessage[]` into v6 `ModelMessage[]` — tool results (pushed by `toolLoop` as `role:'user'` with bare-value tool-result blocks) become a dedicated `role:'tool'` message with structured `output:{type:'json'|'text'|'error-text', value}` parts; `null` output preserved as `{type:'json', value:null}` (not dropped); text/tool-call blocks pass through with v6 field names (`toolCallId`/`toolName`/`input`); applied at the `generateText` call (`messages: toModelMessages(opts.messages)`). The converter is the load-bearing fix for the production subagent path, not just skillopt. Pinned by `test/gateway-model-messages.test.ts`. Companion fix in `src/core/skillopt/rollout.ts`: the inline `paramsToSchema` dropped `items` on array params; it now uses the shared `paramDefToSchema` from `src/mcp/tool-defs.ts` (single source of truth, recursive on items/enum/default). - `src/core/ai/recipes/zeroentropyai.ts` — ZeroEntropy openai-compatible recipe declaring BOTH `embedding` (`zembed-1`, 7 Matryoshka dims: 2560/1280/640/320/160/80/40) AND `reranker` (`zerank-2` flagship + `zerank-1` + `zerank-1-small`, 5MB payload cap) touchpoints. `implementation: 'openai-compatible'` (pinned by regression in `test/ai/zeroentropy-recipe.test.ts`). `base_url_default: 'https://api.zeroentropy.dev/v1'` already ends with `/v1`, so the `zeroEntropyCompatFetch` URL rewrite `/embeddings → /models/embed` produces `…/v1/models/embed` (NOT `…/v1/v1/…` — pinned by regression). `chars_per_token: 1` + `safety_factor: 0.5` match Voyage's dense-content hedge. - `src/core/ai/recipes/llama-server-reranker.ts` — sibling of `llama-server` (the embedding recipe) for llama.cpp in `--reranking` mode. Distinct recipe rather than dual-touchpoint extension because `--reranking` and `--embeddings` are mutually exclusive at server-launch time, so the two backends need independent base URLs (default 8081 here vs 8080 there). Declares `reranker` touchpoint with `models: []` (user-provided id matching the `--alias` the user launched with), `path: '/rerank'` (leaf-only; consumes `RerankerTouchpoint.path` override; gateway concatenates with `base_url_default` which ends in `/v1`, producing `…/v1/rerank`), `default_timeout_ms: 30_000` (consumed by `src/core/search/mode.ts`'s reranker timeout chain — CPU-only first-call warmup headroom; the 5s mode-bundle default would fail-open as `timeout`), `cost_per_1m_tokens_usd: 0` (recognized by `FREE_LOCAL_RERANK_PROVIDERS` in `src/core/budget/budget-tracker.ts` so `--max-cost` callers don't hard-fail on local rerank). Setup hint emphasizes `--alias` because llama-server's `/v1/models` defaults model id to the gguf file path without it. Covers Qwen3-Reranker via llama.cpp AND self-hosted ZE weights via llama.cpp — same recipe, different `--model` at launch. Pinned by `test/ai/recipe-llama-server-reranker.test.ts`. Voyage / Cohere / vLLM rerankers stay out of scope (different wire shapes). Same wave adds: `path?: string` + `default_timeout_ms?: number` on `RerankerTouchpoint` in `src/core/ai/types.ts`; consumed by the URL build at `src/core/ai/gateway.ts:rerank()` and by mode-resolution at `src/core/search/mode.ts:resolveSearchMode` (precedence: per-call > config-key > recipe touchpoint default > mode bundle); `LLAMA_SERVER_RERANKER_BASE_URL` env passthrough in `src/cli.ts:buildGatewayConfig`; `FREE_LOCAL_RERANK_PROVIDERS` set in `src/core/budget/budget-tracker.ts:lookupPricing` (rerank-kind-only zero-pricing for the local provider prefix); doctor-fix at `src/commands/models.ts:probeRerankerConfig` reads `search.reranker.model` via `loadSearchModeConfig` + `resolveSearchMode` (closes file-plane / DB-plane divergence where doctor said "not configured" while live search was actively reranking — the field-plane `getRerankerModel()` read nothing writes); `probeRerankerReachability` reads the recipe's `default_timeout_ms` so CPU-only cold-start doesn't false-fail. - `src/core/ai/recipes/openrouter.ts` — OpenRouter openai-compatible recipe: single key, many providers via `openrouter:/` strings. `base_url_default: 'https://openrouter.ai/api/v1'`. Embedding touchpoint: `openai/text-embedding-3-small` at 1536 dims with Matryoshka `dims_options: [512, 768, 1024, 1536]`; `max_batch_tokens: 300_000` = OpenAI's aggregate-per-request token cap (NOT per-input). Chat touchpoint declares 8 curated entry points (gpt-5.2, gpt-5.2-chat, gpt-5.5, claude-haiku-4.5, claude-sonnet-4.6, claude-opus-4.7, gemini-3-flash-preview, deepseek-chat) but openai-compat tier accepts any model ID; deliberately no `max_context_tokens` because OR's catalog spans 128K to 1M+. `supports_subagent_loop: false` is INFORMATIONAL — the real gate is `isAnthropicProvider()` in `src/core/model-config.ts` which hard-pins gbrain's subagent infra to Anthropic-direct. Declares `resolveDefaultHeaders(env)` returning OR's three attribution headers: `HTTP-Referer` (required for OR app-attribution), `X-OpenRouter-Title` (preferred), `X-Title` (back-compat alias); defaults to `https://gbrain.ai` / `gbrain`; forks override via `OPENROUTER_REFERER` / `OPENROUTER_TITLE` env vars. Smoke-tested by `test/ai/recipe-openrouter.test.ts` (incl. the shape-test regression guard: every model in the chat list matches `^[a-z0-9-]+\/[a-z0-9._-]+$`). @@ -273,13 +274,14 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `test/fixtures/whoknows-eval.jsonl` — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries; placeholder uses obviously-example slugs (`wiki/people/example-alice`). Drives `test/e2e/whoknows.test.ts` (seeds a matching synthetic brain, asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904). `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent: validation-gated (median-of-3 + epsilon=0.05), budget-capped (preflight estimator), per-skill DB-locked (`tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (history-intent-first 5-step commit), body-only mutations (frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload`. Two reflect calls per step; rejected-edit buffer LRU-bounded to 100; bundled-skill gate; bootstrap workflow (sentinel + `--bootstrap-reviewed`); D_sel floor (>=5 with `--split` override); audit JSONL via `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Surface: dream-cycle phase wrapper; `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`); `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`); MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly, validates `skill_name` kebab-only + confines caller-supplied benchmark/held-out paths to skillsDir for remote callers); Minion `skillopt` handler + `--background` with `allowProtectedSubmit: true`; write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag); held-out real-user test set via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`). `--bootstrap-from-skill` → `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts`: reads `SKILL.md` directly (no `routing-eval.jsonl`), makes ONE LLM call emitting a full starter benchmark (tasks + rule judges) as JSONL, parsed line-by-line with skip-bad-line salvage and a min-2-valid-checks-per-task drop; provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`). `--bootstrap-tasks N` (default 15, capped 50); `maxTokens` scales `min(8000, max(4000, N*220))`. The stderr REVIEW line prints the literal `gbrain skillopt --bootstrap-reviewed --split 1:1:1` — load-bearing because the default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark needs `--split 1:1:1`. Both bootstrap generators share `assertBenchmarkAbsent` + `readSkillBodyOrThrow`; `--bootstrap-from-skill` is mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume`. Generated rule judges are explicitly WEAK DRAFTS to be strengthened during the review gate. The F11 held-out gate is wired: `--held-out ` is parsed and threaded through every caller (CLI main + `--background` `held_out_path` + batch/fleet `heldOutPath` + the `run_skillopt` `held_out_path` param), running at CHECKPOINT ACCEPTANCE so no-mutate/fleet paths can't promote a held-out-failing candidate. `assertBundledMutationHeldOut` in bundled-skill-gate.ts: bundled + `--allow-mutate-bundled` requires a NON-EMPTY held-out (`MIN_HELD_OUT_SIZE = D_SEL_MIN_SIZE` = 5, derived so they can't desync) or hard-refuses (exit 2), for ALL callers (they funnel through `runSkillOpt`); held-out must be task_id-DISJOINT from the benchmark (overlap rejected — can't catch overfitting). `receipt.baseline_sel_score` populated + a real final-test eval (`test_score` + `baseline_test_score`) scoring best + baseline on `split.test`; shared `scoreSkillOnTasks` primitive (validate-gate.ts) backs baseline/final-test/held-out scoring. `--no-mutate` writes proposed.md via `writeProposed` in version-store.ts. `maxRuntimeMin` ENFORCED (wall-clock deadline between steps → `skillopt_runtime_exceeded` → outcome aborted). Three eval-internal ablation opts on `SkillOptOpts` (NOT on CLI): `reflectMode` (`'both'`/`'failure-only'`), `disableValidationGate` (greedy-accept), `optimizerMode` (`'reflect'`/`'one-shot-rewrite'`), recorded in `RunReceipt` + audit `run_start` for replayability; `ROLLOUT_SUCCESS_THRESHOLD = 0.5` named constant for the partition; one-shot fence-strip is anchored (`^```...```$`) so an embedded code sample isn't truncated. Budget no-pricing fix: Claude Haiku 4.5's dateless canonical id `claude-haiku-4-5` is in `src/core/anthropic-pricing.ts` (a `BudgetTracker`-capped run on Haiku otherwise threw `no_pricing` on the FIRST `chat()` of every rollout); `runValidationGate` (validate-gate.ts) scans settled results for `isMustAbortError(error)` (from `worker-pool.ts`; `BUDGET_EXHAUSTED` is in `MUST_ABORT_ERROR_TAGS`) and re-throws so the caller aborts loudly instead of recording a hollow `selScore:0` — ordinary non-abort rollout errors still fail-open to `score:0` (judge-hiccup posture preserved). Pinned by 152 tests across 18 files (foundation + adversarial + v2 surface + E2E PGLite serial), `test/skillopt/bootstrap-from-skill.test.ts` (20 cases), `test/skillopt/rollout.test.ts`, `test/skillopt/validate-gate-abort.test.ts` (3 cases), held-out ENFORCE + one-shot-rewrite unit cases, and e2e (F11 block/allow, bundled no-mutate, runtime deadline, receipt honesty, held-out disjointness, no-DB-pollution). Drives the Track B SkillOpt benchmark suite in the sibling `gbrain-evals` repo. - `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge rejecting ideas with resistance >4.5 "too obvious", stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2`. `judges.ts` exports `runJudge(config, ideas)` + two configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` cognitive_load 0.50 + inversion rule). Calibration cold-start fallback: when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back in `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch); internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. The fire-and-forget IIFE is tracked in a module-scoped `Set>`; `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns the pending count. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then a fallback `process.exit(0)` fires ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive) — closes the PGLite CLI search/query/get-hang class where the IIFE raced disconnect and PGLite's WASM kept Bun's event loop alive. `pages.last_retrieved_at TIMESTAMPTZ NULL` has a full (NOT partial) B-tree index covering both NULL and range branches; full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output via `isLsdOutput()` in `src/core/cycle/transcript-discovery.ts` short-circuiting `isDreamOutput()`. `gbrain eval brainstorm ` is a three-axis conjunctive gate (distance + usefulness + grounding — distance alone is gameable). `gbrain doctor` has a `brainstorm_health` check (migration applied, `search.track_retrieval` setting, calibration cold-start status). `judges.ts` computes the judge token budget via `computeJudgeMaxTokens(ideaCount, modelId)` (named constants `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`; `ANTHROPIC_OUTPUT_CAPS` map: Opus 4.7 32K, Sonnet 4.6 / Haiku 4.5 64K, legacy Claude 3.5 8K) so a large multi-call judge doesn't truncate mid-JSON; with no `modelOverride` the cap routes through the gateway's actual configured chat model via `getChatModel()`. `--save` for both commands persists through the canonical ingestion path: `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so search finds it, no embedding cost at save) THEN renders the saved row to disk via the shared `writePageThrough` helper (file rendered FROM the row so the two sinks can't diverge and `gbrain sync` doesn't churn it). `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message (both-sinks, DB-only when no `sync.repo_path`/repo-not-a-dir, DB-saved-but-file-errored, total-failure → loud `save FAILED … NOT persisted` on stderr + nonzero exit) — closes the silent-false-success class where `--save` printed "Saved" unconditionally even when the DB write failed. `buildIdeaSlug(question, label, nonce?)` adds a random nonce suffix (injectable for tests) so two same-day runs sharing the first 60 slug chars don't clobber. `--json` callers stay DB-only. `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (string `buildBrainstormFrontmatter` untouched). Pinned by `test/last-retrieved.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir asserts search/get/query exit 0 in <15s + daemon-survival), `test/fix-wave-structural.test.ts` (asserts the drain `await` is textually BEFORE `engine.disconnect`), `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm,judges-maxtokens,save}.test.ts`. Open Collider source: `github.com/CL-ML/open-collider`. -- `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` reads `sync.repo_path`, re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown` + `resolvePageFilePath`, and writes the `.md` under the repo so the brain has a committable artifact that round-trips through `gbrain sync`. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts`. +- `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` resolves the disk target from the ASSIGNED source's own working tree (`sources.local_path`), re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown`, and writes the `.md` under that tree's root so the brain has a committable artifact that round-trips through `gbrain sync`. A source with its own `local_path` writes there; a source WITHOUT one falls back to the global `sync.repo_path` ONLY when this is the sole source (then that path is unambiguously this source's tree) and otherwise skips with `source_has_no_local_path` rather than leak into a sibling source's git repo. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_has_no_local_path' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts`. - `src/core/model-id.ts` — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface has no parallel re-implementations of `provider:model` splitting — slash-form ids (`anthropic/claude-sonnet-4-6`) classify correctly instead of falling through to "unknown model". Distinct from the gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`, which throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by `test/model-id.test.ts`. - `src/core/ai/model-resolver.ts:parseModelId` — gateway-side resolver accepts both colon and slash form (`provider:model` and `provider/model`) so a slash-form id resolves to the same recipe at every gateway entry point (chat / embed / rerank) instead of throwing `AIConfigError: model id must be in format provider:model`. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). Batch-load fast path on Postgres uses a single SQL query (fixes the PgBouncer round-trip timeout, ~60s → ~6s), gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1`. Batch projection is `SELECT ... ORDER BY source_id, slug` (NOT `SELECT DISTINCT ON (slug)`, which collapsed same-slug-different-source pages into one scan) so multi-source brains scan each `(source, slug)` row independently. Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`; batch + sequential paths report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check at `doctor.ts:1011-1043` consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings in BOTH `doctor.ts` and `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source ` matches. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. IRON-RULE regression banning stale verb names, the sync_freshness boundary matrix, the D4 regression guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). `Migration` interface carries `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses in a transaction; ignored on PGLite). Key migrations: v14 (handler branches on `engine.kind` for CONCURRENTLY-on-Postgres with invalid-remnant pre-drop via `pg_index.indisvalid`, plain `CREATE INDEX` on PGLite); v15 (`minion_jobs.max_stalled` default 1→5 + backfill non-terminal rows); v24 `rls_backfill_missing_tables` (`sqlFor: { pglite: '' }` no-op — PGLite has no RLS engine, targets subagent tables absent from pglite-schema.ts); v30 `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))` (RLS-enabled under BYPASSRLS; synthesize reads/writes to avoid re-judging); v35 auto-RLS event trigger `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` running `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on new `public.*` tables (no FORCE) + one-time backfill on every existing `public.*` base table whose comment doesn't match `^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}` (per-table failure aborts the offending CREATE TABLE; no EXCEPTION wrap; PGLite no-op via `sqlFor.pglite: ''`; breaking change: intentionally-RLS-off public tables need the GBRAIN:RLS_EXEMPT comment before upgrade); v40 `pages_emotional_weight` (`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`, column-only metadata-only); v46 `mcp_request_log_params_jsonb_normalize` (`UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`, idempotent); v60-v65 six-migration chain wiring source-scoping into `oauth_clients` — v60 (`oauth_clients_source_id_fk`: `source_id TEXT` NULL→`'default'` backfill + FK to `sources(id) ON DELETE SET NULL`), v61 (`federated_read TEXT[] NOT NULL DEFAULT '{}'`), v62 (explicit-CASE backfill so `source_id IS NULL` → `'{}'`), v63 (fail-loud check every row's source_id is in its federated_read array), v64 (FK flipped to `ON DELETE RESTRICT`), v65 (GIN index for array-containment); v68 `eval_candidates_embedding_column` (`eval_candidates.embedding_column TEXT NULL` per-row provenance for `gbrain eval replay` to reproduce the same retrieval space; NULL-tolerant); v108 `pages_embedding_signature` (`pages.embedding_signature TEXT NULL` = `:` stamped via `setPageEmbeddingSignature`; GRANDFATHER — stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current` so NULL is NEVER stale and upgrade never re-embeds the whole corpus; no index; metadata-only); v109 `sources_newest_content_at` (`sources.newest_content_at TIMESTAMPTZ` durable newest-COMMIT HEAD committer time written by `writeSyncAnchor`, read by the REMOTE staleness path instead of shelling to git; mirror in pglite-schema.ts + schema.sql + bootstrap probe); v110 `page_aliases` (`(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` + lookup indexes on `(source_id, alias_norm)` and `(source_id, slug)`; `alias_norm` is `normalizeAlias()` output so WRITE/READ key on the same form; also in `src/core/pglite-schema.ts`); v111 `search_telemetry_rank1_columns` (`ADD COLUMN IF NOT EXISTS` on both engines: `sum_rank1_score`, `count_rank1`, three buckets `rank1_lt_solid`/`rank1_solid`/`rank1_high` on `search_telemetry` — aggregate not per-query rows so rank-1 median drift is bounded-growth; ALTERs right after v57 which created the table); v114 `links_link_source_check_kebab_regex` (#1941, opens `link_source` from the closed allowlist to a kebab-case format gate `^[a-z][a-z0-9]*(-[a-z0-9]+)*$` + `char_length<=64`; Postgres branch uses `NOT VALID` + `VALIDATE CONSTRAINT` with `transaction:false`, PGLite plain DROP+ADD; existing built-ins all satisfy the regex so VALIDATE never fails on existing data). +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). `Migration` interface carries `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses in a transaction; ignored on PGLite). Key migrations: v14 (handler branches on `engine.kind` for CONCURRENTLY-on-Postgres with invalid-remnant pre-drop via `pg_index.indisvalid`, plain `CREATE INDEX` on PGLite); v15 (`minion_jobs.max_stalled` default 1→5 + backfill non-terminal rows); v24 `rls_backfill_missing_tables` (`sqlFor: { pglite: '' }` no-op — PGLite has no RLS engine, targets subagent tables absent from pglite-schema.ts); v30 `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))` (RLS-enabled under BYPASSRLS; synthesize reads/writes to avoid re-judging); v35 auto-RLS event trigger `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` running `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on new `public.*` tables (no FORCE) + one-time backfill on every existing `public.*` base table whose comment doesn't match `^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}` (per-table failure aborts the offending CREATE TABLE; no EXCEPTION wrap; PGLite no-op via `sqlFor.pglite: ''`; breaking change: intentionally-RLS-off public tables need the GBRAIN:RLS_EXEMPT comment before upgrade); v40 `pages_emotional_weight` (`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`, column-only metadata-only); v46 `mcp_request_log_params_jsonb_normalize` (`UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`, idempotent); v60-v65 six-migration chain wiring source-scoping into `oauth_clients` — v60 (`oauth_clients_source_id_fk`: `source_id TEXT` NULL→`'default'` backfill + FK to `sources(id) ON DELETE SET NULL`), v61 (`federated_read TEXT[] NOT NULL DEFAULT '{}'`), v62 (explicit-CASE backfill so `source_id IS NULL` → `'{}'`), v63 (fail-loud check every row's source_id is in its federated_read array), v64 (FK flipped to `ON DELETE RESTRICT`), v65 (GIN index for array-containment); v68 `eval_candidates_embedding_column` (`eval_candidates.embedding_column TEXT NULL` per-row provenance for `gbrain eval replay` to reproduce the same retrieval space; NULL-tolerant); v108 `pages_embedding_signature` (`pages.embedding_signature TEXT NULL` = `:` stamped via `setPageEmbeddingSignature`; GRANDFATHER — stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current` so NULL is NEVER stale and upgrade never re-embeds the whole corpus; no index; metadata-only); v109 `sources_newest_content_at` (`sources.newest_content_at TIMESTAMPTZ` durable newest-COMMIT HEAD committer time written by `writeSyncAnchor`, read by the REMOTE staleness path instead of shelling to git; mirror in pglite-schema.ts + schema.sql + bootstrap probe); v110 `page_aliases` (`(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` + lookup indexes on `(source_id, alias_norm)` and `(source_id, slug)`; `alias_norm` is `normalizeAlias()` output so WRITE/READ key on the same form; also in `src/core/pglite-schema.ts`); v111 `search_telemetry_rank1_columns` (`ADD COLUMN IF NOT EXISTS` on both engines: `sum_rank1_score`, `count_rank1`, three buckets `rank1_lt_solid`/`rank1_solid`/`rank1_high` on `search_telemetry` — aggregate not per-query rows so rank-1 median drift is bounded-growth; ALTERs right after v57 which created the table); v114 `links_link_source_check_kebab_regex` (#1941, opens `link_source` from the closed allowlist to a kebab-case format gate `^[a-z][a-z0-9]*(-[a-z0-9]+)*$` + `char_length<=64`; Postgres branch uses `NOT VALID` + `VALIDATE CONSTRAINT` with `transaction:false`, PGLite plain DROP+ADD; existing built-ins all satisfy the regex so VALIDATE never fails on existing data); v116 `code_edges_source_backfill_and_callee_index` (#2073, idempotent: backfills NULL `code_edges_symbol`/`code_edges_chunk` `source_id` from each edge's `from_chunk` page — NULL never matched a scoped `AND source_id = …` filter so scoped `code-callers`/`code-callees` returned 0 rows on multi-source brains — plus plain `CREATE INDEX` on `from_symbol_qualified` for both edge tables, which had no index and seq-scanned per BFS node). The dedup-index self-heal (`timeline_dedup_index`, see `timeline-dedup-repair.ts`) is NOT version-gated: `runMigrations` invokes `repairTimelineDedupIndex` on every pass (including the no-pending early-return path) because a merge-renumbered migration can leave the version counter past the index change while the index stays the old shape. +- `src/core/timeline-dedup-repair.ts` (#2038) — schema-drift self-heal for `idx_timeline_dedup`. The migration that widened the dedup index from `(page_id, date, summary)` to `(page_id, date, summary, source)` was renumbered during a master merge, so a brain that ran the old variant has its version counter stamped past the change while the index keeps the 3-column shape — and every `addTimelineEntry` batch then fails its 4-column `ON CONFLICT`, silently breaking timeline writes brain-wide. The version counter can't detect this, so the repair is keyed off the actual index SHAPE: `checkTimelineDedupIndex(engine)` returns `{tablePresent, indexPresent, columns, needsRepair}` (read-only; powers the `timeline_dedup_index` doctor check) and `repairTimelineDedupIndex(engine)` dedupes-then-rebuilds the index. `runMigrations` invokes the repair on every pass (including the no-pending early-return path); idempotent no-op when the index is already 4-column. `gbrain apply-migrations --force-schema` triggers it on demand. Pinned by `test/timeline-dedup-repair.test.ts`. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY `\r`-rewriting; non-TTY plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. `emitHumanLine` is prefix-aware — inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts` it prepends `[id] ` (and TTY-rewrite mode `\r\x1b[2K` carries the prefix inside the clear-to-EOL escape); `emitJson` is intentionally NOT prefixed so NDJSON consumers don't choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as active prefix; nested wraps replace then restore), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log`/`console.error`). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog`/`serr` fall through to bare `console.log`/`console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form) to defeat log-injection through newline/control-character names. Coverage: `src/commands/sync.ts` performSync + callees, `src/commands/embed.ts` runEmbedCore + helpers, `src/core/progress.ts` emitHumanLine. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -299,7 +301,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/cycle/synthesize.ts` — Synthesize phase: conversation-transcript-to-brain pipeline. Reads `dream.synthesize.session_corpus_dir`, runs a cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at`) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. `--dry-run` runs Haiku, skips Sonnet. Subagent never gets fs-write access. `renderPageToMarkdown` (exported) stamps `dream_generated: true` + `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the summary index — this marker is the explicit identity surface `isDreamOutput` checks in `transcript-discovery.ts`. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` takes a `verdictModel` param loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically; per-chunk char budget = `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token` (non-Anthropic ids fall back to a 180K-token safe default + once-per-process stderr warn); operator overrides `dream.synthesize.max_prompt_tokens` (floor 100K, wins) and `dream.synthesize.max_chunks_per_transcript` (default 24). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts` so raising the cap on next run re-attempts cleanly. Bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by terminal-error classification in `subagent.ts`. Verdict routing is gateway-routed: `makeJudgeClient(verdictModel)` (exported, replacing `makeHaikuClient()`) mirrors `tryBuildGatewayClient` in `src/core/think/index.ts` — a construction-time provider/key probe returns `null` on a clear miss (unknown provider id via `resolveRecipe` AIConfigError, or Anthropic provider with no key via `hasAnthropicKey()`). The verdict loop wraps `judgeSignificance` in try/catch for `AIConfigError` so mid-run provider failures surface as per-transcript `worth=false, reasons=['gateway error: ...']` instead of crashing the phase. Canonical config key `models.dream.synthesize_verdict` (per `PER_TASK_KEYS` in `src/core/model-config.ts`); `JudgeClient` signature preserved verbatim for test-seam stability; CI guard `scripts/check-gateway-routed-no-direct-anthropic.sh` prevents reintroducing `new Anthropic()` here or in `think/index.ts`. At the queue.add boundary (lines 395-404) a conditional `anthropic:` prefix is applied ONLY when the resolved model has no colon AND starts with `claude-` (because `resolveModel` returns bare ids from `TIER_DEFAULTS`/`DEFAULT_ALIASES` and the subagent validator requires `provider:model` form) — avoids changing the shared constants which would ripple across every `resolveModel` caller. Pinned by `test/cycle/synthesize-gateway-adapter.test.ts`, `test/e2e/dream-synthesize-pglite.test.ts` (gateway-adapter mid-run AIConfigError catch), `test/cycle/regression-pr-wave-r1-r2-r4.test.ts`. - `scripts/check-gateway-routed-no-direct-anthropic.sh` — CI guard that fails the build if `src/core/cycle/synthesize.ts` or `src/core/think/index.ts` reintroduces a runtime `new Anthropic()` constructor call or a value-shaped `import Anthropic from '@anthropic-ai/sdk'` import. Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay allowed for adapter types; comment lines (`//` or ` *` prefixes) are excluded so historical JSDoc doesn't false-fire. Mirrors `scripts/check-jsonb-pattern.sh`. Wired into `bun run verify` and `bun run check:all`. Extend `GUARDED_FILES` when migrating another file off direct SDK construction. - `src/core/cycle/patterns.ts` — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh. -- `src/core/cycle/extract-facts.ts` — extract_facts cycle phase. Fence is canonical: per-page wipe (`deleteFactsForPage`) + reinsert from `parseFactsFence` + `extractFactsFromFenceText` + `engine.insertFacts`. Empty-fence guard refuses when legacy rows (`row_num IS NULL AND entity_slug IS NOT NULL`) pend backfill (status: warn, hint: `gbrain apply-migrations --yes`). A phantom-redirect pre-pass runs AFTER the legacy-row guard, BEFORE the main reconcile loop: when `opts.brainDir` is set, `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun)` walks unprefixed-slug pages capped by `GBRAIN_PHANTOM_REDIRECT_LIMIT` (default 50). The pass returns `touched_canonicals` — canonical slugs whose disk fence merged with phantom rows; `runExtractFacts` UNIONs them into the main reconcile slug set so canonical's DB facts derive from the merged fence in the same cycle (handles phantom-had-only-on-disk-fence). `ExtractFactsResult` gains six phantom fields: `phantomsScanned`, `phantomsRedirected`, `phantomsAmbiguous`, `phantomsSkippedDrift`, `phantomsLockBusy`, `phantomsMorePending`. Three bubble to `CycleReport.totals` (`phantoms_redirected`, `phantoms_ambiguous`, `phantoms_skipped_drift`). +- `src/core/cycle/extract-facts.ts` — extract_facts cycle phase. Fence is canonical: per-page wipe (`deleteFactsForPage`) + reinsert from `parseFactsFence` + `extractFactsFromFenceText` + `engine.insertFacts`. #1928: the per-page wipe passes `excludeSourcePrefixes: ['cli:']` so conversation facts (written by `extract-conversation-facts`, on pages with NO `## Facts` fence to recreate them from) survive the reconcile instead of being deleted-with-nothing-to-reinsert. The destructive phase no longer inherits a failed sync's full-brain walk: `slugs: []` (a real incremental no-op) is distinguished from `slugs: undefined` (full-walk intent) by presence, not length. `runPhaseExtractFacts` (cycle.ts) surfaces a `warn` (`net_fact_deletion`) when the reconcile deletes at least `NET_DELETION_WARN_FLOOR` (50) more facts than it reinserts — the exact signature of the conversation-facts wipe, which previously read as a silent `ok`. Empty-fence guard refuses when legacy rows (`row_num IS NULL AND entity_slug IS NOT NULL`) pend backfill (status: warn, hint: `gbrain apply-migrations --yes`). A phantom-redirect pre-pass runs AFTER the legacy-row guard, BEFORE the main reconcile loop: when `opts.brainDir` is set, `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun)` walks unprefixed-slug pages capped by `GBRAIN_PHANTOM_REDIRECT_LIMIT` (default 50). The pass returns `touched_canonicals` — canonical slugs whose disk fence merged with phantom rows; `runExtractFacts` UNIONs them into the main reconcile slug set so canonical's DB facts derive from the merged fence in the same cycle (handles phantom-had-only-on-disk-fence). `ExtractFactsResult` gains six phantom fields: `phantomsScanned`, `phantomsRedirected`, `phantomsAmbiguous`, `phantomsSkippedDrift`, `phantomsLockBusy`, `phantomsMorePending`. Three bubble to `CycleReport.totals` (`phantoms_redirected`, `phantoms_ambiguous`, `phantoms_skipped_drift`). - `src/core/entities/resolve.ts` — Free-form entity name → canonical slug resolution. `resolveEntitySlug(engine, source_id, raw)`: exact slug → fuzzy (pg_trgm @ 0.4 threshold) → bare-name prefix expansion (`people/-%` then `companies/-%`, `connection_count` correlated-subquery tiebreaker) → deterministic `slugify` fallback. Two helpers for the phantom-redirect pass: `resolvePhantomCanonical(engine, sourceId, phantomSlug)` SKIPS the exact-slug step (a phantom slug `'alice'` would exact-match itself and no-op the redirect); returns the canonical only when non-null AND contains `/`. `findPrefixCandidates(engine, sourceId, token)` is a standalone SQL query returning ALL candidates across `PREFIX_EXPANSION_DIRS` (hardcoded `['people', 'companies']`) via `slug LIKE ANY($N::text[])` over patterns `dir/token` + `dir/token-%`, cap of 10 ordered by `connection_count DESC, slug ASC` — NOT a wrapper around `tryPrefixExpansion` (that path returns per-dir top-1 and suppresses ambiguity by design). Pinned by `test/phantom-redirect.test.ts` (resolvePhantomCanonical 3 cases + findPrefixCandidates 6 cases incl. multi-dir ambiguity and the `people/aliceberg`-doesn't-match-`alice` false-positive guard). - `src/core/cycle/phantom-redirect.ts` — Phantom-redirect orchestrator. Exports `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun): Promise` (per-cycle wrapper acquiring the `gbrain-sync` writer lock once for the whole pass, 30s bounded retry, walks up to `GBRAIN_PHANTOM_REDIRECT_LIMIT` unprefixed phantoms) + `tryRedirectPhantom(engine, page, sourceId, brainDir, dryRun): Promise` + `stripFenceAndFrontmatterAndLeadingH1` (pure body-shape gate helper — strips facts fence incl. preceding `## Facts` heading and the leading H1; zero residue = phantom). Handler order: body-shape gate → `resolvePhantomCanonical` (bypasses exact-self-match) → `findPrefixCandidates` ambiguity check → `fenceDbDrift` bi-directional check → dry-run early exit → materialize canonical via `serializeMarkdown` if DB-only → append phantom fence rows to canonical's disk fence with `(claim, valid_from)` dedup-guard + row_num continuation → `engine.refreshPageBody` with SHA-256 content_hash recomputed via the import-file shape → `engine.migrateFactsToCanonical` (lossless) → `engine.rewriteLinks` (DB FK rewrite; wiki-link text rewrite is a documented follow-up) → `engine.softDeletePage` + `engine.deleteFactsForPage(phantom)` + `fs.unlinkSync(phantomPath)`. `RedirectResult.canonical` populated on `'redirected'` (incl. dry-run preview) so the caller builds `touched_canonicals`. Idempotent on re-run: phantom soft-deleted → predicate fails (`deleted_at IS NULL`); migrate UPDATE matches no rows; dedup-guard prevents double-append. - `src/core/facts/phantom-audit.ts` — JSONL audit at `${resolveAuditDir()}/phantoms-YYYY-Www.jsonl`. Pattern copy of `src/core/audit-slug-fallback.ts` (ISO-week rotation, honors `GBRAIN_AUDIT_DIR`). Exports `logPhantomEvent(record)` + `readRecentPhantomEvents(days)` + `computePhantomAuditFilename(now?)`. Records every outcome: `redirected | ambiguous | drift | no_canonical | not_phantom_has_residue | pass_skipped_lock_busy`. Best-effort writes — stderr warn on failure, never throws. Separate file from `stub-guard-audit.ts` (distinct consumer + lifecycle: stub-guard logs PREVENTIVE blocks; phantom-audit logs CLEANUP decisions, to be read by a future `phantoms_pending` doctor check). diff --git a/docs/embedding-migrations.md b/docs/embedding-migrations.md index e9c425128..49e08765b 100644 --- a/docs/embedding-migrations.md +++ b/docs/embedding-migrations.md @@ -43,8 +43,8 @@ genuinely has to change. Switching dimensions requires: 1. Dropping the HNSW vector index (pgvector won't survive an `ALTER COLUMN TYPE`). -2. Altering the column type (Postgres only — PGLite cannot do this). -3. Wiping every existing embedding (the old vectors are unusable in the new space). +2. Wiping every existing embedding (the old vectors are unusable in the new space — and pgvector refuses to cast them across dimensions, so this must happen before the alter). +3. Altering the column type (Postgres only — PGLite cannot do this). 4. Re-embedding the entire corpus (can take hours on a 50K-page brain and costs $1-100 in API calls depending on model). 5. Conditionally recreating the index (HNSW supports up to 2000 dimensions per pgvector; above that you must use exact scans). @@ -115,12 +115,17 @@ BEGIN; -- 1. Drop the HNSW index. It can't survive the column type change. DROP INDEX IF EXISTS idx_chunks_embedding; --- 2. Alter the column type. -ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(); - --- 3. Clear stale embeddings so they don't survive into the new space. +-- 2. Clear stale embeddings FIRST. This must happen BEFORE the column +-- alter: pgvector refuses to cast existing vectors across dimensions +-- ("expected dimensions, not "), so altering a +-- column that still holds old-width vectors aborts the transaction. +-- NULLs cast fine. (The old vectors are unusable in the new space +-- anyway — this is the wipe step from the rationale above.) UPDATE content_chunks SET embedding = NULL, embedded_at = NULL; +-- 3. Alter the column type (all rows are NULL now, so the cast succeeds). +ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(); + -- 4. Recreate the HNSW index ONLY IF dims <= 2000. Above that, leave it -- indexless and rely on exact scans (gbrain searchVector handles this -- automatically — search just gets slower, not broken). diff --git a/package.json b/package.json index 85559b546..cde5c8fda 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.40.0" + "version": "0.42.41.0" } diff --git a/src/cli.ts b/src/cli.ts index 09d187cea..e80d7e4b8 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -360,30 +360,83 @@ async function main() { // // Defense-in-depth (adversarial-review C13): `engine.disconnect()` itself // can hang on PGLite (db.close() or releaseLock racing OS-level FS state). - // Install an unref'd setTimeout hard-exit fallback BEFORE entering the - // try/catch/finally so a hung disconnect cannot defeat the force-exit - // contract. Daemons (`serve`) are excluded so they stay alive. + // The unref'd hard-exit fallback is armed inside the `finally` below, so it + // bounds ONLY the teardown phase (drain + disconnect) — the same placement + // as the fall-through owner-disconnect site later in this file. It used to + // be armed HERE, before the try, which silently killed any op whose BODY ran + // past the deadline: on a slow Postgres pooler (6-10s per fresh connection) + // a healthy `gbrain search` was force-exited mid-handler with code 0 and + // ZERO stdout — an empty "success" indistinguishable from no results. The + // exitCode honor (v0.42.20.0) can't help there: a mid-op kill fires before + // any error path sets exitCode. Op-body wallclock bounds are the read-scope + // withTimeout wrap inside the try below, not this teardown backstop. + // Daemons (`serve`) are excluded so they stay alive. const DISCONNECT_HARD_DEADLINE_MS = 10_000; + // Wallclock bound for READ-scope op handlers. With the hard-deadline timer + // correctly scoped to teardown, a genuinely WEDGED read handler (hung pooler + // connection mid-query) would otherwise hang the CLI forever — the #1633 + // zombie class the old (buggy) pre-try timer accidentally bounded at 10s. + // 180s sits far above any healthy slow-pooler run (6-10s/connection); + // --timeout=Ns overrides. Writes/admin stay unbounded: a long import/embed + // must never be killed by a default deadline. + const READ_OP_TIMEOUT_MS = 180_000; let forceExitTimer: ReturnType | undefined; - if (shouldForceExitAfterMain()) { - forceExitTimer = setTimeout(() => { - console.warn( - `[cli] engine.disconnect() did not return within ${DISCONNECT_HARD_DEADLINE_MS}ms — force-exiting`, - ); - // v0.42.20.0 (codex): honor an exit code an errored op already set — - // a bare process.exit(0) here would mask a failed op as success if the - // drain/disconnect then hangs. - process.exit(process.exitCode ?? 0); - }, DISCONNECT_HARD_DEADLINE_MS); - // unref so the timer itself doesn't keep the event loop alive — only - // the actual pending work (PGLite WASM handle) does. Without unref, - // we'd block a clean exit by 10s on every successful CLI run. - forceExitTimer.unref?.(); - } + // Set when a wallclock bound fired. The abandoned (timed-out but still + // running) handler can hold ref'd sockets/timers that keep Bun's event loop + // alive after main() returns — so the finally must hard-exit after teardown + // on this path, or the timeout print is followed by an immortal process: + // the same zombie class, resurrected through the timeout door (adversarial + // review finding). + let wallclockTimedOut = false; try { - const ctx = await makeContext(engine, params); - const rawResult = await op.handler(ctx, params); + const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts'); + const wallclockMs = getCliOptions().timeoutMs ?? READ_OP_TIMEOUT_MS; + const onWallclockTimeout = (e: InstanceType) => { + const hint = getCliOptions().timeoutMs + ? '' + : ` (default ${e.ms}ms; pass --timeout=Ns to override)`; + console.error(`${e.label} timed out${hint}.`); + process.exitCode = 124; + wallclockTimedOut = true; + }; + + // Context build does DB I/O (resolveSourceId) and runs for EVERY op — + // a wedged pooler connection here would otherwise hang reads, writes, + // and admin alike with no bound at all (adversarial review finding). + let ctx: Awaited>; + try { + ctx = await withTimeout( + makeContext(engine, params), + wallclockMs, + `gbrain ${command}: context`, + ); + } catch (e: unknown) { + if (e instanceof OperationTimeoutError) { + onWallclockTimeout(e); + return; // the finally below still drains + disconnects, then exits + } + throw e; + } + + let rawResult: unknown; + if (op.scope === 'read') { + try { + rawResult = await withTimeout( + op.handler(ctx, params), + wallclockMs, + `gbrain ${command}`, + ); + } catch (e: unknown) { + if (e instanceof OperationTimeoutError) { + onWallclockTimeout(e); + return; // the finally below still drains + disconnects, then exits + } + throw e; + } + } else { + rawResult = await op.handler(ctx, params); + } // ENG-2 (renderer parity by data shape): JSON-round-trip the local-engine // path's return value so renderers see the same shape they'd see on the // routed path. Date → ISO string; bigint → string (postgres.js shape); @@ -396,7 +449,8 @@ async function main() { // STILL runs (drains every background-work sink + disconnects). A bare // process.exit(1) here would skip the finally → skip the drain + disconnect // (leaves facts/cache/eval-capture writes racing teardown). The finally's - // drain bounds teardown; the outer hard-deadline timer bounds a hung one. + // drain bounds teardown; the hard-deadline timer armed at teardown entry + // bounds a hung one. if (e instanceof OperationError) { console.error(`Error [${e.code}]: ${e.message}`); if (e.suggestion) console.error(` Fix: ${e.suggestion}`); @@ -412,11 +466,36 @@ async function main() { // DB logIngest gets the freshest live-engine window. 1s per-sink timeout: // read paths with no pending work pay the ~0ms fast path; capture/import // that DO enqueue pay up to 1s (+ facts shutdown grace) while in-flight - // Haiku finishes. The unref'd hard-deadline timer above is the backstop if - // disconnect or a lingering socket keeps Bun's loop alive. + // Haiku finishes. The unref'd hard-deadline timer armed here is the + // backstop if disconnect or a lingering socket keeps Bun's loop alive — + // armed at teardown entry (NOT before the op body; see the C13 comment + // above) so a slow-but-progressing op handler is never killed mid-flight. + if (shouldForceExitAfterMain()) { + forceExitTimer = setTimeout(() => { + console.warn( + `[cli] engine.disconnect() did not return within ${DISCONNECT_HARD_DEADLINE_MS}ms — force-exiting`, + ); + // v0.42.20.0 (codex): honor an exit code an errored op already set — + // a bare process.exit(0) here would mask a failed op as success if the + // drain/disconnect then hangs. + process.exit(process.exitCode ?? 0); + }, DISCONNECT_HARD_DEADLINE_MS); + // unref so the timer itself doesn't keep the event loop alive — only + // the actual pending work (PGLite WASM handle) does. Without unref, + // we'd block a clean exit by 10s on every successful CLI run. + forceExitTimer.unref?.(); + } await drainAllBackgroundWorkForCliExit({ timeoutMs: 1000 }); await engine.disconnect(); if (forceExitTimer) clearTimeout(forceExitTimer); + // Wallclock-timeout path: teardown is done, but the ABANDONED handler + // (withTimeout races, it does not cancel) can still hold ref'd sockets / + // SDK retry timers that keep Bun's event loop alive indefinitely. With + // the hard-deadline timer just cleared, nothing else bounds that — exit + // explicitly. Safe: drain + disconnect completed on the lines above. + if (wallclockTimedOut) { + process.exit(process.exitCode ?? 124); + } } } diff --git a/src/commands/autopilot.ts b/src/commands/autopilot.ts index d5831df74..7ac05cfbc 100644 --- a/src/commands/autopilot.ts +++ b/src/commands/autopilot.ts @@ -529,8 +529,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) { autopilotReconnectFails = 0; // reset on success } catch (probeErr) { try { - await engine.disconnect(); - await (engine as any).connect?.(); + // #2034: use reconnect() — it restores the config captured at connect() + // and avoids the null-connection window. The previous + // `disconnect()` + bare `connect()` lost the config (throwing + // `database_url undefined` on every retry → FATAL restart-loop on any + // transient DB blip) AND tore down the pool postgres.js can otherwise + // self-heal. + await engine.reconnect({ error: probeErr }); autopilotReconnectFails = 0; } catch (e) { logError('reconnect', e); diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 99fcdda8b..028a8ab4d 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -510,6 +510,33 @@ export async function doctorReportRemote(engine: BrainEngine): Promise { const ageH = Math.floor(s.age_ms / 3600_000); - const source = s.id.startsWith('gbrain-sync:') ? s.id.slice('gbrain-sync:'.length) : null; - const breakHint = source ? `gbrain sync --break-lock --source ${source}` : `gbrain sync --break-lock`; + let breakHint = 'gbrain doctor'; + if (s.id.startsWith('gbrain-sync:')) { + breakHint = `gbrain sync --break-lock --source ${s.id.slice('gbrain-sync:'.length)}`; + } else if (s.id.startsWith('gbrain-cycle:')) { + breakHint = `gbrain dream --break-lock --source ${s.id.slice('gbrain-cycle:'.length)}`; + } else if (s.id === 'gbrain-cycle') { + breakHint = 'gbrain dream --break-lock'; + } return ` ${s.id} (pid ${s.holder_pid} on ${s.holder_host}, age ${ageH}h) → ${breakHint}`; }); const tail = stale.length > 10 ? ` ... and ${stale.length - 10} more.` : null; @@ -2614,7 +2647,7 @@ async function checkEmbeddingEnvOverride(engine: BrainEngine): Promise { }; } -async function checkSubagentCapability(engine: BrainEngine): Promise { +export async function checkSubagentCapability(engine: BrainEngine): Promise { try { const { classifyCapabilities } = await import('../core/ai/capabilities.ts'); const tierSubagent = await engine.getConfig('models.tier.subagent'); @@ -2675,8 +2708,11 @@ async function checkSubagentCapability(engine: BrainEngine): Promise { const { loadConfig } = await import('../core/config.ts'); const cfg = loadConfig(); const chatModel = cfg?.chat_model; + const gatewayLoopRaw = await engine.getConfig('agent.use_gateway_loop').catch(() => null); + const gatewayLoopEnabled = typeof gatewayLoopRaw === 'string' + && ['true', '1', 'yes', 'on'].includes(gatewayLoopRaw.trim().toLowerCase()); const { isAnthropicProvider } = await import('../core/model-config.ts'); - if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY) { + if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY && !gatewayLoopEnabled) { return { name: 'subagent_capability', status: 'warn', @@ -5610,8 +5646,29 @@ export async function buildChecks( "SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization')", ))[0]?.count ?? 0; - const linkPct = ((health.link_coverage ?? 0) * 100).toFixed(0); - const timelinePct = ((health.timeline_coverage ?? 0) * 100).toFixed(0); + // Compute coverage against eligible entities only — exclude test fixtures + // (`tools/gbrain/test/*`) and template stubs (`templates/new-person`) so + // that brains seeded only with code sources don't get spurious warnings + // about missing link/timeline coverage on pages that are test fixtures, not + // real knowledge entities. + const eligibleStats = (await engine.executeRaw<{ entities: number; linked_from: number; timeline: number }>( + `WITH eligible AS ( + SELECT id FROM pages + WHERE type IN ('entity','person','company','organization') + AND slug NOT LIKE 'tools/gbrain/test/%' + AND slug <> 'templates/new-person' + ) + SELECT + (SELECT count(*)::int FROM eligible) AS entities, + (SELECT count(DISTINCT from_page_id)::int FROM links WHERE from_page_id IN (SELECT id FROM eligible)) AS linked_from, + (SELECT count(DISTINCT page_id)::int FROM timeline_entries WHERE page_id IN (SELECT id FROM eligible)) AS timeline`, + ))[0] ?? { entities: entityCount, linked_from: 0, timeline: 0 }; + + const eligibleEntityCount = Number(eligibleStats.entities ?? entityCount); + const linkCoverage = eligibleEntityCount > 0 ? Number(eligibleStats.linked_from ?? 0) / eligibleEntityCount : 0; + const timelineCoverage = eligibleEntityCount > 0 ? Number(eligibleStats.timeline ?? 0) / eligibleEntityCount : 0; + const linkPct = (linkCoverage * 100).toFixed(0); + const timelinePct = (timelineCoverage * 100).toFixed(0); if (entityCount === 0) { // Markdown-only / journal / wiki brain — no entity pages to compute // coverage against. Coverage formula is structurally inapplicable. @@ -5620,13 +5677,19 @@ export async function buildChecks( status: 'ok', message: 'No entity pages — graph_coverage not applicable (markdown-only brain)', }); - } else if ((health.link_coverage ?? 0) >= 0.5 && (health.timeline_coverage ?? 0) >= 0.5) { + } else if (eligibleEntityCount === 0) { + checks.push({ + name: 'graph_coverage', + status: 'ok', + message: `Only code/test fixture entity pages found (${entityCount}); graph_coverage not applicable`, + }); + } else if (linkCoverage >= 0.5 && timelineCoverage >= 0.5) { checks.push({ name: 'graph_coverage', status: 'ok', message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}%` }); } else { checks.push({ name: 'graph_coverage', status: 'warn', - message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${entityCount} entity pages). Run: gbrain extract all`, + message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${eligibleEntityCount} entity pages). Run: gbrain extract all`, }); } @@ -6162,12 +6225,29 @@ export async function buildChecks( .slice(0, 3) .map(([s, n]) => `${s}=${n}`) .join(', '); + // Audit events are evidence, not automatically breakage. A large code + // source can legitimately emit many WARN events (oversize/markup-heavy) + // while remaining searchable and intentionally flagged. Fail on hard + // dispositions (content actually blocked or hidden); warn on soft + // dispositions or volume. This keeps doctor from treating expected + // code-corpus telemetry as an unhealthy brain. + // + // v0.42 renamed the hard path: a rejected page emits `reject` and a + // quarantined (hidden) junk page emits `quarantine`; `hard_block` is now + // only the pre-v0.42 legacy alias. Counting `hard_block` alone let fresh + // junk-ingest evidence (`reject`/`quarantine`) clear as `ok` whenever + // fewer than 10 events landed. `flag` is a warn disposition (still + // searchable, agent warned on retrieval), so it joins `soft_block`. + const hardBlocked = + summary.by_type.hard_block + summary.by_type.reject + summary.by_type.quarantine; + const softBlocked = summary.by_type.soft_block + summary.by_type.flag; const status: 'ok' | 'warn' | 'fail' = - events.length >= 100 ? 'fail' : events.length >= 10 ? 'warn' : 'ok'; + hardBlocked > 0 ? 'fail' : + (softBlocked > 0 || events.length >= 10) ? 'warn' : 'ok'; checks.push({ name: 'content_sanity_audit_recent', status, - message: `${events.length} events (hard=${summary.by_type.hard_block} soft=${summary.by_type.soft_block} warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`, + message: `${events.length} events (hard=${hardBlocked} [hard_block=${summary.by_type.hard_block} reject=${summary.by_type.reject} quarantine=${summary.by_type.quarantine}] soft=${softBlocked} [soft_block=${summary.by_type.soft_block} flag=${summary.by_type.flag}] warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`, }); } } catch (err) { @@ -7145,7 +7225,10 @@ export async function runDoctor( } catch { /* best-effort */ } } - process.exit(hasFail ? 1 : 0); + // Use process.exitCode instead of process.exit() so cleanup handlers + // (e.g. Bun unload events, open database connections) still run before + // the process terminates. process.exit() is a hard kill that bypasses them. + process.exitCode = hasFail ? 1 : 0; } // --------------------------------------------------------------------------- diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 7ef9e337c..e6b5a4077 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -855,6 +855,17 @@ Status (v0.42): if (!jsonMode) { console.log(`Timeline from meetings: ${r.entries_created} entries on ${r.entities_touched} entity pages from ${r.meetings_scanned} meetings`); } + // #2057 (codex): batch failures are no longer swallowed silently — make + // them visible at the command surface (and non-zero exit) instead of + // printing a clean "N entries" success over failed inserts. + if (r.batch_errors > 0) { + console.error( + `[extract timeline] ${r.batch_errors} batch(es) failed to insert` + + (r.first_batch_error ? ` (first error: ${r.first_batch_error})` : '') + + ` — timeline is incomplete.`, + ); + process.exitCode = 1; + } } else if (byMention || ner) { // v0.41.18.0 (T7): combined --by-mention + --ner walk shares one // gazetteer; saves an entire pass on big brains. When only one diff --git a/src/commands/import.ts b/src/commands/import.ts index 1d4575fcb..d9b160a4d 100644 --- a/src/commands/import.ts +++ b/src/commands/import.ts @@ -11,6 +11,7 @@ import { isCodeFilePath, isMarkdownFilePath, isImageFilePath as isImageFilePathFromSync, + pruneDir, type SyncStrategy, } from '../core/sync.ts'; import { sortNewestFirst } from '../core/sort-newest-first.ts'; @@ -512,6 +513,51 @@ function isCollectibleForWalker( } } +/** + * Git-aware fast path for `collectSyncableFiles`. Returns the strategy-filtered + * list of syncable files when `dir` is inside a git work tree (paths absolute, + * sorted), or `null` when `dir` is not a git repo / git is unavailable — in + * which case the caller falls back to the recursive FS walk. + * + * Honors `.gitignore` (the whole point): `git ls-files --cached --others + * --exclude-standard` lists tracked + untracked-not-ignored files, so vendored + * / build / generated trees never reach the importer. `-z` (NUL-delimited) + * survives paths with spaces/newlines. Each path is lstat-checked to preserve + * the walker's no-symlink policy and to drop submodule gitlinks (which surface + * as a single non-regular entry). + */ +function gitListSyncableFiles( + dir: string, + strategy: SyncStrategy, + multimodalOn: boolean, +): string[] | null { + let stdout: string; + try { + stdout = execFileSync( + 'git', + ['-C', dir, 'ls-files', '--cached', '--others', '--exclude-standard', '-z'], + { encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] }, + ); + } catch { + return null; // not a git work tree, or git not on PATH → FS-walk fallback + } + const files: string[] = []; + for (const rel of stdout.split('\0')) { + if (!rel) continue; + if (!isCollectibleForWalker(rel, strategy, multimodalOn)) continue; + const full = join(dir, rel); + let st; + try { + st = lstatSync(full); + } catch { + continue; // ls-files raced a deletion, or unreadable + } + if (st.isSymbolicLink() || !st.isFile()) continue; + files.push(full); + } + return files.sort(); +} + /** * v0.31.2 (codex C4 + C5 + C8): unified walker with five hardenings: * @@ -532,6 +578,19 @@ function isCollectibleForWalker( export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): string[] { const strategy: SyncStrategy = opts.strategy ?? 'markdown'; const multimodalOn = process.env.GBRAIN_EMBEDDING_MULTIMODAL === 'true'; + + // v0.42.x (#1159 --respect-gitignore / #1483 .gbrainignore): when `dir` is a + // git work tree, enumerate via `git ls-files` so the walk honors + // `.gitignore`. Pre-fix the recursive FS walk below descended into every + // git-ignored tree — `vendor/` (PHP Composer), `storage/`, `public/build/`, + // etc. — so a Laravel/PHP repo's `--strategy code` sync tried to import ~50k + // dependency/build files (and bloated DB + embedding cost on any repo with + // vendored data/fixtures). `--cached --others --exclude-standard` = tracked + // PLUS untracked-not-ignored, so uncommitted source is still indexed. Non-git + // dirs (or git unavailable) fall through to the FS walk below. + const gitFiles = gitListSyncableFiles(dir, strategy, multimodalOn); + if (gitFiles) return gitFiles; + const maxDepth = resolveMaxWalkDepth(); const visitedInodes = new Map(); const files: string[] = []; @@ -548,11 +607,11 @@ export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): strin return; } for (const entry of entries) { - // Skip hidden dirs (.git, .claude, .raw, etc.) and `node_modules`/`ops`. - // Same set the legacy walkers honored, surfaced once at the top of - // every iteration. - if (entry.startsWith('.')) continue; - if (entry === 'node_modules' || entry === 'ops') continue; + // Descent-time prune through the canonical gate (single source of truth + // in core/sync.ts) instead of a hand-maintained inline list that drifted + // from it. Skips hidden dirs (`.git`, `.raw`, etc.), `node_modules`, + // `vendor`, `dist`, `build`, `venv` (#2020), `ops`, and git submodules. + if (!pruneDir(entry, d)) continue; const full = join(d, entry); let stat; diff --git a/src/commands/init.ts b/src/commands/init.ts index 08cf8432c..69e3d5b93 100644 --- a/src/commands/init.ts +++ b/src/commands/init.ts @@ -6,7 +6,7 @@ import { homedir } from 'os'; const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); -import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, type GBrainConfig } from '../core/config.ts'; +import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts'; import { createEngine } from '../core/engine-factory.ts'; import { discoverOAuth, mintClientCredentialsToken, smokeTestMcp } from '../core/remote-mcp-probe.ts'; import { runInitEmbedCheck } from '../core/init-embed-check.ts'; @@ -133,7 +133,11 @@ export async function runInit(args: string[]) { if (manualUrl) { databaseUrl = manualUrl; } else if (isNonInteractive) { - const envUrl = process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL; + // effectiveEnvDatabaseUrl applies the #427 guard: a DATABASE_URL that Bun + // auto-loaded from a .env in cwd must not seed a brain config — that is + // the "init inside a web-app checkout writes the app's DB into + // ~/.gbrain/config.json" failure mode. + const envUrl = effectiveEnvDatabaseUrl(); if (envUrl) { databaseUrl = envUrl; } else { diff --git a/src/commands/migrate-engine.ts b/src/commands/migrate-engine.ts index 5ce3027a1..29d04e4c4 100644 --- a/src/commands/migrate-engine.ts +++ b/src/commands/migrate-engine.ts @@ -8,7 +8,7 @@ */ import { createEngine } from '../core/engine-factory.ts'; -import { loadConfig, saveConfig, toEngineConfig, gbrainPath, type GBrainConfig } from '../core/config.ts'; +import { loadConfig, saveConfig, toEngineConfig, gbrainPath, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts'; import type { BrainEngine } from '../core/engine.ts'; import type { EngineConfig } from '../core/types.ts'; import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs'; @@ -91,7 +91,8 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[] // Build target config const targetConfig: EngineConfig = { engine: opts.targetEngine }; if (opts.targetEngine === 'postgres') { - targetConfig.database_url = opts.targetUrl || process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL; + // #427 guard: don't let a cwd-.env DATABASE_URL become a migration target. + targetConfig.database_url = opts.targetUrl || effectiveEnvDatabaseUrl(); if (!targetConfig.database_url) { console.error('Target is Supabase but no connection string provided. Use: --url '); process.exit(1); diff --git a/src/core/ai/gateway.ts b/src/core/ai/gateway.ts index 21c5ce57a..cf41b1ab0 100644 --- a/src/core/ai/gateway.ts +++ b/src/core/ai/gateway.ts @@ -797,6 +797,26 @@ export function isAvailable(touchpoint: TouchpointKind, modelOverride?: string): // ---- Embedding ---- +/** + * Carries the asymmetric-embedding `input_type` ('query' | 'document') + * across the AI SDK boundary, from embedSubBatch() to the per-recipe fetch + * shims below (#1400). + * + * Why this exists: `dimsProviderOptions()` correctly emits `input_type` + * into `providerOptions.openaiCompatible` for asymmetric models (ZE + * zembed-1, Voyage v3+), but the AI SDK's openai-compatible adapter + * validates providerOptions against a fixed schema (`dimensions`, `user`) + * and silently DROPS every other field before building the wire body. By + * the time a compat shim sees the request, the threaded 'query' is gone — + * so every embedQuery() was encoded document-side and asymmetric retrieval + * silently collapsed to surface-token overlap. + * + * embedSubBatch() populates the store only when providerOptions actually + * threads an input_type; shims that need a default (ZE) apply their own. + * Same module-level ALS pattern as `__budgetStore` below. + */ +const __embedInputTypeStore = new AsyncLocalStorage<'query' | 'document'>(); + /** * Voyage AI compatibility shim. Voyage's `/v1/embeddings` endpoint is OpenAI-shaped * but diverges on two parameters: @@ -835,6 +855,15 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit) if (typeof dims === 'number') parsed.output_dimension = dims; mutated = true; } + // Recover the SDK-stripped input_type (#1400) — opt-in, mirroring + // the dims.ts Voyage branch: inject only when a caller actually + // threaded one; when absent, the field stays off the wire + // (pre-v0.35.0.0 behavior preserved). + const threadedInputType = __embedInputTypeStore.getStore(); + if (threadedInputType !== undefined && parsed.input_type === undefined) { + parsed.input_type = threadedInputType; + mutated = true; + } if (mutated) { const newBody = JSON.stringify(parsed); // Drop Content-Length so fetch recomputes from the new body. @@ -1005,10 +1034,13 @@ const zeroEntropyCompatFetch = (async (input: RequestInfo | URL, init?: RequestI parsed.encoding_format = 'float'; mutated = true; } - // Default input_type when caller didn't thread one (document-side - // embedding is the correct default for ingest paths). + // Recover the SDK-stripped input_type (#1400): the threaded value + // arrives via __embedInputTypeStore because the openai-compatible + // adapter drops it from providerOptions before building the body. + // Document-side default preserved for paths that didn't thread one + // (ingest). if (parsed.input_type === undefined) { - parsed.input_type = 'document'; + parsed.input_type = __embedInputTypeStore.getStore() ?? 'document'; mutated = true; } if (mutated) { @@ -1096,6 +1128,56 @@ const zeroEntropyCompatFetch = (async (input: RequestInfo | URL, init?: RequestI } }) as unknown as typeof fetch; +/** + * Generic asymmetric-embedding shim for openai-compatible recipes that + * ship no compat fetch of their own (llama-server, litellm, ollama, ...). + * Those deployments are the canonical local/proxy paths for serving + * asymmetric models (e.g. a zembed-1 behind llama.cpp, vLLM, or an + * OpenAI-compatible proxy), and they need the same `input_type` signal the + * hosted ZE endpoint gets — the serving layer can't apply query-side vs + * document-side encoding without it. + * + * Strictly opt-in at the wire level: when no input_type was threaded (the + * model isn't asymmetric — dims.ts threads it by model id, never for + * Azure/DashScope/Zhipu-style fixed models — or the caller didn't ask), + * the request passes through byte-identical. When one WAS threaded + * (recovered from __embedInputTypeStore — see #1400), inject it into the + * JSON body; OpenAI-compatible servers that don't understand the field + * ignore unknown body keys (llama-server does), and asymmetric-aware + * servers/proxies read it. URL + response shape are untouched — these + * endpoints are already OpenAI-shaped. Recipes with their own compat + * fetch (voyage, zeroentropyai, azure) never reach this shim: the + * `compat.fetch ??` precedence in instantiateEmbedding wins. + */ +const openAICompatAsymmetricFetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const threadedInputType = __embedInputTypeStore.getStore(); + if (threadedInputType === undefined) return fetch(input as any, init); + // Inject only on the string-URL + string-body shape the AI SDK actually + // sends. A Request-shaped input may carry its body on the Request object + // itself, where a rebuild would silently drop it — pass it through + // untouched instead (unreachable via the SDK today; preserves the + // byte-identical contract if it ever becomes reachable). + if (typeof input !== 'string' && !(input instanceof URL)) { + return fetch(input as any, init); + } + let baseInit: RequestInit = init ?? {}; + if (baseInit.body && typeof baseInit.body === 'string') { + try { + const parsed = JSON.parse(baseInit.body); + if (parsed && typeof parsed === 'object' && parsed.input_type === undefined) { + parsed.input_type = threadedInputType; + // Drop Content-Length so fetch recomputes from the new body. + const headers = new Headers(baseInit.headers ?? {}); + headers.delete('content-length'); + baseInit = { ...baseInit, body: JSON.stringify(parsed), headers }; + } + } catch { + // Body wasn't JSON — pass through untouched. + } + } + return fetch(typeof input === 'string' ? input : input.toString(), baseInit); +}) as unknown as typeof fetch; + async function resolveEmbeddingProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> { const { parsed, recipe } = resolveRecipe(modelStr); assertTouchpoint(recipe, 'embedding', parsed.modelId, getExtendedModelsForProvider(parsed.providerId)); @@ -1149,15 +1231,19 @@ function instantiateEmbedding(recipe: Recipe, modelId: string, cfg: AIGatewayCon // wrapper via resolveOpenAICompatConfig. Azure recipes ship their own // fetch (api-version splice); voyage doesn't — use voyageCompatFetch. // ZeroEntropy needs zeroEntropyCompatFetch (URL path + body input_type - // + response shape rewrite + OOM caps). Same per-recipe-id branch - // pattern as voyage so adding a third compat shim is one more case. + // + response shape rewrite + OOM caps). Every other openai-compat + // recipe (llama-server, litellm, ollama, ...) falls through to + // openAICompatAsymmetricFetch so the threaded input_type reaches + // asymmetric models there too (#1400) — a strict pass-through when no + // input_type was threaded, so symmetric deployments see zero wire + // change. const fetchWrapper = compat.fetch ?? (recipe.id === 'voyage' ? voyageCompatFetch : recipe.id === 'zeroentropyai' ? zeroEntropyCompatFetch - : undefined); + : openAICompatAsymmetricFetch); const client = createOpenAICompatible({ name: recipe.id, baseURL: compat.baseURL, @@ -1472,7 +1558,7 @@ async function embedSubBatch( opts?: EmbedOpts, ): Promise { try { - const result = await _embedTransport({ + const callTransport = () => _embedTransport({ model, values: texts, providerOptions: providerOpts, @@ -1483,6 +1569,15 @@ async function embedSubBatch( abortSignal: withDefaultTimeout(opts?.abortSignal, AI_EMBED_TIMEOUT_MS), ...(opts?.maxRetries !== undefined && { maxRetries: opts.maxRetries }), }); + // Carry the threaded input_type across the SDK boundary via + // __embedInputTypeStore (the adapter strips it from providerOptions — + // see the store's doc comment). Populated only when dimsProviderOptions + // actually emitted one, so non-asymmetric paths run store-empty and the + // fetch shims leave their wire bodies untouched. + const threadedInputType = providerOpts?.openaiCompatible?.input_type; + const result = await (threadedInputType === 'query' || threadedInputType === 'document' + ? __embedInputTypeStore.run(threadedInputType, callTransport) + : callTransport()); if (!Array.isArray(result.embeddings) || result.embeddings.length !== texts.length) { throw new AIConfigError( diff --git a/src/core/config.ts b/src/core/config.ts index ff1ee2273..70ddca5b7 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -389,6 +389,96 @@ export function loadConfigFileOnly(): GBrainConfig | null { } } +/** + * #427 guard — DATABASE_URL hijack via Bun's cwd .env auto-load. + * + * Bun merges `.env` files from the process cwd into process.env before any + * user code runs. For a globally-installed tool that is a footgun: running + * gbrain inside any checkout whose `.env` defines DATABASE_URL (Next.js, + * Hono, Supabase, most web apps) silently retargets the brain at that app's + * database. Reads hit the wrong DB; `apply-migrations` can write gbrain's + * schema — including its DDL event trigger — into a production app database + * (see the v0.42.8 report on #427). + * + * Bun gives no way to ask which vars came from a .env file (the merge + * happens before module load), so we re-parse the .env files Bun auto-loads + * from cwd and treat DATABASE_URL as "not operator-provided" when its value + * matches one of them. Deliberate overrides still work two ways: + * - GBRAIN_DATABASE_URL: namespaced to this tool, never auto-ignored; + * - exporting DATABASE_URL in the shell: exported vars win over .env in + * Bun, and a deliberate export that happens to EQUAL the cwd .env value + * would have selected the same database anyway — ignoring it changes + * the outcome only by honoring the brain config, which is the safe + * reading of ambiguous intent. + * + * The file list is a superset of Bun's auto-load set across NODE_ENV values + * so the guard doesn't depend on replicating Bun's exact selection logic. + */ +const CWD_DOTENV_FILES = ['.env', '.env.local', '.env.development', '.env.production', '.env.test']; + +/** + * All values assigned to `key` across the .env files in `dir`. Collecting + * every assignment (rather than emulating override order) keeps the guard + * independent of dotenv precedence rules — a match against ANY assignment + * means the value is file-origin. Exported for tests. + */ +export function dotenvValuesForKey(key: string, dir: string = process.cwd()): Set { + const values = new Set(); + const assignment = /^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$/; + for (const name of CWD_DOTENV_FILES) { + let content: string; + try { + content = readFileSync(join(dir, name), 'utf-8'); + } catch { + continue; // missing/unreadable file — nothing to guard against + } + for (const rawLine of content.split('\n')) { + const line = rawLine.trim(); + if (!line || line.startsWith('#')) continue; + const m = line.match(assignment); + if (!m || m[1] !== key) continue; + let v = m[2].trim(); + if ((v.startsWith('"') && v.endsWith('"') && v.length >= 2) || + (v.startsWith("'") && v.endsWith("'") && v.length >= 2)) { + v = v.slice(1, -1); + } else { + const hash = v.indexOf(' #'); + if (hash !== -1) v = v.slice(0, hash).trim(); + } + if (v) values.add(v); + } + } + return values; +} + +let warnedCwdEnvDbUrlIgnored = false; + +/** + * The env-provided DB URL gbrain should honor, with the #427 guard applied: + * a DATABASE_URL whose value matches an assignment in a cwd .env file is + * treated as belonging to the project in cwd, not to gbrain, and ignored + * with a one-time stderr notice. GBRAIN_DATABASE_URL is always honored. + * `dir` is injectable for tests; callers use the default. + */ +export function effectiveEnvDatabaseUrl(dir: string = process.cwd()): string | undefined { + if (process.env.GBRAIN_DATABASE_URL) return process.env.GBRAIN_DATABASE_URL; + const url = process.env.DATABASE_URL; + if (!url) return undefined; + if (dotenvValuesForKey('DATABASE_URL', dir).has(url)) { + if (!warnedCwdEnvDbUrlIgnored) { + warnedCwdEnvDbUrlIgnored = true; + console.warn( + '[config] Ignoring DATABASE_URL auto-loaded by Bun from a .env file in the current ' + + 'directory — it belongs to the project here, not to gbrain. Using the engine from ' + + '~/.gbrain/config.json instead. To point gbrain at that database deliberately, set ' + + 'GBRAIN_DATABASE_URL.', + ); + } + return undefined; + } + return url; +} + export function loadConfig(): GBrainConfig | null { let fileConfig: GBrainConfig | null = null; try { @@ -397,8 +487,8 @@ export function loadConfig(): GBrainConfig | null { fileConfig = migrateLegacyEmbeddingConfig(parsed) as unknown as GBrainConfig; } catch { /* no config file */ } - // Try env vars - const dbUrl = process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL; + // Try env vars (cwd-.env-origin DATABASE_URL excluded — see #427 guard above) + const dbUrl = effectiveEnvDatabaseUrl(); if (!fileConfig && !dbUrl) return null; @@ -933,7 +1023,12 @@ export function gbrainPath(...segments: string[]): string { */ export function getDbUrlSource(): DbUrlSource { if (process.env.GBRAIN_DATABASE_URL) return 'env:GBRAIN_DATABASE_URL'; - if (process.env.DATABASE_URL) return 'env:DATABASE_URL'; + // Same #427 guard as loadConfig: a DATABASE_URL that Bun auto-loaded from + // a cwd .env file is not an operator-provided source. Keeping this in + // lockstep with loadConfig matters because doctor uses this to tell the + // user where the URL came from — reporting env:DATABASE_URL while + // loadConfig ignores it would send them debugging the wrong layer. + if (effectiveEnvDatabaseUrl()) return 'env:DATABASE_URL'; if (!existsSync(configPath())) return null; try { const raw = readFileSync(configPath(), 'utf-8'); diff --git a/src/core/cycle.ts b/src/core/cycle.ts index a4dce8ffe..116e5420f 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -1033,6 +1033,21 @@ async function runPhaseExtractFacts( || result.phantomsSkippedDrift) ? `, ${result.phantomsRedirected} phantom(s) redirected (${result.phantomsAmbiguous} ambiguous, ${result.phantomsSkippedDrift} drift-skipped)` : ''; + // #1928: a reconcile that deletes far more facts than it reinserts is the + // signature of the conversation-facts wipe (factsDeleted 1829, inserted 0 + // read as a no-op "ok" before this guard). Surface net deletion above a + // floor as `warn` so the daily report and doctor make it visible instead + // of it reading like a clean run. + const NET_DELETION_WARN_FLOOR = 50; + const netDeleted = result.factsDeleted - result.factsInserted; + const netDeletionWarn = netDeleted >= NET_DELETION_WARN_FLOOR; + if (netDeletionWarn) { + result.warnings.push( + `net_fact_deletion: reconcile removed ${result.factsDeleted} fact(s) and ` + + `reinserted ${result.factsInserted} (net -${netDeleted}). If unexpected, a ` + + `destructive full walk may have wiped non-fence facts (#1928).`, + ); + } return { phase: 'extract_facts', status: result.warnings.length > 0 ? 'warn' : 'ok', @@ -1574,6 +1589,11 @@ export async function runCycle( // and which slugs synthesize wrote so recompute_emotional_weight can // pick up the union of (sync ∪ synthesize) for v0.29 incremental mode. let syncPagesAffected: string[] | undefined; + // #1928 (codex): true ONLY when the sync phase actually RAN its work (not + // when it was skipped for no-engine / no-brainDir). The destructive + // extract_facts guard keys off this so a SKIPPED sync still allows a + // legitimate full reconcile — only a sync that ran and failed suppresses it. + let syncAttempted = false; let synthesizeWrittenSlugs: string[] | undefined; if (phases.includes('sync')) { checkAborted(opts.signal); @@ -1589,6 +1609,7 @@ export async function runCycle( phaseResults.push(skipNoBrainDir('sync')); } else { progress.start('cycle.sync'); + syncAttempted = true; // sync ran its work; undefined pagesAffected now means failure const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, brainDir, dryRun, pull, phases.includes('extract'))); result.duration_ms = duration_ms; // Capture changed slugs for incremental extract. @@ -1688,8 +1709,22 @@ export async function runCycle( // the sources table doesn't recognize this brainDir (pre-multi- // source installs). const xfSourceId = cycleSourceId ?? 'default'; + // #1928: extract_facts is DESTRUCTIVE (wipe-and-reinsert per page). It + // must NOT inherit the "sync failed ⇒ undefined ⇒ full walk" fallback + // that's safe for link/timeline extract. When the sync phase RAN but + // failed, syncPagesAffected is undefined (a successful no-op sync + // returns []). In that case pass [] (no-op) so a lock-contention or + // transient sync failure can't escalate into a brain-wide fact wipe. + // undefined still reaches here (intended full reconcile) when the sync + // phase was absent OR skipped (no engine / no brainDir — extract_facts + // supports no-brainDir DB reconciliation). Only a sync that actually + // RAN and came back with undefined pagesAffected is a real failure + // (#1928, codex: keying off phases.includes('sync') wrongly suppressed + // the skipped-sync full reconcile). + const syncRanButFailed = syncAttempted && syncPagesAffected === undefined; + const xfSlugs = syncRanButFailed ? [] : syncPagesAffected; const { result, duration_ms } = await timePhase(() => - runPhaseExtractFacts(engine, brainDir, xfSourceId, dryRun, syncPagesAffected, opts.signal)); + runPhaseExtractFacts(engine, brainDir, xfSourceId, dryRun, xfSlugs, opts.signal)); result.duration_ms = duration_ms; phaseResults.push(result); progress.finish(); diff --git a/src/core/cycle/extract-facts.ts b/src/core/cycle/extract-facts.ts index 306e58a5d..39b15d9ad 100644 --- a/src/core/cycle/extract-facts.ts +++ b/src/core/cycle/extract-facts.ts @@ -222,10 +222,15 @@ export async function runExtractFacts( if (opts.dryRun) continue; - // Wipe-and-reinsert per page. The deleteFactsForPage call targets - // source_markdown_slug = slug only, so NULL-source_markdown_slug - // legacy rows survive (the partial-UNIQUE-index keyspace). - const deleted = await engine.deleteFactsForPage(slug, sourceId); + // Wipe-and-reinsert per page. The delete targets source_markdown_slug = + // slug only, so NULL-source_markdown_slug legacy rows survive (the + // partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts (conversation + // facts from extract-conversation-facts) are NOT fence-owned — the page + // carries no `## Facts` fence to recreate them — so they MUST survive this + // reconcile. Exclude them from the wipe. + const deleted = await engine.deleteFactsForPage(slug, sourceId, { + excludeSourcePrefixes: ['cli:'], + }); result.factsDeleted += deleted.deleted; if (parsed.facts.length === 0) continue; diff --git a/src/core/doctor-categories.ts b/src/core/doctor-categories.ts index 1bbf3b06e..e8adebb71 100644 --- a/src/core/doctor-categories.ts +++ b/src/core/doctor-categories.ts @@ -173,6 +173,7 @@ export const META_CHECK_NAMES: ReadonlySet = new Set([ 'schema_pack_source_drift', 'schema_version', 'slug_fallback_audit', + 'timeline_dedup_index', 'upgrade_errors', ]); diff --git a/src/core/embedding-dim-check.ts b/src/core/embedding-dim-check.ts index 4ca542da2..220bcca05 100644 --- a/src/core/embedding-dim-check.ts +++ b/src/core/embedding-dim-check.ts @@ -217,8 +217,10 @@ export function embeddingMismatchMessage(opts: EmbeddingMismatchOpts): string { ``, ` BEGIN;`, ` DROP INDEX IF EXISTS idx_chunks_embedding;`, - ` ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(${requestedDims});`, + ` -- NULL embeddings BEFORE the alter: pgvector refuses to cast existing`, + ` -- vectors across dimensions and aborts the transaction. NULLs cast fine.`, ` UPDATE content_chunks SET embedding = NULL, embedded_at = NULL;`, + ` ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(${requestedDims});`, ` ${reindexLine.split('\n').join('\n ')}`, ` COMMIT;`, ``, @@ -573,11 +575,25 @@ export function buildFactsAlterRecipe( ): string { const opclass = columnType === 'halfvec' ? 'halfvec_cosine_ops' : 'vector_cosine_ops'; const targetType = columnType === 'halfvec' ? `halfvec(${configuredDims})` : `vector(${configuredDims})`; + const dimsChanged = columnDims !== configuredDims; return [ `-- ALTER ${columnType}(${columnDims}) → ${columnType}(${configuredDims}) on indexed column.`, `-- HOLD a maintenance window: this rewrites every row's embedding.`, `-- Coordinate with any active extract-conversation-facts backfill.`, `DROP INDEX IF EXISTS idx_facts_embedding_hnsw;`, + // Same-dim type swaps (halfvec <-> vector) cast row data losslessly and + // MUST keep it. Cross-dimension changes are different: pgvector refuses + // to cast existing vectors across dimensions ("expected N dimensions, + // not M") and aborts the transaction — and the old-space vectors are + // unusable at the new width anyway. NULL them first; the facts pipeline + // re-embeds on the next write. + ...(dimsChanged + ? [ + `-- Dimension change: NULL embeddings BEFORE the alter — pgvector`, + `-- refuses cross-dimension casts and aborts the transaction.`, + `UPDATE facts SET embedding = NULL;`, + ] + : []), `ALTER TABLE facts ALTER COLUMN embedding TYPE ${targetType}`, ` USING embedding::${targetType};`, `CREATE INDEX idx_facts_embedding_hnsw`, diff --git a/src/core/engine.ts b/src/core/engine.ts index 0f285d498..075870269 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -650,6 +650,15 @@ export interface BrainEngine { // Lifecycle connect(config: EngineConfig): Promise; disconnect(): Promise; + /** + * Recover a dropped connection using the config captured at the last + * `connect()`. Callers (autopilot health probe, batchRetry) MUST use this + * instead of `disconnect()` + bare `connect()`: the latter loses the config + * (#2034 — a bare `connect()` with no args throws `database_url undefined` + * forever) AND opens a null-connection window. Implemented on BOTH engines + * for parity so the call is never a silent no-op. + */ + reconnect(ctx?: { error?: unknown }): Promise; initSchema(): Promise; transaction(fn: (engine: BrainEngine) => Promise): Promise; /** @@ -1678,7 +1687,20 @@ export interface BrainEngine { * until the v0_32_2 migration backfills them. Cycle-phase callers in * commit 7 add the empty-fence-guard as a belt-and-suspenders check. */ - deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }>; + /** + * #1928: `excludeSourcePrefixes` protects facts whose `source` matches any + * given prefix (e.g. `['cli:']` for conversation facts written by + * `extract-conversation-facts`) from a fence reconcile. Those rows are NOT + * fence-owned — a destructive wipe-and-reinsert pass would delete them and + * never recreate them (the page has no `## Facts` fence). Omitted ⇒ legacy + * behavior (delete every fact on the page coordinate). NULL/empty `source` + * rows are always deletable (fence default). + */ + deleteFactsForPage( + slug: string, + source_id: string, + opts?: { excludeSourcePrefixes?: string[] }, + ): Promise<{ deleted: number }>; /** * Mark a fact expired. Never DELETE. Returns true iff a row was updated. diff --git a/src/core/extract-timeline-from-meetings.ts b/src/core/extract-timeline-from-meetings.ts index fb1a3ec40..e63666310 100644 --- a/src/core/extract-timeline-from-meetings.ts +++ b/src/core/extract-timeline-from-meetings.ts @@ -28,6 +28,15 @@ export interface ExtractTimelineFromMeetingsResult { entries_created: number; /** Distinct entity pages that received at least one new timeline entry. */ entities_touched: number; + /** + * #2057: batches that failed to insert. Previously swallowed by a bare + * `catch {}`, which let a brain-wide timeline-write failure read as a clean + * "0 entries" run. Non-zero here means inserts are failing — surfaced on + * stderr too. + */ + batch_errors: number; + /** First batch-insert error message, when batch_errors > 0. */ + first_batch_error?: string; } interface MeetingRow { @@ -71,7 +80,7 @@ export async function extractTimelineFromMeetings( ); if (meetings.length === 0) { - return { meetings_scanned: 0, entries_created: 0, entities_touched: 0 }; + return { meetings_scanned: 0, entries_created: 0, entities_touched: 0, batch_errors: 0 }; } // 2. Fetch all 'attended' edges (one round-trip, scoped to the loaded @@ -106,14 +115,23 @@ export async function extractTimelineFromMeetings( let entriesCreated = 0; const entitiesTouched = new Set(); let meetingsScanned = 0; + let batchErrors = 0; + let firstBatchError: string | undefined; async function flush() { if (batch.length === 0) return; if (!dryRun) { try { entriesCreated += await engine.addTimelineEntriesBatch(batch); - } catch { - // batch error — drop; per-meeting progress continues + } catch (e) { + // #2057: do NOT swallow. A bare `catch {}` here hid a brain-wide + // timeline-write failure (the run reported 0 entries with no error). + // Count + surface it on stderr; the per-meeting loop still continues so + // one bad batch isn't fatal to the rest. + batchErrors += 1; + const msg = e instanceof Error ? e.message : String(e); + if (!firstBatchError) firstBatchError = msg; + console.error(`[extract timeline] batch insert failed (${batch.length} row(s)): ${msg}`); } } else { entriesCreated += batch.length; @@ -182,5 +200,7 @@ export async function extractTimelineFromMeetings( meetings_scanned: meetingsScanned, entries_created: entriesCreated, entities_touched: entitiesTouched.size, + batch_errors: batchErrors, + first_batch_error: firstBatchError, }; } diff --git a/src/core/import-file.ts b/src/core/import-file.ts index fb9190ee4..cbf2d03a0 100644 --- a/src/core/import-file.ts +++ b/src/core/import-file.ts @@ -1193,7 +1193,15 @@ export async function importCodeFile( // chunk IDs are stable. if (extractedEdges.length > 0 && chunks.length > 0) { try { - const persistedChunks = await engine.getChunks(slug, sourceId ? { sourceId } : undefined); + // Normalize ONCE: '' and undefined both mean the schema-default source + // (pages.source_id DEFAULT 'default'). Using the normalized value for + // BOTH the chunk lookup and the edge stamp keeps them in lockstep — + // an unscoped getChunks here could fan out to same-slug chunks from + // another source, and a '' stamp would FK-violate against sources(id) + // and silently drop the file's whole call graph in the best-effort + // catch below (adversarial review findings). + const edgeSourceId = sourceId || 'default'; + const persistedChunks = await engine.getChunks(slug, { sourceId: edgeSourceId }); const byIndex = new Map(); for (const pc of persistedChunks) { byIndex.set(pc.chunk_index, pc); @@ -1231,6 +1239,17 @@ export async function importCodeFile( from_symbol_qualified: from.symbol_name_qualified, to_symbol_qualified: e.toSymbol, edge_type: e.edgeType, + // Stamp the source: getCallersOf/getCalleesOf add + // `AND source_id = ` whenever a worktree pin / --source is + // in play, and a NULL here never matches that filter — so every + // scoped call-graph query silently returned 0 rows on + // multi-source brains even though the edges existed. The fallback + // is 'default', NOT null: an unscoped import lands its pages under + // the schema default (pages.source_id DEFAULT 'default'), so a + // NULL-stamped edge would be invisible to the matching scoped + // query getCallersOf(sym, { sourceId: 'default' }) — the same bug + // through the other door. + source_id: edgeSourceId, }); } diff --git a/src/core/legacy-token-scope.ts b/src/core/legacy-token-scope.ts new file mode 100644 index 000000000..5e616c280 --- /dev/null +++ b/src/core/legacy-token-scope.ts @@ -0,0 +1,22 @@ +/** + * Derive a legacy bearer token's source scope from its stored + * `access_tokens.permissions.source_id` grant. + * + * ARRAY = federated read grant, exposed through `allowedSources` with the + * first granted source as the scalar write floor. STRING = scalar source. + * Missing, empty, or garbage values fail closed to the historical `default` + * floor and NEVER widen to all sources. + */ +export function parseLegacyTokenScope(rawSource: unknown): { sourceId: string; allowedSources?: string[] } { + if (Array.isArray(rawSource)) { + const allowedSources = (rawSource as unknown[]).filter(s => typeof s === 'string' && s.length > 0) as string[]; + if (allowedSources.length > 0) { + return { sourceId: allowedSources[0], allowedSources }; + } + return { sourceId: 'default' }; + } + if (typeof rawSource === 'string' && rawSource.length > 0) { + return { sourceId: rawSource }; + } + return { sourceId: 'default' }; +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 2e31e60ad..31d3a10c1 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -5186,6 +5186,52 @@ export const MIGRATIONS: Migration[] = [ ); `, }, + { + version: 116, + name: 'code_edges_source_backfill_and_callee_index', + // Repair + index pass for the call graph: + // + // 1. BACKFILL: importCodeFile built CodeEdgeInput rows without source_id, + // so every extracted edge landed NULL. getCallersOf/getCalleesOf add + // `AND source_id = ` whenever a worktree pin / --source is in + // play — NULL never matches, so scoped call-graph queries silently + // returned 0 rows on multi-source brains even though the edges + // existed. The write path now stamps `sourceId ?? 'default'`; this + // backfill repairs rows written before the fix by deriving each + // edge's source from its own from_chunk's page (pages.source_id is + // NOT NULL DEFAULT 'default', so COALESCE is belt-and-braces only). + // + // 2. INDEXES: getCalleesOf filters BOTH edge tables on + // from_symbol_qualified, which had no index anywhere — every callee + // lookup was a sequential scan, amplified per-BFS-node by the + // recursive code walk (one getCalleesOf per frontier node, up to + // maxNodes). With NULL edges repaired, scoped walks actually expand, + // so the latent seq-scan cost becomes real. Plain CREATE INDEX (not + // CONCURRENTLY): edge tables are modest (mirrors the v58 resolver + // index). Keep in sync with src/schema.sql. + idempotent: true, + sql: ` + UPDATE code_edges_symbol e + SET source_id = COALESCE(p.source_id, 'default') + FROM content_chunks c + JOIN pages p ON p.id = c.page_id + WHERE c.id = e.from_chunk_id + AND e.source_id IS NULL; + + UPDATE code_edges_chunk e + SET source_id = COALESCE(p.source_id, 'default') + FROM content_chunks c + JOIN pages p ON p.id = c.page_id + WHERE c.id = e.from_chunk_id + AND e.source_id IS NULL; + + CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol + ON code_edges_symbol (from_symbol_qualified); + + CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol + ON code_edges_chunk (from_symbol_qualified); + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 @@ -5524,6 +5570,26 @@ export async function runMigrations(engine: BrainEngine): Promise<{ applied: num const sorted = [...MIGRATIONS].sort((a, b) => a.version - b.version); const pending = sorted.filter(m => m.version > current); + + // #2038: schema-drift self-heal. A migration renumbered during a master + // merge (v102 timeline dedup, originally v99) can be recorded-as-applied + // without its DDL ever running — the version counter can't see it. Repair + // the known drift on EVERY pass, including when nothing is pending (the + // affected brains are stamped AHEAD of the missing migration, so they never + // reach the loop below). Best-effort + idempotent: a no-op on a healthy + // index; `doctor` surfaces it independently if this ever fails. + try { + const { repairTimelineDedupIndex } = await import('./timeline-dedup-repair.ts'); + const r = await repairTimelineDedupIndex(engine); + if (r.repaired) { + console.error( + `[migrate] healed idx_timeline_dedup drift (#2038): ${r.before.join(',') || '(absent)'} ` + + `→ page_id,date,summary,source` + + (r.collapsedDuplicates > 0 ? ` (collapsed ${r.collapsedDuplicates} duplicate row(s))` : ''), + ); + } + } catch { /* best-effort; doctor reports the drift if this couldn't run */ } + if (pending.length === 0) { return { applied: 0, current }; } diff --git a/src/core/minions/supervisor.ts b/src/core/minions/supervisor.ts index 3853a6793..ec68dc499 100644 --- a/src/core/minions/supervisor.ts +++ b/src/core/minions/supervisor.ts @@ -1023,16 +1023,17 @@ export class MinionSupervisor { error: errMsg, queue: this.opts.queue, }); - // Attempt to reconnect the engine if it supports it + // Attempt to reconnect the engine. #2034: reconnect() is now a + // first-class BrainEngine method on both engines (it restores the + // config captured at connect()), so call it directly — the old + // feature-detection cast is no longer needed. try { - if ('reconnect' in this.engine && typeof (this.engine as Record).reconnect === 'function') { - await (this.engine as unknown as { reconnect(): Promise }).reconnect(); - this.consecutiveHealthFailures = 0; - this.emit('health_warn', { - reason: 'db_reconnected', - queue: this.opts.queue, - }); - } + await this.engine.reconnect({ error: errMsg }); + this.consecutiveHealthFailures = 0; + this.emit('health_warn', { + reason: 'db_reconnected', + queue: this.opts.queue, + }); } catch (reconnErr) { this.emit('health_error', { error: `reconnect failed: ${reconnErr instanceof Error ? reconnErr.message : String(reconnErr)}`, diff --git a/src/core/oauth-provider.ts b/src/core/oauth-provider.ts index 8b4ac91d3..83358536b 100644 --- a/src/core/oauth-provider.ts +++ b/src/core/oauth-provider.ts @@ -21,10 +21,12 @@ import type { } from '@modelcontextprotocol/sdk/shared/auth.js'; import type { OAuthServerProvider, AuthorizationParams } from '@modelcontextprotocol/sdk/server/auth/provider.js'; import type { OAuthRegisteredClientsStore } from '@modelcontextprotocol/sdk/server/auth/clients.js'; -import type { AuthInfo } from '@modelcontextprotocol/sdk/server/auth/types.js'; +import type { AuthInfo as SdkAuthInfo } from '@modelcontextprotocol/sdk/server/auth/types.js'; import { InvalidTokenError } from '@modelcontextprotocol/sdk/server/auth/errors.js'; import { hashToken, generateToken, isUndefinedColumnError } from './utils.ts'; import { hasScope, assertAllowedScopes, parseScopeString, InvalidScopeError } from './scope.ts'; +import type { AuthInfo as CoreAuthInfo } from './operations.ts'; +import { parseLegacyTokenScope } from './legacy-token-scope.ts'; import type { SqlQuery, SqlValue } from './sql-query.ts'; export type { SqlQuery, SqlValue }; @@ -389,10 +391,18 @@ export class GBrainOAuthProvider implements OAuthServerProvider { // as a fully-admin access token. Mirrors the filter pattern already used // by exchangeClientCredentials (this file) and exchangeRefreshToken's F3 // subset enforcement (RFC 6749 §6) so all three grant entry points clamp - // consistently. Empty/omitted requested scope inherits the empty-stored - // shape (existing behavior; not a security boundary). + // consistently. When the client requests NO scope, RFC 6749 §3.3 lets the + // server fall back to a default — we default to the client's full + // registered scope (matching exchangeClientCredentials, which already does + // `requestedScope ? ... : allowedScopes`). Previously an omitted request + // granted the empty set, which then propagated into the access+refresh + // tokens and never self-healed: every op failed `insufficient_scope` even + // though the client was registered with `read write`. Clients that omit + // `scope` on /authorize (e.g. some MCP connectors) hit this. Still clamped + // to the allowed set, so an explicit over-broad request can't escalate. const allowedScopes = parseScopeString(client.scope); - const grantedScopes = (params.scopes || []).filter(s => hasScope(allowedScopes, s)); + const requestedScopes = (params.scopes && params.scopes.length) ? params.scopes : allowedScopes; + const grantedScopes = requestedScopes.filter(s => hasScope(allowedScopes, s)); await this.sql` INSERT INTO oauth_codes (code_hash, client_id, scopes, code_challenge, @@ -539,7 +549,7 @@ export class GBrainOAuthProvider implements OAuthServerProvider { // Token Verification // ------------------------------------------------------------------------- - async verifyAccessToken(token: string): Promise { + async verifyAccessToken(token: string): Promise { const tokenHash = hashToken(token); const now = Math.floor(Date.now() / 1000); @@ -629,14 +639,29 @@ export class GBrainOAuthProvider implements OAuthServerProvider { // operations.ts prefers this array over scalar sourceId when set // and non-empty. allowedSources, - } as AuthInfo; + } as CoreAuthInfo as SdkAuthInfo; } - // Fallback: legacy access_tokens table (backward compat) - const legacyRows = await this.sql` - SELECT name FROM access_tokens - WHERE token_hash = ${tokenHash} AND revoked_at IS NULL - `; + // Fallback: legacy access_tokens table (backward compat). Modern legacy + // rows may carry permissions.source_id from the pre-OAuth bearer-token + // path; OAuth transport must preserve that same source grant instead of + // pinning every legacy token to `default`. + let legacyRows: Record[]; + try { + legacyRows = await this.sql` + SELECT name, permissions FROM access_tokens + WHERE token_hash = ${tokenHash} AND revoked_at IS NULL + `; + } catch (err) { + if (isUndefinedColumnError(err, 'permissions')) { + legacyRows = await this.sql` + SELECT name FROM access_tokens + WHERE token_hash = ${tokenHash} AND revoked_at IS NULL + `; + } else { + throw err; + } + } if (legacyRows.length > 0) { // Legacy tokens get full admin access (grandfather in). @@ -646,19 +671,31 @@ export class GBrainOAuthProvider implements OAuthServerProvider { UPDATE access_tokens SET last_used_at = now() WHERE token_hash = ${tokenHash} `; const name = legacyRows[0].name as string; + const permissionsRaw = legacyRows[0].permissions; + let permissions: unknown = permissionsRaw; + if (typeof permissionsRaw === 'string') { + try { + permissions = JSON.parse(permissionsRaw); + } catch { + permissions = undefined; + } + } + const sourceGrant = permissions && typeof permissions === 'object' + ? (permissions as Record).source_id + : undefined; + const { sourceId, allowedSources } = parseLegacyTokenScope(sourceGrant); return { token, clientId: name, clientName: name, scopes: ['read', 'write', 'admin'], expiresAt: Math.floor(Date.now() / 1000) + 365 * 24 * 3600, // Legacy tokens never expire — set 1yr future - // v0.34.1 (#861, D13): legacy bearer tokens default to 'default' - // source — matches the pre-v0.34 effective behavior where the - // serve-http transport fell back to GBRAIN_SOURCE/'default' for - // any caller without explicit scope. Operators who want a - // narrower scope for legacy tokens migrate to OAuth. - sourceId: 'default', - } as AuthInfo; + // Legacy tokens without an explicit permissions.source_id grant keep + // the historical 'default' source floor. Array grants become + // allowedSources for federated reads, matching legacy HTTP transport. + sourceId, + allowedSources, + } as CoreAuthInfo as SdkAuthInfo; } throw new InvalidTokenError('Invalid token'); diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 65c429f36..e2cf4693a 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -199,6 +199,9 @@ export class PGLiteEngine implements BrainEngine { readonly kind = 'pglite' as const; private _db: PGLiteDB | null = null; private _lock: LockHandle | null = null; + // #2034: captured at connect() so reconnect() can restore the same data dir + // after a drop, matching PostgresEngine's _savedConfig contract. + private _savedConfig: EngineConfig | null = null; // Tier 3: when GBRAIN_PGLITE_SNAPSHOT loaded a post-initSchema state into // PGlite.create(loadDataDir), initSchema is a no-op (schema is already // present + migrations already applied). Saves ~1-3s per fresh test PGLite. @@ -211,6 +214,7 @@ export class PGLiteEngine implements BrainEngine { // Lifecycle async connect(config: EngineConfig): Promise { + this._savedConfig = config; // #2034: remember for reconnect() const dataDir = config.database_path || undefined; // undefined = in-memory // Acquire file lock to prevent concurrent PGLite access (crashes with Aborted()) @@ -288,6 +292,27 @@ export class PGLiteEngine implements BrainEngine { } } + /** + * #2034: engine-parity reconnect. PGLite is single-writer in-process so it + * doesn't suffer the pool-drop class PostgresEngine.reconnect() handles, but + * the method MUST exist so callers (autopilot health probe, worker/queue + * claim-error recovery) can call `engine.reconnect()` uniformly. + * + * IN-MEMORY (no `database_path`) is a NO-OP: there is no persistent backing, + * the connection can't recoverably "drop" in-process, and a disconnect+reopen + * would DISCARD all state. This matches the long-standing assumption the + * worker/queue recovery paths are written against ("PGLite has no pooler + * reaping so reconnect is absent" — src/core/minions/queue.ts). A FILE-backed + * engine genuinely re-opens the same data dir (state persists on disk). + */ + async reconnect(_ctx?: { error?: unknown }): Promise { + if (!this._savedConfig) return; // never connected — nothing to restore + if (!this._savedConfig.database_path) return; // in-memory — no-op, preserve state + const config = this._savedConfig; + await this.disconnect(); + await this.connect(config); + } + async initSchema(): Promise { // Tier 3: snapshot was loaded into PGlite — schema + migrations already // applied. Nothing to do. Returns immediately. @@ -3598,7 +3623,25 @@ export class PGLiteEngine implements BrainEngine { return { inserted: ids.length, ids }; } - async deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }> { + async deleteFactsForPage( + slug: string, + source_id: string, + opts?: { excludeSourcePrefixes?: string[] }, + ): Promise<{ deleted: number }> { + const prefixes = opts?.excludeSourcePrefixes; + if (prefixes && prefixes.length > 0) { + // #1928: keep rows whose `source` matches an excluded prefix (e.g. + // `cli:` conversation facts). COALESCE so NULL/empty-source fence rows + // stay deletable — only the explicitly-protected prefixes survive. + const patterns = prefixes.map(p => `${p}%`); + const result = await this.db.query( + `DELETE FROM facts + WHERE source_id = $1 AND source_markdown_slug = $2 + AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))`, + [source_id, slug, patterns], + ); + return { deleted: result.affectedRows ?? 0 }; + } const result = await this.db.query( `DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2`, [source_id, slug], @@ -4908,7 +4951,7 @@ export class PGLiteEngine implements BrainEngine { e.from_chunk_id, e.to_chunk_id, e.from_symbol_qualified, e.to_symbol_qualified, e.edge_type, JSON.stringify(e.edge_metadata ?? {}), - e.source_id ?? null, + e.source_id ?? 'default', ); } const res = await this.db.query( @@ -4929,7 +4972,7 @@ export class PGLiteEngine implements BrainEngine { params.push( e.from_chunk_id, e.from_symbol_qualified, e.to_symbol_qualified, e.edge_type, JSON.stringify(e.edge_metadata ?? {}), - e.source_id ?? null, + e.source_id ?? 'default', ); } const res = await this.db.query( diff --git a/src/core/pglite-lock.ts b/src/core/pglite-lock.ts index 7fc846b77..0fe04952a 100644 --- a/src/core/pglite-lock.ts +++ b/src/core/pglite-lock.ts @@ -19,11 +19,72 @@ import { join } from 'path'; const LOCK_DIR_NAME = '.gbrain-lock'; const LOCK_FILE = 'lock'; -const STALE_THRESHOLD_MS = 5 * 60 * 1000; // 5 minutes — embed jobs can be long + +// #2058: refresh the lock's `refreshed_at` while held so a long-running but +// LIVE holder (embed jobs run for many minutes) is never mistaken for stale. +const HEARTBEAT_INTERVAL_MS = 30_000; + +// #2058: a holder whose heartbeat refreshed within this window is ALIVE and is +// NEVER stolen, regardless of how old the lock is. Only a holder that STOPPED +// refreshing past this grace (hung, crashed without cleanup, or a PID since +// reused by an unrelated process) is reaped. Pairing heartbeat-age with PID +// liveness is what defeats both the WAL-corruption bug (stealing a live +// writer) AND the PID-reuse false-positive (a recycled PID reading as "alive"). +// Env-overridable as an incident escape hatch, matching the sync-lock knobs. +function stealGraceMs(): number { + const env = parseInt(process.env.GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS ?? '', 10); + return Number.isFinite(env) && env > 0 ? env * 1000 : 10 * 60 * 1000; // default 600s +} export interface LockHandle { lockDir: string; acquired: boolean; + /** + * #2058: heartbeat timer + lock-file path, set when a real (on-disk) lock is + * held so `releaseLock` can stop refreshing. Absent for the in-memory engine + * (no lock file, no concurrent access possible). + */ + heartbeat?: ReturnType; + lockPath?: string; + /** + * #2058 (codex): our ownership token (`:`). If we stall + * past the steal grace, another process can reap + re-acquire. When we + * resume, the heartbeat and release MUST verify the on-disk lock is STILL + * ours before touching it — otherwise a resumed stale holder would refresh + * or delete the NEW owner's live lock, re-opening the concurrent-writer hole. + */ + ownerToken?: string; +} + +/** The on-disk lock identity, used to detect "we were reaped and replaced". */ +function tokenOf(lockData: { pid?: unknown; acquired_at?: unknown }): string { + return `${lockData.pid}:${lockData.acquired_at}`; +} + +/** + * #2058: keep the held lock's `refreshed_at` current so a concurrent acquirer + * can tell a live, working holder from a hung/dead one. Best-effort: if the + * file is gone (we're being reaped) the write simply fails. `.unref()` so the + * timer never keeps the process alive on its own. Ownership-checked: if the + * on-disk lock is no longer ours (we were reaped past grace and replaced), stop + * the heartbeat instead of clobbering the new owner's lock. + */ +function startHeartbeat(lockPath: string, ownerToken: string): ReturnType { + const timer = setInterval(() => { + try { + const raw = JSON.parse(readFileSync(lockPath, 'utf-8')); + if (tokenOf(raw) !== ownerToken) { + // We were reaped and someone else owns it now — do NOT refresh their + // lock. Stand down. + clearInterval(timer); + return; + } + raw.refreshed_at = Date.now(); + writeFileSync(lockPath, JSON.stringify(raw), { mode: 0o644 }); + } catch { /* best-effort — file removed or transient FS error */ } + }, HEARTBEAT_INTERVAL_MS); + (timer as { unref?: () => void }).unref?.(); + return timer; } function getLockDir(dataDir: string | undefined): string { @@ -75,16 +136,23 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM const lockPid = lockData.pid as number; const lockTime = lockData.acquired_at as number; - // Is the locking process still alive? - if (!isProcessAlive(lockPid)) { - // Stale lock — clean it up + // #2058: classify by PID liveness AND heartbeat freshness. A holder + // that is alive AND refreshed its heartbeat within the steal grace is + // genuinely working (e.g. a multi-minute embed) and is NEVER reaped — + // force-removing it here is what corrupted the single-writer WAL. + const alive = isProcessAlive(lockPid); + const lastRefresh = (lockData.refreshed_at as number | undefined) ?? lockTime; + const sinceRefresh = Date.now() - lastRefresh; + if (!alive) { + // Holder process is gone — reap. try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition, try again */ } - } else if (Date.now() - lockTime > STALE_THRESHOLD_MS) { - // Lock held for too long — assume stale (e.g., process hung) - // Still alive but probably stuck — force remove + } else if (sinceRefresh > stealGraceMs()) { + // PID is alive but the heartbeat stopped past the grace window: + // either the holder hung, or this PID was reused by an unrelated + // process (the real holder died and stopped refreshing). Reap. try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition */ } } else { - // Lock is held by a live process — wait and retry + // Live holder refreshing within grace — wait and retry. await new Promise(r => setTimeout(r, 1000)); continue; } @@ -97,15 +165,19 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM // Try to acquire lock (atomic mkdir) try { mkdirSync(lockDir, { recursive: false }); - // We got the lock — write our PID + // We got the lock — write our PID. #2058: seed `refreshed_at` and start + // the heartbeat so this holder reads as alive-and-working to others. const lockPath = join(lockDir, LOCK_FILE); + const now = Date.now(); writeFileSync(lockPath, JSON.stringify({ pid: process.pid, - acquired_at: Date.now(), + acquired_at: now, + refreshed_at: now, command: process.argv.slice(1).join(' '), }), { mode: 0o644 }); - return { lockDir, acquired: true }; + const ownerToken = tokenOf({ pid: process.pid, acquired_at: now }); + return { lockDir, acquired: true, lockPath, ownerToken, heartbeat: startHeartbeat(lockPath, ownerToken) }; } catch (e: unknown) { // mkdir failed — someone else grabbed it between our check and mkdir // This is fine, we'll retry @@ -138,8 +210,25 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM * Release a previously acquired lock. */ export async function releaseLock(lock: LockHandle): Promise { + // #2058: stop the heartbeat first so it can't recreate/rewrite the lock file + // after we remove it. + if (lock.heartbeat) { + clearInterval(lock.heartbeat); + lock.heartbeat = undefined; + } if (!lock.lockDir || !lock.acquired) return; + // #2058 (codex): only remove the lock if it is STILL ours. If we were reaped + // past the grace and another process re-acquired, removing its live lock + // would let a third process in alongside it — the corruption this fix exists + // to prevent. Unreadable/absent lock falls through to a best-effort remove. + if (lock.ownerToken) { + try { + const raw = JSON.parse(readFileSync(join(lock.lockDir, LOCK_FILE), 'utf-8')); + if (tokenOf(raw) !== lock.ownerToken) return; // someone else owns it now + } catch { /* unreadable/gone — fall through to best-effort cleanup */ } + } + try { rmSync(lock.lockDir, { recursive: true, force: true }); } catch { diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 3b8c7ce13..ff781eaa3 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -1276,11 +1276,28 @@ export class PostgresEngine implements BrainEngine { } async updateSourceConfig(sourceId: string, patch: Record): Promise { - // v0.38: atomic JSONB merge. `||` is the Postgres concat operator — - // for jsonb, right-side keys overwrite left-side; nested object keys - // are NOT deep-merged (use jsonb_set for nested paths). The patch - // shape this autopilot wave uses is flat (`last_full_cycle_at`, - // `archive_*`, etc.) so concat is sufficient. Idempotent on re-run. + // Atomic single-statement merge. The previous read-then-write form dropped + // concurrent updates: two callers patching different keys could both read + // the same old config and the later `SET config = ...` clobbered the + // earlier patch. These keys are written by background cycle/autopilot + // paths, so the merge must happen inside the UPDATE (parity with + // pglite-engine.updateSourceConfig, which already uses JSONB `||`). + // + // The CASE normalizes historical bad shapes inline (so `config` is re-read + // against the row-locked latest version — a CTE/subquery snapshot would + // reintroduce the lost-update race under READ COMMITTED): older code paths + // could store config as a JSONB string (double-encoded) or as a JSONB array + // of patch objects. We coerce those to a flat object before the `||` merge + // so doctor and source routing keep getting flat keys. + // + // String branch guard: a JSONB string whose inner text is NOT itself valid + // JSON (one of the historical bad shapes this path repairs) would make the + // bare `::jsonb` cast raise `invalid input syntax for type json`, failing + // the whole UPDATE. Postgres has no `try_cast`, so we gate the cast with + // the SQL `IS JSON` predicate (Postgres 16+): parseable inner text is + // double-encoded config and gets parsed; unparseable text falls back to `{}`. + // The guard keeps the merge a single atomic statement (no extra round-trip, + // no lost-update race). // // MUST use sql.json(patch) inside the template tag — postgres-js's // positional executeRaw + `$1::jsonb` cast DOUBLE-ENCODES the @@ -1294,7 +1311,25 @@ export class PostgresEngine implements BrainEngine { const sql = this.sql; const result = await sql` UPDATE sources - SET config = COALESCE(config, '{}'::jsonb) || ${sql.json(patch as Parameters[0])} + SET config = + CASE + WHEN jsonb_typeof(config) = 'object' THEN config + WHEN jsonb_typeof(config) = 'string' + THEN CASE + WHEN (config #>> '{}') IS JSON + THEN COALESCE(NULLIF((config #>> '{}'), '')::jsonb, '{}'::jsonb) + ELSE '{}'::jsonb + END + WHEN jsonb_typeof(config) = 'array' + THEN COALESCE( + (SELECT jsonb_object_agg(kv.key, kv.value) + FROM jsonb_array_elements(config) elem, + jsonb_each(elem) kv), + '{}'::jsonb + ) + ELSE '{}'::jsonb + END + || ${sql.json(patch as Parameters[0])} WHERE id = ${sourceId} `; return (result.count ?? 0) > 0; @@ -3745,8 +3780,26 @@ export class PostgresEngine implements BrainEngine { return { inserted: ids.length, ids }; } - async deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }> { + async deleteFactsForPage( + slug: string, + source_id: string, + opts?: { excludeSourcePrefixes?: string[] }, + ): Promise<{ deleted: number }> { const sql = this.sql; + const prefixes = opts?.excludeSourcePrefixes; + if (prefixes && prefixes.length > 0) { + // #1928: keep rows whose `source` matches an excluded prefix (e.g. + // `cli:` conversation facts). COALESCE so NULL/empty-source fence rows + // stay deletable — only the explicitly-protected prefixes survive. + const patterns = prefixes.map(p => `${p}%`); + const result = await sql` + DELETE FROM facts + WHERE source_id = ${source_id} + AND source_markdown_slug = ${slug} + AND NOT (COALESCE(source, '') LIKE ANY(${patterns})) + `; + return { deleted: result.count ?? 0 }; + } const result = await sql` DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug} `; @@ -5119,7 +5172,7 @@ export class PostgresEngine implements BrainEngine { const toQual = resolved.map(e => e.to_symbol_qualified); const edgeTypes = resolved.map(e => e.edge_type); const metas = resolved.map(e => JSON.stringify(e.edge_metadata ?? {})); - const sources = resolved.map(e => e.source_id ?? null); + const sources = resolved.map(e => e.source_id ?? 'default'); const res = await sql` INSERT INTO code_edges_chunk (from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id) SELECT * FROM unnest( @@ -5139,7 +5192,7 @@ export class PostgresEngine implements BrainEngine { const toQual = unresolved.map(e => e.to_symbol_qualified); const edgeTypes = unresolved.map(e => e.edge_type); const metas = unresolved.map(e => JSON.stringify(e.edge_metadata ?? {})); - const sources = unresolved.map(e => e.source_id ?? null); + const sources = unresolved.map(e => e.source_id ?? 'default'); const res = await sql` INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id) SELECT * FROM unnest( diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index dc288eabd..2e30b49f1 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -390,6 +390,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to ON code_edges_chunk(to_chunk_id, edge_type); CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol ON code_edges_chunk(to_symbol_qualified, edge_type); +-- getCalleesOf filters on from_symbol_qualified; without this index every +-- callee lookup is a sequential scan, amplified per-BFS-node by the +-- recursive code walk. Mirrors migration v116. +CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol + ON code_edges_chunk(from_symbol_qualified); CREATE TABLE IF NOT EXISTS code_edges_symbol ( id SERIAL PRIMARY KEY, @@ -407,16 +412,32 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from ON code_edges_symbol(from_chunk_id, edge_type); CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to ON code_edges_symbol(to_symbol_qualified, edge_type); +-- getCalleesOf companion to idx_code_edges_chunk_from_symbol above. +-- Mirrors migration v116. +CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol + ON code_edges_symbol(from_symbol_qualified); -- ============================================================ -- links: cross-references between pages -- ============================================================ --- Provenance model (v0.13, extended issue #972): --- link_source — 'markdown' | 'frontmatter' | 'manual' | 'wikilink-resolved' | NULL --- 'wikilink-resolved' is the opt-in --- (link_resolution.global_basename) basename-match --- provenance — see issue #972 / migration v113. --- (NULL = legacy row written before v0.13; unknown source) +-- Provenance model (v0.13; opened to kebab provenance in v114 / issue #1941): +-- link_source — open kebab-case provenance tag, NOT a closed allowlist. +-- Format gate (CHECK): ^[a-z][a-z0-9]*(-[a-z0-9]+)*\$ and +-- char_length <= 64. NULL = legacy row (pre-v0.13). +-- Reconciliation-managed built-ins written internally: +-- 'markdown' — body markdown links +-- 'frontmatter' — YAML frontmatter edges (see origin_*) +-- 'mentions' — auto-linked body-text mentions +-- 'wikilink-resolved'— opt-in global-basename [[name]] (#972) +-- User/tool-facing: +-- 'manual' — hand- or tool-created edges (the +-- add_link op default + CLI link-add) +-- '' — external derivers, e.g. 'citation-graph', +-- stamp their own kebab tag (no migration). +-- The add_link OP forbids callers from passing the four +-- managed built-ins (they imply reconciliation semantics a +-- hand-created row can't honor); the DB CHECK still admits +-- them because internal writers use them. See operations.ts. -- origin_page_id — for link_source='frontmatter', the page whose YAML -- frontmatter created this edge; scopes reconciliation -- origin_field — the frontmatter field name (e.g. 'key_people') @@ -424,21 +445,21 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to -- The unique constraint includes link_source + origin_page_id so a manual edge -- and a frontmatter-derived edge with the same (from, to, type) tuple coexist. -- Reconciliation on put_page filters by (link_source='frontmatter' AND --- origin_page_id = written_page) — never touches other pages' edges. +-- origin_page_id = written_page) — never touches other pages' edges. (This is +-- exactly why a CLI-forged 'frontmatter' row with NULL origin would be a phantom +-- edge reconciliation never cleans — hence the op-layer guard.) CREATE TABLE IF NOT EXISTS links ( id SERIAL PRIMARY KEY, from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE, to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE, link_type TEXT NOT NULL DEFAULT '', context TEXT NOT NULL DEFAULT '', - -- v114 (#1941): link_source is an open kebab-case provenance, not a closed - -- allowlist — external derivers (e.g. 'citation-graph') stamp their own tag - -- without a gbrain migration. Format gate only: lowercase kebab, <=64 chars. - -- The reconciliation-managed built-ins ('markdown','frontmatter','mentions', - -- 'wikilink-resolved') still satisfy this and are written internally; the - -- add_link op forbids CALLERS from forging them (see operations.ts). 'manual' - -- is the user-facing default for hand/tool-created edges. - link_source TEXT CHECK (link_source IS NULL OR (link_source ~ '^[a-z][a-z0-9]*(-[a-z0-9]+)*$' AND char_length(link_source) <= 64)), + -- v0.41.18.0: 'mentions' added for auto-linked body-text mentions + -- (gbrain extract links --by-mention). Filtered OUT of backlink-count + -- for search ranking; only counts toward orphan-ratio + graph traversal. + -- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename + -- wikilink resolution (bare [[name]] resolved by slug tail). + link_source TEXT CHECK (link_source IS NULL OR (link_source ~ '^[a-z][a-z0-9]*(-[a-z0-9]+)*\$' AND char_length(link_source) <= 64)), -- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from -- "verb-pattern-derived typed link" within link_source='mentions'. -- Codex finding #12 design: keep link_source stable; add link_kind @@ -675,8 +696,11 @@ CREATE TABLE IF NOT EXISTS op_checkpoints ( CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx ON op_checkpoints (updated_at); --- #1794: append-only delta storage (one row per completed path). FK cascade --- drops children with the parent. Mirrors migration v115 + src/schema.sql. +-- #1794: append-only delta storage. One row per completed path; sync's +-- appendCompleted INSERTs only the delta instead of rewriting the whole +-- completed_keys JSONB array each flush (O(N^2) -> O(delta)). FK cascade drops +-- children with the parent (clearOpCheckpoint + 7-day purge). PK prefix +-- (op,fingerprint) serves all reads; no separate index. Mirrors migration v115. CREATE TABLE IF NOT EXISTS op_checkpoint_paths ( op TEXT NOT NULL, fingerprint TEXT NOT NULL, diff --git a/src/core/search/mode.ts b/src/core/search/mode.ts index d5a8198e6..f851913c0 100644 --- a/src/core/search/mode.ts +++ b/src/core/search/mode.ts @@ -738,7 +738,16 @@ export function attributeKnob( // to take effect immediately (one-time global cache cold-miss on upgrade; refills // within cache.ttl_seconds). Same cache-key-contamination convention as the // autocut / title_boost / graph_signals bumps above. -export const KNOBS_HASH_VERSION = 10; +// +// bump 10→11 (#1400 input_type fix): asymmetric embedding models (zembed-1 +// hosted or local, Voyage v3+) had their query-side input_type stripped by +// the AI SDK before the wire, so every cached row's key embedding AND result +// set were computed with document-side query vectors. The fix changes what +// embedQuery() produces for those providers; pre-fix rows must not be served +// to post-fix lookups. Same one-time global cold-miss pattern as the bumps +// above (the hash is global, not per-provider); refills within +// cache.ttl_seconds (3600s default). +export const KNOBS_HASH_VERSION = 11; /** * v0.36 (D8 / CDX-2) — second-arg context for the cache key. The diff --git a/src/core/sync.ts b/src/core/sync.ts index d92378bc8..c2c46a2e1 100644 --- a/src/core/sync.ts +++ b/src/core/sync.ts @@ -241,6 +241,19 @@ function matchesAnyGlob(path: string, patterns?: string[]): boolean { */ const PRUNE_DIR_NAMES = new Set([ 'node_modules', + // Dependency / build-output trees that are git-ignored on virtually every + // repo and never contain hand-authored source worth indexing. `vendor` + // (PHP Composer / Go / Ruby bundle), `dist` + `build` (compiled output). + // Closes the silent-pollution bug where a Laravel/PHP repo's full code sync + // walked ~50k `vendor/` files (#1483 / #1159 / maintainer #1942). + 'vendor', + 'dist', + 'build', + // Python venv: vendored dependency tree, the `node_modules` analogue (#2020). + // Like `node_modules` it lacks a leading dot so isSyncable's dot-prefix + // exclusion misses it; explicit entry keeps incremental sync consistent + // with the first-sync walker in commands/import.ts. + 'venv', '.raw', 'ops', ]); diff --git a/src/core/timeline-dedup-repair.ts b/src/core/timeline-dedup-repair.ts new file mode 100644 index 000000000..4aff01223 --- /dev/null +++ b/src/core/timeline-dedup-repair.ts @@ -0,0 +1,120 @@ +/** + * #2038 — idx_timeline_dedup schema-drift self-heal. + * + * Migration v102 (`timeline_entries_source_in_dedup`) widens the dedup index + * from (page_id, date, summary) to (page_id, date, summary, source). It was + * renumbered from v99 during a master merge, so a brain that ran the OLD v99 + * variant has its version counter stamped PAST v102 while the index stayed + * 3-column. `runMigrations` then can't see the drift (it early-returns when no + * version is pending), and every `addTimelineEntry(esBatch)` fails with + * "no unique or exclusion constraint matching the ON CONFLICT specification" + * because both insert sites infer on the 4-column tuple — timeline writes + * silently break brain-wide. + * + * The version counter can't detect this, so the repair is keyed off the actual + * index SHAPE and runs on every migrate pass (including the no-pending path). + * Idempotent: a no-op when the index is already 4-column. + */ + +import type { BrainEngine } from './engine.ts'; + +const INDEX_NAME = 'idx_timeline_dedup'; +const EXPECTED_COLUMNS = ['page_id', 'date', 'summary', 'source']; + +export interface TimelineDedupStatus { + /** The timeline_entries table exists (nothing to repair if not). */ + tablePresent: boolean; + /** The index exists. */ + indexPresent: boolean; + /** Indexed columns in order (empty when the index is absent). */ + columns: string[]; + /** Index exists in the wrong (pre-v102) shape — needs a rebuild. */ + needsRepair: boolean; +} + +/** Parse the column list out of a pg_indexes `indexdef` string. */ +function parseIndexColumns(indexdef: string): string[] { + const open = indexdef.lastIndexOf('('); + const close = indexdef.lastIndexOf(')'); + if (open < 0 || close < 0 || close < open) return []; + return indexdef + .slice(open + 1, close) + .split(',') + .map(c => c.trim().split(/\s+/)[0]) // drop any "col DESC"/opclass suffix + .filter(Boolean); +} + +export async function checkTimelineDedupIndex(engine: BrainEngine): Promise { + const tbl = await engine.executeRaw<{ reg: string | null }>( + `SELECT to_regclass('timeline_entries')::text AS reg`, + ); + const tablePresent = !!tbl[0]?.reg; + if (!tablePresent) { + return { tablePresent: false, indexPresent: false, columns: [], needsRepair: false }; + } + const rows = await engine.executeRaw<{ indexdef: string }>( + `SELECT indexdef FROM pg_indexes WHERE indexname = $1`, + [INDEX_NAME], + ); + const indexPresent = rows.length > 0; + const columns = indexPresent ? parseIndexColumns(rows[0].indexdef) : []; + const correct = + columns.length === EXPECTED_COLUMNS.length && + EXPECTED_COLUMNS.every((c, i) => columns[i] === c); + // An ABSENT index is also "needs repair" — the migration that creates it was + // skipped. (A fresh brain always has it, created by the migration chain.) + return { tablePresent, indexPresent, columns, needsRepair: !correct }; +} + +export interface TimelineDedupRepairResult { + repaired: boolean; + before: string[]; + collapsedDuplicates: number; + reason: 'already_correct' | 'no_table' | 'rebuilt'; +} + +/** + * Heal the index if it's missing the v102 4-column shape. Dedupes FIRST — + * the loose 3-column index let rows differing only by `source` coexist, and + * `CREATE UNIQUE INDEX` would throw on those collisions otherwise. Keeps the + * earliest row (min ctid) of each 4-tuple group. + */ +export async function repairTimelineDedupIndex(engine: BrainEngine): Promise { + const status = await checkTimelineDedupIndex(engine); + if (!status.tablePresent) { + return { repaired: false, before: [], collapsedDuplicates: 0, reason: 'no_table' }; + } + if (!status.needsRepair) { + return { repaired: false, before: status.columns, collapsedDuplicates: 0, reason: 'already_correct' }; + } + + // Keep the lowest `id` per 4-tuple group — deterministic and consistent with + // the existing v-migration dedup rule (`a.id > b.id`), unlike `ctid` which is + // a physical tuple location that can preserve an arbitrary duplicate. + const del = await engine.executeRaw<{ n: string }>( + `WITH d AS ( + DELETE FROM timeline_entries t + USING ( + SELECT page_id, date, summary, source, MIN(id) AS keep + FROM timeline_entries + GROUP BY page_id, date, summary, source + HAVING COUNT(*) > 1 + ) dup + WHERE t.page_id = dup.page_id + AND t.date = dup.date + AND t.summary = dup.summary + AND t.source IS NOT DISTINCT FROM dup.source + AND t.id <> dup.keep + RETURNING 1 + ) + SELECT COUNT(*)::text AS n FROM d`, + ); + const collapsedDuplicates = parseInt(del[0]?.n ?? '0', 10); + + await engine.executeRaw(`DROP INDEX IF EXISTS ${INDEX_NAME}`); + await engine.executeRaw( + `CREATE UNIQUE INDEX IF NOT EXISTS ${INDEX_NAME} + ON timeline_entries(page_id, date, summary, source)`, + ); + return { repaired: true, before: status.columns, collapsedDuplicates, reason: 'rebuilt' }; +} diff --git a/src/core/write-through.ts b/src/core/write-through.ts index b21d2cb24..96a1aad9b 100644 --- a/src/core/write-through.ts +++ b/src/core/write-through.ts @@ -22,7 +22,7 @@ */ import { existsSync, statSync, mkdirSync, writeFileSync, renameSync, unlinkSync } from 'fs'; -import { dirname } from 'path'; +import { dirname, join } from 'path'; import { randomBytes } from 'crypto'; import type { BrainEngine } from './engine.ts'; import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts'; @@ -37,12 +37,16 @@ export interface WriteThroughResult { path?: string; /** * Non-error reasons the file was not written: - * - no_repo_configured: `sync.repo_path` is unset (DB-only by design). - * - repo_not_found: `sync.repo_path` set but missing / not a directory. + * - no_repo_configured: the resolved target (source `local_path` or, for a + * sole-source brain, `sync.repo_path`) is unset (DB-only by design). + * - repo_not_found: target set but missing / not a directory. + * - source_repo_belongs_to_other_source: the assigned source has no + * `local_path`, and `sync.repo_path` is another source's own working tree + * — #2018: writing here would pollute that sibling's repo, so we skip. * - page_not_found_after_write: the DB row isn't readable back (the caller's * DB write failed or targeted a different source). */ - skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write'; + skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_repo_belongs_to_other_source' | 'page_not_found_after_write'; /** Set when the render/write/rename itself threw (EACCES, ENOTDIR, disk full). */ error?: string; } @@ -67,12 +71,46 @@ export async function writePageThrough( ): Promise { const sourceId = opts.sourceId ?? 'default'; try { - const repoPath = await engine.getConfig('sync.repo_path'); - if (!repoPath) { - return { written: false, skipped: 'no_repo_configured' }; - } - if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) { - return { written: false, skipped: 'repo_not_found' }; + // #2018: pick the disk target so a page is NEVER written into a different + // source's working tree. Two legitimate topologies, plus the leak guard: + // 1. The assigned source has its OWN `local_path` (a separate working + // tree) → write at that tree's root (matches how `scanOneSource` reads + // it back; never nested under `.sources/`). + // 2. No per-source `local_path` → nest under the host repo + // (`sync.repo_path`): default at the root, non-default under + // `.sources//` (the established multi-source layout). + // 3. LEAK GUARD: if `sync.repo_path` is literally ANOTHER source's own + // `local_path`, nesting this page there would pollute that sibling's + // git repo (the reported bug). Skip instead. + let filePath: string; + const srcRows = await engine.executeRaw<{ local_path: string | null }>( + `SELECT local_path FROM sources WHERE id = $1`, + [sourceId], + ); + const sourceLocalPath = srcRows[0]?.local_path ?? null; + if (sourceLocalPath) { + if (!existsSync(sourceLocalPath) || !statSync(sourceLocalPath).isDirectory()) { + return { written: false, skipped: 'repo_not_found' }; + } + filePath = join(sourceLocalPath, `${slug}.md`); + } else { + const repoPath = await engine.getConfig('sync.repo_path'); + if (!repoPath) { + return { written: false, skipped: 'no_repo_configured' }; + } + if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) { + return { written: false, skipped: 'repo_not_found' }; + } + // Leak guard: refuse to write into a path that is some OTHER source's + // own working tree (#2018). + const collide = await engine.executeRaw<{ one: number }>( + `SELECT 1 AS one FROM sources WHERE id <> $1 AND local_path = $2 LIMIT 1`, + [sourceId, repoPath], + ); + if (collide.length > 0) { + return { written: false, skipped: 'source_repo_belongs_to_other_source' }; + } + filePath = resolvePageFilePath(repoPath, slug, sourceId); } const writtenPage = await engine.getPage(slug, { sourceId }); @@ -85,7 +123,6 @@ export async function writePageThrough( frontmatterOverrides: opts.frontmatterOverrides, }); - const filePath = resolvePageFilePath(repoPath, slug, sourceId); mkdirSync(dirname(filePath), { recursive: true }); // Atomic write: unique temp sibling + rename. Unique name (pid + random) diff --git a/src/mcp/http-transport.ts b/src/mcp/http-transport.ts index c1ccb3152..c3ff28c27 100644 --- a/src/mcp/http-transport.ts +++ b/src/mcp/http-transport.ts @@ -34,6 +34,8 @@ import { VERSION } from '../version.ts'; import { dispatchToolCall } from './dispatch.ts'; import { buildDefaultLimiters, type RateLimiter } from './rate-limit.ts'; import { sqlQueryForEngine } from '../core/sql-query.ts'; +import { parseLegacyTokenScope } from '../core/legacy-token-scope.ts'; +export { parseLegacyTokenScope }; const DEFAULT_BODY_CAP = 1024 * 1024; // 1 MiB @@ -84,28 +86,8 @@ interface AuthResult { auth?: AuthInfo; } -/** - * #1336: derive a legacy bearer token's source scope from its stored - * `permissions.source_id`. An ARRAY value is a federated_read grant → - * `allowedSources` (scoped reads across exactly those sources). A STRING value - * scopes the scalar floor. Anything else → 'default' (preserves pre-v0.34 - * behavior). NEVER widened to "all": an empty/garbage value keeps the 'default' - * floor and no federated grant. - */ -export function parseLegacyTokenScope(rawSource: unknown): { sourceId: string; allowedSources?: string[] } { - if (Array.isArray(rawSource)) { - const allowedSources = (rawSource as unknown[]).filter(s => typeof s === 'string' && s.length > 0) as string[]; - if (allowedSources.length > 0) { - // Scalar floor: the first granted source (write authority); reads span the array. - return { sourceId: allowedSources[0], allowedSources }; - } - return { sourceId: 'default' }; - } - if (typeof rawSource === 'string' && rawSource.length > 0) { - return { sourceId: rawSource }; - } - return { sourceId: 'default' }; -} +/* Legacy token source-scope parsing lives in core/legacy-token-scope.ts and is + * re-exported above so the legacy HTTP transport and OAuth provider cannot drift. */ /** Read up to `cap` bytes off req.body. Returns null if cap exceeded. */ async function readBodyWithCap(req: Request, cap: number): Promise { diff --git a/src/schema.sql b/src/schema.sql index a900c6d1a..3a5f32ef7 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -386,6 +386,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to ON code_edges_chunk(to_chunk_id, edge_type); CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol ON code_edges_chunk(to_symbol_qualified, edge_type); +-- getCalleesOf filters on from_symbol_qualified; without this index every +-- callee lookup is a sequential scan, amplified per-BFS-node by the +-- recursive code walk. Mirrors migration v116. +CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol + ON code_edges_chunk(from_symbol_qualified); CREATE TABLE IF NOT EXISTS code_edges_symbol ( id SERIAL PRIMARY KEY, @@ -403,6 +408,10 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from ON code_edges_symbol(from_chunk_id, edge_type); CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to ON code_edges_symbol(to_symbol_qualified, edge_type); +-- getCalleesOf companion to idx_code_edges_chunk_from_symbol above. +-- Mirrors migration v116. +CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol + ON code_edges_symbol(from_symbol_qualified); -- ============================================================ -- links: cross-references between pages diff --git a/test/ai/zeroentropy-compat-fetch.test.ts b/test/ai/zeroentropy-compat-fetch.test.ts index f2d784b1f..e12824b6d 100644 --- a/test/ai/zeroentropy-compat-fetch.test.ts +++ b/test/ai/zeroentropy-compat-fetch.test.ts @@ -53,13 +53,15 @@ describe('zeroEntropyCompatFetch — shim structural shape', () => { expect(src).not.toContain('/v1/v1/'); }); - test('body injects input_type default "document"', async () => { + test('body recovers threaded input_type, defaulting to "document"', async () => { const src = await Bun.file(GATEWAY_PATH).text(); - // The wrapper defaults input_type to 'document' when caller didn't - // thread one (matches the document-side correctness for sync / - // import / embed CLI paths). + // The wrapper recovers the SDK-stripped input_type from + // __embedInputTypeStore (#1400) and still defaults to 'document' when + // no caller threaded one (document-side correctness for sync / import / + // embed CLI paths). Wire-level behavioral coverage lives in + // test/embed-input-type-wire.test.ts. expect(src).toMatch(/parsed\.input_type\s*===\s*undefined/); - expect(src).toMatch(/parsed\.input_type\s*=\s*['"]document['"]/); + expect(src).toMatch(/parsed\.input_type\s*=\s*__embedInputTypeStore\.getStore\(\)\s*\?\?\s*['"]document['"]/); }); test('body forces encoding_format=float (CDX2-F2)', async () => { diff --git a/test/cli-force-exit-teardown-arming.test.ts b/test/cli-force-exit-teardown-arming.test.ts new file mode 100644 index 000000000..ff9743e30 --- /dev/null +++ b/test/cli-force-exit-teardown-arming.test.ts @@ -0,0 +1,56 @@ +/** + * Structural regression — the DISCONNECT_HARD_DEADLINE_MS force-exit timer in + * cli.ts main() must be armed at TEARDOWN ENTRY (inside the finally, before + * the drain + disconnect), never before the op-dispatch try block. + * + * Pre-fix bug: the 10s unref'd setTimeout was armed BEFORE the try, so any op + * whose handler ran past 10s wall-clock was killed mid-flight with + * process.exit(0) and ZERO stdout — an empty "success" indistinguishable from + * no results (a healthy `gbrain search` on a slow Postgres pooler hit this on + * every run). Armed in the finally, the timer still bounds a hung + * drain/disconnect (the C13 contract) but can no longer kill a + * slow-but-progressing op body. + * + * Source-grep is the right tool here (same rationale as + * fix-wave-structural.test.ts): the rule is "this arming must stay at this + * location". A behavioral test would need >10s of real wall-clock plus a + * deliberately slow op handler in a spawned CLI — slow and flaky by + * construction. + */ + +import { describe, test, expect } from 'bun:test'; +import { readFileSync } from 'fs'; + +describe('cli.ts — disconnect hard-deadline armed at teardown entry, not before the op body', () => { + test('forceExitTimer setTimeout lives inside the finally, gated on the daemon guard, before the drain', () => { + const src = readFileSync('src/cli.ts', 'utf8'); + + const decl = src.indexOf('const DISCONNECT_HARD_DEADLINE_MS'); + expect(decl).toBeGreaterThan(-1); + const tryIdx = src.indexOf('try {', decl); + expect(tryIdx).toBeGreaterThan(-1); + const finallyIdx = src.indexOf('} finally {', tryIdx); + expect(finallyIdx).toBeGreaterThan(-1); + const armIdx = src.indexOf('forceExitTimer = setTimeout', decl); + expect(armIdx).toBeGreaterThan(-1); + const drainIdx = src.indexOf('drainAllBackgroundWorkForCliExit', finallyIdx); + expect(drainIdx).toBeGreaterThan(-1); + + // NO arming between the deadline declaration and the op-body try — a + // pre-try timer kills slow-but-progressing op handlers mid-flight with + // exit 0 and empty stdout. (`setTimeout(` matches only a call site; the + // `ReturnType` type annotation stays allowed.) + expect(src.slice(decl, tryIdx)).not.toContain('setTimeout('); + + // The arming sits AFTER the finally opens (teardown entry) and BEFORE the + // drain + disconnect it exists to bound. + expect(armIdx).toBeGreaterThan(finallyIdx); + expect(armIdx).toBeLessThan(drainIdx); + + // Still gated on the daemon-survival guard so `serve` stays alive, and + // still unref'd + cleared on clean teardown. + expect(src.slice(finallyIdx, drainIdx)).toMatch(/if \(shouldForceExitAfterMain\(\)\)/); + expect(src.slice(finallyIdx, drainIdx)).toContain('forceExitTimer.unref?.()'); + expect(src.slice(drainIdx)).toContain('if (forceExitTimer) clearTimeout(forceExitTimer)'); + }); +}); diff --git a/test/config-env-hijack.test.ts b/test/config-env-hijack.test.ts new file mode 100644 index 000000000..3dafe6e02 --- /dev/null +++ b/test/config-env-hijack.test.ts @@ -0,0 +1,144 @@ +import { mkdtempSync, rmSync, writeFileSync } from 'fs'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { describe, expect, test } from 'bun:test'; +import { dotenvValuesForKey, effectiveEnvDatabaseUrl } from '../src/core/config.ts'; +import { withEnv } from './helpers/with-env.ts'; + +// #427 — Bun auto-loads `.env` from the process cwd, so running gbrain inside +// any web-app checkout whose `.env` defines DATABASE_URL silently retargets +// the brain at that app's database. These tests pin the guard: +// effectiveEnvDatabaseUrl() must treat a DATABASE_URL whose value matches a +// cwd .env assignment as file-origin (ignored), while still honoring +// GBRAIN_DATABASE_URL and genuinely exported DATABASE_URLs. +// +// The dir is injected instead of process.chdir'd so these tests stay safe in +// the parallel shard runner (cwd is process-global, same reason with-env.ts +// exists for process.env). + +function tmpProject(envFiles: Record): string { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-')); + for (const [name, content] of Object.entries(envFiles)) { + writeFileSync(join(dir, name), content); + } + return dir; +} + +describe('dotenvValuesForKey', () => { + test('collects assignments across all auto-loaded .env variants', () => { + const dir = tmpProject({ + '.env': 'DATABASE_URL=postgres://app:pw@prod.example.test:5432/app\n', + '.env.local': "DATABASE_URL='postgres://app:pw@local.example.test:5432/app'\n", + }); + try { + const values = dotenvValuesForKey('DATABASE_URL', dir); + expect(values.has('postgres://app:pw@prod.example.test:5432/app')).toBe(true); + expect(values.has('postgres://app:pw@local.example.test:5432/app')).toBe(true); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('handles export prefix, double quotes, comments, and unrelated keys', () => { + const dir = tmpProject({ + '.env': [ + '# comment line', + 'export DATABASE_URL="postgres://quoted.example.test/db"', + 'OTHER_KEY=not-a-db-url', + 'EMPTY=', + '', + ].join('\n'), + }); + try { + const values = dotenvValuesForKey('DATABASE_URL', dir); + expect(values.has('postgres://quoted.example.test/db')).toBe(true); + expect(values.size).toBe(1); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('returns empty set when no .env files exist', () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-none-')); + try { + expect(dotenvValuesForKey('DATABASE_URL', dir).size).toBe(0); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +describe('effectiveEnvDatabaseUrl (#427 guard)', () => { + const HIJACK_URL = 'postgres://app:pw@victim-prod.example.test:5432/app'; + const OPERATOR_URL = 'postgres://operator@deliberate.example.test:5432/brain'; + + test('ignores DATABASE_URL whose value matches a cwd .env assignment', async () => { + const dir = tmpProject({ '.env': `DATABASE_URL=${HIJACK_URL}\n` }); + try { + await withEnv( + { GBRAIN_DATABASE_URL: undefined, DATABASE_URL: HIJACK_URL }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBeUndefined(); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('honors an exported DATABASE_URL that differs from the cwd .env', async () => { + const dir = tmpProject({ '.env': `DATABASE_URL=${HIJACK_URL}\n` }); + try { + await withEnv( + { GBRAIN_DATABASE_URL: undefined, DATABASE_URL: OPERATOR_URL }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('GBRAIN_DATABASE_URL is never ignored, even when it matches the cwd .env', async () => { + const dir = tmpProject({ '.env': `GBRAIN_DATABASE_URL=${OPERATOR_URL}\nDATABASE_URL=${HIJACK_URL}\n` }); + try { + await withEnv( + { GBRAIN_DATABASE_URL: OPERATOR_URL, DATABASE_URL: HIJACK_URL }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('honors DATABASE_URL when no .env files exist in cwd', async () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-clean-')); + try { + await withEnv( + { GBRAIN_DATABASE_URL: undefined, DATABASE_URL: OPERATOR_URL }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('returns undefined when neither env var is set', async () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-unset-')); + try { + await withEnv( + { GBRAIN_DATABASE_URL: undefined, DATABASE_URL: undefined }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBeUndefined(); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/test/connection-resilience.test.ts b/test/connection-resilience.test.ts index 3f73d7eff..655c8418c 100644 --- a/test/connection-resilience.test.ts +++ b/test/connection-resilience.test.ts @@ -317,8 +317,10 @@ describe('Eng-review D3 — executeRaw has no per-call retry wrapper', () => { it('Supervisor still has the 3-strikes-then-reconnect path', () => { const src = readFileSync(resolve('src/core/minions/supervisor.ts'), 'utf-8'); expect(src).toContain('consecutiveHealthFailures'); - // Supervisor invokes reconnect via a typed cast after 3 consecutive failures. - expect(src).toMatch(/reconnect\(\): Promise/); + // #2034: reconnect() is now a first-class BrainEngine method, so the + // supervisor calls it directly after 3 consecutive failures (the prior + // `(engine as unknown as { reconnect(): Promise })` cast was removed). + expect(src).toContain('this.engine.reconnect('); expect(src).toContain('this.consecutiveHealthFailures >= 3'); }); }); diff --git a/test/cross-modal-phase1.test.ts b/test/cross-modal-phase1.test.ts index eaa245540..4f5243d96 100644 --- a/test/cross-modal-phase1.test.ts +++ b/test/cross-modal-phase1.test.ts @@ -136,15 +136,17 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => { return resolveSearchMode({ mode: 'balanced' }); } - test('KNOBS_HASH_VERSION is 10 (cross-modal still appended; 9→10 relational recall)', () => { + test('KNOBS_HASH_VERSION is 11 (cross-modal still appended; 10→11 asymmetric input_type fix)', () => { // v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3 // with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) + // v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields. // v0.40.3.0 D8 bumps to v=5 (sequenced behind salem's v=4 graph-signals). // v0.41.22.0 (type-unification): 5→6 for alias_resolved post-fusion boost. // T2: 6→7 title_boost. v0.42.3.0: 7→8 autocut. issue #1777: 8→9 archive/ demote. - // v0.43: 9→10 relational recall arm. - expect(KNOBS_HASH_VERSION).toBe(10); + // v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type + // finally reaches asymmetric providers — pre-fix rows were keyed on + // document-side query vectors. + expect(KNOBS_HASH_VERSION).toBe(11); }); test('flipping unified_multimodal changes the hash', () => { diff --git a/test/embed-input-type-wire.serial.test.ts b/test/embed-input-type-wire.serial.test.ts new file mode 100644 index 000000000..3e8fa73e1 --- /dev/null +++ b/test/embed-input-type-wire.serial.test.ts @@ -0,0 +1,247 @@ +/** + * #1400 — asymmetric `input_type` must survive the AI SDK boundary and + * reach the WIRE BODY. + * + * test/asymmetric-encoding-contract.test.ts pins that embedQuery() threads + * `input_type: 'query'` into the transport's providerOptions. This file + * pins the layer BELOW that contract: the AI SDK's openai-compatible + * adapter validates providerOptions against a fixed schema and silently + * drops `input_type` before building the HTTP body. Without the + * `__embedInputTypeStore` recovery in the per-recipe fetch shims, every + * query was encoded document-side (ZE shim's hard default) or with no + * input_type at all (Voyage, llama-server) — asymmetric retrieval silently + * collapsed while the providerOptions-level test stayed green. + * + * These tests run the REAL SDK transport with a mocked global fetch and + * assert on the outbound request body — the only place the regression is + * observable. + */ + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { configureGateway, embed, embedQuery, resetGateway } from '../src/core/ai/gateway.ts'; + +type FetchHandler = (url: string, init: RequestInit) => Promise; +let fetchHandler: FetchHandler | null = null; +const origFetch = globalThis.fetch; + +beforeEach(() => { + fetchHandler = null; + globalThis.fetch = (async (url: string | URL | Request, init?: RequestInit) => { + if (!fetchHandler) { + throw new Error('fetch called but no handler installed'); + } + return fetchHandler(typeof url === 'string' ? url : url.toString(), init ?? {}); + }) as typeof fetch; +}); + +afterEach(() => { + globalThis.fetch = origFetch; + resetGateway(); +}); + +/** OpenAI-shaped /v1/embeddings response (llama-server is already OpenAI-shaped). */ +function openAIShapedResponse(dims: number, count: number): Response { + const vec = Array.from({ length: dims }, () => 0.1); + return new Response( + JSON.stringify({ + data: Array.from({ length: count }, (_, i) => ({ object: 'embedding', index: i, embedding: vec })), + usage: { prompt_tokens: 3, total_tokens: 3 }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ); +} + +/** ZE-shaped /v1/models/embed response (zeroEntropyCompatFetch rewrites results→data). */ +function zeShapedResponse(dims: number, count: number): Response { + const vec = Array.from({ length: dims }, () => 0.1); + return new Response( + JSON.stringify({ + results: Array.from({ length: count }, () => ({ embedding: vec })), + usage: { total_bytes: 12, total_tokens: 3 }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ); +} + +/** Voyage-shaped response: base64 Float32 LE embeddings (rewriter decodes). */ +function voyageShapedResponse(dims: number, count: number): Response { + const b64 = Buffer.from(new Float32Array(dims).fill(0.1).buffer).toString('base64'); + return new Response( + JSON.stringify({ + data: Array.from({ length: count }, (_, i) => ({ object: 'embedding', index: i, embedding: b64 })), + usage: { total_tokens: 3 }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ); +} + +describe('ZeroEntropy hosted — input_type reaches the wire body', () => { + function configureZE() { + configureGateway({ + embedding_model: 'zeroentropyai:zembed-1', + embedding_dimensions: 1280, + env: { ZEROENTROPY_API_KEY: 'sk-fake' }, + }); + } + + test('embedQuery sends input_type=query (not the document default)', async () => { + configureZE(); + let capturedUrl = ''; + let capturedBody: any = null; + fetchHandler = async (url, init) => { + capturedUrl = url; + capturedBody = JSON.parse(init.body as string); + return zeShapedResponse(1280, 1); + }; + + await embedQuery('what does foo bar do?'); + // Sanity: the ZE shim ran (URL path rewritten). + expect(capturedUrl).toContain('/models/embed'); + expect(capturedBody.input_type).toBe('query'); + }); + + test('embed (index path) sends input_type=document', async () => { + configureZE(); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return zeShapedResponse(1280, 1); + }; + + await embed(['this is a document being indexed']); + expect(capturedBody.input_type).toBe('document'); + }); +}); + +describe('openai-compatible recipes (local/proxy asymmetric models) — input_type reaches the wire body', () => { + function configureLlamaServer(modelId: string, dims: number) { + configureGateway({ + embedding_model: `llama-server:${modelId}`, + embedding_dimensions: dims, + env: {}, + }); + } + + test('embedQuery against a local zembed-1 sends input_type=query', async () => { + configureLlamaServer('zembed-1', 1280); + let capturedUrl = ''; + let capturedBody: any = null; + fetchHandler = async (url, init) => { + capturedUrl = url; + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(1280, 1); + }; + + await embedQuery('what does foo bar do?'); + // URL untouched — llama-server's /v1/embeddings is already OpenAI-shaped. + expect(capturedUrl).toContain('/embeddings'); + expect(capturedUrl).not.toContain('/models/embed'); + expect(capturedBody.input_type).toBe('query'); + }); + + test('embed (index path) against a local zembed-1 sends input_type=document', async () => { + configureLlamaServer('zembed-1', 1280); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(1280, 1); + }; + + await embed(['this is a document being indexed']); + expect(capturedBody.input_type).toBe('document'); + }); + + test('non-asymmetric model: wire body carries NO input_type (strict pass-through)', async () => { + // dims.ts only threads input_type for recognized asymmetric models; + // for anything else the shim must leave the body untouched so vanilla + // llama-server deployments see zero wire change. + configureLlamaServer('my-gguf', 768); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(768, 1); + }; + + await embedQuery('hello'); + expect(capturedBody).not.toBeNull(); + expect('input_type' in capturedBody).toBe(false); + }); + + test('litellm proxying an asymmetric model: embedQuery sends input_type=query', async () => { + // The shim is the fallthrough default for every openai-compatible + // recipe without its own compat fetch — dims.ts threads input_type by + // model id, so a zembed-1 behind a LiteLLM proxy (e.g. fronting vLLM) + // gets the same signal as llama-server. + configureGateway({ + embedding_model: 'litellm:zembed-1', + embedding_dimensions: 1280, + env: { LITELLM_API_KEY: 'sk-fake' }, + base_urls: { litellm: 'http://localhost:4000' }, + }); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(1280, 1); + }; + + await embedQuery('what does foo bar do?'); + expect(capturedBody.input_type).toBe('query'); + }); + + test('ollama serving an asymmetric model: embedQuery sends input_type=query', async () => { + configureGateway({ + embedding_model: 'ollama:zembed-1', + embedding_dimensions: 1280, + env: {}, + }); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(1280, 1); + }; + + await embedQuery('what does foo bar do?'); + expect(capturedBody.input_type).toBe('query'); + }); +}); + +describe('Voyage hosted — input_type reaches the wire body (opt-in preserved)', () => { + function configureVoyage() { + configureGateway({ + embedding_model: 'voyage:voyage-3-large', + embedding_dimensions: 1024, + env: { VOYAGE_API_KEY: 'sk-fake' }, + }); + } + + test('embedQuery sends input_type=query', async () => { + configureVoyage(); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return voyageShapedResponse(1024, 1); + }; + + await embedQuery('what does foo bar do?'); + expect(capturedBody.input_type).toBe('query'); + // Existing voyage translation still applies on the same body. + expect(capturedBody.output_dimension).toBe(1024); + expect(capturedBody.encoding_format).toBe('base64'); + }); + + test('embed (index path) keeps input_type OFF the wire (pre-v0.35.0.0 opt-in shape)', async () => { + // dims.ts deliberately emits no input_type for Voyage unless threaded + // (`...(inputType ? { input_type: inputType } : {})`); the shim must + // not invent a default for it. + configureVoyage(); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return voyageShapedResponse(1024, 1); + }; + + await embed(['this is a document being indexed']); + expect(capturedBody).not.toBeNull(); + expect('input_type' in capturedBody).toBe(false); + }); +}); diff --git a/test/embedding-dim-check-facts.test.ts b/test/embedding-dim-check-facts.test.ts index d152e6fd4..2b0a9a8fc 100644 --- a/test/embedding-dim-check-facts.test.ts +++ b/test/embedding-dim-check-facts.test.ts @@ -142,6 +142,27 @@ describe('buildFactsAlterRecipe', () => { const recipe = buildFactsAlterRecipe(1536, 1280, 'halfvec'); expect(recipe).toMatch(/DROP INDEX[\s\S]*ALTER TABLE[\s\S]*CREATE INDEX/); }); + + test('dimension change NULLs embeddings BEFORE the alter (pgvector refuses cross-dim casts)', () => { + // Same defect class fixed for content_chunks in embeddingMismatchMessage: + // pgvector aborts a cross-dimension ALTER while rows still hold old-width + // vectors. The dims-change recipe must wipe first; order pinned. + const recipe = buildFactsAlterRecipe(1536, 1280, 'halfvec'); + const nullIdx = recipe.indexOf('UPDATE facts SET embedding = NULL'); + const alterIdx = recipe.indexOf('ALTER TABLE facts ALTER COLUMN embedding TYPE'); + expect(nullIdx).toBeGreaterThan(-1); + expect(alterIdx).toBeGreaterThan(-1); + expect(nullIdx).toBeLessThan(alterIdx); + }); + + test('same-dim type swap PRESERVES embeddings (no NULL wipe)', () => { + // halfvec(1536) <-> vector(1536): the USING cast is lossless and the + // whole point is keeping the data. A wipe here would destroy valid + // embeddings for no reason. + const recipe = buildFactsAlterRecipe(1536, 1536, 'vector'); + expect(recipe).not.toContain('UPDATE facts SET embedding = NULL'); + expect(recipe).toContain('USING embedding::vector(1536)'); + }); }); describe('FactsEmbeddingDimMismatchError', () => { diff --git a/test/embedding-dim-check.test.ts b/test/embedding-dim-check.test.ts index f74ebc750..6a945b98c 100644 --- a/test/embedding-dim-check.test.ts +++ b/test/embedding-dim-check.test.ts @@ -103,6 +103,26 @@ describe('embeddingMismatchMessage', () => { expect(msg).toContain('docs/embedding-migrations.md'); }); + test('Postgres recipe NULLs embeddings BEFORE the column alter (pgvector refuses cross-dim casts)', () => { + // pgvector aborts `ALTER COLUMN TYPE vector(N)` with "expected N + // dimensions, not M" while rows still hold old-width vectors — which is + // every brain running this recipe. The UPDATE must precede the ALTER + // (NULLs cast fine). Order pinned so the printed recipe can't drift from + // the corrected docs/embedding-migrations.md again. + const msg = embeddingMismatchMessage({ + currentDims: 1536, + requestedDims: 768, + requestedModel: 'nomic-embed-text', + source: 'init', + engineKind: 'postgres', + }); + const nullIdx = msg.indexOf('UPDATE content_chunks SET embedding = NULL'); + const alterIdx = msg.indexOf('ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(768)'); + expect(nullIdx).toBeGreaterThan(-1); + expect(alterIdx).toBeGreaterThan(-1); + expect(nullIdx).toBeLessThan(alterIdx); + }); + test('Postgres branch skips HNSW recreate when requested dims exceed pgvector cap', () => { // Codex finding #8: 2048d (Voyage 4 Large) cannot be HNSW-indexed in pgvector. // The recipe must NOT instruct a CREATE INDEX HNSW for that dim. diff --git a/test/facts-reconcile-protect-cli.test.ts b/test/facts-reconcile-protect-cli.test.ts new file mode 100644 index 000000000..b4ee032f7 --- /dev/null +++ b/test/facts-reconcile-protect-cli.test.ts @@ -0,0 +1,78 @@ +/** + * #1928 regression — extract_facts must NOT wipe conversation facts. + * + * `extract-conversation-facts` writes facts with source='cli:...' and + * source_markdown_slug=. Those pages carry no `## Facts` + * fence, so the cycle's wipe-and-reinsert reconcile (deleteFactsForPage + + * insertFactsBatch) used to delete them and reinsert nothing — a brain-wide + * conversation-facts wipe on a failed-sync full walk (status `ok`, 0 + * inserted, 1829 rows gone in the original report). + * + * The fix scopes the cycle delete with excludeSourcePrefixes: ['cli:']. + * These tests pin: (a) the exclusion protects cli:-origin rows while still + * deleting fence-owned rows on the same page coordinate, and (b) the default + * (no opts) behavior is unchanged — every fact on the coordinate is deleted. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +async function seedPageWithMixedFacts(slug: string, sourceId: string) { + await engine.insertFacts( + [ + // Fence-owned row (what extract_facts recreates from the page fence). + { fact: 'fence fact', kind: 'fact', source: 'fence', row_num: 1, source_markdown_slug: slug }, + // Empty-source row — also fence-default; must stay deletable. + { fact: 'blank-source fact', kind: 'fact', source: '', row_num: 2, source_markdown_slug: slug }, + // Conversation fact — NOT fence-owned; must survive the reconcile. + { fact: 'conversation fact', kind: 'fact', source: 'cli:extract-conversation-facts', row_num: 3, source_markdown_slug: slug }, + ], + { source_id: sourceId }, + ); +} + +async function factSourcesOnPage(slug: string, sourceId: string): Promise { + const rows = await engine.executeRaw<{ source: string }>( + `SELECT COALESCE(source, '') AS source FROM facts + WHERE source_id = $1 AND source_markdown_slug = $2 ORDER BY row_num`, + [sourceId, slug], + ); + return rows.map(r => r.source); +} + +describe('#1928 deleteFactsForPage excludeSourcePrefixes', () => { + test('protects cli:-origin facts, still deletes fence + empty-source rows', async () => { + await seedPageWithMixedFacts('transcripts/2026-06-01', 'default'); + expect((await factSourcesOnPage('transcripts/2026-06-01', 'default')).length).toBe(3); + + const { deleted } = await engine.deleteFactsForPage('transcripts/2026-06-01', 'default', { + excludeSourcePrefixes: ['cli:'], + }); + + expect(deleted).toBe(2); // fence + blank-source removed + const survivors = await factSourcesOnPage('transcripts/2026-06-01', 'default'); + expect(survivors).toEqual(['cli:extract-conversation-facts']); + }); + + test('default behavior (no opts) deletes every fact on the coordinate', async () => { + await seedPageWithMixedFacts('transcripts/2026-06-02', 'default'); + expect((await factSourcesOnPage('transcripts/2026-06-02', 'default')).length).toBe(3); + + const { deleted } = await engine.deleteFactsForPage('transcripts/2026-06-02', 'default'); + + expect(deleted).toBe(3); + expect((await factSourcesOnPage('transcripts/2026-06-02', 'default')).length).toBe(0); + }); +}); diff --git a/test/import-code-edges-source-id.test.ts b/test/import-code-edges-source-id.test.ts new file mode 100644 index 000000000..69c46be0f --- /dev/null +++ b/test/import-code-edges-source-id.test.ts @@ -0,0 +1,112 @@ +/** + * Regression — importCodeFile stamps source_id on extracted call-graph edges. + * + * Pre-fix bug: importCodeFile built CodeEdgeInput rows WITHOUT source_id, so + * every extracted edge landed NULL in code_edges_symbol. getCallersOf / + * getCalleesOf add `AND source_id = ` whenever a worktree pin or + * --source is in play — NULL never matches that filter, so scoped call-graph + * queries silently returned 0 rows on multi-source brains even though the + * edges existed. Pre-existing coverage (cathedral-ii-brainbench.test.ts, + * code-edges.test.ts) only ever queried with { allSources: true }, which + * bypasses the filter — exactly why the NULL never surfaced. + * + * This test imports a caller/callee pair under a non-default source and + * asserts (a) the persisted code_edges_symbol rows carry the source_id, and + * (b) the SCOPED getCallersOf/getCalleesOf — the user-visible path that + * returned 0 — now find the edge, while a different source scope does not. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { importCodeFile } from '../src/core/import-file.ts'; +import { runSources } from '../src/commands/sources.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + await runSources(engine, ['add', 'testsrc', '--no-federated']); + + // Same fixture shape as cathedral-ii-brainbench: runner() calls helper(), + // Layer 5 edge extraction captures the unresolved 'calls' edge — but here + // the import is pinned to a non-default source. + await importCodeFile( + engine, + 'src/a.ts', + 'export function runner() { return helper(); }\n', + { noEmbed: true, sourceId: 'testsrc' }, + ); + await importCodeFile( + engine, + 'src/b.ts', + 'export function helper() { return 42; }\n', + { noEmbed: true, sourceId: 'testsrc' }, + ); +}, 60_000); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}, 30_000); + +describe('importCodeFile — source_id stamped on extracted call-graph edges', () => { + test('edges land with the import source_id and scoped caller/callee queries match', async () => { + // (a) The persisted unresolved edge rows carry the source, not NULL. + const rows = await engine.executeRaw<{ source_id: string | null }>( + `SELECT source_id FROM code_edges_symbol WHERE from_symbol_qualified = $1`, + ['runner'], + ); + expect(rows.length).toBeGreaterThanOrEqual(1); + for (const r of rows) { + expect(r.source_id).toBe('testsrc'); + } + + // (b) The scoped queries — the path that silently returned 0 pre-fix. + const callers = await engine.getCallersOf('helper', { sourceId: 'testsrc' }); + const fromRunner = callers.find(r => r.from_symbol_qualified === 'runner'); + expect(fromRunner).toBeDefined(); + expect(fromRunner!.edge_type).toBe('calls'); + expect(fromRunner!.source_id).toBe('testsrc'); + + const callees = await engine.getCalleesOf('runner', { sourceId: 'testsrc' }); + expect(callees.some(r => r.to_symbol_qualified === 'helper')).toBe(true); + + // Source isolation still holds: a different scope must NOT see the edge. + const otherScope = await engine.getCallersOf('helper', { sourceId: 'default' }); + expect(otherScope.find(r => r.from_symbol_qualified === 'runner')).toBeUndefined(); + }); + + test('UNSCOPED import stamps edges with the schema-default source, not NULL', async () => { + // The other door of the same bug: an import WITHOUT opts.sourceId (legacy + // unscoped callers — `gbrain reindex --code` with no --source) lands its + // pages under the schema default (pages.source_id DEFAULT 'default'). + // If its edges were stamped NULL, the matching scoped query + // getCallersOf(sym, { sourceId: 'default' }) — a worktree pinned to + // default, --source default, GBRAIN_SOURCE=default — would miss them. + await importCodeFile( + engine, + 'src/c.ts', + 'export function unscopedRunner() { return unscopedHelper(); }\n', + { noEmbed: true }, + ); + await importCodeFile( + engine, + 'src/d.ts', + 'export function unscopedHelper() { return 7; }\n', + { noEmbed: true }, + ); + + const rows = await engine.executeRaw<{ source_id: string | null }>( + `SELECT source_id FROM code_edges_symbol WHERE from_symbol_qualified = $1`, + ['unscopedRunner'], + ); + expect(rows.length).toBeGreaterThanOrEqual(1); + for (const r of rows) { + expect(r.source_id).toBe('default'); + } + + const callers = await engine.getCallersOf('unscopedHelper', { sourceId: 'default' }); + expect(callers.some(r => r.from_symbol_qualified === 'unscopedRunner')).toBe(true); + }); +}); diff --git a/test/list-all-sources.test.ts b/test/list-all-sources.test.ts index 4d0f9ddb5..fddfc9b0a 100644 --- a/test/list-all-sources.test.ts +++ b/test/list-all-sources.test.ts @@ -135,4 +135,51 @@ describe('engine.updateSourceConfig', () => { const all = await engine.listAllSources(); expect(all.find(s => s.id === 'delta')!.config.last_full_cycle_at).toBe('2026-05-22T11:00:00.000Z'); }); + + // IS JSON guard: the postgres-engine atomic merge gates its `::jsonb` cast + // behind the SQL `IS JSON` predicate so a historical bad row whose config is + // a JSONB string of NON-JSON text normalizes to `{}` instead of raising + // `invalid input syntax for type json` and aborting the UPDATE. + // + // We exercise the SQL CASE expression directly via executeRaw against PGLite + // (which ships Postgres 17 + IS JSON parity) rather than calling + // PostgresEngine.updateSourceConfig (which requires a live Postgres pool). + test('IS-JSON guard: non-JSON string config normalizes to {} on merge', async () => { + const patch = { merged_key: 'v' }; + + const guarded = (configExpr: string) => ` + SELECT ( + CASE + WHEN jsonb_typeof(${configExpr}) = 'object' THEN ${configExpr} + WHEN jsonb_typeof(${configExpr}) = 'string' + THEN CASE + WHEN (${configExpr} #>> '{}') IS JSON + THEN COALESCE(NULLIF((${configExpr} #>> '{}'), '')::jsonb, '{}'::jsonb) + ELSE '{}'::jsonb + END + WHEN jsonb_typeof(${configExpr}) = 'array' + THEN COALESCE( + (SELECT jsonb_object_agg(kv.key, kv.value) + FROM jsonb_array_elements(${configExpr}) elem, + jsonb_each(elem) kv), + '{}'::jsonb + ) + ELSE '{}'::jsonb + END || $1::jsonb + ) AS result`; + + // 1. JSONB string holding NON-JSON text → normalizes to {} then merges patch. + const bad = await engine.executeRaw<{ result: Record }>( + guarded(`to_jsonb('garbage text'::text)`), + [JSON.stringify(patch)], + ); + expect(bad[0].result).toEqual({ merged_key: 'v' }); + + // 2. JSONB string holding double-encoded valid JSON object → parsed + merged. + const good = await engine.executeRaw<{ result: Record }>( + guarded(`to_jsonb('{"x":1}'::text)`), + [JSON.stringify(patch)], + ); + expect(good[0].result).toEqual({ x: 1, merged_key: 'v' }); + }); }); diff --git a/test/oauth-authorize-scope-default.test.ts b/test/oauth-authorize-scope-default.test.ts new file mode 100644 index 000000000..4dc2c3d81 --- /dev/null +++ b/test/oauth-authorize-scope-default.test.ts @@ -0,0 +1,89 @@ +/** + * Authorize-grant scope default (RFC 6749 §3.3). + * + * When a client omits `scope` on /authorize, the granted scope must default to + * the client's full registered scope — NOT the empty set. Regression guard for + * the bug where an omitted request granted [], which then propagated into the + * access + refresh tokens and never self-healed: every op failed + * `insufficient_scope` even though the client was registered `read write` + * (some MCP connectors omit `scope` on /authorize). The clamp must still hold — + * an explicit over-broad request cannot escalate past the client's allowed set. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts'; +import { sqlQueryForEngine } from '../src/core/sql-query.ts'; + +let engine: PGLiteEngine; +let provider: GBrainOAuthProvider; +let sql: ReturnType; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + sql = sqlQueryForEngine(engine); + provider = new GBrainOAuthProvider({ sql }); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await (engine as any).db.exec('DELETE FROM oauth_tokens'); + await (engine as any).db.exec('DELETE FROM oauth_codes'); + await (engine as any).db.exec('DELETE FROM oauth_clients'); +}); + +// authorize() writes the granted scope into oauth_codes then redirects; we +// assert on the stored grant directly, so the redirect is a no-op. +const noopRes = { redirect() {} } as any; + +async function authorizeAndReadScopes( + scope: string, + requested: string[] | undefined, +): Promise { + const reg = await provider.registerClientManual( + 'authz-test', ['authorization_code'], scope, ['https://example.test/cb'], + ); + const client = await provider.clientsStore.getClient(reg.clientId); + expect(client).toBeTruthy(); + await provider.authorize( + client!, + { + scopes: requested, + codeChallenge: 'test-challenge', + redirectUri: 'https://example.test/cb', + state: 'xyz', + } as any, + noopRes, + ); + const rows = (await sql` + SELECT scopes FROM oauth_codes WHERE client_id = ${reg.clientId} + `) as Array<{ scopes: string[] }>; + expect(rows.length).toBe(1); + return rows[0].scopes ?? []; +} + +describe('authorize() scope default — omitted scope inherits client grant', () => { + test('omitted scope → inherits full registered scope', async () => { + expect((await authorizeAndReadScopes('read write', undefined)).sort()).toEqual(['read', 'write']); + }); + + test('empty scope array → inherits full registered scope', async () => { + expect((await authorizeAndReadScopes('read write', [])).sort()).toEqual(['read', 'write']); + }); + + test('explicit subset is honored (not overridden to full)', async () => { + expect(await authorizeAndReadScopes('read write admin', ['read'])).toEqual(['read']); + }); + + test('clamp preserved: over-broad request cannot escalate', async () => { + expect(await authorizeAndReadScopes('read', ['read', 'admin'])).toEqual(['read']); + }); + + test('clamp preserved: requesting only a disallowed scope grants nothing (no inheritance)', async () => { + expect(await authorizeAndReadScopes('read write', ['admin'])).toEqual([]); + }); +}); diff --git a/test/oauth.test.ts b/test/oauth.test.ts index 257ebb2d7..3c466ec5b 100644 --- a/test/oauth.test.ts +++ b/test/oauth.test.ts @@ -12,6 +12,7 @@ import { import { hashToken, generateToken } from '../src/core/utils.ts'; import { PGLITE_SCHEMA_SQL } from '../src/core/pglite-schema.ts'; import { InvalidTokenError } from '@modelcontextprotocol/sdk/server/auth/errors.js'; +import type { AuthInfo as CoreAuthInfo } from '../src/core/operations.ts'; // --------------------------------------------------------------------------- // Test setup: in-memory PGLite with OAuth tables @@ -310,6 +311,33 @@ describe('verifyAccessToken', () => { expect(authInfo.clientId).toBe('legacy-agent'); expect(authInfo.scopes).toEqual(['read', 'write', 'admin']); // grandfathered full access }); + + test('legacy access_tokens fallback honors permissions.source_id array grants', async () => { + // oauth.test.ts initializes the static PGLite schema blob, not the full + // migration stack. Add the v38 permissions column here so the row matches + // a modern brain carrying a legacy-token source grant. + await sql` + ALTER TABLE access_tokens + ADD COLUMN IF NOT EXISTS permissions JSONB NOT NULL DEFAULT '{"takes_holders":["world"]}'::jsonb + `; + + const legacyToken = generateToken('gbrain_'); + const hash = hashToken(legacyToken); + await sql` + INSERT INTO access_tokens (id, name, token_hash, permissions) + VALUES ( + ${crypto.randomUUID()}, + ${'legacy-federated-agent'}, + ${hash}, + ${JSON.stringify({ source_id: ['default', 'src-a', 'src-b'] })}::jsonb + ) + `; + + const authInfo = await provider.verifyAccessToken(legacyToken) as CoreAuthInfo; + expect(authInfo.clientId).toBe('legacy-federated-agent'); + expect(authInfo.sourceId).toBe('default'); + expect(authInfo.allowedSources).toEqual(['default', 'src-a', 'src-b']); + }); }); // --------------------------------------------------------------------------- diff --git a/test/pglite-lock.test.ts b/test/pglite-lock.test.ts index 3253aa41b..e9b603bbf 100644 --- a/test/pglite-lock.test.ts +++ b/test/pglite-lock.test.ts @@ -3,6 +3,7 @@ import { mkdirSync, rmSync, existsSync, readFileSync, writeFileSync } from 'fs'; import { join } from 'path'; import { tmpdir } from 'os'; import { acquireLock, releaseLock, type LockHandle } from '../src/core/pglite-lock'; +import { withEnv } from './helpers/with-env.ts'; const TEST_DIR = join(tmpdir(), 'gbrain-lock-test-' + process.pid); @@ -99,3 +100,96 @@ describe('pglite-lock', () => { await releaseLock(lock2); }); }); + +describe('pglite-lock #2058 heartbeat + steal-grace', () => { + beforeEach(() => { + if (existsSync(TEST_DIR)) rmSync(TEST_DIR, { recursive: true, force: true }); + mkdirSync(TEST_DIR, { recursive: true }); + }); + afterEach(() => { + if (existsSync(TEST_DIR)) rmSync(TEST_DIR, { recursive: true, force: true }); + }); + + function writeHolder(fields: { pid: number; acquiredAgoMs: number; refreshedAgoMs: number }) { + const lockDir = join(TEST_DIR, '.gbrain-lock'); + mkdirSync(lockDir, { recursive: true }); + const now = Date.now(); + writeFileSync(join(lockDir, 'lock'), JSON.stringify({ + pid: fields.pid, + acquired_at: now - fields.acquiredAgoMs, + refreshed_at: now - fields.refreshedAgoMs, + command: 'test holder', + })); + } + + test('[REGRESSION] a LIVE holder with a fresh heartbeat is NOT stolen even when the lock is old', async () => { + // The WAL-corruption bug: a >5min embed used to get its lock force-removed. + // Now an alive holder that heartbeated recently is left alone regardless of + // age. acquired 20min ago, but refreshed just now → must wait, not steal. + writeHolder({ pid: process.pid, acquiredAgoMs: 20 * 60_000, refreshedAgoMs: 0 }); + + await expect(acquireLock(TEST_DIR, { timeoutMs: 1200 })).rejects.toThrow(/Timed out/); + // Holder's lock still present (was never stolen). + expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(true); + }); + + test('a LIVE PID whose heartbeat went stale past the grace window IS reaped', async () => { + // PID is alive (our own) but hasn't refreshed in 20min (> 600s grace): + // hung holder, or a reused PID whose real holder is gone. Reap + acquire. + writeHolder({ pid: process.pid, acquiredAgoMs: 25 * 60_000, refreshedAgoMs: 20 * 60_000 }); + + const lock = await acquireLock(TEST_DIR, { timeoutMs: 2000 }); + expect(lock.acquired).toBe(true); + await releaseLock(lock); + }); + + test('GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS tunes the grace window', async () => { + // withEnv keeps the process-global mutation isolated across shard files. + await withEnv({ GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS: '5' }, async () => { + // Refreshed 30s ago — fresh under the 600s default, STALE under 5s. + writeHolder({ pid: process.pid, acquiredAgoMs: 60_000, refreshedAgoMs: 30_000 }); + const lock = await acquireLock(TEST_DIR, { timeoutMs: 2000 }); + expect(lock.acquired).toBe(true); + await releaseLock(lock); + }); + }); + + test('[REGRESSION] releaseLock does NOT remove a lock that was stolen + re-acquired by another process', async () => { + // We acquire, then simulate a steal: another process reaped us past grace + // and now owns the lock (different pid + acquired_at). Our releaseLock must + // NOT delete their live lock — doing so would let a third process in + // alongside the new owner (the #2058 corruption class). + const lock: LockHandle = await acquireLock(TEST_DIR); + expect(lock.acquired).toBe(true); + expect(lock.ownerToken).toBeDefined(); + if (lock.heartbeat) clearInterval(lock.heartbeat); // stop our heartbeat for a deterministic test + + // Overwrite the lock file as if process B re-acquired it. + const lockFile = join(TEST_DIR, '.gbrain-lock', 'lock'); + const bNow = Date.now() + 1; + writeFileSync(lockFile, JSON.stringify({ pid: 999999, acquired_at: bNow, refreshed_at: bNow, command: 'process B' })); + + await releaseLock(lock); // our (stale) handle + + // B's lock survives — we did not clobber it. + expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(true); + const after = JSON.parse(readFileSync(lockFile, 'utf-8')); + expect(after.pid).toBe(999999); + + // Cleanup for afterEach. + rmSync(join(TEST_DIR, '.gbrain-lock'), { recursive: true, force: true }); + }); + + test('acquire starts a heartbeat and seeds refreshed_at; release clears it', async () => { + const lock: LockHandle = await acquireLock(TEST_DIR); + expect(lock.acquired).toBe(true); + expect(lock.heartbeat).toBeDefined(); + const data = JSON.parse(readFileSync(join(TEST_DIR, '.gbrain-lock', 'lock'), 'utf-8')); + expect(data.refreshed_at).toBeDefined(); + expect(typeof data.refreshed_at).toBe('number'); + + await releaseLock(lock); + expect(lock.heartbeat).toBeUndefined(); + expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(false); + }); +}); diff --git a/test/pglite-reconnect.serial.test.ts b/test/pglite-reconnect.serial.test.ts new file mode 100644 index 000000000..15eb2fa6d --- /dev/null +++ b/test/pglite-reconnect.serial.test.ts @@ -0,0 +1,45 @@ +/** + * #2034 — engine-parity reconnect(). + * + * The autopilot health-probe recovery path used `disconnect()` + bare + * `connect()`, which (a) threw `database_url undefined` forever on Postgres + * because the config was lost, and (b) was a silent no-op on PGLite which had + * no reconnect() at all. Both engines now expose `reconnect()`; this pins the + * PGLite side: it restores connectivity against the config captured at the + * last connect(), and is a safe no-op when never connected. + */ + +import { describe, test, expect } from 'bun:test'; +import { mkdtempSync, rmSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; + +describe('#2034 PGLiteEngine.reconnect', () => { + test('restores connectivity and persisted state against the saved config', async () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-reconnect-')); + const engine = new PGLiteEngine(); + try { + await engine.connect({ database_path: dir }); + await engine.initSchema(); + await engine.setConfig('reconnect.probe', 'pre'); + + // Reconnect with no args — the #2034 contract: it must reuse the config + // captured at connect(), not require it to be re-passed. + await engine.reconnect(); + + // Still queryable, and persisted state survived (same data dir). + const rows = await engine.executeRaw<{ x: number }>('SELECT 1 AS x'); + expect(rows[0]?.x).toBe(1); + expect(await engine.getConfig('reconnect.probe')).toBe('pre'); + } finally { + await engine.disconnect(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('reconnect() before any connect() is a safe no-op', async () => { + const engine = new PGLiteEngine(); + await expect(engine.reconnect()).resolves.toBeUndefined(); + }); +}); diff --git a/test/search-alias-resolved-boost.test.ts b/test/search-alias-resolved-boost.test.ts index e67bb0300..57fe7ace0 100644 --- a/test/search-alias-resolved-boost.test.ts +++ b/test/search-alias-resolved-boost.test.ts @@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => { }); describe('KNOBS_HASH_VERSION', () => { - it('is 10 (9→10 relational recall arm invalidates rel-off cache rows, v0.43)', () => { - expect(KNOBS_HASH_VERSION).toBe(10); + it('is 11 (10→11 asymmetric input_type fix invalidates document-side query-vector rows, #1400)', () => { + expect(KNOBS_HASH_VERSION).toBe(11); }); }); diff --git a/test/search-mode.test.ts b/test/search-mode.test.ts index c2a3516e1..a8e9ecf18 100644 --- a/test/search-mode.test.ts +++ b/test/search-mode.test.ts @@ -403,7 +403,11 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => { // must not be served from a cache row written before the policy change. // v0.43: bumped 9→10 for the relational recall arm (rel=/reld=) — a // relational-on write must not be served to a relational-off lookup. - expect(KNOBS_HASH_VERSION).toBe(10); + // #1400: bumped 10→11 for the asymmetric input_type fix — embedQuery() + // now produces query-side vectors for asymmetric providers (zembed-1, + // Voyage v3+), so rows keyed on pre-fix document-side query vectors + // must not be served to post-fix lookups. + expect(KNOBS_HASH_VERSION).toBe(11); }); test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => { @@ -568,8 +572,8 @@ describe('v0.40.4 — graph_signals knob', () => { }); describe('v0.42.3.0 — autocut knobs', () => { - test('KNOBS_HASH_VERSION is 10 (9→10 relational recall arm, v0.43)', () => { - expect(KNOBS_HASH_VERSION).toBe(10); + test('KNOBS_HASH_VERSION is 11 (10→11 asymmetric input_type fix, #1400)', () => { + expect(KNOBS_HASH_VERSION).toBe(11); }); test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => { diff --git a/test/search/knobs-hash-reranker.test.ts b/test/search/knobs-hash-reranker.test.ts index 509e98249..bd793d427 100644 --- a/test/search/knobs-hash-reranker.test.ts +++ b/test/search/knobs-hash-reranker.test.ts @@ -43,7 +43,7 @@ function baseKnobs(): ResolvedSearchKnobs { } describe('KNOBS_HASH_VERSION + version invariants', () => { - test('version is 10 (…; 7→8 autocut; 8→9 archive-demote #1777; 9→10 relational recall)', () => { + test('version is 11 (…; 8→9 archive-demote #1777; 9→10 relational recall; 10→11 asymmetric input_type #1400)', () => { // v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold // floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs // (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation). @@ -58,7 +58,10 @@ describe('KNOBS_HASH_VERSION + version invariants', () => { // autocut. issue #1777: 8→9 archive/ demote (search-exclude policy change // isn't in the hash, so the bump invalidates archive-excluded cache rows). // v0.43: 9→10 relational recall arm (rel=/reld=). - expect(KNOBS_HASH_VERSION).toBe(10); + // #1400: 10→11 asymmetric input_type fix — embedQuery() now produces + // query-side vectors for asymmetric providers, so rows keyed on + // pre-fix document-side query vectors must not be served. + expect(KNOBS_HASH_VERSION).toBe(11); }); test('hash is 16 hex chars regardless of reranker config', () => { diff --git a/test/timeline-date-batch.test.ts b/test/timeline-date-batch.test.ts new file mode 100644 index 000000000..eba9e009b --- /dev/null +++ b/test/timeline-date-batch.test.ts @@ -0,0 +1,76 @@ +/** + * #2057 regression — addTimelineEntriesBatch must accept JS Date `date` values. + * + * The original bug: callers source `date` from SQL rows (e.g. + * `meeting.effective_date`), which arrive as JS Date objects. The OLD insert + * bound them into `${dates}::text[]`, which threw `cannot cast type timestamp + * with time zone to text[]`, and a bare `catch {}` in the meetings extractor + * swallowed it — timeline stayed empty forever. + * + * The #1861 refactor moved the insert to `jsonb_to_recordset` + `v.date::date`, + * where a Date serializes to an ISO string that casts cleanly. This test pins + * that a Date-bearing batch round-trips, so a future refactor can't silently + * reintroduce the cast failure. (The companion fix un-silences the extractor's + * catch so any future failure is visible rather than a phantom "0 entries".) + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { importFromContent } from '../src/core/import-file.ts'; +import type { TimelineBatchInput } from '../src/core/engine.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + await importFromContent(engine, 'people/alice-example', `---\ntitle: Alice\ntype: note\n---\n\n# Alice\n`, { + noEmbed: true, + sourceId: 'default', + sourcePath: 'people/alice-example.md', + }); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +describe('#2057 addTimelineEntriesBatch with Date date values', () => { + test('a Date `date` is inserted and round-trips as the right calendar day', async () => { + // Mimic the real caller: `date` typed string on the interface, but a JS + // Date at runtime (straight off a TIMESTAMPTZ column). The cast must hold. + const effectiveDate = new Date('2026-04-03T00:00:00.000Z'); + const batch: TimelineBatchInput[] = [ + { + slug: 'people/alice-example', + date: effectiveDate as unknown as string, + source: 'cli:test', + summary: 'met alice', + source_id: 'default', + }, + ]; + + const inserted = await engine.addTimelineEntriesBatch(batch); + expect(inserted).toBe(1); + + const rows = await engine.executeRaw<{ date: string }>( + `SELECT date::text AS date FROM timeline_entries WHERE summary = 'met alice'`, + ); + expect(rows.length).toBe(1); + expect(rows[0].date).toBe('2026-04-03'); + }); + + test('a plain ISO string `date` still works (no regression)', async () => { + const inserted = await engine.addTimelineEntriesBatch([ + { + slug: 'people/alice-example', + date: '2026-05-01', + source: 'cli:test', + summary: 'string-dated entry', + source_id: 'default', + }, + ]); + expect(inserted).toBe(1); + }); +}); diff --git a/test/timeline-dedup-repair.test.ts b/test/timeline-dedup-repair.test.ts new file mode 100644 index 000000000..bfacab68a --- /dev/null +++ b/test/timeline-dedup-repair.test.ts @@ -0,0 +1,119 @@ +/** + * #2038 — idx_timeline_dedup schema-drift self-heal. + * + * A brain that ran the pre-renumber v99 variant of the dedup migration is + * stamped past v102 with the OLD 3-column index. `runMigrations` early-returns + * (nothing pending) so a migration verify-hook can't fix it. The repair is + * keyed off the index SHAPE and runs regardless. These tests simulate the + * drifted states directly and pin: detection, rebuild, dedupe-before-rebuild + * (only possible when the index was absent), and idempotency. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { + checkTimelineDedupIndex, + repairTimelineDedupIndex, +} from '../src/core/timeline-dedup-repair.ts'; +import { importFromContent } from '../src/core/import-file.ts'; + +let engine: PGLiteEngine; +let pageId: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + await importFromContent(engine, 'people/alice-example', `---\ntitle: Alice\ntype: note\n---\n\n# Alice\n`, { + noEmbed: true, + sourceId: 'default', + sourcePath: 'people/alice-example.md', + }); + const pid = await engine.executeRaw<{ id: string }>( + `SELECT id::text AS id FROM pages WHERE slug = 'people/alice-example' AND source_id = 'default'`, + ); + pageId = pid[0].id; +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +/** Force the index back to the broken pre-v102 3-column shape. */ +async function regressTo3Col() { + await engine.executeRaw(`DELETE FROM timeline_entries`); + await engine.executeRaw(`DROP INDEX IF EXISTS idx_timeline_dedup`); + await engine.executeRaw( + `CREATE UNIQUE INDEX idx_timeline_dedup ON timeline_entries(page_id, date, summary)`, + ); +} + +/** The other drift shape: the index was dropped entirely, letting true + * 4-tuple duplicates accumulate that would block a naive CREATE UNIQUE INDEX. */ +async function regressToAbsentWithDupes() { + await engine.executeRaw(`DELETE FROM timeline_entries`); + await engine.executeRaw(`DROP INDEX IF EXISTS idx_timeline_dedup`); + await engine.executeRaw( + `INSERT INTO timeline_entries (page_id, date, summary, source, detail) + VALUES ($1, '2026-04-03', 'met alice', 'meeting', ''), + ($1, '2026-04-03', 'met alice', 'meeting', ''), + ($1, '2026-04-03', 'met alice', 'cli:extract', '')`, + [pageId], + ); +} + +describe('#2038 idx_timeline_dedup drift repair', () => { + test('detects the 3-column drift', async () => { + await regressTo3Col(); + const status = await checkTimelineDedupIndex(engine); + expect(status.tablePresent).toBe(true); + expect(status.indexPresent).toBe(true); + expect(status.columns).toEqual(['page_id', 'date', 'summary']); + expect(status.needsRepair).toBe(true); + }); + + test('rebuilds the 3-column index to 4 columns (no dupes to collapse)', async () => { + await regressTo3Col(); + await engine.executeRaw( + `INSERT INTO timeline_entries (page_id, date, summary, source, detail) + VALUES ($1, '2026-04-03', 'met alice', 'meeting', '')`, + [pageId], + ); + + const res = await repairTimelineDedupIndex(engine); + expect(res.repaired).toBe(true); + expect(res.reason).toBe('rebuilt'); + expect(res.collapsedDuplicates).toBe(0); + + const after = await checkTimelineDedupIndex(engine); + expect(after.columns).toEqual(['page_id', 'date', 'summary', 'source']); + expect(after.needsRepair).toBe(false); + }); + + test('dedupes true 4-tuple duplicates before building the unique index', async () => { + await regressToAbsentWithDupes(); // index absent + a real (meeting) dup + + const before = await checkTimelineDedupIndex(engine); + expect(before.indexPresent).toBe(false); + expect(before.needsRepair).toBe(true); + + const res = await repairTimelineDedupIndex(engine); + expect(res.repaired).toBe(true); + expect(res.collapsedDuplicates).toBe(1); // one of the two 'meeting' rows removed + + const after = await checkTimelineDedupIndex(engine); + expect(after.columns).toEqual(['page_id', 'date', 'summary', 'source']); + const rows = await engine.executeRaw<{ n: string }>( + `SELECT COUNT(*)::text AS n FROM timeline_entries`, + ); + expect(parseInt(rows[0].n, 10)).toBe(2); // meeting (deduped) + cli:extract + }); + + test('idempotent — a second repair is a no-op', async () => { + await regressTo3Col(); + await repairTimelineDedupIndex(engine); + const second = await repairTimelineDedupIndex(engine); + expect(second.repaired).toBe(false); + expect(second.reason).toBe('already_correct'); + }); +}); diff --git a/test/write-through.test.ts b/test/write-through.test.ts index 67ed55117..238082a60 100644 --- a/test/write-through.test.ts +++ b/test/write-through.test.ts @@ -120,6 +120,58 @@ describe('writePageThrough', () => { expect(res).toEqual({ written: false, skipped: 'page_not_found_after_write' }); }); + test('[REGRESSION #2018] default page (null local_path) in a multi-source brain → skipped, no leak into a sibling source repo', async () => { + // A sibling federated source with its OWN working tree. + const siblingDir = path.join(tmpRoot, 'housefax'); + fs.mkdirSync(siblingDir, { recursive: true }); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, config) VALUES ('housefax', 'Housefax', $1, '{}'::jsonb)`, + [siblingDir], + ); + // The leak trigger: global sync.repo_path points at the sibling's tree + // while the default source (re-seeded by resetPgliteState) has no + // local_path of its own. + await engine.setConfig('sync.repo_path', siblingDir); + + const slug = 'internal/cross-cutting-note'; + await seedPage(slug); // sourceId 'default' + + const res = await writePageThrough(engine, slug, { sourceId: 'default' }); + + expect(res).toEqual({ written: false, skipped: 'source_repo_belongs_to_other_source' }); + // The sibling source's repo stays clean — the whole point of #2018. + expect(walkFiles(siblingDir).some((f) => f.endsWith('.md'))).toBe(false); + }); + + test('[#2018] page assigned to a source with its own local_path writes to that tree root, not the global path', async () => { + const alphaDir = path.join(tmpRoot, 'alpha-repo'); + fs.mkdirSync(alphaDir, { recursive: true }); + const globalDir = path.join(tmpRoot, 'global-repo'); + fs.mkdirSync(globalDir, { recursive: true }); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, config) VALUES ('alpha', 'Alpha', $1, '{}'::jsonb)`, + [alphaDir], + ); + // Must NOT be used — the assigned source has its own tree. + await engine.setConfig('sync.repo_path', globalDir); + + const slug = 'notes/alpha-thing'; + await importFromContent(engine, slug, `---\ntitle: T\ntype: note\n---\n\n# Body\n`, { + noEmbed: true, + sourceId: 'alpha', + sourcePath: `${slug}.md`, + }); + + const res = await writePageThrough(engine, slug, { sourceId: 'alpha' }); + + expect(res.written).toBe(true); + // File at the source's tree ROOT, never nested under `.sources//`. + expect(res.path).toBe(path.join(alphaDir, `${slug}.md`)); + expect(fs.existsSync(path.join(alphaDir, `${slug}.md`))).toBe(true); + // The global repo path is untouched. + expect(walkFiles(globalDir).some((f) => f.endsWith('.md'))).toBe(false); + }); + test('[REGRESSION] mkdir ENOTDIR (parent is a file) → error, no partial .md, no .tmp', async () => { await engine.setConfig('sync.repo_path', brainDir); // Block the `wiki/` directory by putting a FILE named "wiki" under the repo,