diff --git a/CHANGELOG.md b/CHANGELOG.md index 6aeaf6617..a620e693d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,39 @@ All notable changes to GBrain will be documented in this file. +## [0.42.41.0] - 2026-06-11 + +**A correctness-and-reliability wave: your conversation facts survive a cycle, write-through stops polluting other repos, autopilot rides out a DB blip instead of crash-looping, and concurrent PGLite processes stop corrupting each other.** A triage of open reports surfaced six bugs with no fix yet plus a batch of community PRs; this ships them together, each with a regression test. + +The headline is data durability. The `extract_facts` cycle phase reconciles a page's facts from its `## Facts` fence by deleting then reinserting — but conversation facts (written by `extract-conversation-facts`) live on pages that have no fence, so a cycle could delete them and reinsert nothing. A failed sync made it worse by escalating the phase to a full-brain walk. Both are fixed: the reconcile now protects non-fence (`cli:`-origin) facts, the destructive phase no longer inherits the failed-sync full-walk, and a reconcile that removes far more than it adds now reports `warn` instead of a silent `ok`. + +Three more silent failures, each found in production and now self-healing or loud: +- **Timeline writes that silently stopped.** A migration renumbered during a merge could be recorded as applied without its index change ever running, so every timeline insert failed its `ON CONFLICT`. `gbrain` now repairs the index shape on every migrate pass (dedupe-then-rebuild, even when nothing is pending), `gbrain doctor` reports the drift, and the meetings extractor no longer swallows batch errors. +- **Autopilot crash-loop on transient DB errors.** The health-probe recovery called `connect()` without its config, so every reconnect threw and the process exited on any blip. It now uses `reconnect()`, which restores the captured config; `reconnect()` is a first-class method on both engines. +- **WAL corruption from concurrent PGLite processes.** A live, working process holding the data-dir lock could have it stolen after five minutes. The lock now heartbeats while held and is only reclaimed from a dead or genuinely stalled holder. + +### Added +- **`timeline_dedup_index` doctor check + always-run repair (#2038).** Detects and heals a stale `idx_timeline_dedup` shape; `gbrain apply-migrations --force-schema` triggers the repair on demand. Reported by @jbarol. +- **`BrainEngine.reconnect()` on both engines (#2034).** Config-restoring reconnect, replacing the `disconnect()`+bare-`connect()` pattern. + +### Fixed +- **Conversation facts survive a fence reconcile (#1928).** The cycle's per-page wipe now excludes non-fence (`cli:`) facts, the destructive phase no longer full-walks on a failed sync, and net-negative reconciles surface as `warn`. +- **`put_page` write-through no longer leaks into an unrelated source's repo (#2018).** It mirrors to the assigned source's own `local_path`; a source without one is skipped rather than written into the global repo path. +- **Autopilot survives transient DB errors instead of crash-looping (#2034).** +- **Concurrent PGLite processes no longer corrupt the WAL (#2058).** Heartbeat + steal-grace replaces the age-only stale-lock check. +- **Timeline migration drift self-heals; the extractor stops swallowing errors (#2038, #2057).** A Date-typed batch date round-trips correctly (verified by test). +- **A cwd `.env` `DATABASE_URL` no longer silently retargets the brain (#2064, closes #427).** Reported by @bomliu. +- **`gbrain sync --strategy code` honors `.gitignore` and skips `vendor`/`dist`/`build`/`venv` (#2052, #2020).** Reported by @aphaiboon and @dMac716. +- **Asymmetric embedding `input_type` reaches the wire across every openai-compatible recipe (#2033, supersedes #1400).** Query vectors are no longer document-typed. By @pabloglzg; original diagnosis by @billy-armstrong. +- **`updateSourceConfig` JSONB merge is atomic (#2074).** Eliminates a concurrent-writer lost-update race. Reported by @pai-scaffolde. +- **`gbrain doctor` correctness pass (#2075):** stale-lock hints, content sanity, graph coverage, exit code, gateway guard. Reported by @pai-scaffolde. +- **`gbrain search` returns results instead of exiting 0 empty on slow poolers; scoped `code-callers`/`code-callees` find their edges (#2073).** Migration v116 backfills NULL edge `source_id` and indexes `from_symbol_qualified`. Reported by @jbarol. +- **OAuth scope handling (#2009, #2072):** an omitted authorize scope now defaults to the client's registered grant (clamped to it, so no widening) instead of an empty grant that never self-heals; legacy token source grants are honored through a single shared scope parser. By @austinrarnett and @maxpetrusenkoagent. + +### To take advantage of v0.42.41.0 + +`gbrain upgrade`. The timeline-index repair and migration v116 run automatically on the next migrate pass — if `gbrain doctor` flagged a timeline or call-graph problem before, re-run it after upgrade. No config changes required; the facts, write-through, autopilot, and lock fixes apply on restart. + ## [0.42.40.0] - 2026-06-09 **`gbrain extract --stale` no longer aborts partway through a brain that contains emoji or other non-BMP characters.** On a large brain, link/timeline extraction could die with `invalid input syntax for type json` and commit nothing — and because the staleness bookmark only advances on a clean finish, every retry re-hit the same point and extraction stayed wedged. The cause: the link-context excerpt was sliced by raw UTF-16 index, so a window boundary landing inside an emoji's surrogate pair left an unpaired surrogate half in the text, which Postgres rejects when the batch is serialized to JSONB — taking down the whole batch, not just the one row. (PGLite is more permissive here, so this primarily bit the managed-Postgres engine.) diff --git a/TODOS.md b/TODOS.md index ba96fc81a..cf8baa784 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,31 @@ # TODOS +## gbrain triage wave follow-ups (filed v0.42.41.0) + +Deferred from the v0.42.41.0 fix wave (eng-reviewed as separate scope, not hotfixes). +See plan + GSTACK REVIEW REPORT at +`~/.claude/plans/system-instruction-you-are-working-zany-thacker.md`. + +- [ ] **P1 — supervisor: retry-with-backoff instead of hard stop on transient DB outages (#1994).** + `max_crashes_exceeded` gives up permanently; a transient pooler blip that trips the + counter wedges the supervisor until manual restart. **Why:** the #2034 reconnect fix + makes the engine recover, but the supervisor still hard-stops. **Where:** + `src/core/minions/supervisor.ts` crash-count loop — add exponential backoff with a + much higher (or no) permanent-give-up threshold for recoverable errors. +- [ ] **P2 — PGLite `reindex-frontmatter` / backfill statement_timeout boost (#1963).** + Community RCA: `SET LOCAL statement_timeout` is gated on `engine.kind === 'postgres'`, + so PGLite inherits the 30s session default and trips on non-trivial batches; the CLI + then swallows the error and exits 0. **Where:** `src/core/backfill-effective-date.ts` + (boost on PGLite too, or per-row updates) + the cli.ts catch that hides it. +- [ ] **P2 — autopilot drain-worker concurrency self-deadlock (#2050).** Drain-worker + runs at concurrency=1, so any cycle phase that spawns a subagent (patterns, synthesize) + deadlocks waiting on a worker slot it can't get. **Where:** autopilot drain-worker + dispatch — raise concurrency or exempt subagent-spawning phases. +- [ ] **P3 — name-keyed migration ledger (#2038 structural follow-up).** The always-run + index drift probe heals the one known case; the general fix is keying applied-migration + tracking by stable name rather than version integer so a renumber can't strand a + migration as recorded-but-not-executed. **Where:** `src/core/migrate.ts` ledger. + ## gbrain#1981 Retrieval Reflex follow-ups (v0.43+) Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extractor @@ -1800,11 +1826,6 @@ Three items deferred: self-heals via stale-reclaim). The common sync SUCCESS path already drains via handleCliOnly's finally. Convert for graceful drain on sync error exits. -- [ ] **(v0.42.20.0 follow-up) Decouple the op-dispatch force-exit timer** so it - wraps `engine.disconnect()` only (it's armed before the handler today, doubling - as a blanket handler watchdog) and fix its misleading "engine.disconnect() did - not return…" message that fires even when the handler (not disconnect) was slow. - - [ ] **(v0.42.20.0 follow-up) Gateway idle-timeout (vs absolute) for streaming chat.** `withDefaultTimeout` uses an absolute `AbortSignal.timeout`; a streaming generation actively producing tokens past the chat default (300s) would abort. @@ -3574,6 +3595,18 @@ keeping both skills' triggers intact for chaining. ## Completed +### ~~(v0.42.20.0 follow-up) Decouple the op-dispatch force-exit timer~~ +**Completed:** v0.42.39.0 (2026-06-10) + +The timer now arms at teardown entry (inside the op-dispatch finally, before +drain + disconnect) so it bounds ONLY disconnect — no longer doubling as a +blanket handler watchdog that killed slow-but-healthy ops at 10s with exit 0 +and empty stdout. Its "engine.disconnect() did not return…" message is now +accurate by construction (it can only fire during teardown). Read-scope +handlers + context build got their own explicit wallclock bound (180s default, +`--timeout=Ns`, exit 124, hard-exit after teardown) in the same wave. Pinned by +`test/cli-force-exit-teardown-arming.test.ts`. + ### ~~Checks 5 + 6 for check-resolvable~~ **Completed:** v0.19.0 (2026-04-22) diff --git a/VERSION b/VERSION index cc968e114..06c56e14a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.40.0 \ No newline at end of file +0.42.41.0 diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 198b2759a..3491e3d5c 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -9,7 +9,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this (`scripts/check-key-files-current-state.sh`). - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Exports upload validators `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` is a REQUIRED field flagging untrusted callers; `OperationContext.allowedSlugPrefixes` is the trusted-workspace allow-list set by the dream cycle; `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link skipped only when `remote=true && !trustedWorkspace`. Every `Operation` carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`; `sync_brain`, `file_upload`, `file_list`, `file_url` are `admin + localOnly` (rejected over HTTP). Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) use FAIL-CLOSED semantics: `ctx.remote === false` for trusted-only sites, `ctx.remote !== false` for "untrust unless explicit-false" — anything not strictly `false` is treated as remote (closes the HTTP MCP shell-job RCE where a read+write OAuth token could submit `shell` jobs). `sourceScopeOpts(ctx)` encodes the source-scoped read precedence ladder — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId`/`ctx.auth.sourceId`) over nothing; every read-side op handler routes through it so a source-bound OAuth client can't see neighboring sources via `search`/`query`/`list_pages`/`get_page`/`find_experts`/query's image path. `put_page`'s inline disk write-through is the shared `writePageThrough` helper (`src/core/write-through.ts`), ATOMIC via temp-sibling + rename so a crash or concurrent `gbrain sync` can't read a half-written `.md`; same helper backs `gbrain brainstorm/lsd --save`. Link provenance surface (#1941): `add_link` (`gbrain link`/`link-add`) + `remove_link` (`gbrain unlink`/`link-rm`) expose `link_source`/`link_type`; `add_link` rejects the reconciliation-managed built-ins via `MANAGED_LINK_SOURCES` (`markdown`/`frontmatter`/`mentions`/`wikilink-resolved`) and defaults omitted provenance to `'manual'` (the engine's own default stays `'markdown'` for internal callers); `list_link_sources` (`gbrain link-sources`, read) lists provenances via `sourceScopeOpts`. CLI aliases register through `cliHints.aliases` (collision-guarded in `src/cli.ts`). -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput`/`TimelineBatchInput` for the bulk-insert API (`addLinksBatch`/`addTimelineEntriesBatch`). `readonly kind: 'postgres' | 'pglite'` discriminator lets `src/core/migrate.ts` and others branch without `instanceof` + dynamic imports. Methods: `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[],$2::text[],$3::real[])` composite-keyed on `(slug, source_id)`), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` has `sort?: 'updated_desc'|'updated_asc'|'created_desc'|'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines. `listAllPageRefs(): Promise>` ordered by `(source_id, slug)` — cheap cross-source enumeration replacing the `getAllSlugs()→getPage(slug)` N+1 (which silently defaulted to `source_id='default'`); parity across postgres-engine.ts + pglite-engine.ts; Pinned by `test/e2e/multi-source-bug-class.test.ts`. `SearchOpts`+`PageFilters` add `sourceIds?: string[]` (federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when set, preserve scalar `sourceId` fast path when unset); `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId`/`opts.sourceIds`. `traverseGraph` opts has `frontierCap?: number` (per-iteration recursive-CTE cap, approx per-BFS-layer); return type `Promise` for MCP wire stability; export `TraverseGraphOpts`; Postgres uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, PGLite mirrors with positional params; Pinned by `test/regressions/v0_36_frontier_cap.test.ts`. Phantom-redirect methods: `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (content_hash refresh required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug`+`source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence; parity at `test/phantom-redirect-engine-parity.test.ts`. `getAdjacencyBoosts(pageIds): Promise>` powers the per-query graph-signals stage — one SQL query returning inbound-link counts among top-K plus a cross-source count (links from differing `source_id`); `COALESCE(p.source_id,'default')` null safety, `HAVING >= 1`, cross-source CASE-WHEN excludes the target's own source; parity SQL across both engines; `SearchResult` gains optional `base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta` + internal staging fields; Pinned by `test/e2e/graph-signals-engine.test.ts`. Two REQUIRED methods: `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup); `sourceId` REQUIRED on both at the type level (asymmetric with single-row `deletePage` which keeps optional/'default'); both short-circuit on empty input and throw when `> DELETE_BATCH_SIZE`. Embedding-signature stale-detection quartet: `countStaleChunks(opts?)` gains optional `signature?: string` widening the stale predicate from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (NULL signature is GRANDFATHERED, never counted; omit `signature` for the legacy NULL-only count); `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` = `SUM(LENGTH(chunk_text))` over stale chunks (same predicate + embed_skip filter + optional sourceId scope), used by `gbrain sync --all` cost preview via `estimateCostFromChars`; `setPageEmbeddingSignature(slug, {sourceId?, signature})` stamps `pages.embedding_signature` after a page's chunks (re)embed, idempotent no-op when page absent; `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` NULLs `embedding`+`embedded_at` on every chunk whose page signature is set AND differs, returning the count, called BEFORE `listStaleChunks` so signature-drift pages flow through the NULL-embedding keyset cursor unchanged (NULL never invalidated). Widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. Free-text alias layer: `resolveAliases(aliasNorms, opts?): Promise>>` (READ; maps each normalized alias to declaring `(slug, source_id)` pairs, source-scoped) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE; replaces the full alias set, delete-then-insert, empty clears, idempotent on the unique triple), called by the `importFromContent` ingest projection and the `reindex --aliases` backfill; parity across both engines, Pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines injects the shared `buildBestPerPagePoolCte` per-page max-pool so a page surfaces on its strongest chunk. `executeRawDirect(sql, params?, opts?)` is the lock-hot-path sibling of `executeRaw`: same single-statement contract, but routes to the direct session-mode pool when dual-pool is active (Postgres/Supabase port 5432) so a long-held lock heartbeat survives the transaction pooler's per-transaction connection recycling; PGLite delegates straight to `executeRaw` (no pooler). Both engines implement it; the Minion lock path (`claim`/`renewLock`) is the consumer. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput`/`TimelineBatchInput` for the bulk-insert API (`addLinksBatch`/`addTimelineEntriesBatch`). `readonly kind: 'postgres' | 'pglite'` discriminator lets `src/core/migrate.ts` and others branch without `instanceof` + dynamic imports. Methods: `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[],$2::text[],$3::real[])` composite-keyed on `(slug, source_id)`), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` has `sort?: 'updated_desc'|'updated_asc'|'created_desc'|'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines. `listAllPageRefs(): Promise>` ordered by `(source_id, slug)` — cheap cross-source enumeration replacing the `getAllSlugs()→getPage(slug)` N+1 (which silently defaulted to `source_id='default'`); parity across postgres-engine.ts + pglite-engine.ts; Pinned by `test/e2e/multi-source-bug-class.test.ts`. `SearchOpts`+`PageFilters` add `sourceIds?: string[]` (federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when set, preserve scalar `sourceId` fast path when unset); `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId`/`opts.sourceIds`. `traverseGraph` opts has `frontierCap?: number` (per-iteration recursive-CTE cap, approx per-BFS-layer); return type `Promise` for MCP wire stability; export `TraverseGraphOpts`; Postgres uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, PGLite mirrors with positional params; Pinned by `test/regressions/v0_36_frontier_cap.test.ts`. Phantom-redirect methods: `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (content_hash refresh required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug`+`source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence; parity at `test/phantom-redirect-engine-parity.test.ts`. `getAdjacencyBoosts(pageIds): Promise>` powers the per-query graph-signals stage — one SQL query returning inbound-link counts among top-K plus a cross-source count (links from differing `source_id`); `COALESCE(p.source_id,'default')` null safety, `HAVING >= 1`, cross-source CASE-WHEN excludes the target's own source; parity SQL across both engines; `SearchResult` gains optional `base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta` + internal staging fields; Pinned by `test/e2e/graph-signals-engine.test.ts`. Two REQUIRED methods: `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup); `sourceId` REQUIRED on both at the type level (asymmetric with single-row `deletePage` which keeps optional/'default'); both short-circuit on empty input and throw when `> DELETE_BATCH_SIZE`. Embedding-signature stale-detection quartet: `countStaleChunks(opts?)` gains optional `signature?: string` widening the stale predicate from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (NULL signature is GRANDFATHERED, never counted; omit `signature` for the legacy NULL-only count); `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` = `SUM(LENGTH(chunk_text))` over stale chunks (same predicate + embed_skip filter + optional sourceId scope), used by `gbrain sync --all` cost preview via `estimateCostFromChars`; `setPageEmbeddingSignature(slug, {sourceId?, signature})` stamps `pages.embedding_signature` after a page's chunks (re)embed, idempotent no-op when page absent; `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` NULLs `embedding`+`embedded_at` on every chunk whose page signature is set AND differs, returning the count, called BEFORE `listStaleChunks` so signature-drift pages flow through the NULL-embedding keyset cursor unchanged (NULL never invalidated). Widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. Free-text alias layer: `resolveAliases(aliasNorms, opts?): Promise>>` (READ; maps each normalized alias to declaring `(slug, source_id)` pairs, source-scoped) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE; replaces the full alias set, delete-then-insert, empty clears, idempotent on the unique triple), called by the `importFromContent` ingest projection and the `reindex --aliases` backfill; parity across both engines, Pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines injects the shared `buildBestPerPagePoolCte` per-page max-pool so a page surfaces on its strongest chunk. `executeRawDirect(sql, params?, opts?)` is the lock-hot-path sibling of `executeRaw`: same single-statement contract, but routes to the direct session-mode pool when dual-pool is active (Postgres/Supabase port 5432) so a long-held lock heartbeat survives the transaction pooler's per-transaction connection recycling; PGLite delegates straight to `executeRaw` (no pooler). Both engines implement it; the Minion lock path (`claim`/`renewLock`) is the consumer. `reconnect(ctx?: {error?})` is a REQUIRED lifecycle method on BOTH engines: it recovers a dropped connection using the config captured at the last `connect()`, so callers (autopilot health probe, `batchRetry`) never `disconnect()` + bare `connect()` (which loses the config and throws `database_url undefined` forever, and opens a null-connection window). PostgresEngine rebuilds its pool with a `_reconnecting` reentrancy guard and emits pool-recovery audit; PGLite is single-writer in-process so it just restores the saved data dir for parity. - `src/core/engine-constants.ts` — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/background-work.ts` (#1762/#1745/#1775) — process background-work registry: the single owner of "drain every fire-and-forget DB-write sink before the CLI disconnects." `registerBackgroundWorkDrainer({name, order, drain(timeoutMs), abort?})` + `drainAllBackgroundWorkForCliExit({timeoutMs})` over a `Map` (idempotent registration by name; `__registerDrainerForTest` returns an unregister handle). Drains in explicit `(order, name)` order — facts FIRST (order 0) so its abort-path DB `logIngest` runs against the freshest live engine — and AWAITS `abort()` only when `drain()` reports `unfinished>0`. Best-effort per drainer: one sink's failure never blocks the others or the disconnect. FOUR sinks register at module import (rule-of-four): `facts/queue.ts` (order 0; `abort`=`shutdown()` cancels a hung facts:absorb Haiku via internalAbort), `last-retrieved.ts` (order 1), `search/hybrid.ts` (order 2; `awaitPendingSearchCacheWrites` bounded via `Promise.race`), `eval-capture.ts` (order 3; `captureEvalCandidate` self-tracks its promise via `awaitPendingEvalCaptures`). Both CLI-exit paths (`src/cli.ts` op-dispatch finally + `handleCliOnly` finally) call it before `engine.disconnect()` — closing the PGLite busy-loop where `db.close()` raced an in-flight job and pinned the single-writer lock (#1762). CLI-EXIT-ONLY: the facts `shutdown()` abort is permanent process state, never call in a long-lived `gbrain serve`. Companion changes: `src/core/ai/gateway.ts` `withDefaultTimeout(caller, ms)` bounds every outbound AI call (chat 300s, embed+multimodal 60s; env `GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS`; composed with caller signals via `AbortSignal.any`) and the op-dispatch + handleCliOnly force-exit timers `process.exit(process.exitCode ?? 0)` so a hung disconnect can't mask an errored op as success; `src/core/postgres-engine.ts` `reconnect()` module-mode branch re-establishes via idempotent `db.connect()` + `connectionManager.setReadPool` refresh instead of `db.disconnect()` (no null window for concurrent ops; fail-loud on real connect failure — #1745); `src/core/search/hybrid.ts` `embedQueryBounded` + a shared `QueryEmbedDeadline` (6s, floored 2s per embed via `MIN_QUERY_EMBED_BUDGET_MS`; env `GBRAIN_QUERY_EMBED_TIMEOUT_MS`) bounds the cache-lookup AND inner query embeds so a stalled provider falls back to keyword instead of hanging past the 10s force-exit (#1775). Incorporates + hardens PR #1763 (@ElliotDrel). Pinned by `test/core/background-work.test.ts`, `test/search/query-embed-deadline.test.ts`, `test/eval-capture-drain.test.ts`, `test/e2e/postgres-reconnect-singleton.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts`, `test/fix-wave-structural.test.ts`. - `src/core/search/graph-signals.ts` — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring at full score, demote the rest). All three inherit the floor-ratio gate preventing weak pages from being boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width`. `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (incl. the IRON-RULE floor-gate regression). @@ -25,6 +25,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/doctor.ts` extension — `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded. Pinned by cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`). - `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all BrainEngine methods. `listLinkSources({sourceId?, sourceIds?})` returns distinct `link_source` provenances + counts (`ORDER BY count DESC, link_source ASC NULLS LAST`; scalar + federated scoped; parity with postgres-engine.ts) powering `gbrain link-sources`. `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the whole batch as one JSONB document via `jsonb_to_recordset(($1::jsonb)->'rows')` (bound through `executeRawJsonb` with a `{ rows }` wrapper; rows built by the shared `src/core/batch-rows.ts` helpers, NUL-stripped), and are `batchRetry`-wrapped. `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error (macOS 26.3 WASM bug #223, points at `gbrain doctor`); the lock is released on failure so the next process can retry cleanly. `searchKeyword`/`searchKeywordChunks` multiply `ts_rank` by the source-factor CASE at chunk grain; `searchVector` is a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`, inner LIMIT scales with offset to preserve pagination. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for forward-referenced state the embedded blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target, plus `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`) and adds only what's missing; threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope; no-op on fresh installs and modern brains (closes the upgrade-wedge bug class #239/#243/#266/#357/#366/#374/#375/#378/#395/#396/#1018/#974/#820). `getBrainScore` returns 100/100 with full breakdown (35/25/15/15/10) when `pageCount === 0` (vacuous truth — empty brain has no coverage problem); Pinned by `test/brain-score-breakdown.test.ts` empty-brain assertion + `test/doctor-report-remote.serial.test.ts`. `disconnect()` uses snapshot+early-null (snapshot `_db`/`_lock`, null instance fields BEFORE any `await` so a concurrent `connect()` can't see a partial mid-close state) wrapped in try/finally guaranteeing lock-release even if `db.close()` throws; KEEPS close-then-release order (release-then-close was rejected: it would widen the window where a sibling process connects to a still-closing brain); Pinned by `test/pglite-engine-disconnect.serial.test.ts`. Exports `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` routing the catch-block hint by failure shape (`bunfs` matches literal `$$bunfs` OR `ENOENT[\s\S]*pglite\.data` co-occurrence, surfaces a paste-ready `bun upgrade` + Node fallback; `macos-26-3` keeps the #223 link; `unknown` falls through); Pinned by `test/pglite-init-classifier.test.ts`. Implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding, caller-chunking primitive throwing when input exceeds `DELETE_BATCH_SIZE`, `deletePages` returns `RETURNING slug` rows so callers filter `pagesAffected` to confirmed deletes. Implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, widened `countStaleChunks({sourceId?, signature?})` (the `signature` opt widens via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)`, NULL grandfathered); parity SQL with postgres-engine.ts. PGLite-specific DDL (pgvector, pg_trgm, triggers). +- `src/core/pglite-lock.ts` (#2058) — advisory data-dir lock so two processes never open the same embedded PGLite (single-connection WASM Postgres) at once. Atomic `mkdir` of `.gbrain-lock/` + a `lock` file carrying `{pid, acquired_at, refreshed_at}`. A held lock HEARTBEATS its `refreshed_at` every 30s (`.unref()`ed timer) so a long-running but LIVE holder (an `embed` job can run for minutes) is never mistaken for stale. A waiting acquirer reaps the holder only when it is dead (PID gone) OR has stopped refreshing past the steal grace (`GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS`, default 600s) — pairing PID liveness with heartbeat freshness defeats BOTH the WAL-corruption bug (stealing a live writer) and the PID-reuse false positive (a recycled PID reading as alive). Each holder carries an ownership token (`:`); the heartbeat and `releaseLock` verify the on-disk lock is STILL theirs before touching it, so a stalled-then-resumed holder that was already reaped + replaced can't clobber the new owner. In-memory engines take no lock (no file, no concurrent access). Pinned by `test/pglite-lock.test.ts`. - `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the batch as one JSONB document — `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages ...` bound through `executeRawJsonb({ rows })` — which encodes arbitrary free text safely (the old `unnest(${arr}::text[])` array-literal path crashed Postgres with "malformed array literal" on calendar/Zoom context, gbrain#1861) and sidesteps the 65535-parameter cap; takes declares native recordset column types (`page_id int, weight real, active boolean, …`) so no per-element casts; all three are `batchRetry`-wrapped. `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection. `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by `batchRetry` on a retryable connection error). Concurrent callers share one in-flight `_reconnectPromise` (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic `db.connect()` token on the connect leg. `reconnect(ctx?)` accepts the triggering error and records a pool-recovery audit event (`reap_detected`/`reconnect_other`/`reconnect_succeeded`/`reconnect_failed`) for the `pool_reap_health` doctor check. `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same probe set as PGLite (extended for column-only forward-reference cases: `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`/`archived_at`/`archive_expires_at`); the entire probe path runs on the DDL connection threaded from `initSchema` (closing a concurrent-bootstrap race for Supabase pooler users); closes #1018/#974/#820. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls `db.disconnect()` when it owns the singleton (`_ownsModuleSingleton`, set from the `db.connect()` creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` + `test/postgres-engine-singleton-ownership.test.ts`. `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`. - `src/core/cjk.ts` — Single source of truth for CJK detection. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables). Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()`. Separate surface from `sync-failures.jsonl` — that file carries bookmark-gating semantics that info events shouldn't trigger. @@ -101,7 +102,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. `BATCH_SIZE=100` (per-recipe pre-split + recursive halving + adaptive shrink-on-miss live in the gateway; the outer paginator is for progress-callback granularity, not batch protection). `estimateEmbeddingCostUsd(tokens)` prices against the currently-configured model's rate via `currentEmbeddingPricePerMTok()` (resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate 0.13 only when the gateway is unconfigured or the model is unknown to the pricing table). `EMBEDDING_COST_PER_1K_TOKENS` retained for back-compat with direct importers/tests. `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from current and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so cost gate and embed decision can't drift. `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. - `src/core/ai/dims.ts` — per-provider `providerOptions` resolver for embed-time dimension passthrough; the single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, `isValidVoyageOutputDim(dims)`. Voyage path uses the SDK-supported `dimensions` field (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built (the AI SDK's openai-compatible adapter doesn't recognize the wire-key, so sending it from here would be silently dropped and Voyage would return its default 1024-dim). Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary (most common trigger: `embedding_model: voyage:voyage-4-large` without `embedding_dimensions`, falling back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). - `src/core/ai/types.ts` — provider/recipe types. `EmbeddingTouchpoint` has optional `chars_per_token` (default 4, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling), both consulted only when `max_batch_tokens` is also set; Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64). Pre-split budget = `max_batch_tokens × safety_factor / chars_per_token`. `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes mixing text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`); when omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` (OpenAI text + Voyage images without flipping the primary pipeline). `Recipe.default_headers?: Record` (static) and `Recipe.resolveDefaultHeaders?(env)` (env-templated) seam for per-recipe headers riding alongside auth on every openai-compat touchpoint; mutually exclusive (declaring both throws `AIConfigError` at gateway-configure time); keys conflicting with the resolved auth header (`Authorization`, the resolver's custom header) rejected at `applyResolveAuth` call time so defaults can't shadow auth. Used by OpenRouter for the `HTTP-Referer` + `X-OpenRouter-Title` + `X-Title` attribution triple. -- `src/core/ai/gateway.ts` — unified seam for every AI call. `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path embeds via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default; the hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}` and the gateway resolves the matching recipe via `instantiateEmbedding()`. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down. `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL `/embeddings → /models/embed`, injects `input_type` (default `'document'`; `'query'` when threaded via `providerOptions.openaiCompatible.input_type`) and explicit `encoding_format: 'float'`, and rewrites the response `{results: [{embedding}], usage: {total_bytes, total_tokens}}` → `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates. Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name). Wired in `instantiateEmbedding()` via the `recipe.id === 'zeroentropyai'` branch. `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'`. `_rerankTransport` test seam mirrors `_embedTransport`. `embedQuery(text)` threads `inputType: 'query'` through `dimsProviderOptions()` (4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch; `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model`; `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries; the Voyage `/multimodalembeddings` path is unchanged (gateway selects by recipe `implementation` tag). Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions`. Pinned by `test/openai-compat-multimodal.test.ts`. Module-scoped `_embedTransport` defaults to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive `embed()` with a stubbed transport. `splitByTokenBudget` and `isTokenLimitError` exported `@internal` (pure functions reused by the test file). Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model`); after the recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared (closes the Voyage-text-only-into-multimodal-endpoint footgun before any HTTP call). `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel`. Exported `VoyageResponseTooLargeError` tagged class: `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length at `:595`, Layer 2 per-embedding base64 at `:619`) throw it; the inbound response-rewriter's try/catch (which swallows parse failures so misshaped responses fall through to the SDK parser) checks `instanceof VoyageResponseTooLargeError` and rethrows so the cap is actually effective (regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line). AI SDK v6 toolLoop compat (`gbrain skillopt` rollouts AND production background `subagent` jobs both route through `chat()` / `toolLoop`): in `chat()`, tool defs wrap the raw JSON Schema with the SDK's `jsonSchema()` helper (`inputSchema: jsonSchema(t.inputSchema)`) — v6's `asSchema()` treats a bare `{jsonSchema: ...}` object as a thunk and throws "schema is not a function"; new exported pure `toModelMessages(messages: ChatMessage[]): unknown[]` converts gbrain's provider-neutral `ChatMessage[]` into v6 `ModelMessage[]` — tool results (pushed by `toolLoop` as `role:'user'` with bare-value tool-result blocks) become a dedicated `role:'tool'` message with structured `output:{type:'json'|'text'|'error-text', value}` parts; `null` output preserved as `{type:'json', value:null}` (not dropped); text/tool-call blocks pass through with v6 field names (`toolCallId`/`toolName`/`input`); applied at the `generateText` call (`messages: toModelMessages(opts.messages)`). The converter is the load-bearing fix for the production subagent path, not just skillopt. Pinned by `test/gateway-model-messages.test.ts`. Companion fix in `src/core/skillopt/rollout.ts`: the inline `paramsToSchema` dropped `items` on array params; it now uses the shared `paramDefToSchema` from `src/mcp/tool-defs.ts` (single source of truth, recursive on items/enum/default). +- `src/core/ai/gateway.ts` — unified seam for every AI call. `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path embeds via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default; the hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}` and the gateway resolves the matching recipe via `instantiateEmbedding()`. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down. `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL `/embeddings → /models/embed`, injects `input_type` (default `'document'`; the threaded `'query'|'document'` crosses the SDK boundary via the module-level `__embedInputTypeStore` AsyncLocalStorage populated in `embedSubBatch()`, because the AI SDK's openai-compatible adapter strips `input_type` from `providerOptions` before building the wire body — #1400; `voyageCompatFetch` injects it opt-in the same way, and `openAICompatAsymmetricFetch` is the fallthrough shim for every other openai-compat recipe — llama-server/litellm/ollama — a strict pass-through when nothing was threaded) and explicit `encoding_format: 'float'`, and rewrites the response `{results: [{embedding}], usage: {total_bytes, total_tokens}}` → `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates. Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name). Wired in `instantiateEmbedding()` via the `recipe.id === 'zeroentropyai'` branch. `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'`. `_rerankTransport` test seam mirrors `_embedTransport`. `embedQuery(text)` threads `inputType: 'query'` through `dimsProviderOptions()` (4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch; `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model`; `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries; the Voyage `/multimodalembeddings` path is unchanged (gateway selects by recipe `implementation` tag). Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions`. Pinned by `test/openai-compat-multimodal.test.ts`. Module-scoped `_embedTransport` defaults to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive `embed()` with a stubbed transport. `splitByTokenBudget` and `isTokenLimitError` exported `@internal` (pure functions reused by the test file). Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model`); after the recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared (closes the Voyage-text-only-into-multimodal-endpoint footgun before any HTTP call). `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel`. Exported `VoyageResponseTooLargeError` tagged class: `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length at `:595`, Layer 2 per-embedding base64 at `:619`) throw it; the inbound response-rewriter's try/catch (which swallows parse failures so misshaped responses fall through to the SDK parser) checks `instanceof VoyageResponseTooLargeError` and rethrows so the cap is actually effective (regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line). AI SDK v6 toolLoop compat (`gbrain skillopt` rollouts AND production background `subagent` jobs both route through `chat()` / `toolLoop`): in `chat()`, tool defs wrap the raw JSON Schema with the SDK's `jsonSchema()` helper (`inputSchema: jsonSchema(t.inputSchema)`) — v6's `asSchema()` treats a bare `{jsonSchema: ...}` object as a thunk and throws "schema is not a function"; new exported pure `toModelMessages(messages: ChatMessage[]): unknown[]` converts gbrain's provider-neutral `ChatMessage[]` into v6 `ModelMessage[]` — tool results (pushed by `toolLoop` as `role:'user'` with bare-value tool-result blocks) become a dedicated `role:'tool'` message with structured `output:{type:'json'|'text'|'error-text', value}` parts; `null` output preserved as `{type:'json', value:null}` (not dropped); text/tool-call blocks pass through with v6 field names (`toolCallId`/`toolName`/`input`); applied at the `generateText` call (`messages: toModelMessages(opts.messages)`). The converter is the load-bearing fix for the production subagent path, not just skillopt. Pinned by `test/gateway-model-messages.test.ts`. Companion fix in `src/core/skillopt/rollout.ts`: the inline `paramsToSchema` dropped `items` on array params; it now uses the shared `paramDefToSchema` from `src/mcp/tool-defs.ts` (single source of truth, recursive on items/enum/default). - `src/core/ai/recipes/zeroentropyai.ts` — ZeroEntropy openai-compatible recipe declaring BOTH `embedding` (`zembed-1`, 7 Matryoshka dims: 2560/1280/640/320/160/80/40) AND `reranker` (`zerank-2` flagship + `zerank-1` + `zerank-1-small`, 5MB payload cap) touchpoints. `implementation: 'openai-compatible'` (pinned by regression in `test/ai/zeroentropy-recipe.test.ts`). `base_url_default: 'https://api.zeroentropy.dev/v1'` already ends with `/v1`, so the `zeroEntropyCompatFetch` URL rewrite `/embeddings → /models/embed` produces `…/v1/models/embed` (NOT `…/v1/v1/…` — pinned by regression). `chars_per_token: 1` + `safety_factor: 0.5` match Voyage's dense-content hedge. - `src/core/ai/recipes/llama-server-reranker.ts` — sibling of `llama-server` (the embedding recipe) for llama.cpp in `--reranking` mode. Distinct recipe rather than dual-touchpoint extension because `--reranking` and `--embeddings` are mutually exclusive at server-launch time, so the two backends need independent base URLs (default 8081 here vs 8080 there). Declares `reranker` touchpoint with `models: []` (user-provided id matching the `--alias` the user launched with), `path: '/rerank'` (leaf-only; consumes `RerankerTouchpoint.path` override; gateway concatenates with `base_url_default` which ends in `/v1`, producing `…/v1/rerank`), `default_timeout_ms: 30_000` (consumed by `src/core/search/mode.ts`'s reranker timeout chain — CPU-only first-call warmup headroom; the 5s mode-bundle default would fail-open as `timeout`), `cost_per_1m_tokens_usd: 0` (recognized by `FREE_LOCAL_RERANK_PROVIDERS` in `src/core/budget/budget-tracker.ts` so `--max-cost` callers don't hard-fail on local rerank). Setup hint emphasizes `--alias` because llama-server's `/v1/models` defaults model id to the gguf file path without it. Covers Qwen3-Reranker via llama.cpp AND self-hosted ZE weights via llama.cpp — same recipe, different `--model` at launch. Pinned by `test/ai/recipe-llama-server-reranker.test.ts`. Voyage / Cohere / vLLM rerankers stay out of scope (different wire shapes). Same wave adds: `path?: string` + `default_timeout_ms?: number` on `RerankerTouchpoint` in `src/core/ai/types.ts`; consumed by the URL build at `src/core/ai/gateway.ts:rerank()` and by mode-resolution at `src/core/search/mode.ts:resolveSearchMode` (precedence: per-call > config-key > recipe touchpoint default > mode bundle); `LLAMA_SERVER_RERANKER_BASE_URL` env passthrough in `src/cli.ts:buildGatewayConfig`; `FREE_LOCAL_RERANK_PROVIDERS` set in `src/core/budget/budget-tracker.ts:lookupPricing` (rerank-kind-only zero-pricing for the local provider prefix); doctor-fix at `src/commands/models.ts:probeRerankerConfig` reads `search.reranker.model` via `loadSearchModeConfig` + `resolveSearchMode` (closes file-plane / DB-plane divergence where doctor said "not configured" while live search was actively reranking — the field-plane `getRerankerModel()` read nothing writes); `probeRerankerReachability` reads the recipe's `default_timeout_ms` so CPU-only cold-start doesn't false-fail. - `src/core/ai/recipes/openrouter.ts` — OpenRouter openai-compatible recipe: single key, many providers via `openrouter:/` strings. `base_url_default: 'https://openrouter.ai/api/v1'`. Embedding touchpoint: `openai/text-embedding-3-small` at 1536 dims with Matryoshka `dims_options: [512, 768, 1024, 1536]`; `max_batch_tokens: 300_000` = OpenAI's aggregate-per-request token cap (NOT per-input). Chat touchpoint declares 8 curated entry points (gpt-5.2, gpt-5.2-chat, gpt-5.5, claude-haiku-4.5, claude-sonnet-4.6, claude-opus-4.7, gemini-3-flash-preview, deepseek-chat) but openai-compat tier accepts any model ID; deliberately no `max_context_tokens` because OR's catalog spans 128K to 1M+. `supports_subagent_loop: false` is INFORMATIONAL — the real gate is `isAnthropicProvider()` in `src/core/model-config.ts` which hard-pins gbrain's subagent infra to Anthropic-direct. Declares `resolveDefaultHeaders(env)` returning OR's three attribution headers: `HTTP-Referer` (required for OR app-attribution), `X-OpenRouter-Title` (preferred), `X-Title` (back-compat alias); defaults to `https://gbrain.ai` / `gbrain`; forks override via `OPENROUTER_REFERER` / `OPENROUTER_TITLE` env vars. Smoke-tested by `test/ai/recipe-openrouter.test.ts` (incl. the shape-test regression guard: every model in the chat list matches `^[a-z0-9-]+\/[a-z0-9._-]+$`). @@ -273,13 +274,14 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `test/fixtures/whoknows-eval.jsonl` — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries; placeholder uses obviously-example slugs (`wiki/people/example-alice`). Drives `test/e2e/whoknows.test.ts` (seeds a matching synthetic brain, asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/core/skillopt/` + `src/commands/skillopt.ts` + `skills/skill-optimizer/` — self-evolving skill optimization grounded in the SkillOpt paper (arXiv 2605.23904). `gbrain skillopt ` treats `SKILL.md` as trainable parameters of a frozen agent: validation-gated (median-of-3 + epsilon=0.05), budget-capped (preflight estimator), per-skill DB-locked (`tryAcquireDbLock('skillopt:', 60min)`), atomic-versioned (history-intent-first 5-step commit), body-only mutations (frontmatter forbidden). Rollouts use `gateway.toolLoop` directly with no-op persistence callbacks (zero `subagent_messages` pollution) + a read-only tool allowlist derived from `BRAIN_TOOL_ALLOWLIST` minus `put_page`/`submit_job`/`file_upload`. Two reflect calls per step; rejected-edit buffer LRU-bounded to 100; bundled-skill gate; bootstrap workflow (sentinel + `--bootstrap-reviewed`); D_sel floor (>=5 with `--split` override); audit JSONL via `audit-writer.ts`. Added to `ALL_PHASES` after `patterns` (default OFF; opt-in via `gbrain config set cycle.skillopt.enabled true`); cycle phase wrapper at `src/core/skillopt/cycle-phase.ts` walks stale skills with per-skill ($0.50) + brain-wide ($2.00) caps. Added to `PROTECTED_JOB_NAMES`. Surface: dream-cycle phase wrapper; `--all` batch mode (`src/core/skillopt/batch.ts:runBatchAll`); `--target-models` fleet (`runFleet` parallel per-model receipts under `skillopt/fleet//`); MCP op `run_skillopt` (admin scope + per-skill `skillopt.allowed_skills` allowlist, NOT localOnly, validates `skill_name` kebab-only + confines caller-supplied benchmark/held-out paths to skillsDir for remote callers); Minion `skillopt` handler + `--background` with `allowProtectedSubmit: true`; write-flavored optimization via `src/core/skillopt/write-capture.ts:buildWriteCaptureRegistry` (virtual `put_page`/`submit_job`/`file_upload` captured in-memory; `--write-capture` flag); held-out real-user test set via `src/core/skillopt/held-out.ts` (capture infra at `~/.gbrain/skillopt-captures//.jsonl`, `--held-out ` flag, `runHeldOutGate` candidate >= baseline). Hermetic via DI seams (`opts.chatFn` for optimizer + judge; `opts.toolLoopFn` for rollouts; no `mock.module`). `--bootstrap-from-skill` → `runBootstrapFromSkill` in `src/core/skillopt/bootstrap-benchmark.ts`: reads `SKILL.md` directly (no `routing-eval.jsonl`), makes ONE LLM call emitting a full starter benchmark (tasks + rule judges) as JSONL, parsed line-by-line with skip-bad-line salvage and a min-2-valid-checks-per-task drop; provider/transport errors PROPAGATE (not collapsed to `bootstrap_empty`). `--bootstrap-tasks N` (default 15, capped 50); `maxTokens` scales `min(8000, max(4000, N*220))`. The stderr REVIEW line prints the literal `gbrain skillopt --bootstrap-reviewed --split 1:1:1` — load-bearing because the default `4:1:5` split makes a 15-task starter's `D_sel = floor(15/10) = 1`, below the `>=5` floor, so a 15-task benchmark needs `--split 1:1:1`. Both bootstrap generators share `assertBenchmarkAbsent` + `readSkillBodyOrThrow`; `--bootstrap-from-skill` is mutually exclusive with `--bootstrap-from-routing`/`--benchmark`/`--all`/`--target-models`/`--resume`. Generated rule judges are explicitly WEAK DRAFTS to be strengthened during the review gate. The F11 held-out gate is wired: `--held-out ` is parsed and threaded through every caller (CLI main + `--background` `held_out_path` + batch/fleet `heldOutPath` + the `run_skillopt` `held_out_path` param), running at CHECKPOINT ACCEPTANCE so no-mutate/fleet paths can't promote a held-out-failing candidate. `assertBundledMutationHeldOut` in bundled-skill-gate.ts: bundled + `--allow-mutate-bundled` requires a NON-EMPTY held-out (`MIN_HELD_OUT_SIZE = D_SEL_MIN_SIZE` = 5, derived so they can't desync) or hard-refuses (exit 2), for ALL callers (they funnel through `runSkillOpt`); held-out must be task_id-DISJOINT from the benchmark (overlap rejected — can't catch overfitting). `receipt.baseline_sel_score` populated + a real final-test eval (`test_score` + `baseline_test_score`) scoring best + baseline on `split.test`; shared `scoreSkillOnTasks` primitive (validate-gate.ts) backs baseline/final-test/held-out scoring. `--no-mutate` writes proposed.md via `writeProposed` in version-store.ts. `maxRuntimeMin` ENFORCED (wall-clock deadline between steps → `skillopt_runtime_exceeded` → outcome aborted). Three eval-internal ablation opts on `SkillOptOpts` (NOT on CLI): `reflectMode` (`'both'`/`'failure-only'`), `disableValidationGate` (greedy-accept), `optimizerMode` (`'reflect'`/`'one-shot-rewrite'`), recorded in `RunReceipt` + audit `run_start` for replayability; `ROLLOUT_SUCCESS_THRESHOLD = 0.5` named constant for the partition; one-shot fence-strip is anchored (`^```...```$`) so an embedded code sample isn't truncated. Budget no-pricing fix: Claude Haiku 4.5's dateless canonical id `claude-haiku-4-5` is in `src/core/anthropic-pricing.ts` (a `BudgetTracker`-capped run on Haiku otherwise threw `no_pricing` on the FIRST `chat()` of every rollout); `runValidationGate` (validate-gate.ts) scans settled results for `isMustAbortError(error)` (from `worker-pool.ts`; `BUDGET_EXHAUSTED` is in `MUST_ABORT_ERROR_TAGS`) and re-throws so the caller aborts loudly instead of recording a hollow `selScore:0` — ordinary non-abort rollout errors still fail-open to `score:0` (judge-hiccup posture preserved). Pinned by 152 tests across 18 files (foundation + adversarial + v2 surface + E2E PGLite serial), `test/skillopt/bootstrap-from-skill.test.ts` (20 cases), `test/skillopt/rollout.test.ts`, `test/skillopt/validate-gate-abort.test.ts` (3 cases), held-out ENFORCE + one-shot-rewrite unit cases, and e2e (F11 block/allow, bundled no-mutate, runtime deadline, receipt honesty, held-out disjointness, no-DB-pollution). Drives the Track B SkillOpt benchmark suite in the sibling `gbrain-evals` repo. - `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge rejecting ideas with resistance >4.5 "too obvious", stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2`. `judges.ts` exports `runJudge(config, ideas)` + two configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` cognitive_load 0.50 + inversion rule). Calibration cold-start fallback: when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back in `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch); internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. The fire-and-forget IIFE is tracked in a module-scoped `Set>`; `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns the pending count. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then a fallback `process.exit(0)` fires ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive) — closes the PGLite CLI search/query/get-hang class where the IIFE raced disconnect and PGLite's WASM kept Bun's event loop alive. `pages.last_retrieved_at TIMESTAMPTZ NULL` has a full (NOT partial) B-tree index covering both NULL and range branches; full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output via `isLsdOutput()` in `src/core/cycle/transcript-discovery.ts` short-circuiting `isDreamOutput()`. `gbrain eval brainstorm ` is a three-axis conjunctive gate (distance + usefulness + grounding — distance alone is gameable). `gbrain doctor` has a `brainstorm_health` check (migration applied, `search.track_retrieval` setting, calibration cold-start status). `judges.ts` computes the judge token budget via `computeJudgeMaxTokens(ideaCount, modelId)` (named constants `TOKEN_BUDGET_PER_IDEA`, `TOKEN_BUDGET_ENVELOPE`, `LEGACY_MIN_MAX_TOKENS`, `MAX_OUTPUT_TOKENS_CEIL`; `ANTHROPIC_OUTPUT_CAPS` map: Opus 4.7 32K, Sonnet 4.6 / Haiku 4.5 64K, legacy Claude 3.5 8K) so a large multi-call judge doesn't truncate mid-JSON; with no `modelOverride` the cap routes through the gateway's actual configured chat model via `getChatModel()`. `--save` for both commands persists through the canonical ingestion path: `persistSavedIdea(engine, {slug, content, provenanceVia})` calls `importFromContent({noEmbed:true, sourcePath})` (chunked + tagged + content_hash so search finds it, no embedding cost at save) THEN renders the saved row to disk via the shared `writePageThrough` helper (file rendered FROM the row so the two sinks can't diverge and `gbrain sync` doesn't churn it). `formatSaveOutcome(outcome, ctx)` returns an honest per-branch message (both-sinks, DB-only when no `sync.repo_path`/repo-not-a-dir, DB-saved-but-file-errored, total-failure → loud `save FAILED … NOT persisted` on stderr + nonzero exit) — closes the silent-false-success class where `--save` printed "Saved" unconditionally even when the DB write failed. `buildIdeaSlug(question, label, nonce?)` adds a random nonce suffix (injectable for tests) so two same-day runs sharing the first 60 slug chars don't clobber. `--json` callers stay DB-only. `buildBrainstormFrontmatterObject(result)` in orchestrator.ts returns the object form for `serializeMarkdown` (string `buildBrainstormFrontmatter` untouched). Pinned by `test/last-retrieved.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir asserts search/get/query exit 0 in <15s + daemon-survival), `test/fix-wave-structural.test.ts` (asserts the drain `await` is textually BEFORE `engine.disconnect`), `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm,judges-maxtokens,save}.test.ts`. Open Collider source: `github.com/CL-ML/open-collider`. -- `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` reads `sync.repo_path`, re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown` + `resolvePageFilePath`, and writes the `.md` under the repo so the brain has a committable artifact that round-trips through `gbrain sync`. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts`. +- `src/core/write-through.ts` — shared atomic disk write-through for the canonical ingestion path. `writePageThrough(engine, slug, {sourceId?, frontmatterOverrides?, logger?})` resolves the disk target from the ASSIGNED source's own working tree (`sources.local_path`), re-reads the just-written DB row (`getPage`), renders it via `serializePageToMarkdown`, and writes the `.md` under that tree's root so the brain has a committable artifact that round-trips through `gbrain sync`. A source with its own `local_path` writes there; a source WITHOUT one falls back to the global `sync.repo_path` ONLY when this is the sole source (then that path is unambiguously this source's tree) and otherwise skips with `source_has_no_local_path` rather than leak into a sibling source's git repo. Rendering FROM the row means file and row cannot diverge. ATOMIC: writes to a unique temp sibling (`.tmp..`) + `renameSync`, cleaning up temp on any failure, so a crash or concurrent `gbrain sync`/autopilot walking the live git tree never reads a half-written `.md` (matches the `.tmp + rename` convention in import-checkpoint.ts / op-checkpoint.ts). Never throws — returns `WriteThroughResult { written, path?, skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_has_no_local_path' | 'page_not_found_after_write', error? }` so the caller decides messaging + exit codes. Trust gating (subagent sandbox, dry-run) stays at the CALLER. Consumers: `put_page` op and `gbrain brainstorm/lsd --save` via `persistSavedIdea`. Pinned by `test/write-through.test.ts`. - `src/core/model-id.ts` — `splitProviderModelId(input: string | null | undefined): {provider: string | null, model: string}` shared parser for the pricing side. Splits on `:` first, then `/`. Defensive contract: null/undefined/empty/whitespace returns `{provider: null, model: ''}`. Five sites consume it (`src/core/anthropic-pricing.ts:estimateMaxCostUsd`, `src/core/budget/budget-tracker.ts:lookupPricing`, `src/core/eval-contradictions/cost-tracker.ts:pricingFor`, `src/core/minions/batch-projection.ts` at two call sites, `src/core/model-config.ts:isAnthropicProvider`) so the pricing + classification surface has no parallel re-implementations of `provider:model` splitting — slash-form ids (`anthropic/claude-sonnet-4-6`) classify correctly instead of falling through to "unknown model". Distinct from the gateway-side `parseModelId` in `src/core/ai/model-resolver.ts`, which throws on bare names because routing needs an explicit provider; this one returns `{provider: null, model: 'bare'}` because pricing lookups happen against bare model ids. Pinned by `test/model-id.test.ts`. - `src/core/ai/model-resolver.ts:parseModelId` — gateway-side resolver accepts both colon and slash form (`provider:model` and `provider/model`) so a slash-form id resolves to the same recipe at every gateway entry point (chat / embed / rerank) instead of throwing `AIConfigError: model id must be in format provider:model`. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). Batch-load fast path on Postgres uses a single SQL query (fixes the PgBouncer round-trip timeout, ~60s → ~6s), gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1`. Batch projection is `SELECT ... ORDER BY source_id, slug` (NOT `SELECT DISTINCT ON (slug)`, which collapsed same-slug-different-source pages into one scan) so multi-source brains scan each `(source, slug)` row independently. Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`; batch + sequential paths report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check at `doctor.ts:1011-1043` consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings in BOTH `doctor.ts` and `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source ` matches. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. IRON-RULE regression banning stale verb names, the sync_freshness boundary matrix, the D4 regression guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases). -- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). `Migration` interface carries `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses in a transaction; ignored on PGLite). Key migrations: v14 (handler branches on `engine.kind` for CONCURRENTLY-on-Postgres with invalid-remnant pre-drop via `pg_index.indisvalid`, plain `CREATE INDEX` on PGLite); v15 (`minion_jobs.max_stalled` default 1→5 + backfill non-terminal rows); v24 `rls_backfill_missing_tables` (`sqlFor: { pglite: '' }` no-op — PGLite has no RLS engine, targets subagent tables absent from pglite-schema.ts); v30 `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))` (RLS-enabled under BYPASSRLS; synthesize reads/writes to avoid re-judging); v35 auto-RLS event trigger `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` running `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on new `public.*` tables (no FORCE) + one-time backfill on every existing `public.*` base table whose comment doesn't match `^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}` (per-table failure aborts the offending CREATE TABLE; no EXCEPTION wrap; PGLite no-op via `sqlFor.pglite: ''`; breaking change: intentionally-RLS-off public tables need the GBRAIN:RLS_EXEMPT comment before upgrade); v40 `pages_emotional_weight` (`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`, column-only metadata-only); v46 `mcp_request_log_params_jsonb_normalize` (`UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`, idempotent); v60-v65 six-migration chain wiring source-scoping into `oauth_clients` — v60 (`oauth_clients_source_id_fk`: `source_id TEXT` NULL→`'default'` backfill + FK to `sources(id) ON DELETE SET NULL`), v61 (`federated_read TEXT[] NOT NULL DEFAULT '{}'`), v62 (explicit-CASE backfill so `source_id IS NULL` → `'{}'`), v63 (fail-loud check every row's source_id is in its federated_read array), v64 (FK flipped to `ON DELETE RESTRICT`), v65 (GIN index for array-containment); v68 `eval_candidates_embedding_column` (`eval_candidates.embedding_column TEXT NULL` per-row provenance for `gbrain eval replay` to reproduce the same retrieval space; NULL-tolerant); v108 `pages_embedding_signature` (`pages.embedding_signature TEXT NULL` = `:` stamped via `setPageEmbeddingSignature`; GRANDFATHER — stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current` so NULL is NEVER stale and upgrade never re-embeds the whole corpus; no index; metadata-only); v109 `sources_newest_content_at` (`sources.newest_content_at TIMESTAMPTZ` durable newest-COMMIT HEAD committer time written by `writeSyncAnchor`, read by the REMOTE staleness path instead of shelling to git; mirror in pglite-schema.ts + schema.sql + bootstrap probe); v110 `page_aliases` (`(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` + lookup indexes on `(source_id, alias_norm)` and `(source_id, slug)`; `alias_norm` is `normalizeAlias()` output so WRITE/READ key on the same form; also in `src/core/pglite-schema.ts`); v111 `search_telemetry_rank1_columns` (`ADD COLUMN IF NOT EXISTS` on both engines: `sum_rank1_score`, `count_rank1`, three buckets `rank1_lt_solid`/`rank1_solid`/`rank1_high` on `search_telemetry` — aggregate not per-query rows so rank-1 median drift is bounded-growth; ALTERs right after v57 which created the table); v114 `links_link_source_check_kebab_regex` (#1941, opens `link_source` from the closed allowlist to a kebab-case format gate `^[a-z][a-z0-9]*(-[a-z0-9]+)*$` + `char_length<=64`; Postgres branch uses `NOT VALID` + `VALIDATE CONSTRAINT` with `transaction:false`, PGLite plain DROP+ADD; existing built-ins all satisfy the regex so VALIDATE never fails on existing data). +- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). `Migration` interface carries `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses in a transaction; ignored on PGLite). Key migrations: v14 (handler branches on `engine.kind` for CONCURRENTLY-on-Postgres with invalid-remnant pre-drop via `pg_index.indisvalid`, plain `CREATE INDEX` on PGLite); v15 (`minion_jobs.max_stalled` default 1→5 + backfill non-terminal rows); v24 `rls_backfill_missing_tables` (`sqlFor: { pglite: '' }` no-op — PGLite has no RLS engine, targets subagent tables absent from pglite-schema.ts); v30 `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))` (RLS-enabled under BYPASSRLS; synthesize reads/writes to avoid re-judging); v35 auto-RLS event trigger `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` running `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on new `public.*` tables (no FORCE) + one-time backfill on every existing `public.*` base table whose comment doesn't match `^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}` (per-table failure aborts the offending CREATE TABLE; no EXCEPTION wrap; PGLite no-op via `sqlFor.pglite: ''`; breaking change: intentionally-RLS-off public tables need the GBRAIN:RLS_EXEMPT comment before upgrade); v40 `pages_emotional_weight` (`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`, column-only metadata-only); v46 `mcp_request_log_params_jsonb_normalize` (`UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`, idempotent); v60-v65 six-migration chain wiring source-scoping into `oauth_clients` — v60 (`oauth_clients_source_id_fk`: `source_id TEXT` NULL→`'default'` backfill + FK to `sources(id) ON DELETE SET NULL`), v61 (`federated_read TEXT[] NOT NULL DEFAULT '{}'`), v62 (explicit-CASE backfill so `source_id IS NULL` → `'{}'`), v63 (fail-loud check every row's source_id is in its federated_read array), v64 (FK flipped to `ON DELETE RESTRICT`), v65 (GIN index for array-containment); v68 `eval_candidates_embedding_column` (`eval_candidates.embedding_column TEXT NULL` per-row provenance for `gbrain eval replay` to reproduce the same retrieval space; NULL-tolerant); v108 `pages_embedding_signature` (`pages.embedding_signature TEXT NULL` = `:` stamped via `setPageEmbeddingSignature`; GRANDFATHER — stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current` so NULL is NEVER stale and upgrade never re-embeds the whole corpus; no index; metadata-only); v109 `sources_newest_content_at` (`sources.newest_content_at TIMESTAMPTZ` durable newest-COMMIT HEAD committer time written by `writeSyncAnchor`, read by the REMOTE staleness path instead of shelling to git; mirror in pglite-schema.ts + schema.sql + bootstrap probe); v110 `page_aliases` (`(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` + lookup indexes on `(source_id, alias_norm)` and `(source_id, slug)`; `alias_norm` is `normalizeAlias()` output so WRITE/READ key on the same form; also in `src/core/pglite-schema.ts`); v111 `search_telemetry_rank1_columns` (`ADD COLUMN IF NOT EXISTS` on both engines: `sum_rank1_score`, `count_rank1`, three buckets `rank1_lt_solid`/`rank1_solid`/`rank1_high` on `search_telemetry` — aggregate not per-query rows so rank-1 median drift is bounded-growth; ALTERs right after v57 which created the table); v114 `links_link_source_check_kebab_regex` (#1941, opens `link_source` from the closed allowlist to a kebab-case format gate `^[a-z][a-z0-9]*(-[a-z0-9]+)*$` + `char_length<=64`; Postgres branch uses `NOT VALID` + `VALIDATE CONSTRAINT` with `transaction:false`, PGLite plain DROP+ADD; existing built-ins all satisfy the regex so VALIDATE never fails on existing data); v116 `code_edges_source_backfill_and_callee_index` (#2073, idempotent: backfills NULL `code_edges_symbol`/`code_edges_chunk` `source_id` from each edge's `from_chunk` page — NULL never matched a scoped `AND source_id = …` filter so scoped `code-callers`/`code-callees` returned 0 rows on multi-source brains — plus plain `CREATE INDEX` on `from_symbol_qualified` for both edge tables, which had no index and seq-scanned per BFS node). The dedup-index self-heal (`timeline_dedup_index`, see `timeline-dedup-repair.ts`) is NOT version-gated: `runMigrations` invokes `repairTimelineDedupIndex` on every pass (including the no-pending early-return path) because a merge-renumbered migration can leave the version counter past the index change while the index stays the old shape. +- `src/core/timeline-dedup-repair.ts` (#2038) — schema-drift self-heal for `idx_timeline_dedup`. The migration that widened the dedup index from `(page_id, date, summary)` to `(page_id, date, summary, source)` was renumbered during a master merge, so a brain that ran the old variant has its version counter stamped past the change while the index keeps the 3-column shape — and every `addTimelineEntry` batch then fails its 4-column `ON CONFLICT`, silently breaking timeline writes brain-wide. The version counter can't detect this, so the repair is keyed off the actual index SHAPE: `checkTimelineDedupIndex(engine)` returns `{tablePresent, indexPresent, columns, needsRepair}` (read-only; powers the `timeline_dedup_index` doctor check) and `repairTimelineDedupIndex(engine)` dedupes-then-rebuilds the index. `runMigrations` invokes the repair on every pass (including the no-pending early-return path); idempotent no-op when the index is already 4-column. `gbrain apply-migrations --force-schema` triggers it on demand. Pinned by `test/timeline-dedup-repair.test.ts`. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY `\r`-rewriting; non-TTY plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. `emitHumanLine` is prefix-aware — inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts` it prepends `[id] ` (and TTY-rewrite mode `\r\x1b[2K` carries the prefix inside the clear-to-EOL escape); `emitJson` is intentionally NOT prefixed so NDJSON consumers don't choke on a `[id] {...}` shape. - `src/core/console-prefix.ts` — `AsyncLocalStorage`-backed per-source line-prefix helper. Exports `withSourcePrefix(id, fn)` (runs `fn` with `id` as active prefix; nested wraps replace then restore), `getSourcePrefix()` (read-only accessor; test seam), and `slog(...)` / `serr(...)` (prefix-aware `console.log`/`console.error`). Embedded-newline-safe: a multi-line string under prefix `[foo]` emits `[foo] line1\n[foo] line2`. Outside a wrap, `slog`/`serr` fall through to bare `console.log`/`console.error` so single-source callers see identical output (back-compat invariant). Use `src.id` (slug-validated by `sources add`) NOT `src.name` (free-form) to defeat log-injection through newline/control-character names. Coverage: `src/commands/sync.ts` performSync + callees, `src/commands/embed.ts` runEmbedCore + helpers, `src/core/progress.ts` emitHumanLine. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. @@ -299,7 +301,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/cycle/synthesize.ts` — Synthesize phase: conversation-transcript-to-brain pipeline. Reads `dream.synthesize.session_corpus_dir`, runs a cheap Haiku verdict (cached in `dream_verdicts`), then fans out one Sonnet subagent per worth-processing transcript with `allowed_slug_prefixes` (sourced from `skills/_brain-filing-rules.json` `dream_synthesize_paths.globs`). Orchestrator collects slugs from `subagent_tool_executions` (NOT `pages.updated_at`) and reverse-renders DB → markdown via `serializeMarkdown`. Cooldown via `dream.synthesize.last_completion_ts`, written ONLY on success. Idempotency key `dream:synth::`. `--dry-run` runs Haiku, skips Sonnet. Subagent never gets fs-write access. `renderPageToMarkdown` (exported) stamps `dream_generated: true` + `dream_cycle_date` into every reverse-write's frontmatter; `writeSummaryPage` does the same on the summary index — this marker is the explicit identity surface `isDreamOutput` checks in `transcript-discovery.ts`. `judgeSignificance` and `JudgeClient` are exported; `judgeSignificance` takes a `verdictModel` param loaded from `dream.synthesize.verdict_model` via `loadSynthConfig`. `splitTranscriptByBudget(content, contentHash, maxChars)` splits oversized transcripts at paragraph boundaries (`## Topic:` → `---` → `\n` ladder) using a deterministic offset seeded from the first 32 bits of `contentHash` so retries chunk identically; per-chunk char budget = `MODEL_CONTEXT_TOKENS[resolvedModel] × 0.9 × 3.5 chars/token` (non-Anthropic ids fall back to a 180K-token safe default + once-per-process stderr warn); operator overrides `dream.synthesize.max_prompt_tokens` (floor 100K, wins) and `dream.synthesize.max_chunks_per_transcript` (default 24). Per-chunk idempotency keys `dream:synth:::cof`; single-chunk transcripts preserve the legacy `dream:synth::` key byte-for-byte so existing brains skip with `already_synthesized_legacy_single_chunk` instead of re-spending Sonnet. `collectChildPutPageSlugs` raw-fetches every (job_id, slug) pair (not `SELECT DISTINCT`) and rewrites bare-hash6 slugs to `-c` for chunked children (orchestrator-side, zero Sonnet trust). Cap-hit skips don't write to `dream_verdicts` so raising the cap on next run re-attempts cleanly. Bounds INITIAL prompt size only; tool-loop turn-N accumulation is caught by terminal-error classification in `subagent.ts`. Verdict routing is gateway-routed: `makeJudgeClient(verdictModel)` (exported, replacing `makeHaikuClient()`) mirrors `tryBuildGatewayClient` in `src/core/think/index.ts` — a construction-time provider/key probe returns `null` on a clear miss (unknown provider id via `resolveRecipe` AIConfigError, or Anthropic provider with no key via `hasAnthropicKey()`). The verdict loop wraps `judgeSignificance` in try/catch for `AIConfigError` so mid-run provider failures surface as per-transcript `worth=false, reasons=['gateway error: ...']` instead of crashing the phase. Canonical config key `models.dream.synthesize_verdict` (per `PER_TASK_KEYS` in `src/core/model-config.ts`); `JudgeClient` signature preserved verbatim for test-seam stability; CI guard `scripts/check-gateway-routed-no-direct-anthropic.sh` prevents reintroducing `new Anthropic()` here or in `think/index.ts`. At the queue.add boundary (lines 395-404) a conditional `anthropic:` prefix is applied ONLY when the resolved model has no colon AND starts with `claude-` (because `resolveModel` returns bare ids from `TIER_DEFAULTS`/`DEFAULT_ALIASES` and the subagent validator requires `provider:model` form) — avoids changing the shared constants which would ripple across every `resolveModel` caller. Pinned by `test/cycle/synthesize-gateway-adapter.test.ts`, `test/e2e/dream-synthesize-pglite.test.ts` (gateway-adapter mid-run AIConfigError catch), `test/cycle/regression-pr-wave-r1-r2-r4.test.ts`. - `scripts/check-gateway-routed-no-direct-anthropic.sh` — CI guard that fails the build if `src/core/cycle/synthesize.ts` or `src/core/think/index.ts` reintroduces a runtime `new Anthropic()` constructor call or a value-shaped `import Anthropic from '@anthropic-ai/sdk'` import. Type-only imports (`import type Anthropic from '@anthropic-ai/sdk'`) stay allowed for adapter types; comment lines (`//` or ` *` prefixes) are excluded so historical JSDoc doesn't false-fire. Mirrors `scripts/check-jsonb-pattern.sh`. Wired into `bun run verify` and `bun run check:all`. Extend `GUARDED_FILES` when migrating another file off direct SDK construction. - `src/core/cycle/patterns.ts` — Patterns phase: cross-session theme detection over reflections within `dream.patterns.lookback_days` (default 30). Names a pattern only when ≥`dream.patterns.min_evidence` (default 3) reflections support it. Single Sonnet subagent; same allow-list path as synthesize. Runs AFTER `extract` so the graph is fresh. -- `src/core/cycle/extract-facts.ts` — extract_facts cycle phase. Fence is canonical: per-page wipe (`deleteFactsForPage`) + reinsert from `parseFactsFence` + `extractFactsFromFenceText` + `engine.insertFacts`. Empty-fence guard refuses when legacy rows (`row_num IS NULL AND entity_slug IS NOT NULL`) pend backfill (status: warn, hint: `gbrain apply-migrations --yes`). A phantom-redirect pre-pass runs AFTER the legacy-row guard, BEFORE the main reconcile loop: when `opts.brainDir` is set, `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun)` walks unprefixed-slug pages capped by `GBRAIN_PHANTOM_REDIRECT_LIMIT` (default 50). The pass returns `touched_canonicals` — canonical slugs whose disk fence merged with phantom rows; `runExtractFacts` UNIONs them into the main reconcile slug set so canonical's DB facts derive from the merged fence in the same cycle (handles phantom-had-only-on-disk-fence). `ExtractFactsResult` gains six phantom fields: `phantomsScanned`, `phantomsRedirected`, `phantomsAmbiguous`, `phantomsSkippedDrift`, `phantomsLockBusy`, `phantomsMorePending`. Three bubble to `CycleReport.totals` (`phantoms_redirected`, `phantoms_ambiguous`, `phantoms_skipped_drift`). +- `src/core/cycle/extract-facts.ts` — extract_facts cycle phase. Fence is canonical: per-page wipe (`deleteFactsForPage`) + reinsert from `parseFactsFence` + `extractFactsFromFenceText` + `engine.insertFacts`. #1928: the per-page wipe passes `excludeSourcePrefixes: ['cli:']` so conversation facts (written by `extract-conversation-facts`, on pages with NO `## Facts` fence to recreate them from) survive the reconcile instead of being deleted-with-nothing-to-reinsert. The destructive phase no longer inherits a failed sync's full-brain walk: `slugs: []` (a real incremental no-op) is distinguished from `slugs: undefined` (full-walk intent) by presence, not length. `runPhaseExtractFacts` (cycle.ts) surfaces a `warn` (`net_fact_deletion`) when the reconcile deletes at least `NET_DELETION_WARN_FLOOR` (50) more facts than it reinserts — the exact signature of the conversation-facts wipe, which previously read as a silent `ok`. Empty-fence guard refuses when legacy rows (`row_num IS NULL AND entity_slug IS NOT NULL`) pend backfill (status: warn, hint: `gbrain apply-migrations --yes`). A phantom-redirect pre-pass runs AFTER the legacy-row guard, BEFORE the main reconcile loop: when `opts.brainDir` is set, `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun)` walks unprefixed-slug pages capped by `GBRAIN_PHANTOM_REDIRECT_LIMIT` (default 50). The pass returns `touched_canonicals` — canonical slugs whose disk fence merged with phantom rows; `runExtractFacts` UNIONs them into the main reconcile slug set so canonical's DB facts derive from the merged fence in the same cycle (handles phantom-had-only-on-disk-fence). `ExtractFactsResult` gains six phantom fields: `phantomsScanned`, `phantomsRedirected`, `phantomsAmbiguous`, `phantomsSkippedDrift`, `phantomsLockBusy`, `phantomsMorePending`. Three bubble to `CycleReport.totals` (`phantoms_redirected`, `phantoms_ambiguous`, `phantoms_skipped_drift`). - `src/core/entities/resolve.ts` — Free-form entity name → canonical slug resolution. `resolveEntitySlug(engine, source_id, raw)`: exact slug → fuzzy (pg_trgm @ 0.4 threshold) → bare-name prefix expansion (`people/-%` then `companies/-%`, `connection_count` correlated-subquery tiebreaker) → deterministic `slugify` fallback. Two helpers for the phantom-redirect pass: `resolvePhantomCanonical(engine, sourceId, phantomSlug)` SKIPS the exact-slug step (a phantom slug `'alice'` would exact-match itself and no-op the redirect); returns the canonical only when non-null AND contains `/`. `findPrefixCandidates(engine, sourceId, token)` is a standalone SQL query returning ALL candidates across `PREFIX_EXPANSION_DIRS` (hardcoded `['people', 'companies']`) via `slug LIKE ANY($N::text[])` over patterns `dir/token` + `dir/token-%`, cap of 10 ordered by `connection_count DESC, slug ASC` — NOT a wrapper around `tryPrefixExpansion` (that path returns per-dir top-1 and suppresses ambiguity by design). Pinned by `test/phantom-redirect.test.ts` (resolvePhantomCanonical 3 cases + findPrefixCandidates 6 cases incl. multi-dir ambiguity and the `people/aliceberg`-doesn't-match-`alice` false-positive guard). - `src/core/cycle/phantom-redirect.ts` — Phantom-redirect orchestrator. Exports `runPhantomRedirectPass(engine, brainDir, sourceId, dryRun): Promise` (per-cycle wrapper acquiring the `gbrain-sync` writer lock once for the whole pass, 30s bounded retry, walks up to `GBRAIN_PHANTOM_REDIRECT_LIMIT` unprefixed phantoms) + `tryRedirectPhantom(engine, page, sourceId, brainDir, dryRun): Promise` + `stripFenceAndFrontmatterAndLeadingH1` (pure body-shape gate helper — strips facts fence incl. preceding `## Facts` heading and the leading H1; zero residue = phantom). Handler order: body-shape gate → `resolvePhantomCanonical` (bypasses exact-self-match) → `findPrefixCandidates` ambiguity check → `fenceDbDrift` bi-directional check → dry-run early exit → materialize canonical via `serializeMarkdown` if DB-only → append phantom fence rows to canonical's disk fence with `(claim, valid_from)` dedup-guard + row_num continuation → `engine.refreshPageBody` with SHA-256 content_hash recomputed via the import-file shape → `engine.migrateFactsToCanonical` (lossless) → `engine.rewriteLinks` (DB FK rewrite; wiki-link text rewrite is a documented follow-up) → `engine.softDeletePage` + `engine.deleteFactsForPage(phantom)` + `fs.unlinkSync(phantomPath)`. `RedirectResult.canonical` populated on `'redirected'` (incl. dry-run preview) so the caller builds `touched_canonicals`. Idempotent on re-run: phantom soft-deleted → predicate fails (`deleted_at IS NULL`); migrate UPDATE matches no rows; dedup-guard prevents double-append. - `src/core/facts/phantom-audit.ts` — JSONL audit at `${resolveAuditDir()}/phantoms-YYYY-Www.jsonl`. Pattern copy of `src/core/audit-slug-fallback.ts` (ISO-week rotation, honors `GBRAIN_AUDIT_DIR`). Exports `logPhantomEvent(record)` + `readRecentPhantomEvents(days)` + `computePhantomAuditFilename(now?)`. Records every outcome: `redirected | ambiguous | drift | no_canonical | not_phantom_has_residue | pass_skipped_lock_busy`. Best-effort writes — stderr warn on failure, never throws. Separate file from `stub-guard-audit.ts` (distinct consumer + lifecycle: stub-guard logs PREVENTIVE blocks; phantom-audit logs CLEANUP decisions, to be read by a future `phantoms_pending` doctor check). diff --git a/docs/embedding-migrations.md b/docs/embedding-migrations.md index e9c425128..49e08765b 100644 --- a/docs/embedding-migrations.md +++ b/docs/embedding-migrations.md @@ -43,8 +43,8 @@ genuinely has to change. Switching dimensions requires: 1. Dropping the HNSW vector index (pgvector won't survive an `ALTER COLUMN TYPE`). -2. Altering the column type (Postgres only — PGLite cannot do this). -3. Wiping every existing embedding (the old vectors are unusable in the new space). +2. Wiping every existing embedding (the old vectors are unusable in the new space — and pgvector refuses to cast them across dimensions, so this must happen before the alter). +3. Altering the column type (Postgres only — PGLite cannot do this). 4. Re-embedding the entire corpus (can take hours on a 50K-page brain and costs $1-100 in API calls depending on model). 5. Conditionally recreating the index (HNSW supports up to 2000 dimensions per pgvector; above that you must use exact scans). @@ -115,12 +115,17 @@ BEGIN; -- 1. Drop the HNSW index. It can't survive the column type change. DROP INDEX IF EXISTS idx_chunks_embedding; --- 2. Alter the column type. -ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(); - --- 3. Clear stale embeddings so they don't survive into the new space. +-- 2. Clear stale embeddings FIRST. This must happen BEFORE the column +-- alter: pgvector refuses to cast existing vectors across dimensions +-- ("expected dimensions, not "), so altering a +-- column that still holds old-width vectors aborts the transaction. +-- NULLs cast fine. (The old vectors are unusable in the new space +-- anyway — this is the wipe step from the rationale above.) UPDATE content_chunks SET embedding = NULL, embedded_at = NULL; +-- 3. Alter the column type (all rows are NULL now, so the cast succeeds). +ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(); + -- 4. Recreate the HNSW index ONLY IF dims <= 2000. Above that, leave it -- indexless and rely on exact scans (gbrain searchVector handles this -- automatically — search just gets slower, not broken). diff --git a/package.json b/package.json index 85559b546..cde5c8fda 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.40.0" + "version": "0.42.41.0" } diff --git a/src/cli.ts b/src/cli.ts index 09d187cea..e80d7e4b8 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -360,30 +360,83 @@ async function main() { // // Defense-in-depth (adversarial-review C13): `engine.disconnect()` itself // can hang on PGLite (db.close() or releaseLock racing OS-level FS state). - // Install an unref'd setTimeout hard-exit fallback BEFORE entering the - // try/catch/finally so a hung disconnect cannot defeat the force-exit - // contract. Daemons (`serve`) are excluded so they stay alive. + // The unref'd hard-exit fallback is armed inside the `finally` below, so it + // bounds ONLY the teardown phase (drain + disconnect) — the same placement + // as the fall-through owner-disconnect site later in this file. It used to + // be armed HERE, before the try, which silently killed any op whose BODY ran + // past the deadline: on a slow Postgres pooler (6-10s per fresh connection) + // a healthy `gbrain search` was force-exited mid-handler with code 0 and + // ZERO stdout — an empty "success" indistinguishable from no results. The + // exitCode honor (v0.42.20.0) can't help there: a mid-op kill fires before + // any error path sets exitCode. Op-body wallclock bounds are the read-scope + // withTimeout wrap inside the try below, not this teardown backstop. + // Daemons (`serve`) are excluded so they stay alive. const DISCONNECT_HARD_DEADLINE_MS = 10_000; + // Wallclock bound for READ-scope op handlers. With the hard-deadline timer + // correctly scoped to teardown, a genuinely WEDGED read handler (hung pooler + // connection mid-query) would otherwise hang the CLI forever — the #1633 + // zombie class the old (buggy) pre-try timer accidentally bounded at 10s. + // 180s sits far above any healthy slow-pooler run (6-10s/connection); + // --timeout=Ns overrides. Writes/admin stay unbounded: a long import/embed + // must never be killed by a default deadline. + const READ_OP_TIMEOUT_MS = 180_000; let forceExitTimer: ReturnType | undefined; - if (shouldForceExitAfterMain()) { - forceExitTimer = setTimeout(() => { - console.warn( - `[cli] engine.disconnect() did not return within ${DISCONNECT_HARD_DEADLINE_MS}ms — force-exiting`, - ); - // v0.42.20.0 (codex): honor an exit code an errored op already set — - // a bare process.exit(0) here would mask a failed op as success if the - // drain/disconnect then hangs. - process.exit(process.exitCode ?? 0); - }, DISCONNECT_HARD_DEADLINE_MS); - // unref so the timer itself doesn't keep the event loop alive — only - // the actual pending work (PGLite WASM handle) does. Without unref, - // we'd block a clean exit by 10s on every successful CLI run. - forceExitTimer.unref?.(); - } + // Set when a wallclock bound fired. The abandoned (timed-out but still + // running) handler can hold ref'd sockets/timers that keep Bun's event loop + // alive after main() returns — so the finally must hard-exit after teardown + // on this path, or the timeout print is followed by an immortal process: + // the same zombie class, resurrected through the timeout door (adversarial + // review finding). + let wallclockTimedOut = false; try { - const ctx = await makeContext(engine, params); - const rawResult = await op.handler(ctx, params); + const { withTimeout, OperationTimeoutError } = await import('./core/timeout.ts'); + const wallclockMs = getCliOptions().timeoutMs ?? READ_OP_TIMEOUT_MS; + const onWallclockTimeout = (e: InstanceType) => { + const hint = getCliOptions().timeoutMs + ? '' + : ` (default ${e.ms}ms; pass --timeout=Ns to override)`; + console.error(`${e.label} timed out${hint}.`); + process.exitCode = 124; + wallclockTimedOut = true; + }; + + // Context build does DB I/O (resolveSourceId) and runs for EVERY op — + // a wedged pooler connection here would otherwise hang reads, writes, + // and admin alike with no bound at all (adversarial review finding). + let ctx: Awaited>; + try { + ctx = await withTimeout( + makeContext(engine, params), + wallclockMs, + `gbrain ${command}: context`, + ); + } catch (e: unknown) { + if (e instanceof OperationTimeoutError) { + onWallclockTimeout(e); + return; // the finally below still drains + disconnects, then exits + } + throw e; + } + + let rawResult: unknown; + if (op.scope === 'read') { + try { + rawResult = await withTimeout( + op.handler(ctx, params), + wallclockMs, + `gbrain ${command}`, + ); + } catch (e: unknown) { + if (e instanceof OperationTimeoutError) { + onWallclockTimeout(e); + return; // the finally below still drains + disconnects, then exits + } + throw e; + } + } else { + rawResult = await op.handler(ctx, params); + } // ENG-2 (renderer parity by data shape): JSON-round-trip the local-engine // path's return value so renderers see the same shape they'd see on the // routed path. Date → ISO string; bigint → string (postgres.js shape); @@ -396,7 +449,8 @@ async function main() { // STILL runs (drains every background-work sink + disconnects). A bare // process.exit(1) here would skip the finally → skip the drain + disconnect // (leaves facts/cache/eval-capture writes racing teardown). The finally's - // drain bounds teardown; the outer hard-deadline timer bounds a hung one. + // drain bounds teardown; the hard-deadline timer armed at teardown entry + // bounds a hung one. if (e instanceof OperationError) { console.error(`Error [${e.code}]: ${e.message}`); if (e.suggestion) console.error(` Fix: ${e.suggestion}`); @@ -412,11 +466,36 @@ async function main() { // DB logIngest gets the freshest live-engine window. 1s per-sink timeout: // read paths with no pending work pay the ~0ms fast path; capture/import // that DO enqueue pay up to 1s (+ facts shutdown grace) while in-flight - // Haiku finishes. The unref'd hard-deadline timer above is the backstop if - // disconnect or a lingering socket keeps Bun's loop alive. + // Haiku finishes. The unref'd hard-deadline timer armed here is the + // backstop if disconnect or a lingering socket keeps Bun's loop alive — + // armed at teardown entry (NOT before the op body; see the C13 comment + // above) so a slow-but-progressing op handler is never killed mid-flight. + if (shouldForceExitAfterMain()) { + forceExitTimer = setTimeout(() => { + console.warn( + `[cli] engine.disconnect() did not return within ${DISCONNECT_HARD_DEADLINE_MS}ms — force-exiting`, + ); + // v0.42.20.0 (codex): honor an exit code an errored op already set — + // a bare process.exit(0) here would mask a failed op as success if the + // drain/disconnect then hangs. + process.exit(process.exitCode ?? 0); + }, DISCONNECT_HARD_DEADLINE_MS); + // unref so the timer itself doesn't keep the event loop alive — only + // the actual pending work (PGLite WASM handle) does. Without unref, + // we'd block a clean exit by 10s on every successful CLI run. + forceExitTimer.unref?.(); + } await drainAllBackgroundWorkForCliExit({ timeoutMs: 1000 }); await engine.disconnect(); if (forceExitTimer) clearTimeout(forceExitTimer); + // Wallclock-timeout path: teardown is done, but the ABANDONED handler + // (withTimeout races, it does not cancel) can still hold ref'd sockets / + // SDK retry timers that keep Bun's event loop alive indefinitely. With + // the hard-deadline timer just cleared, nothing else bounds that — exit + // explicitly. Safe: drain + disconnect completed on the lines above. + if (wallclockTimedOut) { + process.exit(process.exitCode ?? 124); + } } } diff --git a/src/commands/autopilot.ts b/src/commands/autopilot.ts index d5831df74..7ac05cfbc 100644 --- a/src/commands/autopilot.ts +++ b/src/commands/autopilot.ts @@ -529,8 +529,13 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) { autopilotReconnectFails = 0; // reset on success } catch (probeErr) { try { - await engine.disconnect(); - await (engine as any).connect?.(); + // #2034: use reconnect() — it restores the config captured at connect() + // and avoids the null-connection window. The previous + // `disconnect()` + bare `connect()` lost the config (throwing + // `database_url undefined` on every retry → FATAL restart-loop on any + // transient DB blip) AND tore down the pool postgres.js can otherwise + // self-heal. + await engine.reconnect({ error: probeErr }); autopilotReconnectFails = 0; } catch (e) { logError('reconnect', e); diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 99fcdda8b..028a8ab4d 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -510,6 +510,33 @@ export async function doctorReportRemote(engine: BrainEngine): Promise { const ageH = Math.floor(s.age_ms / 3600_000); - const source = s.id.startsWith('gbrain-sync:') ? s.id.slice('gbrain-sync:'.length) : null; - const breakHint = source ? `gbrain sync --break-lock --source ${source}` : `gbrain sync --break-lock`; + let breakHint = 'gbrain doctor'; + if (s.id.startsWith('gbrain-sync:')) { + breakHint = `gbrain sync --break-lock --source ${s.id.slice('gbrain-sync:'.length)}`; + } else if (s.id.startsWith('gbrain-cycle:')) { + breakHint = `gbrain dream --break-lock --source ${s.id.slice('gbrain-cycle:'.length)}`; + } else if (s.id === 'gbrain-cycle') { + breakHint = 'gbrain dream --break-lock'; + } return ` ${s.id} (pid ${s.holder_pid} on ${s.holder_host}, age ${ageH}h) → ${breakHint}`; }); const tail = stale.length > 10 ? ` ... and ${stale.length - 10} more.` : null; @@ -2614,7 +2647,7 @@ async function checkEmbeddingEnvOverride(engine: BrainEngine): Promise { }; } -async function checkSubagentCapability(engine: BrainEngine): Promise { +export async function checkSubagentCapability(engine: BrainEngine): Promise { try { const { classifyCapabilities } = await import('../core/ai/capabilities.ts'); const tierSubagent = await engine.getConfig('models.tier.subagent'); @@ -2675,8 +2708,11 @@ async function checkSubagentCapability(engine: BrainEngine): Promise { const { loadConfig } = await import('../core/config.ts'); const cfg = loadConfig(); const chatModel = cfg?.chat_model; + const gatewayLoopRaw = await engine.getConfig('agent.use_gateway_loop').catch(() => null); + const gatewayLoopEnabled = typeof gatewayLoopRaw === 'string' + && ['true', '1', 'yes', 'on'].includes(gatewayLoopRaw.trim().toLowerCase()); const { isAnthropicProvider } = await import('../core/model-config.ts'); - if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY) { + if (chatModel && !isAnthropicProvider(chatModel) && !process.env.ANTHROPIC_API_KEY && !gatewayLoopEnabled) { return { name: 'subagent_capability', status: 'warn', @@ -5610,8 +5646,29 @@ export async function buildChecks( "SELECT COUNT(*)::int AS count FROM pages WHERE type IN ('entity', 'person', 'company', 'organization')", ))[0]?.count ?? 0; - const linkPct = ((health.link_coverage ?? 0) * 100).toFixed(0); - const timelinePct = ((health.timeline_coverage ?? 0) * 100).toFixed(0); + // Compute coverage against eligible entities only — exclude test fixtures + // (`tools/gbrain/test/*`) and template stubs (`templates/new-person`) so + // that brains seeded only with code sources don't get spurious warnings + // about missing link/timeline coverage on pages that are test fixtures, not + // real knowledge entities. + const eligibleStats = (await engine.executeRaw<{ entities: number; linked_from: number; timeline: number }>( + `WITH eligible AS ( + SELECT id FROM pages + WHERE type IN ('entity','person','company','organization') + AND slug NOT LIKE 'tools/gbrain/test/%' + AND slug <> 'templates/new-person' + ) + SELECT + (SELECT count(*)::int FROM eligible) AS entities, + (SELECT count(DISTINCT from_page_id)::int FROM links WHERE from_page_id IN (SELECT id FROM eligible)) AS linked_from, + (SELECT count(DISTINCT page_id)::int FROM timeline_entries WHERE page_id IN (SELECT id FROM eligible)) AS timeline`, + ))[0] ?? { entities: entityCount, linked_from: 0, timeline: 0 }; + + const eligibleEntityCount = Number(eligibleStats.entities ?? entityCount); + const linkCoverage = eligibleEntityCount > 0 ? Number(eligibleStats.linked_from ?? 0) / eligibleEntityCount : 0; + const timelineCoverage = eligibleEntityCount > 0 ? Number(eligibleStats.timeline ?? 0) / eligibleEntityCount : 0; + const linkPct = (linkCoverage * 100).toFixed(0); + const timelinePct = (timelineCoverage * 100).toFixed(0); if (entityCount === 0) { // Markdown-only / journal / wiki brain — no entity pages to compute // coverage against. Coverage formula is structurally inapplicable. @@ -5620,13 +5677,19 @@ export async function buildChecks( status: 'ok', message: 'No entity pages — graph_coverage not applicable (markdown-only brain)', }); - } else if ((health.link_coverage ?? 0) >= 0.5 && (health.timeline_coverage ?? 0) >= 0.5) { + } else if (eligibleEntityCount === 0) { + checks.push({ + name: 'graph_coverage', + status: 'ok', + message: `Only code/test fixture entity pages found (${entityCount}); graph_coverage not applicable`, + }); + } else if (linkCoverage >= 0.5 && timelineCoverage >= 0.5) { checks.push({ name: 'graph_coverage', status: 'ok', message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}%` }); } else { checks.push({ name: 'graph_coverage', status: 'warn', - message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${entityCount} entity pages). Run: gbrain extract all`, + message: `Entity link coverage ${linkPct}%, timeline ${timelinePct}% (${eligibleEntityCount} entity pages). Run: gbrain extract all`, }); } @@ -6162,12 +6225,29 @@ export async function buildChecks( .slice(0, 3) .map(([s, n]) => `${s}=${n}`) .join(', '); + // Audit events are evidence, not automatically breakage. A large code + // source can legitimately emit many WARN events (oversize/markup-heavy) + // while remaining searchable and intentionally flagged. Fail on hard + // dispositions (content actually blocked or hidden); warn on soft + // dispositions or volume. This keeps doctor from treating expected + // code-corpus telemetry as an unhealthy brain. + // + // v0.42 renamed the hard path: a rejected page emits `reject` and a + // quarantined (hidden) junk page emits `quarantine`; `hard_block` is now + // only the pre-v0.42 legacy alias. Counting `hard_block` alone let fresh + // junk-ingest evidence (`reject`/`quarantine`) clear as `ok` whenever + // fewer than 10 events landed. `flag` is a warn disposition (still + // searchable, agent warned on retrieval), so it joins `soft_block`. + const hardBlocked = + summary.by_type.hard_block + summary.by_type.reject + summary.by_type.quarantine; + const softBlocked = summary.by_type.soft_block + summary.by_type.flag; const status: 'ok' | 'warn' | 'fail' = - events.length >= 100 ? 'fail' : events.length >= 10 ? 'warn' : 'ok'; + hardBlocked > 0 ? 'fail' : + (softBlocked > 0 || events.length >= 10) ? 'warn' : 'ok'; checks.push({ name: 'content_sanity_audit_recent', status, - message: `${events.length} events (hard=${summary.by_type.hard_block} soft=${summary.by_type.soft_block} warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`, + message: `${events.length} events (hard=${hardBlocked} [hard_block=${summary.by_type.hard_block} reject=${summary.by_type.reject} quarantine=${summary.by_type.quarantine}] soft=${softBlocked} [soft_block=${summary.by_type.soft_block} flag=${summary.by_type.flag}] warn=${summary.by_type.warn})${topPatterns ? ', patterns: ' + topPatterns : ''}${topSources ? ', sources: ' + topSources : ''}. (Local audit only — multi-host operators set GBRAIN_AUDIT_DIR.)`, }); } } catch (err) { @@ -7145,7 +7225,10 @@ export async function runDoctor( } catch { /* best-effort */ } } - process.exit(hasFail ? 1 : 0); + // Use process.exitCode instead of process.exit() so cleanup handlers + // (e.g. Bun unload events, open database connections) still run before + // the process terminates. process.exit() is a hard kill that bypasses them. + process.exitCode = hasFail ? 1 : 0; } // --------------------------------------------------------------------------- diff --git a/src/commands/extract.ts b/src/commands/extract.ts index 7ef9e337c..e6b5a4077 100644 --- a/src/commands/extract.ts +++ b/src/commands/extract.ts @@ -855,6 +855,17 @@ Status (v0.42): if (!jsonMode) { console.log(`Timeline from meetings: ${r.entries_created} entries on ${r.entities_touched} entity pages from ${r.meetings_scanned} meetings`); } + // #2057 (codex): batch failures are no longer swallowed silently — make + // them visible at the command surface (and non-zero exit) instead of + // printing a clean "N entries" success over failed inserts. + if (r.batch_errors > 0) { + console.error( + `[extract timeline] ${r.batch_errors} batch(es) failed to insert` + + (r.first_batch_error ? ` (first error: ${r.first_batch_error})` : '') + + ` — timeline is incomplete.`, + ); + process.exitCode = 1; + } } else if (byMention || ner) { // v0.41.18.0 (T7): combined --by-mention + --ner walk shares one // gazetteer; saves an entire pass on big brains. When only one diff --git a/src/commands/import.ts b/src/commands/import.ts index 1d4575fcb..d9b160a4d 100644 --- a/src/commands/import.ts +++ b/src/commands/import.ts @@ -11,6 +11,7 @@ import { isCodeFilePath, isMarkdownFilePath, isImageFilePath as isImageFilePathFromSync, + pruneDir, type SyncStrategy, } from '../core/sync.ts'; import { sortNewestFirst } from '../core/sort-newest-first.ts'; @@ -512,6 +513,51 @@ function isCollectibleForWalker( } } +/** + * Git-aware fast path for `collectSyncableFiles`. Returns the strategy-filtered + * list of syncable files when `dir` is inside a git work tree (paths absolute, + * sorted), or `null` when `dir` is not a git repo / git is unavailable — in + * which case the caller falls back to the recursive FS walk. + * + * Honors `.gitignore` (the whole point): `git ls-files --cached --others + * --exclude-standard` lists tracked + untracked-not-ignored files, so vendored + * / build / generated trees never reach the importer. `-z` (NUL-delimited) + * survives paths with spaces/newlines. Each path is lstat-checked to preserve + * the walker's no-symlink policy and to drop submodule gitlinks (which surface + * as a single non-regular entry). + */ +function gitListSyncableFiles( + dir: string, + strategy: SyncStrategy, + multimodalOn: boolean, +): string[] | null { + let stdout: string; + try { + stdout = execFileSync( + 'git', + ['-C', dir, 'ls-files', '--cached', '--others', '--exclude-standard', '-z'], + { encoding: 'utf8', maxBuffer: 512 * 1024 * 1024, stdio: ['ignore', 'pipe', 'ignore'] }, + ); + } catch { + return null; // not a git work tree, or git not on PATH → FS-walk fallback + } + const files: string[] = []; + for (const rel of stdout.split('\0')) { + if (!rel) continue; + if (!isCollectibleForWalker(rel, strategy, multimodalOn)) continue; + const full = join(dir, rel); + let st; + try { + st = lstatSync(full); + } catch { + continue; // ls-files raced a deletion, or unreadable + } + if (st.isSymbolicLink() || !st.isFile()) continue; + files.push(full); + } + return files.sort(); +} + /** * v0.31.2 (codex C4 + C5 + C8): unified walker with five hardenings: * @@ -532,6 +578,19 @@ function isCollectibleForWalker( export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): string[] { const strategy: SyncStrategy = opts.strategy ?? 'markdown'; const multimodalOn = process.env.GBRAIN_EMBEDDING_MULTIMODAL === 'true'; + + // v0.42.x (#1159 --respect-gitignore / #1483 .gbrainignore): when `dir` is a + // git work tree, enumerate via `git ls-files` so the walk honors + // `.gitignore`. Pre-fix the recursive FS walk below descended into every + // git-ignored tree — `vendor/` (PHP Composer), `storage/`, `public/build/`, + // etc. — so a Laravel/PHP repo's `--strategy code` sync tried to import ~50k + // dependency/build files (and bloated DB + embedding cost on any repo with + // vendored data/fixtures). `--cached --others --exclude-standard` = tracked + // PLUS untracked-not-ignored, so uncommitted source is still indexed. Non-git + // dirs (or git unavailable) fall through to the FS walk below. + const gitFiles = gitListSyncableFiles(dir, strategy, multimodalOn); + if (gitFiles) return gitFiles; + const maxDepth = resolveMaxWalkDepth(); const visitedInodes = new Map(); const files: string[] = []; @@ -548,11 +607,11 @@ export function collectSyncableFiles(dir: string, opts: CollectOpts = {}): strin return; } for (const entry of entries) { - // Skip hidden dirs (.git, .claude, .raw, etc.) and `node_modules`/`ops`. - // Same set the legacy walkers honored, surfaced once at the top of - // every iteration. - if (entry.startsWith('.')) continue; - if (entry === 'node_modules' || entry === 'ops') continue; + // Descent-time prune through the canonical gate (single source of truth + // in core/sync.ts) instead of a hand-maintained inline list that drifted + // from it. Skips hidden dirs (`.git`, `.raw`, etc.), `node_modules`, + // `vendor`, `dist`, `build`, `venv` (#2020), `ops`, and git submodules. + if (!pruneDir(entry, d)) continue; const full = join(d, entry); let stat; diff --git a/src/commands/init.ts b/src/commands/init.ts index 08cf8432c..69e3d5b93 100644 --- a/src/commands/init.ts +++ b/src/commands/init.ts @@ -6,7 +6,7 @@ import { homedir } from 'os'; const __filename = fileURLToPath(import.meta.url); const __dirname = dirname(__filename); -import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, type GBrainConfig } from '../core/config.ts'; +import { saveConfig, loadConfig, loadConfigFileOnly, toEngineConfig, gbrainPath, configPath, isThinClient, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts'; import { createEngine } from '../core/engine-factory.ts'; import { discoverOAuth, mintClientCredentialsToken, smokeTestMcp } from '../core/remote-mcp-probe.ts'; import { runInitEmbedCheck } from '../core/init-embed-check.ts'; @@ -133,7 +133,11 @@ export async function runInit(args: string[]) { if (manualUrl) { databaseUrl = manualUrl; } else if (isNonInteractive) { - const envUrl = process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL; + // effectiveEnvDatabaseUrl applies the #427 guard: a DATABASE_URL that Bun + // auto-loaded from a .env in cwd must not seed a brain config — that is + // the "init inside a web-app checkout writes the app's DB into + // ~/.gbrain/config.json" failure mode. + const envUrl = effectiveEnvDatabaseUrl(); if (envUrl) { databaseUrl = envUrl; } else { diff --git a/src/commands/migrate-engine.ts b/src/commands/migrate-engine.ts index 5ce3027a1..29d04e4c4 100644 --- a/src/commands/migrate-engine.ts +++ b/src/commands/migrate-engine.ts @@ -8,7 +8,7 @@ */ import { createEngine } from '../core/engine-factory.ts'; -import { loadConfig, saveConfig, toEngineConfig, gbrainPath, type GBrainConfig } from '../core/config.ts'; +import { loadConfig, saveConfig, toEngineConfig, gbrainPath, effectiveEnvDatabaseUrl, type GBrainConfig } from '../core/config.ts'; import type { BrainEngine } from '../core/engine.ts'; import type { EngineConfig } from '../core/types.ts'; import { writeFileSync, readFileSync, existsSync, unlinkSync } from 'fs'; @@ -91,7 +91,8 @@ export async function runMigrateEngine(sourceEngine: BrainEngine, args: string[] // Build target config const targetConfig: EngineConfig = { engine: opts.targetEngine }; if (opts.targetEngine === 'postgres') { - targetConfig.database_url = opts.targetUrl || process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL; + // #427 guard: don't let a cwd-.env DATABASE_URL become a migration target. + targetConfig.database_url = opts.targetUrl || effectiveEnvDatabaseUrl(); if (!targetConfig.database_url) { console.error('Target is Supabase but no connection string provided. Use: --url '); process.exit(1); diff --git a/src/core/ai/gateway.ts b/src/core/ai/gateway.ts index 21c5ce57a..cf41b1ab0 100644 --- a/src/core/ai/gateway.ts +++ b/src/core/ai/gateway.ts @@ -797,6 +797,26 @@ export function isAvailable(touchpoint: TouchpointKind, modelOverride?: string): // ---- Embedding ---- +/** + * Carries the asymmetric-embedding `input_type` ('query' | 'document') + * across the AI SDK boundary, from embedSubBatch() to the per-recipe fetch + * shims below (#1400). + * + * Why this exists: `dimsProviderOptions()` correctly emits `input_type` + * into `providerOptions.openaiCompatible` for asymmetric models (ZE + * zembed-1, Voyage v3+), but the AI SDK's openai-compatible adapter + * validates providerOptions against a fixed schema (`dimensions`, `user`) + * and silently DROPS every other field before building the wire body. By + * the time a compat shim sees the request, the threaded 'query' is gone — + * so every embedQuery() was encoded document-side and asymmetric retrieval + * silently collapsed to surface-token overlap. + * + * embedSubBatch() populates the store only when providerOptions actually + * threads an input_type; shims that need a default (ZE) apply their own. + * Same module-level ALS pattern as `__budgetStore` below. + */ +const __embedInputTypeStore = new AsyncLocalStorage<'query' | 'document'>(); + /** * Voyage AI compatibility shim. Voyage's `/v1/embeddings` endpoint is OpenAI-shaped * but diverges on two parameters: @@ -835,6 +855,15 @@ const voyageCompatFetch = (async (input: RequestInfo | URL, init?: RequestInit) if (typeof dims === 'number') parsed.output_dimension = dims; mutated = true; } + // Recover the SDK-stripped input_type (#1400) — opt-in, mirroring + // the dims.ts Voyage branch: inject only when a caller actually + // threaded one; when absent, the field stays off the wire + // (pre-v0.35.0.0 behavior preserved). + const threadedInputType = __embedInputTypeStore.getStore(); + if (threadedInputType !== undefined && parsed.input_type === undefined) { + parsed.input_type = threadedInputType; + mutated = true; + } if (mutated) { const newBody = JSON.stringify(parsed); // Drop Content-Length so fetch recomputes from the new body. @@ -1005,10 +1034,13 @@ const zeroEntropyCompatFetch = (async (input: RequestInfo | URL, init?: RequestI parsed.encoding_format = 'float'; mutated = true; } - // Default input_type when caller didn't thread one (document-side - // embedding is the correct default for ingest paths). + // Recover the SDK-stripped input_type (#1400): the threaded value + // arrives via __embedInputTypeStore because the openai-compatible + // adapter drops it from providerOptions before building the body. + // Document-side default preserved for paths that didn't thread one + // (ingest). if (parsed.input_type === undefined) { - parsed.input_type = 'document'; + parsed.input_type = __embedInputTypeStore.getStore() ?? 'document'; mutated = true; } if (mutated) { @@ -1096,6 +1128,56 @@ const zeroEntropyCompatFetch = (async (input: RequestInfo | URL, init?: RequestI } }) as unknown as typeof fetch; +/** + * Generic asymmetric-embedding shim for openai-compatible recipes that + * ship no compat fetch of their own (llama-server, litellm, ollama, ...). + * Those deployments are the canonical local/proxy paths for serving + * asymmetric models (e.g. a zembed-1 behind llama.cpp, vLLM, or an + * OpenAI-compatible proxy), and they need the same `input_type` signal the + * hosted ZE endpoint gets — the serving layer can't apply query-side vs + * document-side encoding without it. + * + * Strictly opt-in at the wire level: when no input_type was threaded (the + * model isn't asymmetric — dims.ts threads it by model id, never for + * Azure/DashScope/Zhipu-style fixed models — or the caller didn't ask), + * the request passes through byte-identical. When one WAS threaded + * (recovered from __embedInputTypeStore — see #1400), inject it into the + * JSON body; OpenAI-compatible servers that don't understand the field + * ignore unknown body keys (llama-server does), and asymmetric-aware + * servers/proxies read it. URL + response shape are untouched — these + * endpoints are already OpenAI-shaped. Recipes with their own compat + * fetch (voyage, zeroentropyai, azure) never reach this shim: the + * `compat.fetch ??` precedence in instantiateEmbedding wins. + */ +const openAICompatAsymmetricFetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const threadedInputType = __embedInputTypeStore.getStore(); + if (threadedInputType === undefined) return fetch(input as any, init); + // Inject only on the string-URL + string-body shape the AI SDK actually + // sends. A Request-shaped input may carry its body on the Request object + // itself, where a rebuild would silently drop it — pass it through + // untouched instead (unreachable via the SDK today; preserves the + // byte-identical contract if it ever becomes reachable). + if (typeof input !== 'string' && !(input instanceof URL)) { + return fetch(input as any, init); + } + let baseInit: RequestInit = init ?? {}; + if (baseInit.body && typeof baseInit.body === 'string') { + try { + const parsed = JSON.parse(baseInit.body); + if (parsed && typeof parsed === 'object' && parsed.input_type === undefined) { + parsed.input_type = threadedInputType; + // Drop Content-Length so fetch recomputes from the new body. + const headers = new Headers(baseInit.headers ?? {}); + headers.delete('content-length'); + baseInit = { ...baseInit, body: JSON.stringify(parsed), headers }; + } + } catch { + // Body wasn't JSON — pass through untouched. + } + } + return fetch(typeof input === 'string' ? input : input.toString(), baseInit); +}) as unknown as typeof fetch; + async function resolveEmbeddingProvider(modelStr: string): Promise<{ model: any; recipe: Recipe; modelId: string }> { const { parsed, recipe } = resolveRecipe(modelStr); assertTouchpoint(recipe, 'embedding', parsed.modelId, getExtendedModelsForProvider(parsed.providerId)); @@ -1149,15 +1231,19 @@ function instantiateEmbedding(recipe: Recipe, modelId: string, cfg: AIGatewayCon // wrapper via resolveOpenAICompatConfig. Azure recipes ship their own // fetch (api-version splice); voyage doesn't — use voyageCompatFetch. // ZeroEntropy needs zeroEntropyCompatFetch (URL path + body input_type - // + response shape rewrite + OOM caps). Same per-recipe-id branch - // pattern as voyage so adding a third compat shim is one more case. + // + response shape rewrite + OOM caps). Every other openai-compat + // recipe (llama-server, litellm, ollama, ...) falls through to + // openAICompatAsymmetricFetch so the threaded input_type reaches + // asymmetric models there too (#1400) — a strict pass-through when no + // input_type was threaded, so symmetric deployments see zero wire + // change. const fetchWrapper = compat.fetch ?? (recipe.id === 'voyage' ? voyageCompatFetch : recipe.id === 'zeroentropyai' ? zeroEntropyCompatFetch - : undefined); + : openAICompatAsymmetricFetch); const client = createOpenAICompatible({ name: recipe.id, baseURL: compat.baseURL, @@ -1472,7 +1558,7 @@ async function embedSubBatch( opts?: EmbedOpts, ): Promise { try { - const result = await _embedTransport({ + const callTransport = () => _embedTransport({ model, values: texts, providerOptions: providerOpts, @@ -1483,6 +1569,15 @@ async function embedSubBatch( abortSignal: withDefaultTimeout(opts?.abortSignal, AI_EMBED_TIMEOUT_MS), ...(opts?.maxRetries !== undefined && { maxRetries: opts.maxRetries }), }); + // Carry the threaded input_type across the SDK boundary via + // __embedInputTypeStore (the adapter strips it from providerOptions — + // see the store's doc comment). Populated only when dimsProviderOptions + // actually emitted one, so non-asymmetric paths run store-empty and the + // fetch shims leave their wire bodies untouched. + const threadedInputType = providerOpts?.openaiCompatible?.input_type; + const result = await (threadedInputType === 'query' || threadedInputType === 'document' + ? __embedInputTypeStore.run(threadedInputType, callTransport) + : callTransport()); if (!Array.isArray(result.embeddings) || result.embeddings.length !== texts.length) { throw new AIConfigError( diff --git a/src/core/config.ts b/src/core/config.ts index ff1ee2273..70ddca5b7 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -389,6 +389,96 @@ export function loadConfigFileOnly(): GBrainConfig | null { } } +/** + * #427 guard — DATABASE_URL hijack via Bun's cwd .env auto-load. + * + * Bun merges `.env` files from the process cwd into process.env before any + * user code runs. For a globally-installed tool that is a footgun: running + * gbrain inside any checkout whose `.env` defines DATABASE_URL (Next.js, + * Hono, Supabase, most web apps) silently retargets the brain at that app's + * database. Reads hit the wrong DB; `apply-migrations` can write gbrain's + * schema — including its DDL event trigger — into a production app database + * (see the v0.42.8 report on #427). + * + * Bun gives no way to ask which vars came from a .env file (the merge + * happens before module load), so we re-parse the .env files Bun auto-loads + * from cwd and treat DATABASE_URL as "not operator-provided" when its value + * matches one of them. Deliberate overrides still work two ways: + * - GBRAIN_DATABASE_URL: namespaced to this tool, never auto-ignored; + * - exporting DATABASE_URL in the shell: exported vars win over .env in + * Bun, and a deliberate export that happens to EQUAL the cwd .env value + * would have selected the same database anyway — ignoring it changes + * the outcome only by honoring the brain config, which is the safe + * reading of ambiguous intent. + * + * The file list is a superset of Bun's auto-load set across NODE_ENV values + * so the guard doesn't depend on replicating Bun's exact selection logic. + */ +const CWD_DOTENV_FILES = ['.env', '.env.local', '.env.development', '.env.production', '.env.test']; + +/** + * All values assigned to `key` across the .env files in `dir`. Collecting + * every assignment (rather than emulating override order) keeps the guard + * independent of dotenv precedence rules — a match against ANY assignment + * means the value is file-origin. Exported for tests. + */ +export function dotenvValuesForKey(key: string, dir: string = process.cwd()): Set { + const values = new Set(); + const assignment = /^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$/; + for (const name of CWD_DOTENV_FILES) { + let content: string; + try { + content = readFileSync(join(dir, name), 'utf-8'); + } catch { + continue; // missing/unreadable file — nothing to guard against + } + for (const rawLine of content.split('\n')) { + const line = rawLine.trim(); + if (!line || line.startsWith('#')) continue; + const m = line.match(assignment); + if (!m || m[1] !== key) continue; + let v = m[2].trim(); + if ((v.startsWith('"') && v.endsWith('"') && v.length >= 2) || + (v.startsWith("'") && v.endsWith("'") && v.length >= 2)) { + v = v.slice(1, -1); + } else { + const hash = v.indexOf(' #'); + if (hash !== -1) v = v.slice(0, hash).trim(); + } + if (v) values.add(v); + } + } + return values; +} + +let warnedCwdEnvDbUrlIgnored = false; + +/** + * The env-provided DB URL gbrain should honor, with the #427 guard applied: + * a DATABASE_URL whose value matches an assignment in a cwd .env file is + * treated as belonging to the project in cwd, not to gbrain, and ignored + * with a one-time stderr notice. GBRAIN_DATABASE_URL is always honored. + * `dir` is injectable for tests; callers use the default. + */ +export function effectiveEnvDatabaseUrl(dir: string = process.cwd()): string | undefined { + if (process.env.GBRAIN_DATABASE_URL) return process.env.GBRAIN_DATABASE_URL; + const url = process.env.DATABASE_URL; + if (!url) return undefined; + if (dotenvValuesForKey('DATABASE_URL', dir).has(url)) { + if (!warnedCwdEnvDbUrlIgnored) { + warnedCwdEnvDbUrlIgnored = true; + console.warn( + '[config] Ignoring DATABASE_URL auto-loaded by Bun from a .env file in the current ' + + 'directory — it belongs to the project here, not to gbrain. Using the engine from ' + + '~/.gbrain/config.json instead. To point gbrain at that database deliberately, set ' + + 'GBRAIN_DATABASE_URL.', + ); + } + return undefined; + } + return url; +} + export function loadConfig(): GBrainConfig | null { let fileConfig: GBrainConfig | null = null; try { @@ -397,8 +487,8 @@ export function loadConfig(): GBrainConfig | null { fileConfig = migrateLegacyEmbeddingConfig(parsed) as unknown as GBrainConfig; } catch { /* no config file */ } - // Try env vars - const dbUrl = process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL; + // Try env vars (cwd-.env-origin DATABASE_URL excluded — see #427 guard above) + const dbUrl = effectiveEnvDatabaseUrl(); if (!fileConfig && !dbUrl) return null; @@ -933,7 +1023,12 @@ export function gbrainPath(...segments: string[]): string { */ export function getDbUrlSource(): DbUrlSource { if (process.env.GBRAIN_DATABASE_URL) return 'env:GBRAIN_DATABASE_URL'; - if (process.env.DATABASE_URL) return 'env:DATABASE_URL'; + // Same #427 guard as loadConfig: a DATABASE_URL that Bun auto-loaded from + // a cwd .env file is not an operator-provided source. Keeping this in + // lockstep with loadConfig matters because doctor uses this to tell the + // user where the URL came from — reporting env:DATABASE_URL while + // loadConfig ignores it would send them debugging the wrong layer. + if (effectiveEnvDatabaseUrl()) return 'env:DATABASE_URL'; if (!existsSync(configPath())) return null; try { const raw = readFileSync(configPath(), 'utf-8'); diff --git a/src/core/cycle.ts b/src/core/cycle.ts index a4dce8ffe..116e5420f 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -1033,6 +1033,21 @@ async function runPhaseExtractFacts( || result.phantomsSkippedDrift) ? `, ${result.phantomsRedirected} phantom(s) redirected (${result.phantomsAmbiguous} ambiguous, ${result.phantomsSkippedDrift} drift-skipped)` : ''; + // #1928: a reconcile that deletes far more facts than it reinserts is the + // signature of the conversation-facts wipe (factsDeleted 1829, inserted 0 + // read as a no-op "ok" before this guard). Surface net deletion above a + // floor as `warn` so the daily report and doctor make it visible instead + // of it reading like a clean run. + const NET_DELETION_WARN_FLOOR = 50; + const netDeleted = result.factsDeleted - result.factsInserted; + const netDeletionWarn = netDeleted >= NET_DELETION_WARN_FLOOR; + if (netDeletionWarn) { + result.warnings.push( + `net_fact_deletion: reconcile removed ${result.factsDeleted} fact(s) and ` + + `reinserted ${result.factsInserted} (net -${netDeleted}). If unexpected, a ` + + `destructive full walk may have wiped non-fence facts (#1928).`, + ); + } return { phase: 'extract_facts', status: result.warnings.length > 0 ? 'warn' : 'ok', @@ -1574,6 +1589,11 @@ export async function runCycle( // and which slugs synthesize wrote so recompute_emotional_weight can // pick up the union of (sync ∪ synthesize) for v0.29 incremental mode. let syncPagesAffected: string[] | undefined; + // #1928 (codex): true ONLY when the sync phase actually RAN its work (not + // when it was skipped for no-engine / no-brainDir). The destructive + // extract_facts guard keys off this so a SKIPPED sync still allows a + // legitimate full reconcile — only a sync that ran and failed suppresses it. + let syncAttempted = false; let synthesizeWrittenSlugs: string[] | undefined; if (phases.includes('sync')) { checkAborted(opts.signal); @@ -1589,6 +1609,7 @@ export async function runCycle( phaseResults.push(skipNoBrainDir('sync')); } else { progress.start('cycle.sync'); + syncAttempted = true; // sync ran its work; undefined pagesAffected now means failure const { result, duration_ms } = await timePhase(() => runPhaseSync(engine, brainDir, dryRun, pull, phases.includes('extract'))); result.duration_ms = duration_ms; // Capture changed slugs for incremental extract. @@ -1688,8 +1709,22 @@ export async function runCycle( // the sources table doesn't recognize this brainDir (pre-multi- // source installs). const xfSourceId = cycleSourceId ?? 'default'; + // #1928: extract_facts is DESTRUCTIVE (wipe-and-reinsert per page). It + // must NOT inherit the "sync failed ⇒ undefined ⇒ full walk" fallback + // that's safe for link/timeline extract. When the sync phase RAN but + // failed, syncPagesAffected is undefined (a successful no-op sync + // returns []). In that case pass [] (no-op) so a lock-contention or + // transient sync failure can't escalate into a brain-wide fact wipe. + // undefined still reaches here (intended full reconcile) when the sync + // phase was absent OR skipped (no engine / no brainDir — extract_facts + // supports no-brainDir DB reconciliation). Only a sync that actually + // RAN and came back with undefined pagesAffected is a real failure + // (#1928, codex: keying off phases.includes('sync') wrongly suppressed + // the skipped-sync full reconcile). + const syncRanButFailed = syncAttempted && syncPagesAffected === undefined; + const xfSlugs = syncRanButFailed ? [] : syncPagesAffected; const { result, duration_ms } = await timePhase(() => - runPhaseExtractFacts(engine, brainDir, xfSourceId, dryRun, syncPagesAffected, opts.signal)); + runPhaseExtractFacts(engine, brainDir, xfSourceId, dryRun, xfSlugs, opts.signal)); result.duration_ms = duration_ms; phaseResults.push(result); progress.finish(); diff --git a/src/core/cycle/extract-facts.ts b/src/core/cycle/extract-facts.ts index 306e58a5d..39b15d9ad 100644 --- a/src/core/cycle/extract-facts.ts +++ b/src/core/cycle/extract-facts.ts @@ -222,10 +222,15 @@ export async function runExtractFacts( if (opts.dryRun) continue; - // Wipe-and-reinsert per page. The deleteFactsForPage call targets - // source_markdown_slug = slug only, so NULL-source_markdown_slug - // legacy rows survive (the partial-UNIQUE-index keyspace). - const deleted = await engine.deleteFactsForPage(slug, sourceId); + // Wipe-and-reinsert per page. The delete targets source_markdown_slug = + // slug only, so NULL-source_markdown_slug legacy rows survive (the + // partial-UNIQUE-index keyspace). #1928: `cli:`-origin facts (conversation + // facts from extract-conversation-facts) are NOT fence-owned — the page + // carries no `## Facts` fence to recreate them — so they MUST survive this + // reconcile. Exclude them from the wipe. + const deleted = await engine.deleteFactsForPage(slug, sourceId, { + excludeSourcePrefixes: ['cli:'], + }); result.factsDeleted += deleted.deleted; if (parsed.facts.length === 0) continue; diff --git a/src/core/doctor-categories.ts b/src/core/doctor-categories.ts index 1bbf3b06e..e8adebb71 100644 --- a/src/core/doctor-categories.ts +++ b/src/core/doctor-categories.ts @@ -173,6 +173,7 @@ export const META_CHECK_NAMES: ReadonlySet = new Set([ 'schema_pack_source_drift', 'schema_version', 'slug_fallback_audit', + 'timeline_dedup_index', 'upgrade_errors', ]); diff --git a/src/core/embedding-dim-check.ts b/src/core/embedding-dim-check.ts index 4ca542da2..220bcca05 100644 --- a/src/core/embedding-dim-check.ts +++ b/src/core/embedding-dim-check.ts @@ -217,8 +217,10 @@ export function embeddingMismatchMessage(opts: EmbeddingMismatchOpts): string { ``, ` BEGIN;`, ` DROP INDEX IF EXISTS idx_chunks_embedding;`, - ` ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(${requestedDims});`, + ` -- NULL embeddings BEFORE the alter: pgvector refuses to cast existing`, + ` -- vectors across dimensions and aborts the transaction. NULLs cast fine.`, ` UPDATE content_chunks SET embedding = NULL, embedded_at = NULL;`, + ` ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(${requestedDims});`, ` ${reindexLine.split('\n').join('\n ')}`, ` COMMIT;`, ``, @@ -573,11 +575,25 @@ export function buildFactsAlterRecipe( ): string { const opclass = columnType === 'halfvec' ? 'halfvec_cosine_ops' : 'vector_cosine_ops'; const targetType = columnType === 'halfvec' ? `halfvec(${configuredDims})` : `vector(${configuredDims})`; + const dimsChanged = columnDims !== configuredDims; return [ `-- ALTER ${columnType}(${columnDims}) → ${columnType}(${configuredDims}) on indexed column.`, `-- HOLD a maintenance window: this rewrites every row's embedding.`, `-- Coordinate with any active extract-conversation-facts backfill.`, `DROP INDEX IF EXISTS idx_facts_embedding_hnsw;`, + // Same-dim type swaps (halfvec <-> vector) cast row data losslessly and + // MUST keep it. Cross-dimension changes are different: pgvector refuses + // to cast existing vectors across dimensions ("expected N dimensions, + // not M") and aborts the transaction — and the old-space vectors are + // unusable at the new width anyway. NULL them first; the facts pipeline + // re-embeds on the next write. + ...(dimsChanged + ? [ + `-- Dimension change: NULL embeddings BEFORE the alter — pgvector`, + `-- refuses cross-dimension casts and aborts the transaction.`, + `UPDATE facts SET embedding = NULL;`, + ] + : []), `ALTER TABLE facts ALTER COLUMN embedding TYPE ${targetType}`, ` USING embedding::${targetType};`, `CREATE INDEX idx_facts_embedding_hnsw`, diff --git a/src/core/engine.ts b/src/core/engine.ts index 0f285d498..075870269 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -650,6 +650,15 @@ export interface BrainEngine { // Lifecycle connect(config: EngineConfig): Promise; disconnect(): Promise; + /** + * Recover a dropped connection using the config captured at the last + * `connect()`. Callers (autopilot health probe, batchRetry) MUST use this + * instead of `disconnect()` + bare `connect()`: the latter loses the config + * (#2034 — a bare `connect()` with no args throws `database_url undefined` + * forever) AND opens a null-connection window. Implemented on BOTH engines + * for parity so the call is never a silent no-op. + */ + reconnect(ctx?: { error?: unknown }): Promise; initSchema(): Promise; transaction(fn: (engine: BrainEngine) => Promise): Promise; /** @@ -1678,7 +1687,20 @@ export interface BrainEngine { * until the v0_32_2 migration backfills them. Cycle-phase callers in * commit 7 add the empty-fence-guard as a belt-and-suspenders check. */ - deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }>; + /** + * #1928: `excludeSourcePrefixes` protects facts whose `source` matches any + * given prefix (e.g. `['cli:']` for conversation facts written by + * `extract-conversation-facts`) from a fence reconcile. Those rows are NOT + * fence-owned — a destructive wipe-and-reinsert pass would delete them and + * never recreate them (the page has no `## Facts` fence). Omitted ⇒ legacy + * behavior (delete every fact on the page coordinate). NULL/empty `source` + * rows are always deletable (fence default). + */ + deleteFactsForPage( + slug: string, + source_id: string, + opts?: { excludeSourcePrefixes?: string[] }, + ): Promise<{ deleted: number }>; /** * Mark a fact expired. Never DELETE. Returns true iff a row was updated. diff --git a/src/core/extract-timeline-from-meetings.ts b/src/core/extract-timeline-from-meetings.ts index fb1a3ec40..e63666310 100644 --- a/src/core/extract-timeline-from-meetings.ts +++ b/src/core/extract-timeline-from-meetings.ts @@ -28,6 +28,15 @@ export interface ExtractTimelineFromMeetingsResult { entries_created: number; /** Distinct entity pages that received at least one new timeline entry. */ entities_touched: number; + /** + * #2057: batches that failed to insert. Previously swallowed by a bare + * `catch {}`, which let a brain-wide timeline-write failure read as a clean + * "0 entries" run. Non-zero here means inserts are failing — surfaced on + * stderr too. + */ + batch_errors: number; + /** First batch-insert error message, when batch_errors > 0. */ + first_batch_error?: string; } interface MeetingRow { @@ -71,7 +80,7 @@ export async function extractTimelineFromMeetings( ); if (meetings.length === 0) { - return { meetings_scanned: 0, entries_created: 0, entities_touched: 0 }; + return { meetings_scanned: 0, entries_created: 0, entities_touched: 0, batch_errors: 0 }; } // 2. Fetch all 'attended' edges (one round-trip, scoped to the loaded @@ -106,14 +115,23 @@ export async function extractTimelineFromMeetings( let entriesCreated = 0; const entitiesTouched = new Set(); let meetingsScanned = 0; + let batchErrors = 0; + let firstBatchError: string | undefined; async function flush() { if (batch.length === 0) return; if (!dryRun) { try { entriesCreated += await engine.addTimelineEntriesBatch(batch); - } catch { - // batch error — drop; per-meeting progress continues + } catch (e) { + // #2057: do NOT swallow. A bare `catch {}` here hid a brain-wide + // timeline-write failure (the run reported 0 entries with no error). + // Count + surface it on stderr; the per-meeting loop still continues so + // one bad batch isn't fatal to the rest. + batchErrors += 1; + const msg = e instanceof Error ? e.message : String(e); + if (!firstBatchError) firstBatchError = msg; + console.error(`[extract timeline] batch insert failed (${batch.length} row(s)): ${msg}`); } } else { entriesCreated += batch.length; @@ -182,5 +200,7 @@ export async function extractTimelineFromMeetings( meetings_scanned: meetingsScanned, entries_created: entriesCreated, entities_touched: entitiesTouched.size, + batch_errors: batchErrors, + first_batch_error: firstBatchError, }; } diff --git a/src/core/import-file.ts b/src/core/import-file.ts index fb9190ee4..cbf2d03a0 100644 --- a/src/core/import-file.ts +++ b/src/core/import-file.ts @@ -1193,7 +1193,15 @@ export async function importCodeFile( // chunk IDs are stable. if (extractedEdges.length > 0 && chunks.length > 0) { try { - const persistedChunks = await engine.getChunks(slug, sourceId ? { sourceId } : undefined); + // Normalize ONCE: '' and undefined both mean the schema-default source + // (pages.source_id DEFAULT 'default'). Using the normalized value for + // BOTH the chunk lookup and the edge stamp keeps them in lockstep — + // an unscoped getChunks here could fan out to same-slug chunks from + // another source, and a '' stamp would FK-violate against sources(id) + // and silently drop the file's whole call graph in the best-effort + // catch below (adversarial review findings). + const edgeSourceId = sourceId || 'default'; + const persistedChunks = await engine.getChunks(slug, { sourceId: edgeSourceId }); const byIndex = new Map(); for (const pc of persistedChunks) { byIndex.set(pc.chunk_index, pc); @@ -1231,6 +1239,17 @@ export async function importCodeFile( from_symbol_qualified: from.symbol_name_qualified, to_symbol_qualified: e.toSymbol, edge_type: e.edgeType, + // Stamp the source: getCallersOf/getCalleesOf add + // `AND source_id = ` whenever a worktree pin / --source is + // in play, and a NULL here never matches that filter — so every + // scoped call-graph query silently returned 0 rows on + // multi-source brains even though the edges existed. The fallback + // is 'default', NOT null: an unscoped import lands its pages under + // the schema default (pages.source_id DEFAULT 'default'), so a + // NULL-stamped edge would be invisible to the matching scoped + // query getCallersOf(sym, { sourceId: 'default' }) — the same bug + // through the other door. + source_id: edgeSourceId, }); } diff --git a/src/core/legacy-token-scope.ts b/src/core/legacy-token-scope.ts new file mode 100644 index 000000000..5e616c280 --- /dev/null +++ b/src/core/legacy-token-scope.ts @@ -0,0 +1,22 @@ +/** + * Derive a legacy bearer token's source scope from its stored + * `access_tokens.permissions.source_id` grant. + * + * ARRAY = federated read grant, exposed through `allowedSources` with the + * first granted source as the scalar write floor. STRING = scalar source. + * Missing, empty, or garbage values fail closed to the historical `default` + * floor and NEVER widen to all sources. + */ +export function parseLegacyTokenScope(rawSource: unknown): { sourceId: string; allowedSources?: string[] } { + if (Array.isArray(rawSource)) { + const allowedSources = (rawSource as unknown[]).filter(s => typeof s === 'string' && s.length > 0) as string[]; + if (allowedSources.length > 0) { + return { sourceId: allowedSources[0], allowedSources }; + } + return { sourceId: 'default' }; + } + if (typeof rawSource === 'string' && rawSource.length > 0) { + return { sourceId: rawSource }; + } + return { sourceId: 'default' }; +} diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 2e31e60ad..31d3a10c1 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -5186,6 +5186,52 @@ export const MIGRATIONS: Migration[] = [ ); `, }, + { + version: 116, + name: 'code_edges_source_backfill_and_callee_index', + // Repair + index pass for the call graph: + // + // 1. BACKFILL: importCodeFile built CodeEdgeInput rows without source_id, + // so every extracted edge landed NULL. getCallersOf/getCalleesOf add + // `AND source_id = ` whenever a worktree pin / --source is in + // play — NULL never matches, so scoped call-graph queries silently + // returned 0 rows on multi-source brains even though the edges + // existed. The write path now stamps `sourceId ?? 'default'`; this + // backfill repairs rows written before the fix by deriving each + // edge's source from its own from_chunk's page (pages.source_id is + // NOT NULL DEFAULT 'default', so COALESCE is belt-and-braces only). + // + // 2. INDEXES: getCalleesOf filters BOTH edge tables on + // from_symbol_qualified, which had no index anywhere — every callee + // lookup was a sequential scan, amplified per-BFS-node by the + // recursive code walk (one getCalleesOf per frontier node, up to + // maxNodes). With NULL edges repaired, scoped walks actually expand, + // so the latent seq-scan cost becomes real. Plain CREATE INDEX (not + // CONCURRENTLY): edge tables are modest (mirrors the v58 resolver + // index). Keep in sync with src/schema.sql. + idempotent: true, + sql: ` + UPDATE code_edges_symbol e + SET source_id = COALESCE(p.source_id, 'default') + FROM content_chunks c + JOIN pages p ON p.id = c.page_id + WHERE c.id = e.from_chunk_id + AND e.source_id IS NULL; + + UPDATE code_edges_chunk e + SET source_id = COALESCE(p.source_id, 'default') + FROM content_chunks c + JOIN pages p ON p.id = c.page_id + WHERE c.id = e.from_chunk_id + AND e.source_id IS NULL; + + CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol + ON code_edges_symbol (from_symbol_qualified); + + CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol + ON code_edges_chunk (from_symbol_qualified); + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 @@ -5524,6 +5570,26 @@ export async function runMigrations(engine: BrainEngine): Promise<{ applied: num const sorted = [...MIGRATIONS].sort((a, b) => a.version - b.version); const pending = sorted.filter(m => m.version > current); + + // #2038: schema-drift self-heal. A migration renumbered during a master + // merge (v102 timeline dedup, originally v99) can be recorded-as-applied + // without its DDL ever running — the version counter can't see it. Repair + // the known drift on EVERY pass, including when nothing is pending (the + // affected brains are stamped AHEAD of the missing migration, so they never + // reach the loop below). Best-effort + idempotent: a no-op on a healthy + // index; `doctor` surfaces it independently if this ever fails. + try { + const { repairTimelineDedupIndex } = await import('./timeline-dedup-repair.ts'); + const r = await repairTimelineDedupIndex(engine); + if (r.repaired) { + console.error( + `[migrate] healed idx_timeline_dedup drift (#2038): ${r.before.join(',') || '(absent)'} ` + + `→ page_id,date,summary,source` + + (r.collapsedDuplicates > 0 ? ` (collapsed ${r.collapsedDuplicates} duplicate row(s))` : ''), + ); + } + } catch { /* best-effort; doctor reports the drift if this couldn't run */ } + if (pending.length === 0) { return { applied: 0, current }; } diff --git a/src/core/minions/supervisor.ts b/src/core/minions/supervisor.ts index 3853a6793..ec68dc499 100644 --- a/src/core/minions/supervisor.ts +++ b/src/core/minions/supervisor.ts @@ -1023,16 +1023,17 @@ export class MinionSupervisor { error: errMsg, queue: this.opts.queue, }); - // Attempt to reconnect the engine if it supports it + // Attempt to reconnect the engine. #2034: reconnect() is now a + // first-class BrainEngine method on both engines (it restores the + // config captured at connect()), so call it directly — the old + // feature-detection cast is no longer needed. try { - if ('reconnect' in this.engine && typeof (this.engine as Record).reconnect === 'function') { - await (this.engine as unknown as { reconnect(): Promise }).reconnect(); - this.consecutiveHealthFailures = 0; - this.emit('health_warn', { - reason: 'db_reconnected', - queue: this.opts.queue, - }); - } + await this.engine.reconnect({ error: errMsg }); + this.consecutiveHealthFailures = 0; + this.emit('health_warn', { + reason: 'db_reconnected', + queue: this.opts.queue, + }); } catch (reconnErr) { this.emit('health_error', { error: `reconnect failed: ${reconnErr instanceof Error ? reconnErr.message : String(reconnErr)}`, diff --git a/src/core/oauth-provider.ts b/src/core/oauth-provider.ts index 8b4ac91d3..83358536b 100644 --- a/src/core/oauth-provider.ts +++ b/src/core/oauth-provider.ts @@ -21,10 +21,12 @@ import type { } from '@modelcontextprotocol/sdk/shared/auth.js'; import type { OAuthServerProvider, AuthorizationParams } from '@modelcontextprotocol/sdk/server/auth/provider.js'; import type { OAuthRegisteredClientsStore } from '@modelcontextprotocol/sdk/server/auth/clients.js'; -import type { AuthInfo } from '@modelcontextprotocol/sdk/server/auth/types.js'; +import type { AuthInfo as SdkAuthInfo } from '@modelcontextprotocol/sdk/server/auth/types.js'; import { InvalidTokenError } from '@modelcontextprotocol/sdk/server/auth/errors.js'; import { hashToken, generateToken, isUndefinedColumnError } from './utils.ts'; import { hasScope, assertAllowedScopes, parseScopeString, InvalidScopeError } from './scope.ts'; +import type { AuthInfo as CoreAuthInfo } from './operations.ts'; +import { parseLegacyTokenScope } from './legacy-token-scope.ts'; import type { SqlQuery, SqlValue } from './sql-query.ts'; export type { SqlQuery, SqlValue }; @@ -389,10 +391,18 @@ export class GBrainOAuthProvider implements OAuthServerProvider { // as a fully-admin access token. Mirrors the filter pattern already used // by exchangeClientCredentials (this file) and exchangeRefreshToken's F3 // subset enforcement (RFC 6749 §6) so all three grant entry points clamp - // consistently. Empty/omitted requested scope inherits the empty-stored - // shape (existing behavior; not a security boundary). + // consistently. When the client requests NO scope, RFC 6749 §3.3 lets the + // server fall back to a default — we default to the client's full + // registered scope (matching exchangeClientCredentials, which already does + // `requestedScope ? ... : allowedScopes`). Previously an omitted request + // granted the empty set, which then propagated into the access+refresh + // tokens and never self-healed: every op failed `insufficient_scope` even + // though the client was registered with `read write`. Clients that omit + // `scope` on /authorize (e.g. some MCP connectors) hit this. Still clamped + // to the allowed set, so an explicit over-broad request can't escalate. const allowedScopes = parseScopeString(client.scope); - const grantedScopes = (params.scopes || []).filter(s => hasScope(allowedScopes, s)); + const requestedScopes = (params.scopes && params.scopes.length) ? params.scopes : allowedScopes; + const grantedScopes = requestedScopes.filter(s => hasScope(allowedScopes, s)); await this.sql` INSERT INTO oauth_codes (code_hash, client_id, scopes, code_challenge, @@ -539,7 +549,7 @@ export class GBrainOAuthProvider implements OAuthServerProvider { // Token Verification // ------------------------------------------------------------------------- - async verifyAccessToken(token: string): Promise { + async verifyAccessToken(token: string): Promise { const tokenHash = hashToken(token); const now = Math.floor(Date.now() / 1000); @@ -629,14 +639,29 @@ export class GBrainOAuthProvider implements OAuthServerProvider { // operations.ts prefers this array over scalar sourceId when set // and non-empty. allowedSources, - } as AuthInfo; + } as CoreAuthInfo as SdkAuthInfo; } - // Fallback: legacy access_tokens table (backward compat) - const legacyRows = await this.sql` - SELECT name FROM access_tokens - WHERE token_hash = ${tokenHash} AND revoked_at IS NULL - `; + // Fallback: legacy access_tokens table (backward compat). Modern legacy + // rows may carry permissions.source_id from the pre-OAuth bearer-token + // path; OAuth transport must preserve that same source grant instead of + // pinning every legacy token to `default`. + let legacyRows: Record[]; + try { + legacyRows = await this.sql` + SELECT name, permissions FROM access_tokens + WHERE token_hash = ${tokenHash} AND revoked_at IS NULL + `; + } catch (err) { + if (isUndefinedColumnError(err, 'permissions')) { + legacyRows = await this.sql` + SELECT name FROM access_tokens + WHERE token_hash = ${tokenHash} AND revoked_at IS NULL + `; + } else { + throw err; + } + } if (legacyRows.length > 0) { // Legacy tokens get full admin access (grandfather in). @@ -646,19 +671,31 @@ export class GBrainOAuthProvider implements OAuthServerProvider { UPDATE access_tokens SET last_used_at = now() WHERE token_hash = ${tokenHash} `; const name = legacyRows[0].name as string; + const permissionsRaw = legacyRows[0].permissions; + let permissions: unknown = permissionsRaw; + if (typeof permissionsRaw === 'string') { + try { + permissions = JSON.parse(permissionsRaw); + } catch { + permissions = undefined; + } + } + const sourceGrant = permissions && typeof permissions === 'object' + ? (permissions as Record).source_id + : undefined; + const { sourceId, allowedSources } = parseLegacyTokenScope(sourceGrant); return { token, clientId: name, clientName: name, scopes: ['read', 'write', 'admin'], expiresAt: Math.floor(Date.now() / 1000) + 365 * 24 * 3600, // Legacy tokens never expire — set 1yr future - // v0.34.1 (#861, D13): legacy bearer tokens default to 'default' - // source — matches the pre-v0.34 effective behavior where the - // serve-http transport fell back to GBRAIN_SOURCE/'default' for - // any caller without explicit scope. Operators who want a - // narrower scope for legacy tokens migrate to OAuth. - sourceId: 'default', - } as AuthInfo; + // Legacy tokens without an explicit permissions.source_id grant keep + // the historical 'default' source floor. Array grants become + // allowedSources for federated reads, matching legacy HTTP transport. + sourceId, + allowedSources, + } as CoreAuthInfo as SdkAuthInfo; } throw new InvalidTokenError('Invalid token'); diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 65c429f36..e2cf4693a 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -199,6 +199,9 @@ export class PGLiteEngine implements BrainEngine { readonly kind = 'pglite' as const; private _db: PGLiteDB | null = null; private _lock: LockHandle | null = null; + // #2034: captured at connect() so reconnect() can restore the same data dir + // after a drop, matching PostgresEngine's _savedConfig contract. + private _savedConfig: EngineConfig | null = null; // Tier 3: when GBRAIN_PGLITE_SNAPSHOT loaded a post-initSchema state into // PGlite.create(loadDataDir), initSchema is a no-op (schema is already // present + migrations already applied). Saves ~1-3s per fresh test PGLite. @@ -211,6 +214,7 @@ export class PGLiteEngine implements BrainEngine { // Lifecycle async connect(config: EngineConfig): Promise { + this._savedConfig = config; // #2034: remember for reconnect() const dataDir = config.database_path || undefined; // undefined = in-memory // Acquire file lock to prevent concurrent PGLite access (crashes with Aborted()) @@ -288,6 +292,27 @@ export class PGLiteEngine implements BrainEngine { } } + /** + * #2034: engine-parity reconnect. PGLite is single-writer in-process so it + * doesn't suffer the pool-drop class PostgresEngine.reconnect() handles, but + * the method MUST exist so callers (autopilot health probe, worker/queue + * claim-error recovery) can call `engine.reconnect()` uniformly. + * + * IN-MEMORY (no `database_path`) is a NO-OP: there is no persistent backing, + * the connection can't recoverably "drop" in-process, and a disconnect+reopen + * would DISCARD all state. This matches the long-standing assumption the + * worker/queue recovery paths are written against ("PGLite has no pooler + * reaping so reconnect is absent" — src/core/minions/queue.ts). A FILE-backed + * engine genuinely re-opens the same data dir (state persists on disk). + */ + async reconnect(_ctx?: { error?: unknown }): Promise { + if (!this._savedConfig) return; // never connected — nothing to restore + if (!this._savedConfig.database_path) return; // in-memory — no-op, preserve state + const config = this._savedConfig; + await this.disconnect(); + await this.connect(config); + } + async initSchema(): Promise { // Tier 3: snapshot was loaded into PGlite — schema + migrations already // applied. Nothing to do. Returns immediately. @@ -3598,7 +3623,25 @@ export class PGLiteEngine implements BrainEngine { return { inserted: ids.length, ids }; } - async deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }> { + async deleteFactsForPage( + slug: string, + source_id: string, + opts?: { excludeSourcePrefixes?: string[] }, + ): Promise<{ deleted: number }> { + const prefixes = opts?.excludeSourcePrefixes; + if (prefixes && prefixes.length > 0) { + // #1928: keep rows whose `source` matches an excluded prefix (e.g. + // `cli:` conversation facts). COALESCE so NULL/empty-source fence rows + // stay deletable — only the explicitly-protected prefixes survive. + const patterns = prefixes.map(p => `${p}%`); + const result = await this.db.query( + `DELETE FROM facts + WHERE source_id = $1 AND source_markdown_slug = $2 + AND NOT (COALESCE(source, '') LIKE ANY($3::text[]))`, + [source_id, slug, patterns], + ); + return { deleted: result.affectedRows ?? 0 }; + } const result = await this.db.query( `DELETE FROM facts WHERE source_id = $1 AND source_markdown_slug = $2`, [source_id, slug], @@ -4908,7 +4951,7 @@ export class PGLiteEngine implements BrainEngine { e.from_chunk_id, e.to_chunk_id, e.from_symbol_qualified, e.to_symbol_qualified, e.edge_type, JSON.stringify(e.edge_metadata ?? {}), - e.source_id ?? null, + e.source_id ?? 'default', ); } const res = await this.db.query( @@ -4929,7 +4972,7 @@ export class PGLiteEngine implements BrainEngine { params.push( e.from_chunk_id, e.from_symbol_qualified, e.to_symbol_qualified, e.edge_type, JSON.stringify(e.edge_metadata ?? {}), - e.source_id ?? null, + e.source_id ?? 'default', ); } const res = await this.db.query( diff --git a/src/core/pglite-lock.ts b/src/core/pglite-lock.ts index 7fc846b77..0fe04952a 100644 --- a/src/core/pglite-lock.ts +++ b/src/core/pglite-lock.ts @@ -19,11 +19,72 @@ import { join } from 'path'; const LOCK_DIR_NAME = '.gbrain-lock'; const LOCK_FILE = 'lock'; -const STALE_THRESHOLD_MS = 5 * 60 * 1000; // 5 minutes — embed jobs can be long + +// #2058: refresh the lock's `refreshed_at` while held so a long-running but +// LIVE holder (embed jobs run for many minutes) is never mistaken for stale. +const HEARTBEAT_INTERVAL_MS = 30_000; + +// #2058: a holder whose heartbeat refreshed within this window is ALIVE and is +// NEVER stolen, regardless of how old the lock is. Only a holder that STOPPED +// refreshing past this grace (hung, crashed without cleanup, or a PID since +// reused by an unrelated process) is reaped. Pairing heartbeat-age with PID +// liveness is what defeats both the WAL-corruption bug (stealing a live +// writer) AND the PID-reuse false-positive (a recycled PID reading as "alive"). +// Env-overridable as an incident escape hatch, matching the sync-lock knobs. +function stealGraceMs(): number { + const env = parseInt(process.env.GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS ?? '', 10); + return Number.isFinite(env) && env > 0 ? env * 1000 : 10 * 60 * 1000; // default 600s +} export interface LockHandle { lockDir: string; acquired: boolean; + /** + * #2058: heartbeat timer + lock-file path, set when a real (on-disk) lock is + * held so `releaseLock` can stop refreshing. Absent for the in-memory engine + * (no lock file, no concurrent access possible). + */ + heartbeat?: ReturnType; + lockPath?: string; + /** + * #2058 (codex): our ownership token (`:`). If we stall + * past the steal grace, another process can reap + re-acquire. When we + * resume, the heartbeat and release MUST verify the on-disk lock is STILL + * ours before touching it — otherwise a resumed stale holder would refresh + * or delete the NEW owner's live lock, re-opening the concurrent-writer hole. + */ + ownerToken?: string; +} + +/** The on-disk lock identity, used to detect "we were reaped and replaced". */ +function tokenOf(lockData: { pid?: unknown; acquired_at?: unknown }): string { + return `${lockData.pid}:${lockData.acquired_at}`; +} + +/** + * #2058: keep the held lock's `refreshed_at` current so a concurrent acquirer + * can tell a live, working holder from a hung/dead one. Best-effort: if the + * file is gone (we're being reaped) the write simply fails. `.unref()` so the + * timer never keeps the process alive on its own. Ownership-checked: if the + * on-disk lock is no longer ours (we were reaped past grace and replaced), stop + * the heartbeat instead of clobbering the new owner's lock. + */ +function startHeartbeat(lockPath: string, ownerToken: string): ReturnType { + const timer = setInterval(() => { + try { + const raw = JSON.parse(readFileSync(lockPath, 'utf-8')); + if (tokenOf(raw) !== ownerToken) { + // We were reaped and someone else owns it now — do NOT refresh their + // lock. Stand down. + clearInterval(timer); + return; + } + raw.refreshed_at = Date.now(); + writeFileSync(lockPath, JSON.stringify(raw), { mode: 0o644 }); + } catch { /* best-effort — file removed or transient FS error */ } + }, HEARTBEAT_INTERVAL_MS); + (timer as { unref?: () => void }).unref?.(); + return timer; } function getLockDir(dataDir: string | undefined): string { @@ -75,16 +136,23 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM const lockPid = lockData.pid as number; const lockTime = lockData.acquired_at as number; - // Is the locking process still alive? - if (!isProcessAlive(lockPid)) { - // Stale lock — clean it up + // #2058: classify by PID liveness AND heartbeat freshness. A holder + // that is alive AND refreshed its heartbeat within the steal grace is + // genuinely working (e.g. a multi-minute embed) and is NEVER reaped — + // force-removing it here is what corrupted the single-writer WAL. + const alive = isProcessAlive(lockPid); + const lastRefresh = (lockData.refreshed_at as number | undefined) ?? lockTime; + const sinceRefresh = Date.now() - lastRefresh; + if (!alive) { + // Holder process is gone — reap. try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition, try again */ } - } else if (Date.now() - lockTime > STALE_THRESHOLD_MS) { - // Lock held for too long — assume stale (e.g., process hung) - // Still alive but probably stuck — force remove + } else if (sinceRefresh > stealGraceMs()) { + // PID is alive but the heartbeat stopped past the grace window: + // either the holder hung, or this PID was reused by an unrelated + // process (the real holder died and stopped refreshing). Reap. try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition */ } } else { - // Lock is held by a live process — wait and retry + // Live holder refreshing within grace — wait and retry. await new Promise(r => setTimeout(r, 1000)); continue; } @@ -97,15 +165,19 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM // Try to acquire lock (atomic mkdir) try { mkdirSync(lockDir, { recursive: false }); - // We got the lock — write our PID + // We got the lock — write our PID. #2058: seed `refreshed_at` and start + // the heartbeat so this holder reads as alive-and-working to others. const lockPath = join(lockDir, LOCK_FILE); + const now = Date.now(); writeFileSync(lockPath, JSON.stringify({ pid: process.pid, - acquired_at: Date.now(), + acquired_at: now, + refreshed_at: now, command: process.argv.slice(1).join(' '), }), { mode: 0o644 }); - return { lockDir, acquired: true }; + const ownerToken = tokenOf({ pid: process.pid, acquired_at: now }); + return { lockDir, acquired: true, lockPath, ownerToken, heartbeat: startHeartbeat(lockPath, ownerToken) }; } catch (e: unknown) { // mkdir failed — someone else grabbed it between our check and mkdir // This is fine, we'll retry @@ -138,8 +210,25 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM * Release a previously acquired lock. */ export async function releaseLock(lock: LockHandle): Promise { + // #2058: stop the heartbeat first so it can't recreate/rewrite the lock file + // after we remove it. + if (lock.heartbeat) { + clearInterval(lock.heartbeat); + lock.heartbeat = undefined; + } if (!lock.lockDir || !lock.acquired) return; + // #2058 (codex): only remove the lock if it is STILL ours. If we were reaped + // past the grace and another process re-acquired, removing its live lock + // would let a third process in alongside it — the corruption this fix exists + // to prevent. Unreadable/absent lock falls through to a best-effort remove. + if (lock.ownerToken) { + try { + const raw = JSON.parse(readFileSync(join(lock.lockDir, LOCK_FILE), 'utf-8')); + if (tokenOf(raw) !== lock.ownerToken) return; // someone else owns it now + } catch { /* unreadable/gone — fall through to best-effort cleanup */ } + } + try { rmSync(lock.lockDir, { recursive: true, force: true }); } catch { diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 3b8c7ce13..ff781eaa3 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -1276,11 +1276,28 @@ export class PostgresEngine implements BrainEngine { } async updateSourceConfig(sourceId: string, patch: Record): Promise { - // v0.38: atomic JSONB merge. `||` is the Postgres concat operator — - // for jsonb, right-side keys overwrite left-side; nested object keys - // are NOT deep-merged (use jsonb_set for nested paths). The patch - // shape this autopilot wave uses is flat (`last_full_cycle_at`, - // `archive_*`, etc.) so concat is sufficient. Idempotent on re-run. + // Atomic single-statement merge. The previous read-then-write form dropped + // concurrent updates: two callers patching different keys could both read + // the same old config and the later `SET config = ...` clobbered the + // earlier patch. These keys are written by background cycle/autopilot + // paths, so the merge must happen inside the UPDATE (parity with + // pglite-engine.updateSourceConfig, which already uses JSONB `||`). + // + // The CASE normalizes historical bad shapes inline (so `config` is re-read + // against the row-locked latest version — a CTE/subquery snapshot would + // reintroduce the lost-update race under READ COMMITTED): older code paths + // could store config as a JSONB string (double-encoded) or as a JSONB array + // of patch objects. We coerce those to a flat object before the `||` merge + // so doctor and source routing keep getting flat keys. + // + // String branch guard: a JSONB string whose inner text is NOT itself valid + // JSON (one of the historical bad shapes this path repairs) would make the + // bare `::jsonb` cast raise `invalid input syntax for type json`, failing + // the whole UPDATE. Postgres has no `try_cast`, so we gate the cast with + // the SQL `IS JSON` predicate (Postgres 16+): parseable inner text is + // double-encoded config and gets parsed; unparseable text falls back to `{}`. + // The guard keeps the merge a single atomic statement (no extra round-trip, + // no lost-update race). // // MUST use sql.json(patch) inside the template tag — postgres-js's // positional executeRaw + `$1::jsonb` cast DOUBLE-ENCODES the @@ -1294,7 +1311,25 @@ export class PostgresEngine implements BrainEngine { const sql = this.sql; const result = await sql` UPDATE sources - SET config = COALESCE(config, '{}'::jsonb) || ${sql.json(patch as Parameters[0])} + SET config = + CASE + WHEN jsonb_typeof(config) = 'object' THEN config + WHEN jsonb_typeof(config) = 'string' + THEN CASE + WHEN (config #>> '{}') IS JSON + THEN COALESCE(NULLIF((config #>> '{}'), '')::jsonb, '{}'::jsonb) + ELSE '{}'::jsonb + END + WHEN jsonb_typeof(config) = 'array' + THEN COALESCE( + (SELECT jsonb_object_agg(kv.key, kv.value) + FROM jsonb_array_elements(config) elem, + jsonb_each(elem) kv), + '{}'::jsonb + ) + ELSE '{}'::jsonb + END + || ${sql.json(patch as Parameters[0])} WHERE id = ${sourceId} `; return (result.count ?? 0) > 0; @@ -3745,8 +3780,26 @@ export class PostgresEngine implements BrainEngine { return { inserted: ids.length, ids }; } - async deleteFactsForPage(slug: string, source_id: string): Promise<{ deleted: number }> { + async deleteFactsForPage( + slug: string, + source_id: string, + opts?: { excludeSourcePrefixes?: string[] }, + ): Promise<{ deleted: number }> { const sql = this.sql; + const prefixes = opts?.excludeSourcePrefixes; + if (prefixes && prefixes.length > 0) { + // #1928: keep rows whose `source` matches an excluded prefix (e.g. + // `cli:` conversation facts). COALESCE so NULL/empty-source fence rows + // stay deletable — only the explicitly-protected prefixes survive. + const patterns = prefixes.map(p => `${p}%`); + const result = await sql` + DELETE FROM facts + WHERE source_id = ${source_id} + AND source_markdown_slug = ${slug} + AND NOT (COALESCE(source, '') LIKE ANY(${patterns})) + `; + return { deleted: result.count ?? 0 }; + } const result = await sql` DELETE FROM facts WHERE source_id = ${source_id} AND source_markdown_slug = ${slug} `; @@ -5119,7 +5172,7 @@ export class PostgresEngine implements BrainEngine { const toQual = resolved.map(e => e.to_symbol_qualified); const edgeTypes = resolved.map(e => e.edge_type); const metas = resolved.map(e => JSON.stringify(e.edge_metadata ?? {})); - const sources = resolved.map(e => e.source_id ?? null); + const sources = resolved.map(e => e.source_id ?? 'default'); const res = await sql` INSERT INTO code_edges_chunk (from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id) SELECT * FROM unnest( @@ -5139,7 +5192,7 @@ export class PostgresEngine implements BrainEngine { const toQual = unresolved.map(e => e.to_symbol_qualified); const edgeTypes = unresolved.map(e => e.edge_type); const metas = unresolved.map(e => JSON.stringify(e.edge_metadata ?? {})); - const sources = unresolved.map(e => e.source_id ?? null); + const sources = unresolved.map(e => e.source_id ?? 'default'); const res = await sql` INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id) SELECT * FROM unnest( diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index dc288eabd..2e30b49f1 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -390,6 +390,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to ON code_edges_chunk(to_chunk_id, edge_type); CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol ON code_edges_chunk(to_symbol_qualified, edge_type); +-- getCalleesOf filters on from_symbol_qualified; without this index every +-- callee lookup is a sequential scan, amplified per-BFS-node by the +-- recursive code walk. Mirrors migration v116. +CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol + ON code_edges_chunk(from_symbol_qualified); CREATE TABLE IF NOT EXISTS code_edges_symbol ( id SERIAL PRIMARY KEY, @@ -407,16 +412,32 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from ON code_edges_symbol(from_chunk_id, edge_type); CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to ON code_edges_symbol(to_symbol_qualified, edge_type); +-- getCalleesOf companion to idx_code_edges_chunk_from_symbol above. +-- Mirrors migration v116. +CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol + ON code_edges_symbol(from_symbol_qualified); -- ============================================================ -- links: cross-references between pages -- ============================================================ --- Provenance model (v0.13, extended issue #972): --- link_source — 'markdown' | 'frontmatter' | 'manual' | 'wikilink-resolved' | NULL --- 'wikilink-resolved' is the opt-in --- (link_resolution.global_basename) basename-match --- provenance — see issue #972 / migration v113. --- (NULL = legacy row written before v0.13; unknown source) +-- Provenance model (v0.13; opened to kebab provenance in v114 / issue #1941): +-- link_source — open kebab-case provenance tag, NOT a closed allowlist. +-- Format gate (CHECK): ^[a-z][a-z0-9]*(-[a-z0-9]+)*\$ and +-- char_length <= 64. NULL = legacy row (pre-v0.13). +-- Reconciliation-managed built-ins written internally: +-- 'markdown' — body markdown links +-- 'frontmatter' — YAML frontmatter edges (see origin_*) +-- 'mentions' — auto-linked body-text mentions +-- 'wikilink-resolved'— opt-in global-basename [[name]] (#972) +-- User/tool-facing: +-- 'manual' — hand- or tool-created edges (the +-- add_link op default + CLI link-add) +-- '' — external derivers, e.g. 'citation-graph', +-- stamp their own kebab tag (no migration). +-- The add_link OP forbids callers from passing the four +-- managed built-ins (they imply reconciliation semantics a +-- hand-created row can't honor); the DB CHECK still admits +-- them because internal writers use them. See operations.ts. -- origin_page_id — for link_source='frontmatter', the page whose YAML -- frontmatter created this edge; scopes reconciliation -- origin_field — the frontmatter field name (e.g. 'key_people') @@ -424,21 +445,21 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to -- The unique constraint includes link_source + origin_page_id so a manual edge -- and a frontmatter-derived edge with the same (from, to, type) tuple coexist. -- Reconciliation on put_page filters by (link_source='frontmatter' AND --- origin_page_id = written_page) — never touches other pages' edges. +-- origin_page_id = written_page) — never touches other pages' edges. (This is +-- exactly why a CLI-forged 'frontmatter' row with NULL origin would be a phantom +-- edge reconciliation never cleans — hence the op-layer guard.) CREATE TABLE IF NOT EXISTS links ( id SERIAL PRIMARY KEY, from_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE, to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE, link_type TEXT NOT NULL DEFAULT '', context TEXT NOT NULL DEFAULT '', - -- v114 (#1941): link_source is an open kebab-case provenance, not a closed - -- allowlist — external derivers (e.g. 'citation-graph') stamp their own tag - -- without a gbrain migration. Format gate only: lowercase kebab, <=64 chars. - -- The reconciliation-managed built-ins ('markdown','frontmatter','mentions', - -- 'wikilink-resolved') still satisfy this and are written internally; the - -- add_link op forbids CALLERS from forging them (see operations.ts). 'manual' - -- is the user-facing default for hand/tool-created edges. - link_source TEXT CHECK (link_source IS NULL OR (link_source ~ '^[a-z][a-z0-9]*(-[a-z0-9]+)*$' AND char_length(link_source) <= 64)), + -- v0.41.18.0: 'mentions' added for auto-linked body-text mentions + -- (gbrain extract links --by-mention). Filtered OUT of backlink-count + -- for search ranking; only counts toward orphan-ratio + graph traversal. + -- v0.40.8.2 (#972): 'wikilink-resolved' added for opt-in global-basename + -- wikilink resolution (bare [[name]] resolved by slug tail). + link_source TEXT CHECK (link_source IS NULL OR (link_source ~ '^[a-z][a-z0-9]*(-[a-z0-9]+)*\$' AND char_length(link_source) <= 64)), -- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from -- "verb-pattern-derived typed link" within link_source='mentions'. -- Codex finding #12 design: keep link_source stable; add link_kind @@ -675,8 +696,11 @@ CREATE TABLE IF NOT EXISTS op_checkpoints ( CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx ON op_checkpoints (updated_at); --- #1794: append-only delta storage (one row per completed path). FK cascade --- drops children with the parent. Mirrors migration v115 + src/schema.sql. +-- #1794: append-only delta storage. One row per completed path; sync's +-- appendCompleted INSERTs only the delta instead of rewriting the whole +-- completed_keys JSONB array each flush (O(N^2) -> O(delta)). FK cascade drops +-- children with the parent (clearOpCheckpoint + 7-day purge). PK prefix +-- (op,fingerprint) serves all reads; no separate index. Mirrors migration v115. CREATE TABLE IF NOT EXISTS op_checkpoint_paths ( op TEXT NOT NULL, fingerprint TEXT NOT NULL, diff --git a/src/core/search/mode.ts b/src/core/search/mode.ts index d5a8198e6..f851913c0 100644 --- a/src/core/search/mode.ts +++ b/src/core/search/mode.ts @@ -738,7 +738,16 @@ export function attributeKnob( // to take effect immediately (one-time global cache cold-miss on upgrade; refills // within cache.ttl_seconds). Same cache-key-contamination convention as the // autocut / title_boost / graph_signals bumps above. -export const KNOBS_HASH_VERSION = 10; +// +// bump 10→11 (#1400 input_type fix): asymmetric embedding models (zembed-1 +// hosted or local, Voyage v3+) had their query-side input_type stripped by +// the AI SDK before the wire, so every cached row's key embedding AND result +// set were computed with document-side query vectors. The fix changes what +// embedQuery() produces for those providers; pre-fix rows must not be served +// to post-fix lookups. Same one-time global cold-miss pattern as the bumps +// above (the hash is global, not per-provider); refills within +// cache.ttl_seconds (3600s default). +export const KNOBS_HASH_VERSION = 11; /** * v0.36 (D8 / CDX-2) — second-arg context for the cache key. The diff --git a/src/core/sync.ts b/src/core/sync.ts index d92378bc8..c2c46a2e1 100644 --- a/src/core/sync.ts +++ b/src/core/sync.ts @@ -241,6 +241,19 @@ function matchesAnyGlob(path: string, patterns?: string[]): boolean { */ const PRUNE_DIR_NAMES = new Set([ 'node_modules', + // Dependency / build-output trees that are git-ignored on virtually every + // repo and never contain hand-authored source worth indexing. `vendor` + // (PHP Composer / Go / Ruby bundle), `dist` + `build` (compiled output). + // Closes the silent-pollution bug where a Laravel/PHP repo's full code sync + // walked ~50k `vendor/` files (#1483 / #1159 / maintainer #1942). + 'vendor', + 'dist', + 'build', + // Python venv: vendored dependency tree, the `node_modules` analogue (#2020). + // Like `node_modules` it lacks a leading dot so isSyncable's dot-prefix + // exclusion misses it; explicit entry keeps incremental sync consistent + // with the first-sync walker in commands/import.ts. + 'venv', '.raw', 'ops', ]); diff --git a/src/core/timeline-dedup-repair.ts b/src/core/timeline-dedup-repair.ts new file mode 100644 index 000000000..4aff01223 --- /dev/null +++ b/src/core/timeline-dedup-repair.ts @@ -0,0 +1,120 @@ +/** + * #2038 — idx_timeline_dedup schema-drift self-heal. + * + * Migration v102 (`timeline_entries_source_in_dedup`) widens the dedup index + * from (page_id, date, summary) to (page_id, date, summary, source). It was + * renumbered from v99 during a master merge, so a brain that ran the OLD v99 + * variant has its version counter stamped PAST v102 while the index stayed + * 3-column. `runMigrations` then can't see the drift (it early-returns when no + * version is pending), and every `addTimelineEntry(esBatch)` fails with + * "no unique or exclusion constraint matching the ON CONFLICT specification" + * because both insert sites infer on the 4-column tuple — timeline writes + * silently break brain-wide. + * + * The version counter can't detect this, so the repair is keyed off the actual + * index SHAPE and runs on every migrate pass (including the no-pending path). + * Idempotent: a no-op when the index is already 4-column. + */ + +import type { BrainEngine } from './engine.ts'; + +const INDEX_NAME = 'idx_timeline_dedup'; +const EXPECTED_COLUMNS = ['page_id', 'date', 'summary', 'source']; + +export interface TimelineDedupStatus { + /** The timeline_entries table exists (nothing to repair if not). */ + tablePresent: boolean; + /** The index exists. */ + indexPresent: boolean; + /** Indexed columns in order (empty when the index is absent). */ + columns: string[]; + /** Index exists in the wrong (pre-v102) shape — needs a rebuild. */ + needsRepair: boolean; +} + +/** Parse the column list out of a pg_indexes `indexdef` string. */ +function parseIndexColumns(indexdef: string): string[] { + const open = indexdef.lastIndexOf('('); + const close = indexdef.lastIndexOf(')'); + if (open < 0 || close < 0 || close < open) return []; + return indexdef + .slice(open + 1, close) + .split(',') + .map(c => c.trim().split(/\s+/)[0]) // drop any "col DESC"/opclass suffix + .filter(Boolean); +} + +export async function checkTimelineDedupIndex(engine: BrainEngine): Promise { + const tbl = await engine.executeRaw<{ reg: string | null }>( + `SELECT to_regclass('timeline_entries')::text AS reg`, + ); + const tablePresent = !!tbl[0]?.reg; + if (!tablePresent) { + return { tablePresent: false, indexPresent: false, columns: [], needsRepair: false }; + } + const rows = await engine.executeRaw<{ indexdef: string }>( + `SELECT indexdef FROM pg_indexes WHERE indexname = $1`, + [INDEX_NAME], + ); + const indexPresent = rows.length > 0; + const columns = indexPresent ? parseIndexColumns(rows[0].indexdef) : []; + const correct = + columns.length === EXPECTED_COLUMNS.length && + EXPECTED_COLUMNS.every((c, i) => columns[i] === c); + // An ABSENT index is also "needs repair" — the migration that creates it was + // skipped. (A fresh brain always has it, created by the migration chain.) + return { tablePresent, indexPresent, columns, needsRepair: !correct }; +} + +export interface TimelineDedupRepairResult { + repaired: boolean; + before: string[]; + collapsedDuplicates: number; + reason: 'already_correct' | 'no_table' | 'rebuilt'; +} + +/** + * Heal the index if it's missing the v102 4-column shape. Dedupes FIRST — + * the loose 3-column index let rows differing only by `source` coexist, and + * `CREATE UNIQUE INDEX` would throw on those collisions otherwise. Keeps the + * earliest row (min ctid) of each 4-tuple group. + */ +export async function repairTimelineDedupIndex(engine: BrainEngine): Promise { + const status = await checkTimelineDedupIndex(engine); + if (!status.tablePresent) { + return { repaired: false, before: [], collapsedDuplicates: 0, reason: 'no_table' }; + } + if (!status.needsRepair) { + return { repaired: false, before: status.columns, collapsedDuplicates: 0, reason: 'already_correct' }; + } + + // Keep the lowest `id` per 4-tuple group — deterministic and consistent with + // the existing v-migration dedup rule (`a.id > b.id`), unlike `ctid` which is + // a physical tuple location that can preserve an arbitrary duplicate. + const del = await engine.executeRaw<{ n: string }>( + `WITH d AS ( + DELETE FROM timeline_entries t + USING ( + SELECT page_id, date, summary, source, MIN(id) AS keep + FROM timeline_entries + GROUP BY page_id, date, summary, source + HAVING COUNT(*) > 1 + ) dup + WHERE t.page_id = dup.page_id + AND t.date = dup.date + AND t.summary = dup.summary + AND t.source IS NOT DISTINCT FROM dup.source + AND t.id <> dup.keep + RETURNING 1 + ) + SELECT COUNT(*)::text AS n FROM d`, + ); + const collapsedDuplicates = parseInt(del[0]?.n ?? '0', 10); + + await engine.executeRaw(`DROP INDEX IF EXISTS ${INDEX_NAME}`); + await engine.executeRaw( + `CREATE UNIQUE INDEX IF NOT EXISTS ${INDEX_NAME} + ON timeline_entries(page_id, date, summary, source)`, + ); + return { repaired: true, before: status.columns, collapsedDuplicates, reason: 'rebuilt' }; +} diff --git a/src/core/write-through.ts b/src/core/write-through.ts index b21d2cb24..96a1aad9b 100644 --- a/src/core/write-through.ts +++ b/src/core/write-through.ts @@ -22,7 +22,7 @@ */ import { existsSync, statSync, mkdirSync, writeFileSync, renameSync, unlinkSync } from 'fs'; -import { dirname } from 'path'; +import { dirname, join } from 'path'; import { randomBytes } from 'crypto'; import type { BrainEngine } from './engine.ts'; import { serializePageToMarkdown, resolvePageFilePath } from './markdown.ts'; @@ -37,12 +37,16 @@ export interface WriteThroughResult { path?: string; /** * Non-error reasons the file was not written: - * - no_repo_configured: `sync.repo_path` is unset (DB-only by design). - * - repo_not_found: `sync.repo_path` set but missing / not a directory. + * - no_repo_configured: the resolved target (source `local_path` or, for a + * sole-source brain, `sync.repo_path`) is unset (DB-only by design). + * - repo_not_found: target set but missing / not a directory. + * - source_repo_belongs_to_other_source: the assigned source has no + * `local_path`, and `sync.repo_path` is another source's own working tree + * — #2018: writing here would pollute that sibling's repo, so we skip. * - page_not_found_after_write: the DB row isn't readable back (the caller's * DB write failed or targeted a different source). */ - skipped?: 'no_repo_configured' | 'repo_not_found' | 'page_not_found_after_write'; + skipped?: 'no_repo_configured' | 'repo_not_found' | 'source_repo_belongs_to_other_source' | 'page_not_found_after_write'; /** Set when the render/write/rename itself threw (EACCES, ENOTDIR, disk full). */ error?: string; } @@ -67,12 +71,46 @@ export async function writePageThrough( ): Promise { const sourceId = opts.sourceId ?? 'default'; try { - const repoPath = await engine.getConfig('sync.repo_path'); - if (!repoPath) { - return { written: false, skipped: 'no_repo_configured' }; - } - if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) { - return { written: false, skipped: 'repo_not_found' }; + // #2018: pick the disk target so a page is NEVER written into a different + // source's working tree. Two legitimate topologies, plus the leak guard: + // 1. The assigned source has its OWN `local_path` (a separate working + // tree) → write at that tree's root (matches how `scanOneSource` reads + // it back; never nested under `.sources/`). + // 2. No per-source `local_path` → nest under the host repo + // (`sync.repo_path`): default at the root, non-default under + // `.sources//` (the established multi-source layout). + // 3. LEAK GUARD: if `sync.repo_path` is literally ANOTHER source's own + // `local_path`, nesting this page there would pollute that sibling's + // git repo (the reported bug). Skip instead. + let filePath: string; + const srcRows = await engine.executeRaw<{ local_path: string | null }>( + `SELECT local_path FROM sources WHERE id = $1`, + [sourceId], + ); + const sourceLocalPath = srcRows[0]?.local_path ?? null; + if (sourceLocalPath) { + if (!existsSync(sourceLocalPath) || !statSync(sourceLocalPath).isDirectory()) { + return { written: false, skipped: 'repo_not_found' }; + } + filePath = join(sourceLocalPath, `${slug}.md`); + } else { + const repoPath = await engine.getConfig('sync.repo_path'); + if (!repoPath) { + return { written: false, skipped: 'no_repo_configured' }; + } + if (!existsSync(repoPath) || !statSync(repoPath).isDirectory()) { + return { written: false, skipped: 'repo_not_found' }; + } + // Leak guard: refuse to write into a path that is some OTHER source's + // own working tree (#2018). + const collide = await engine.executeRaw<{ one: number }>( + `SELECT 1 AS one FROM sources WHERE id <> $1 AND local_path = $2 LIMIT 1`, + [sourceId, repoPath], + ); + if (collide.length > 0) { + return { written: false, skipped: 'source_repo_belongs_to_other_source' }; + } + filePath = resolvePageFilePath(repoPath, slug, sourceId); } const writtenPage = await engine.getPage(slug, { sourceId }); @@ -85,7 +123,6 @@ export async function writePageThrough( frontmatterOverrides: opts.frontmatterOverrides, }); - const filePath = resolvePageFilePath(repoPath, slug, sourceId); mkdirSync(dirname(filePath), { recursive: true }); // Atomic write: unique temp sibling + rename. Unique name (pid + random) diff --git a/src/mcp/http-transport.ts b/src/mcp/http-transport.ts index c1ccb3152..c3ff28c27 100644 --- a/src/mcp/http-transport.ts +++ b/src/mcp/http-transport.ts @@ -34,6 +34,8 @@ import { VERSION } from '../version.ts'; import { dispatchToolCall } from './dispatch.ts'; import { buildDefaultLimiters, type RateLimiter } from './rate-limit.ts'; import { sqlQueryForEngine } from '../core/sql-query.ts'; +import { parseLegacyTokenScope } from '../core/legacy-token-scope.ts'; +export { parseLegacyTokenScope }; const DEFAULT_BODY_CAP = 1024 * 1024; // 1 MiB @@ -84,28 +86,8 @@ interface AuthResult { auth?: AuthInfo; } -/** - * #1336: derive a legacy bearer token's source scope from its stored - * `permissions.source_id`. An ARRAY value is a federated_read grant → - * `allowedSources` (scoped reads across exactly those sources). A STRING value - * scopes the scalar floor. Anything else → 'default' (preserves pre-v0.34 - * behavior). NEVER widened to "all": an empty/garbage value keeps the 'default' - * floor and no federated grant. - */ -export function parseLegacyTokenScope(rawSource: unknown): { sourceId: string; allowedSources?: string[] } { - if (Array.isArray(rawSource)) { - const allowedSources = (rawSource as unknown[]).filter(s => typeof s === 'string' && s.length > 0) as string[]; - if (allowedSources.length > 0) { - // Scalar floor: the first granted source (write authority); reads span the array. - return { sourceId: allowedSources[0], allowedSources }; - } - return { sourceId: 'default' }; - } - if (typeof rawSource === 'string' && rawSource.length > 0) { - return { sourceId: rawSource }; - } - return { sourceId: 'default' }; -} +/* Legacy token source-scope parsing lives in core/legacy-token-scope.ts and is + * re-exported above so the legacy HTTP transport and OAuth provider cannot drift. */ /** Read up to `cap` bytes off req.body. Returns null if cap exceeded. */ async function readBodyWithCap(req: Request, cap: number): Promise { diff --git a/src/schema.sql b/src/schema.sql index a900c6d1a..3a5f32ef7 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -386,6 +386,11 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to ON code_edges_chunk(to_chunk_id, edge_type); CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_to_symbol ON code_edges_chunk(to_symbol_qualified, edge_type); +-- getCalleesOf filters on from_symbol_qualified; without this index every +-- callee lookup is a sequential scan, amplified per-BFS-node by the +-- recursive code walk. Mirrors migration v116. +CREATE INDEX IF NOT EXISTS idx_code_edges_chunk_from_symbol + ON code_edges_chunk(from_symbol_qualified); CREATE TABLE IF NOT EXISTS code_edges_symbol ( id SERIAL PRIMARY KEY, @@ -403,6 +408,10 @@ CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from ON code_edges_symbol(from_chunk_id, edge_type); CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_to ON code_edges_symbol(to_symbol_qualified, edge_type); +-- getCalleesOf companion to idx_code_edges_chunk_from_symbol above. +-- Mirrors migration v116. +CREATE INDEX IF NOT EXISTS idx_code_edges_symbol_from_symbol + ON code_edges_symbol(from_symbol_qualified); -- ============================================================ -- links: cross-references between pages diff --git a/test/ai/zeroentropy-compat-fetch.test.ts b/test/ai/zeroentropy-compat-fetch.test.ts index f2d784b1f..e12824b6d 100644 --- a/test/ai/zeroentropy-compat-fetch.test.ts +++ b/test/ai/zeroentropy-compat-fetch.test.ts @@ -53,13 +53,15 @@ describe('zeroEntropyCompatFetch — shim structural shape', () => { expect(src).not.toContain('/v1/v1/'); }); - test('body injects input_type default "document"', async () => { + test('body recovers threaded input_type, defaulting to "document"', async () => { const src = await Bun.file(GATEWAY_PATH).text(); - // The wrapper defaults input_type to 'document' when caller didn't - // thread one (matches the document-side correctness for sync / - // import / embed CLI paths). + // The wrapper recovers the SDK-stripped input_type from + // __embedInputTypeStore (#1400) and still defaults to 'document' when + // no caller threaded one (document-side correctness for sync / import / + // embed CLI paths). Wire-level behavioral coverage lives in + // test/embed-input-type-wire.test.ts. expect(src).toMatch(/parsed\.input_type\s*===\s*undefined/); - expect(src).toMatch(/parsed\.input_type\s*=\s*['"]document['"]/); + expect(src).toMatch(/parsed\.input_type\s*=\s*__embedInputTypeStore\.getStore\(\)\s*\?\?\s*['"]document['"]/); }); test('body forces encoding_format=float (CDX2-F2)', async () => { diff --git a/test/cli-force-exit-teardown-arming.test.ts b/test/cli-force-exit-teardown-arming.test.ts new file mode 100644 index 000000000..ff9743e30 --- /dev/null +++ b/test/cli-force-exit-teardown-arming.test.ts @@ -0,0 +1,56 @@ +/** + * Structural regression — the DISCONNECT_HARD_DEADLINE_MS force-exit timer in + * cli.ts main() must be armed at TEARDOWN ENTRY (inside the finally, before + * the drain + disconnect), never before the op-dispatch try block. + * + * Pre-fix bug: the 10s unref'd setTimeout was armed BEFORE the try, so any op + * whose handler ran past 10s wall-clock was killed mid-flight with + * process.exit(0) and ZERO stdout — an empty "success" indistinguishable from + * no results (a healthy `gbrain search` on a slow Postgres pooler hit this on + * every run). Armed in the finally, the timer still bounds a hung + * drain/disconnect (the C13 contract) but can no longer kill a + * slow-but-progressing op body. + * + * Source-grep is the right tool here (same rationale as + * fix-wave-structural.test.ts): the rule is "this arming must stay at this + * location". A behavioral test would need >10s of real wall-clock plus a + * deliberately slow op handler in a spawned CLI — slow and flaky by + * construction. + */ + +import { describe, test, expect } from 'bun:test'; +import { readFileSync } from 'fs'; + +describe('cli.ts — disconnect hard-deadline armed at teardown entry, not before the op body', () => { + test('forceExitTimer setTimeout lives inside the finally, gated on the daemon guard, before the drain', () => { + const src = readFileSync('src/cli.ts', 'utf8'); + + const decl = src.indexOf('const DISCONNECT_HARD_DEADLINE_MS'); + expect(decl).toBeGreaterThan(-1); + const tryIdx = src.indexOf('try {', decl); + expect(tryIdx).toBeGreaterThan(-1); + const finallyIdx = src.indexOf('} finally {', tryIdx); + expect(finallyIdx).toBeGreaterThan(-1); + const armIdx = src.indexOf('forceExitTimer = setTimeout', decl); + expect(armIdx).toBeGreaterThan(-1); + const drainIdx = src.indexOf('drainAllBackgroundWorkForCliExit', finallyIdx); + expect(drainIdx).toBeGreaterThan(-1); + + // NO arming between the deadline declaration and the op-body try — a + // pre-try timer kills slow-but-progressing op handlers mid-flight with + // exit 0 and empty stdout. (`setTimeout(` matches only a call site; the + // `ReturnType` type annotation stays allowed.) + expect(src.slice(decl, tryIdx)).not.toContain('setTimeout('); + + // The arming sits AFTER the finally opens (teardown entry) and BEFORE the + // drain + disconnect it exists to bound. + expect(armIdx).toBeGreaterThan(finallyIdx); + expect(armIdx).toBeLessThan(drainIdx); + + // Still gated on the daemon-survival guard so `serve` stays alive, and + // still unref'd + cleared on clean teardown. + expect(src.slice(finallyIdx, drainIdx)).toMatch(/if \(shouldForceExitAfterMain\(\)\)/); + expect(src.slice(finallyIdx, drainIdx)).toContain('forceExitTimer.unref?.()'); + expect(src.slice(drainIdx)).toContain('if (forceExitTimer) clearTimeout(forceExitTimer)'); + }); +}); diff --git a/test/config-env-hijack.test.ts b/test/config-env-hijack.test.ts new file mode 100644 index 000000000..3dafe6e02 --- /dev/null +++ b/test/config-env-hijack.test.ts @@ -0,0 +1,144 @@ +import { mkdtempSync, rmSync, writeFileSync } from 'fs'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { describe, expect, test } from 'bun:test'; +import { dotenvValuesForKey, effectiveEnvDatabaseUrl } from '../src/core/config.ts'; +import { withEnv } from './helpers/with-env.ts'; + +// #427 — Bun auto-loads `.env` from the process cwd, so running gbrain inside +// any web-app checkout whose `.env` defines DATABASE_URL silently retargets +// the brain at that app's database. These tests pin the guard: +// effectiveEnvDatabaseUrl() must treat a DATABASE_URL whose value matches a +// cwd .env assignment as file-origin (ignored), while still honoring +// GBRAIN_DATABASE_URL and genuinely exported DATABASE_URLs. +// +// The dir is injected instead of process.chdir'd so these tests stay safe in +// the parallel shard runner (cwd is process-global, same reason with-env.ts +// exists for process.env). + +function tmpProject(envFiles: Record): string { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-')); + for (const [name, content] of Object.entries(envFiles)) { + writeFileSync(join(dir, name), content); + } + return dir; +} + +describe('dotenvValuesForKey', () => { + test('collects assignments across all auto-loaded .env variants', () => { + const dir = tmpProject({ + '.env': 'DATABASE_URL=postgres://app:pw@prod.example.test:5432/app\n', + '.env.local': "DATABASE_URL='postgres://app:pw@local.example.test:5432/app'\n", + }); + try { + const values = dotenvValuesForKey('DATABASE_URL', dir); + expect(values.has('postgres://app:pw@prod.example.test:5432/app')).toBe(true); + expect(values.has('postgres://app:pw@local.example.test:5432/app')).toBe(true); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('handles export prefix, double quotes, comments, and unrelated keys', () => { + const dir = tmpProject({ + '.env': [ + '# comment line', + 'export DATABASE_URL="postgres://quoted.example.test/db"', + 'OTHER_KEY=not-a-db-url', + 'EMPTY=', + '', + ].join('\n'), + }); + try { + const values = dotenvValuesForKey('DATABASE_URL', dir); + expect(values.has('postgres://quoted.example.test/db')).toBe(true); + expect(values.size).toBe(1); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('returns empty set when no .env files exist', () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-none-')); + try { + expect(dotenvValuesForKey('DATABASE_URL', dir).size).toBe(0); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +describe('effectiveEnvDatabaseUrl (#427 guard)', () => { + const HIJACK_URL = 'postgres://app:pw@victim-prod.example.test:5432/app'; + const OPERATOR_URL = 'postgres://operator@deliberate.example.test:5432/brain'; + + test('ignores DATABASE_URL whose value matches a cwd .env assignment', async () => { + const dir = tmpProject({ '.env': `DATABASE_URL=${HIJACK_URL}\n` }); + try { + await withEnv( + { GBRAIN_DATABASE_URL: undefined, DATABASE_URL: HIJACK_URL }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBeUndefined(); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('honors an exported DATABASE_URL that differs from the cwd .env', async () => { + const dir = tmpProject({ '.env': `DATABASE_URL=${HIJACK_URL}\n` }); + try { + await withEnv( + { GBRAIN_DATABASE_URL: undefined, DATABASE_URL: OPERATOR_URL }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('GBRAIN_DATABASE_URL is never ignored, even when it matches the cwd .env', async () => { + const dir = tmpProject({ '.env': `GBRAIN_DATABASE_URL=${OPERATOR_URL}\nDATABASE_URL=${HIJACK_URL}\n` }); + try { + await withEnv( + { GBRAIN_DATABASE_URL: OPERATOR_URL, DATABASE_URL: HIJACK_URL }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('honors DATABASE_URL when no .env files exist in cwd', async () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-clean-')); + try { + await withEnv( + { GBRAIN_DATABASE_URL: undefined, DATABASE_URL: OPERATOR_URL }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBe(OPERATOR_URL); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('returns undefined when neither env var is set', async () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-env-hijack-unset-')); + try { + await withEnv( + { GBRAIN_DATABASE_URL: undefined, DATABASE_URL: undefined }, + () => { + expect(effectiveEnvDatabaseUrl(dir)).toBeUndefined(); + }, + ); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +}); diff --git a/test/connection-resilience.test.ts b/test/connection-resilience.test.ts index 3f73d7eff..655c8418c 100644 --- a/test/connection-resilience.test.ts +++ b/test/connection-resilience.test.ts @@ -317,8 +317,10 @@ describe('Eng-review D3 — executeRaw has no per-call retry wrapper', () => { it('Supervisor still has the 3-strikes-then-reconnect path', () => { const src = readFileSync(resolve('src/core/minions/supervisor.ts'), 'utf-8'); expect(src).toContain('consecutiveHealthFailures'); - // Supervisor invokes reconnect via a typed cast after 3 consecutive failures. - expect(src).toMatch(/reconnect\(\): Promise/); + // #2034: reconnect() is now a first-class BrainEngine method, so the + // supervisor calls it directly after 3 consecutive failures (the prior + // `(engine as unknown as { reconnect(): Promise })` cast was removed). + expect(src).toContain('this.engine.reconnect('); expect(src).toContain('this.consecutiveHealthFailures >= 3'); }); }); diff --git a/test/cross-modal-phase1.test.ts b/test/cross-modal-phase1.test.ts index eaa245540..4f5243d96 100644 --- a/test/cross-modal-phase1.test.ts +++ b/test/cross-modal-phase1.test.ts @@ -136,15 +136,17 @@ describe('D2 — knobsHash differs across cross-modal knob values', () => { return resolveSearchMode({ mode: 'balanced' }); } - test('KNOBS_HASH_VERSION is 10 (cross-modal still appended; 9→10 relational recall)', () => { + test('KNOBS_HASH_VERSION is 11 (cross-modal still appended; 10→11 asymmetric input_type fix)', () => { // v0.35 ladder: 1→2 reranker, 2→3 floor_ratio. v0.36 piggybacks on v=3 // with 7 cross-modal knobs + column/provider context. v0.40.4 (salem) + // v0.39 T21 (master) bump to v=4 for graph_signals + schema-pack fields. // v0.40.3.0 D8 bumps to v=5 (sequenced behind salem's v=4 graph-signals). // v0.41.22.0 (type-unification): 5→6 for alias_resolved post-fusion boost. // T2: 6→7 title_boost. v0.42.3.0: 7→8 autocut. issue #1777: 8→9 archive/ demote. - // v0.43: 9→10 relational recall arm. - expect(KNOBS_HASH_VERSION).toBe(10); + // v0.43: 9→10 relational recall arm. #1400: 10→11 query-side input_type + // finally reaches asymmetric providers — pre-fix rows were keyed on + // document-side query vectors. + expect(KNOBS_HASH_VERSION).toBe(11); }); test('flipping unified_multimodal changes the hash', () => { diff --git a/test/embed-input-type-wire.serial.test.ts b/test/embed-input-type-wire.serial.test.ts new file mode 100644 index 000000000..3e8fa73e1 --- /dev/null +++ b/test/embed-input-type-wire.serial.test.ts @@ -0,0 +1,247 @@ +/** + * #1400 — asymmetric `input_type` must survive the AI SDK boundary and + * reach the WIRE BODY. + * + * test/asymmetric-encoding-contract.test.ts pins that embedQuery() threads + * `input_type: 'query'` into the transport's providerOptions. This file + * pins the layer BELOW that contract: the AI SDK's openai-compatible + * adapter validates providerOptions against a fixed schema and silently + * drops `input_type` before building the HTTP body. Without the + * `__embedInputTypeStore` recovery in the per-recipe fetch shims, every + * query was encoded document-side (ZE shim's hard default) or with no + * input_type at all (Voyage, llama-server) — asymmetric retrieval silently + * collapsed while the providerOptions-level test stayed green. + * + * These tests run the REAL SDK transport with a mocked global fetch and + * assert on the outbound request body — the only place the regression is + * observable. + */ + +import { afterEach, beforeEach, describe, expect, test } from 'bun:test'; +import { configureGateway, embed, embedQuery, resetGateway } from '../src/core/ai/gateway.ts'; + +type FetchHandler = (url: string, init: RequestInit) => Promise; +let fetchHandler: FetchHandler | null = null; +const origFetch = globalThis.fetch; + +beforeEach(() => { + fetchHandler = null; + globalThis.fetch = (async (url: string | URL | Request, init?: RequestInit) => { + if (!fetchHandler) { + throw new Error('fetch called but no handler installed'); + } + return fetchHandler(typeof url === 'string' ? url : url.toString(), init ?? {}); + }) as typeof fetch; +}); + +afterEach(() => { + globalThis.fetch = origFetch; + resetGateway(); +}); + +/** OpenAI-shaped /v1/embeddings response (llama-server is already OpenAI-shaped). */ +function openAIShapedResponse(dims: number, count: number): Response { + const vec = Array.from({ length: dims }, () => 0.1); + return new Response( + JSON.stringify({ + data: Array.from({ length: count }, (_, i) => ({ object: 'embedding', index: i, embedding: vec })), + usage: { prompt_tokens: 3, total_tokens: 3 }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ); +} + +/** ZE-shaped /v1/models/embed response (zeroEntropyCompatFetch rewrites results→data). */ +function zeShapedResponse(dims: number, count: number): Response { + const vec = Array.from({ length: dims }, () => 0.1); + return new Response( + JSON.stringify({ + results: Array.from({ length: count }, () => ({ embedding: vec })), + usage: { total_bytes: 12, total_tokens: 3 }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ); +} + +/** Voyage-shaped response: base64 Float32 LE embeddings (rewriter decodes). */ +function voyageShapedResponse(dims: number, count: number): Response { + const b64 = Buffer.from(new Float32Array(dims).fill(0.1).buffer).toString('base64'); + return new Response( + JSON.stringify({ + data: Array.from({ length: count }, (_, i) => ({ object: 'embedding', index: i, embedding: b64 })), + usage: { total_tokens: 3 }, + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ); +} + +describe('ZeroEntropy hosted — input_type reaches the wire body', () => { + function configureZE() { + configureGateway({ + embedding_model: 'zeroentropyai:zembed-1', + embedding_dimensions: 1280, + env: { ZEROENTROPY_API_KEY: 'sk-fake' }, + }); + } + + test('embedQuery sends input_type=query (not the document default)', async () => { + configureZE(); + let capturedUrl = ''; + let capturedBody: any = null; + fetchHandler = async (url, init) => { + capturedUrl = url; + capturedBody = JSON.parse(init.body as string); + return zeShapedResponse(1280, 1); + }; + + await embedQuery('what does foo bar do?'); + // Sanity: the ZE shim ran (URL path rewritten). + expect(capturedUrl).toContain('/models/embed'); + expect(capturedBody.input_type).toBe('query'); + }); + + test('embed (index path) sends input_type=document', async () => { + configureZE(); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return zeShapedResponse(1280, 1); + }; + + await embed(['this is a document being indexed']); + expect(capturedBody.input_type).toBe('document'); + }); +}); + +describe('openai-compatible recipes (local/proxy asymmetric models) — input_type reaches the wire body', () => { + function configureLlamaServer(modelId: string, dims: number) { + configureGateway({ + embedding_model: `llama-server:${modelId}`, + embedding_dimensions: dims, + env: {}, + }); + } + + test('embedQuery against a local zembed-1 sends input_type=query', async () => { + configureLlamaServer('zembed-1', 1280); + let capturedUrl = ''; + let capturedBody: any = null; + fetchHandler = async (url, init) => { + capturedUrl = url; + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(1280, 1); + }; + + await embedQuery('what does foo bar do?'); + // URL untouched — llama-server's /v1/embeddings is already OpenAI-shaped. + expect(capturedUrl).toContain('/embeddings'); + expect(capturedUrl).not.toContain('/models/embed'); + expect(capturedBody.input_type).toBe('query'); + }); + + test('embed (index path) against a local zembed-1 sends input_type=document', async () => { + configureLlamaServer('zembed-1', 1280); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(1280, 1); + }; + + await embed(['this is a document being indexed']); + expect(capturedBody.input_type).toBe('document'); + }); + + test('non-asymmetric model: wire body carries NO input_type (strict pass-through)', async () => { + // dims.ts only threads input_type for recognized asymmetric models; + // for anything else the shim must leave the body untouched so vanilla + // llama-server deployments see zero wire change. + configureLlamaServer('my-gguf', 768); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(768, 1); + }; + + await embedQuery('hello'); + expect(capturedBody).not.toBeNull(); + expect('input_type' in capturedBody).toBe(false); + }); + + test('litellm proxying an asymmetric model: embedQuery sends input_type=query', async () => { + // The shim is the fallthrough default for every openai-compatible + // recipe without its own compat fetch — dims.ts threads input_type by + // model id, so a zembed-1 behind a LiteLLM proxy (e.g. fronting vLLM) + // gets the same signal as llama-server. + configureGateway({ + embedding_model: 'litellm:zembed-1', + embedding_dimensions: 1280, + env: { LITELLM_API_KEY: 'sk-fake' }, + base_urls: { litellm: 'http://localhost:4000' }, + }); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(1280, 1); + }; + + await embedQuery('what does foo bar do?'); + expect(capturedBody.input_type).toBe('query'); + }); + + test('ollama serving an asymmetric model: embedQuery sends input_type=query', async () => { + configureGateway({ + embedding_model: 'ollama:zembed-1', + embedding_dimensions: 1280, + env: {}, + }); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return openAIShapedResponse(1280, 1); + }; + + await embedQuery('what does foo bar do?'); + expect(capturedBody.input_type).toBe('query'); + }); +}); + +describe('Voyage hosted — input_type reaches the wire body (opt-in preserved)', () => { + function configureVoyage() { + configureGateway({ + embedding_model: 'voyage:voyage-3-large', + embedding_dimensions: 1024, + env: { VOYAGE_API_KEY: 'sk-fake' }, + }); + } + + test('embedQuery sends input_type=query', async () => { + configureVoyage(); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return voyageShapedResponse(1024, 1); + }; + + await embedQuery('what does foo bar do?'); + expect(capturedBody.input_type).toBe('query'); + // Existing voyage translation still applies on the same body. + expect(capturedBody.output_dimension).toBe(1024); + expect(capturedBody.encoding_format).toBe('base64'); + }); + + test('embed (index path) keeps input_type OFF the wire (pre-v0.35.0.0 opt-in shape)', async () => { + // dims.ts deliberately emits no input_type for Voyage unless threaded + // (`...(inputType ? { input_type: inputType } : {})`); the shim must + // not invent a default for it. + configureVoyage(); + let capturedBody: any = null; + fetchHandler = async (_url, init) => { + capturedBody = JSON.parse(init.body as string); + return voyageShapedResponse(1024, 1); + }; + + await embed(['this is a document being indexed']); + expect(capturedBody).not.toBeNull(); + expect('input_type' in capturedBody).toBe(false); + }); +}); diff --git a/test/embedding-dim-check-facts.test.ts b/test/embedding-dim-check-facts.test.ts index d152e6fd4..2b0a9a8fc 100644 --- a/test/embedding-dim-check-facts.test.ts +++ b/test/embedding-dim-check-facts.test.ts @@ -142,6 +142,27 @@ describe('buildFactsAlterRecipe', () => { const recipe = buildFactsAlterRecipe(1536, 1280, 'halfvec'); expect(recipe).toMatch(/DROP INDEX[\s\S]*ALTER TABLE[\s\S]*CREATE INDEX/); }); + + test('dimension change NULLs embeddings BEFORE the alter (pgvector refuses cross-dim casts)', () => { + // Same defect class fixed for content_chunks in embeddingMismatchMessage: + // pgvector aborts a cross-dimension ALTER while rows still hold old-width + // vectors. The dims-change recipe must wipe first; order pinned. + const recipe = buildFactsAlterRecipe(1536, 1280, 'halfvec'); + const nullIdx = recipe.indexOf('UPDATE facts SET embedding = NULL'); + const alterIdx = recipe.indexOf('ALTER TABLE facts ALTER COLUMN embedding TYPE'); + expect(nullIdx).toBeGreaterThan(-1); + expect(alterIdx).toBeGreaterThan(-1); + expect(nullIdx).toBeLessThan(alterIdx); + }); + + test('same-dim type swap PRESERVES embeddings (no NULL wipe)', () => { + // halfvec(1536) <-> vector(1536): the USING cast is lossless and the + // whole point is keeping the data. A wipe here would destroy valid + // embeddings for no reason. + const recipe = buildFactsAlterRecipe(1536, 1536, 'vector'); + expect(recipe).not.toContain('UPDATE facts SET embedding = NULL'); + expect(recipe).toContain('USING embedding::vector(1536)'); + }); }); describe('FactsEmbeddingDimMismatchError', () => { diff --git a/test/embedding-dim-check.test.ts b/test/embedding-dim-check.test.ts index f74ebc750..6a945b98c 100644 --- a/test/embedding-dim-check.test.ts +++ b/test/embedding-dim-check.test.ts @@ -103,6 +103,26 @@ describe('embeddingMismatchMessage', () => { expect(msg).toContain('docs/embedding-migrations.md'); }); + test('Postgres recipe NULLs embeddings BEFORE the column alter (pgvector refuses cross-dim casts)', () => { + // pgvector aborts `ALTER COLUMN TYPE vector(N)` with "expected N + // dimensions, not M" while rows still hold old-width vectors — which is + // every brain running this recipe. The UPDATE must precede the ALTER + // (NULLs cast fine). Order pinned so the printed recipe can't drift from + // the corrected docs/embedding-migrations.md again. + const msg = embeddingMismatchMessage({ + currentDims: 1536, + requestedDims: 768, + requestedModel: 'nomic-embed-text', + source: 'init', + engineKind: 'postgres', + }); + const nullIdx = msg.indexOf('UPDATE content_chunks SET embedding = NULL'); + const alterIdx = msg.indexOf('ALTER TABLE content_chunks ALTER COLUMN embedding TYPE vector(768)'); + expect(nullIdx).toBeGreaterThan(-1); + expect(alterIdx).toBeGreaterThan(-1); + expect(nullIdx).toBeLessThan(alterIdx); + }); + test('Postgres branch skips HNSW recreate when requested dims exceed pgvector cap', () => { // Codex finding #8: 2048d (Voyage 4 Large) cannot be HNSW-indexed in pgvector. // The recipe must NOT instruct a CREATE INDEX HNSW for that dim. diff --git a/test/facts-reconcile-protect-cli.test.ts b/test/facts-reconcile-protect-cli.test.ts new file mode 100644 index 000000000..b4ee032f7 --- /dev/null +++ b/test/facts-reconcile-protect-cli.test.ts @@ -0,0 +1,78 @@ +/** + * #1928 regression — extract_facts must NOT wipe conversation facts. + * + * `extract-conversation-facts` writes facts with source='cli:...' and + * source_markdown_slug=. Those pages carry no `## Facts` + * fence, so the cycle's wipe-and-reinsert reconcile (deleteFactsForPage + + * insertFactsBatch) used to delete them and reinsert nothing — a brain-wide + * conversation-facts wipe on a failed-sync full walk (status `ok`, 0 + * inserted, 1829 rows gone in the original report). + * + * The fix scopes the cycle delete with excludeSourcePrefixes: ['cli:']. + * These tests pin: (a) the exclusion protects cli:-origin rows while still + * deleting fence-owned rows on the same page coordinate, and (b) the default + * (no opts) behavior is unchanged — every fact on the coordinate is deleted. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +async function seedPageWithMixedFacts(slug: string, sourceId: string) { + await engine.insertFacts( + [ + // Fence-owned row (what extract_facts recreates from the page fence). + { fact: 'fence fact', kind: 'fact', source: 'fence', row_num: 1, source_markdown_slug: slug }, + // Empty-source row — also fence-default; must stay deletable. + { fact: 'blank-source fact', kind: 'fact', source: '', row_num: 2, source_markdown_slug: slug }, + // Conversation fact — NOT fence-owned; must survive the reconcile. + { fact: 'conversation fact', kind: 'fact', source: 'cli:extract-conversation-facts', row_num: 3, source_markdown_slug: slug }, + ], + { source_id: sourceId }, + ); +} + +async function factSourcesOnPage(slug: string, sourceId: string): Promise { + const rows = await engine.executeRaw<{ source: string }>( + `SELECT COALESCE(source, '') AS source FROM facts + WHERE source_id = $1 AND source_markdown_slug = $2 ORDER BY row_num`, + [sourceId, slug], + ); + return rows.map(r => r.source); +} + +describe('#1928 deleteFactsForPage excludeSourcePrefixes', () => { + test('protects cli:-origin facts, still deletes fence + empty-source rows', async () => { + await seedPageWithMixedFacts('transcripts/2026-06-01', 'default'); + expect((await factSourcesOnPage('transcripts/2026-06-01', 'default')).length).toBe(3); + + const { deleted } = await engine.deleteFactsForPage('transcripts/2026-06-01', 'default', { + excludeSourcePrefixes: ['cli:'], + }); + + expect(deleted).toBe(2); // fence + blank-source removed + const survivors = await factSourcesOnPage('transcripts/2026-06-01', 'default'); + expect(survivors).toEqual(['cli:extract-conversation-facts']); + }); + + test('default behavior (no opts) deletes every fact on the coordinate', async () => { + await seedPageWithMixedFacts('transcripts/2026-06-02', 'default'); + expect((await factSourcesOnPage('transcripts/2026-06-02', 'default')).length).toBe(3); + + const { deleted } = await engine.deleteFactsForPage('transcripts/2026-06-02', 'default'); + + expect(deleted).toBe(3); + expect((await factSourcesOnPage('transcripts/2026-06-02', 'default')).length).toBe(0); + }); +}); diff --git a/test/import-code-edges-source-id.test.ts b/test/import-code-edges-source-id.test.ts new file mode 100644 index 000000000..69c46be0f --- /dev/null +++ b/test/import-code-edges-source-id.test.ts @@ -0,0 +1,112 @@ +/** + * Regression — importCodeFile stamps source_id on extracted call-graph edges. + * + * Pre-fix bug: importCodeFile built CodeEdgeInput rows WITHOUT source_id, so + * every extracted edge landed NULL in code_edges_symbol. getCallersOf / + * getCalleesOf add `AND source_id = ` whenever a worktree pin or + * --source is in play — NULL never matches that filter, so scoped call-graph + * queries silently returned 0 rows on multi-source brains even though the + * edges existed. Pre-existing coverage (cathedral-ii-brainbench.test.ts, + * code-edges.test.ts) only ever queried with { allSources: true }, which + * bypasses the filter — exactly why the NULL never surfaced. + * + * This test imports a caller/callee pair under a non-default source and + * asserts (a) the persisted code_edges_symbol rows carry the source_id, and + * (b) the SCOPED getCallersOf/getCalleesOf — the user-visible path that + * returned 0 — now find the edge, while a different source scope does not. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { importCodeFile } from '../src/core/import-file.ts'; +import { runSources } from '../src/commands/sources.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + await runSources(engine, ['add', 'testsrc', '--no-federated']); + + // Same fixture shape as cathedral-ii-brainbench: runner() calls helper(), + // Layer 5 edge extraction captures the unresolved 'calls' edge — but here + // the import is pinned to a non-default source. + await importCodeFile( + engine, + 'src/a.ts', + 'export function runner() { return helper(); }\n', + { noEmbed: true, sourceId: 'testsrc' }, + ); + await importCodeFile( + engine, + 'src/b.ts', + 'export function helper() { return 42; }\n', + { noEmbed: true, sourceId: 'testsrc' }, + ); +}, 60_000); + +afterAll(async () => { + if (engine) await engine.disconnect(); +}, 30_000); + +describe('importCodeFile — source_id stamped on extracted call-graph edges', () => { + test('edges land with the import source_id and scoped caller/callee queries match', async () => { + // (a) The persisted unresolved edge rows carry the source, not NULL. + const rows = await engine.executeRaw<{ source_id: string | null }>( + `SELECT source_id FROM code_edges_symbol WHERE from_symbol_qualified = $1`, + ['runner'], + ); + expect(rows.length).toBeGreaterThanOrEqual(1); + for (const r of rows) { + expect(r.source_id).toBe('testsrc'); + } + + // (b) The scoped queries — the path that silently returned 0 pre-fix. + const callers = await engine.getCallersOf('helper', { sourceId: 'testsrc' }); + const fromRunner = callers.find(r => r.from_symbol_qualified === 'runner'); + expect(fromRunner).toBeDefined(); + expect(fromRunner!.edge_type).toBe('calls'); + expect(fromRunner!.source_id).toBe('testsrc'); + + const callees = await engine.getCalleesOf('runner', { sourceId: 'testsrc' }); + expect(callees.some(r => r.to_symbol_qualified === 'helper')).toBe(true); + + // Source isolation still holds: a different scope must NOT see the edge. + const otherScope = await engine.getCallersOf('helper', { sourceId: 'default' }); + expect(otherScope.find(r => r.from_symbol_qualified === 'runner')).toBeUndefined(); + }); + + test('UNSCOPED import stamps edges with the schema-default source, not NULL', async () => { + // The other door of the same bug: an import WITHOUT opts.sourceId (legacy + // unscoped callers — `gbrain reindex --code` with no --source) lands its + // pages under the schema default (pages.source_id DEFAULT 'default'). + // If its edges were stamped NULL, the matching scoped query + // getCallersOf(sym, { sourceId: 'default' }) — a worktree pinned to + // default, --source default, GBRAIN_SOURCE=default — would miss them. + await importCodeFile( + engine, + 'src/c.ts', + 'export function unscopedRunner() { return unscopedHelper(); }\n', + { noEmbed: true }, + ); + await importCodeFile( + engine, + 'src/d.ts', + 'export function unscopedHelper() { return 7; }\n', + { noEmbed: true }, + ); + + const rows = await engine.executeRaw<{ source_id: string | null }>( + `SELECT source_id FROM code_edges_symbol WHERE from_symbol_qualified = $1`, + ['unscopedRunner'], + ); + expect(rows.length).toBeGreaterThanOrEqual(1); + for (const r of rows) { + expect(r.source_id).toBe('default'); + } + + const callers = await engine.getCallersOf('unscopedHelper', { sourceId: 'default' }); + expect(callers.some(r => r.from_symbol_qualified === 'unscopedRunner')).toBe(true); + }); +}); diff --git a/test/list-all-sources.test.ts b/test/list-all-sources.test.ts index 4d0f9ddb5..fddfc9b0a 100644 --- a/test/list-all-sources.test.ts +++ b/test/list-all-sources.test.ts @@ -135,4 +135,51 @@ describe('engine.updateSourceConfig', () => { const all = await engine.listAllSources(); expect(all.find(s => s.id === 'delta')!.config.last_full_cycle_at).toBe('2026-05-22T11:00:00.000Z'); }); + + // IS JSON guard: the postgres-engine atomic merge gates its `::jsonb` cast + // behind the SQL `IS JSON` predicate so a historical bad row whose config is + // a JSONB string of NON-JSON text normalizes to `{}` instead of raising + // `invalid input syntax for type json` and aborting the UPDATE. + // + // We exercise the SQL CASE expression directly via executeRaw against PGLite + // (which ships Postgres 17 + IS JSON parity) rather than calling + // PostgresEngine.updateSourceConfig (which requires a live Postgres pool). + test('IS-JSON guard: non-JSON string config normalizes to {} on merge', async () => { + const patch = { merged_key: 'v' }; + + const guarded = (configExpr: string) => ` + SELECT ( + CASE + WHEN jsonb_typeof(${configExpr}) = 'object' THEN ${configExpr} + WHEN jsonb_typeof(${configExpr}) = 'string' + THEN CASE + WHEN (${configExpr} #>> '{}') IS JSON + THEN COALESCE(NULLIF((${configExpr} #>> '{}'), '')::jsonb, '{}'::jsonb) + ELSE '{}'::jsonb + END + WHEN jsonb_typeof(${configExpr}) = 'array' + THEN COALESCE( + (SELECT jsonb_object_agg(kv.key, kv.value) + FROM jsonb_array_elements(${configExpr}) elem, + jsonb_each(elem) kv), + '{}'::jsonb + ) + ELSE '{}'::jsonb + END || $1::jsonb + ) AS result`; + + // 1. JSONB string holding NON-JSON text → normalizes to {} then merges patch. + const bad = await engine.executeRaw<{ result: Record }>( + guarded(`to_jsonb('garbage text'::text)`), + [JSON.stringify(patch)], + ); + expect(bad[0].result).toEqual({ merged_key: 'v' }); + + // 2. JSONB string holding double-encoded valid JSON object → parsed + merged. + const good = await engine.executeRaw<{ result: Record }>( + guarded(`to_jsonb('{"x":1}'::text)`), + [JSON.stringify(patch)], + ); + expect(good[0].result).toEqual({ x: 1, merged_key: 'v' }); + }); }); diff --git a/test/oauth-authorize-scope-default.test.ts b/test/oauth-authorize-scope-default.test.ts new file mode 100644 index 000000000..4dc2c3d81 --- /dev/null +++ b/test/oauth-authorize-scope-default.test.ts @@ -0,0 +1,89 @@ +/** + * Authorize-grant scope default (RFC 6749 §3.3). + * + * When a client omits `scope` on /authorize, the granted scope must default to + * the client's full registered scope — NOT the empty set. Regression guard for + * the bug where an omitted request granted [], which then propagated into the + * access + refresh tokens and never self-healed: every op failed + * `insufficient_scope` even though the client was registered `read write` + * (some MCP connectors omit `scope` on /authorize). The clamp must still hold — + * an explicit over-broad request cannot escalate past the client's allowed set. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { GBrainOAuthProvider } from '../src/core/oauth-provider.ts'; +import { sqlQueryForEngine } from '../src/core/sql-query.ts'; + +let engine: PGLiteEngine; +let provider: GBrainOAuthProvider; +let sql: ReturnType; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + sql = sqlQueryForEngine(engine); + provider = new GBrainOAuthProvider({ sql }); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await (engine as any).db.exec('DELETE FROM oauth_tokens'); + await (engine as any).db.exec('DELETE FROM oauth_codes'); + await (engine as any).db.exec('DELETE FROM oauth_clients'); +}); + +// authorize() writes the granted scope into oauth_codes then redirects; we +// assert on the stored grant directly, so the redirect is a no-op. +const noopRes = { redirect() {} } as any; + +async function authorizeAndReadScopes( + scope: string, + requested: string[] | undefined, +): Promise { + const reg = await provider.registerClientManual( + 'authz-test', ['authorization_code'], scope, ['https://example.test/cb'], + ); + const client = await provider.clientsStore.getClient(reg.clientId); + expect(client).toBeTruthy(); + await provider.authorize( + client!, + { + scopes: requested, + codeChallenge: 'test-challenge', + redirectUri: 'https://example.test/cb', + state: 'xyz', + } as any, + noopRes, + ); + const rows = (await sql` + SELECT scopes FROM oauth_codes WHERE client_id = ${reg.clientId} + `) as Array<{ scopes: string[] }>; + expect(rows.length).toBe(1); + return rows[0].scopes ?? []; +} + +describe('authorize() scope default — omitted scope inherits client grant', () => { + test('omitted scope → inherits full registered scope', async () => { + expect((await authorizeAndReadScopes('read write', undefined)).sort()).toEqual(['read', 'write']); + }); + + test('empty scope array → inherits full registered scope', async () => { + expect((await authorizeAndReadScopes('read write', [])).sort()).toEqual(['read', 'write']); + }); + + test('explicit subset is honored (not overridden to full)', async () => { + expect(await authorizeAndReadScopes('read write admin', ['read'])).toEqual(['read']); + }); + + test('clamp preserved: over-broad request cannot escalate', async () => { + expect(await authorizeAndReadScopes('read', ['read', 'admin'])).toEqual(['read']); + }); + + test('clamp preserved: requesting only a disallowed scope grants nothing (no inheritance)', async () => { + expect(await authorizeAndReadScopes('read write', ['admin'])).toEqual([]); + }); +}); diff --git a/test/oauth.test.ts b/test/oauth.test.ts index 257ebb2d7..3c466ec5b 100644 --- a/test/oauth.test.ts +++ b/test/oauth.test.ts @@ -12,6 +12,7 @@ import { import { hashToken, generateToken } from '../src/core/utils.ts'; import { PGLITE_SCHEMA_SQL } from '../src/core/pglite-schema.ts'; import { InvalidTokenError } from '@modelcontextprotocol/sdk/server/auth/errors.js'; +import type { AuthInfo as CoreAuthInfo } from '../src/core/operations.ts'; // --------------------------------------------------------------------------- // Test setup: in-memory PGLite with OAuth tables @@ -310,6 +311,33 @@ describe('verifyAccessToken', () => { expect(authInfo.clientId).toBe('legacy-agent'); expect(authInfo.scopes).toEqual(['read', 'write', 'admin']); // grandfathered full access }); + + test('legacy access_tokens fallback honors permissions.source_id array grants', async () => { + // oauth.test.ts initializes the static PGLite schema blob, not the full + // migration stack. Add the v38 permissions column here so the row matches + // a modern brain carrying a legacy-token source grant. + await sql` + ALTER TABLE access_tokens + ADD COLUMN IF NOT EXISTS permissions JSONB NOT NULL DEFAULT '{"takes_holders":["world"]}'::jsonb + `; + + const legacyToken = generateToken('gbrain_'); + const hash = hashToken(legacyToken); + await sql` + INSERT INTO access_tokens (id, name, token_hash, permissions) + VALUES ( + ${crypto.randomUUID()}, + ${'legacy-federated-agent'}, + ${hash}, + ${JSON.stringify({ source_id: ['default', 'src-a', 'src-b'] })}::jsonb + ) + `; + + const authInfo = await provider.verifyAccessToken(legacyToken) as CoreAuthInfo; + expect(authInfo.clientId).toBe('legacy-federated-agent'); + expect(authInfo.sourceId).toBe('default'); + expect(authInfo.allowedSources).toEqual(['default', 'src-a', 'src-b']); + }); }); // --------------------------------------------------------------------------- diff --git a/test/pglite-lock.test.ts b/test/pglite-lock.test.ts index 3253aa41b..e9b603bbf 100644 --- a/test/pglite-lock.test.ts +++ b/test/pglite-lock.test.ts @@ -3,6 +3,7 @@ import { mkdirSync, rmSync, existsSync, readFileSync, writeFileSync } from 'fs'; import { join } from 'path'; import { tmpdir } from 'os'; import { acquireLock, releaseLock, type LockHandle } from '../src/core/pglite-lock'; +import { withEnv } from './helpers/with-env.ts'; const TEST_DIR = join(tmpdir(), 'gbrain-lock-test-' + process.pid); @@ -99,3 +100,96 @@ describe('pglite-lock', () => { await releaseLock(lock2); }); }); + +describe('pglite-lock #2058 heartbeat + steal-grace', () => { + beforeEach(() => { + if (existsSync(TEST_DIR)) rmSync(TEST_DIR, { recursive: true, force: true }); + mkdirSync(TEST_DIR, { recursive: true }); + }); + afterEach(() => { + if (existsSync(TEST_DIR)) rmSync(TEST_DIR, { recursive: true, force: true }); + }); + + function writeHolder(fields: { pid: number; acquiredAgoMs: number; refreshedAgoMs: number }) { + const lockDir = join(TEST_DIR, '.gbrain-lock'); + mkdirSync(lockDir, { recursive: true }); + const now = Date.now(); + writeFileSync(join(lockDir, 'lock'), JSON.stringify({ + pid: fields.pid, + acquired_at: now - fields.acquiredAgoMs, + refreshed_at: now - fields.refreshedAgoMs, + command: 'test holder', + })); + } + + test('[REGRESSION] a LIVE holder with a fresh heartbeat is NOT stolen even when the lock is old', async () => { + // The WAL-corruption bug: a >5min embed used to get its lock force-removed. + // Now an alive holder that heartbeated recently is left alone regardless of + // age. acquired 20min ago, but refreshed just now → must wait, not steal. + writeHolder({ pid: process.pid, acquiredAgoMs: 20 * 60_000, refreshedAgoMs: 0 }); + + await expect(acquireLock(TEST_DIR, { timeoutMs: 1200 })).rejects.toThrow(/Timed out/); + // Holder's lock still present (was never stolen). + expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(true); + }); + + test('a LIVE PID whose heartbeat went stale past the grace window IS reaped', async () => { + // PID is alive (our own) but hasn't refreshed in 20min (> 600s grace): + // hung holder, or a reused PID whose real holder is gone. Reap + acquire. + writeHolder({ pid: process.pid, acquiredAgoMs: 25 * 60_000, refreshedAgoMs: 20 * 60_000 }); + + const lock = await acquireLock(TEST_DIR, { timeoutMs: 2000 }); + expect(lock.acquired).toBe(true); + await releaseLock(lock); + }); + + test('GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS tunes the grace window', async () => { + // withEnv keeps the process-global mutation isolated across shard files. + await withEnv({ GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS: '5' }, async () => { + // Refreshed 30s ago — fresh under the 600s default, STALE under 5s. + writeHolder({ pid: process.pid, acquiredAgoMs: 60_000, refreshedAgoMs: 30_000 }); + const lock = await acquireLock(TEST_DIR, { timeoutMs: 2000 }); + expect(lock.acquired).toBe(true); + await releaseLock(lock); + }); + }); + + test('[REGRESSION] releaseLock does NOT remove a lock that was stolen + re-acquired by another process', async () => { + // We acquire, then simulate a steal: another process reaped us past grace + // and now owns the lock (different pid + acquired_at). Our releaseLock must + // NOT delete their live lock — doing so would let a third process in + // alongside the new owner (the #2058 corruption class). + const lock: LockHandle = await acquireLock(TEST_DIR); + expect(lock.acquired).toBe(true); + expect(lock.ownerToken).toBeDefined(); + if (lock.heartbeat) clearInterval(lock.heartbeat); // stop our heartbeat for a deterministic test + + // Overwrite the lock file as if process B re-acquired it. + const lockFile = join(TEST_DIR, '.gbrain-lock', 'lock'); + const bNow = Date.now() + 1; + writeFileSync(lockFile, JSON.stringify({ pid: 999999, acquired_at: bNow, refreshed_at: bNow, command: 'process B' })); + + await releaseLock(lock); // our (stale) handle + + // B's lock survives — we did not clobber it. + expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(true); + const after = JSON.parse(readFileSync(lockFile, 'utf-8')); + expect(after.pid).toBe(999999); + + // Cleanup for afterEach. + rmSync(join(TEST_DIR, '.gbrain-lock'), { recursive: true, force: true }); + }); + + test('acquire starts a heartbeat and seeds refreshed_at; release clears it', async () => { + const lock: LockHandle = await acquireLock(TEST_DIR); + expect(lock.acquired).toBe(true); + expect(lock.heartbeat).toBeDefined(); + const data = JSON.parse(readFileSync(join(TEST_DIR, '.gbrain-lock', 'lock'), 'utf-8')); + expect(data.refreshed_at).toBeDefined(); + expect(typeof data.refreshed_at).toBe('number'); + + await releaseLock(lock); + expect(lock.heartbeat).toBeUndefined(); + expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(false); + }); +}); diff --git a/test/pglite-reconnect.serial.test.ts b/test/pglite-reconnect.serial.test.ts new file mode 100644 index 000000000..15eb2fa6d --- /dev/null +++ b/test/pglite-reconnect.serial.test.ts @@ -0,0 +1,45 @@ +/** + * #2034 — engine-parity reconnect(). + * + * The autopilot health-probe recovery path used `disconnect()` + bare + * `connect()`, which (a) threw `database_url undefined` forever on Postgres + * because the config was lost, and (b) was a silent no-op on PGLite which had + * no reconnect() at all. Both engines now expose `reconnect()`; this pins the + * PGLite side: it restores connectivity against the config captured at the + * last connect(), and is a safe no-op when never connected. + */ + +import { describe, test, expect } from 'bun:test'; +import { mkdtempSync, rmSync } from 'fs'; +import { join } from 'path'; +import { tmpdir } from 'os'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; + +describe('#2034 PGLiteEngine.reconnect', () => { + test('restores connectivity and persisted state against the saved config', async () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-reconnect-')); + const engine = new PGLiteEngine(); + try { + await engine.connect({ database_path: dir }); + await engine.initSchema(); + await engine.setConfig('reconnect.probe', 'pre'); + + // Reconnect with no args — the #2034 contract: it must reuse the config + // captured at connect(), not require it to be re-passed. + await engine.reconnect(); + + // Still queryable, and persisted state survived (same data dir). + const rows = await engine.executeRaw<{ x: number }>('SELECT 1 AS x'); + expect(rows[0]?.x).toBe(1); + expect(await engine.getConfig('reconnect.probe')).toBe('pre'); + } finally { + await engine.disconnect(); + rmSync(dir, { recursive: true, force: true }); + } + }); + + test('reconnect() before any connect() is a safe no-op', async () => { + const engine = new PGLiteEngine(); + await expect(engine.reconnect()).resolves.toBeUndefined(); + }); +}); diff --git a/test/search-alias-resolved-boost.test.ts b/test/search-alias-resolved-boost.test.ts index e67bb0300..57fe7ace0 100644 --- a/test/search-alias-resolved-boost.test.ts +++ b/test/search-alias-resolved-boost.test.ts @@ -89,7 +89,7 @@ describe('alias_resolved boost stage', () => { }); describe('KNOBS_HASH_VERSION', () => { - it('is 10 (9→10 relational recall arm invalidates rel-off cache rows, v0.43)', () => { - expect(KNOBS_HASH_VERSION).toBe(10); + it('is 11 (10→11 asymmetric input_type fix invalidates document-side query-vector rows, #1400)', () => { + expect(KNOBS_HASH_VERSION).toBe(11); }); }); diff --git a/test/search-mode.test.ts b/test/search-mode.test.ts index c2a3516e1..a8e9ecf18 100644 --- a/test/search-mode.test.ts +++ b/test/search-mode.test.ts @@ -403,7 +403,11 @@ describe('knobsHash determinism + cross-mode separation (CDX-4)', () => { // must not be served from a cache row written before the policy change. // v0.43: bumped 9→10 for the relational recall arm (rel=/reld=) — a // relational-on write must not be served to a relational-off lookup. - expect(KNOBS_HASH_VERSION).toBe(10); + // #1400: bumped 10→11 for the asymmetric input_type fix — embedQuery() + // now produces query-side vectors for asymmetric providers (zembed-1, + // Voyage v3+), so rows keyed on pre-fix document-side query vectors + // must not be served to post-fix lookups. + expect(KNOBS_HASH_VERSION).toBe(11); }); test('T1 (codex): floor_ratio set vs unset produces DIFFERENT hashes (cache contamination prevention)', () => { @@ -568,8 +572,8 @@ describe('v0.40.4 — graph_signals knob', () => { }); describe('v0.42.3.0 — autocut knobs', () => { - test('KNOBS_HASH_VERSION is 10 (9→10 relational recall arm, v0.43)', () => { - expect(KNOBS_HASH_VERSION).toBe(10); + test('KNOBS_HASH_VERSION is 11 (10→11 asymmetric input_type fix, #1400)', () => { + expect(KNOBS_HASH_VERSION).toBe(11); }); test('bundle defaults: conservative off, balanced/tokenmax on @0.20', () => { diff --git a/test/search/knobs-hash-reranker.test.ts b/test/search/knobs-hash-reranker.test.ts index 509e98249..bd793d427 100644 --- a/test/search/knobs-hash-reranker.test.ts +++ b/test/search/knobs-hash-reranker.test.ts @@ -43,7 +43,7 @@ function baseKnobs(): ResolvedSearchKnobs { } describe('KNOBS_HASH_VERSION + version invariants', () => { - test('version is 10 (…; 7→8 autocut; 8→9 archive-demote #1777; 9→10 relational recall)', () => { + test('version is 11 (…; 8→9 archive-demote #1777; 9→10 relational recall; 10→11 asymmetric input_type #1400)', () => { // v0.35.0.0: 1→2 to fold reranker fields. v0.35.6.0: 2→3 to fold // floor_ratio. v0.36 wave: piggybacks on v=3 with 7 cross-modal knobs // (D2) PLUS column + provider context (D8/CDX-2 cross-column isolation). @@ -58,7 +58,10 @@ describe('KNOBS_HASH_VERSION + version invariants', () => { // autocut. issue #1777: 8→9 archive/ demote (search-exclude policy change // isn't in the hash, so the bump invalidates archive-excluded cache rows). // v0.43: 9→10 relational recall arm (rel=/reld=). - expect(KNOBS_HASH_VERSION).toBe(10); + // #1400: 10→11 asymmetric input_type fix — embedQuery() now produces + // query-side vectors for asymmetric providers, so rows keyed on + // pre-fix document-side query vectors must not be served. + expect(KNOBS_HASH_VERSION).toBe(11); }); test('hash is 16 hex chars regardless of reranker config', () => { diff --git a/test/timeline-date-batch.test.ts b/test/timeline-date-batch.test.ts new file mode 100644 index 000000000..eba9e009b --- /dev/null +++ b/test/timeline-date-batch.test.ts @@ -0,0 +1,76 @@ +/** + * #2057 regression — addTimelineEntriesBatch must accept JS Date `date` values. + * + * The original bug: callers source `date` from SQL rows (e.g. + * `meeting.effective_date`), which arrive as JS Date objects. The OLD insert + * bound them into `${dates}::text[]`, which threw `cannot cast type timestamp + * with time zone to text[]`, and a bare `catch {}` in the meetings extractor + * swallowed it — timeline stayed empty forever. + * + * The #1861 refactor moved the insert to `jsonb_to_recordset` + `v.date::date`, + * where a Date serializes to an ISO string that casts cleanly. This test pins + * that a Date-bearing batch round-trips, so a future refactor can't silently + * reintroduce the cast failure. (The companion fix un-silences the extractor's + * catch so any future failure is visible rather than a phantom "0 entries".) + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { importFromContent } from '../src/core/import-file.ts'; +import type { TimelineBatchInput } from '../src/core/engine.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + await importFromContent(engine, 'people/alice-example', `---\ntitle: Alice\ntype: note\n---\n\n# Alice\n`, { + noEmbed: true, + sourceId: 'default', + sourcePath: 'people/alice-example.md', + }); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +describe('#2057 addTimelineEntriesBatch with Date date values', () => { + test('a Date `date` is inserted and round-trips as the right calendar day', async () => { + // Mimic the real caller: `date` typed string on the interface, but a JS + // Date at runtime (straight off a TIMESTAMPTZ column). The cast must hold. + const effectiveDate = new Date('2026-04-03T00:00:00.000Z'); + const batch: TimelineBatchInput[] = [ + { + slug: 'people/alice-example', + date: effectiveDate as unknown as string, + source: 'cli:test', + summary: 'met alice', + source_id: 'default', + }, + ]; + + const inserted = await engine.addTimelineEntriesBatch(batch); + expect(inserted).toBe(1); + + const rows = await engine.executeRaw<{ date: string }>( + `SELECT date::text AS date FROM timeline_entries WHERE summary = 'met alice'`, + ); + expect(rows.length).toBe(1); + expect(rows[0].date).toBe('2026-04-03'); + }); + + test('a plain ISO string `date` still works (no regression)', async () => { + const inserted = await engine.addTimelineEntriesBatch([ + { + slug: 'people/alice-example', + date: '2026-05-01', + source: 'cli:test', + summary: 'string-dated entry', + source_id: 'default', + }, + ]); + expect(inserted).toBe(1); + }); +}); diff --git a/test/timeline-dedup-repair.test.ts b/test/timeline-dedup-repair.test.ts new file mode 100644 index 000000000..bfacab68a --- /dev/null +++ b/test/timeline-dedup-repair.test.ts @@ -0,0 +1,119 @@ +/** + * #2038 — idx_timeline_dedup schema-drift self-heal. + * + * A brain that ran the pre-renumber v99 variant of the dedup migration is + * stamped past v102 with the OLD 3-column index. `runMigrations` early-returns + * (nothing pending) so a migration verify-hook can't fix it. The repair is + * keyed off the index SHAPE and runs regardless. These tests simulate the + * drifted states directly and pin: detection, rebuild, dedupe-before-rebuild + * (only possible when the index was absent), and idempotency. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { + checkTimelineDedupIndex, + repairTimelineDedupIndex, +} from '../src/core/timeline-dedup-repair.ts'; +import { importFromContent } from '../src/core/import-file.ts'; + +let engine: PGLiteEngine; +let pageId: string; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + await importFromContent(engine, 'people/alice-example', `---\ntitle: Alice\ntype: note\n---\n\n# Alice\n`, { + noEmbed: true, + sourceId: 'default', + sourcePath: 'people/alice-example.md', + }); + const pid = await engine.executeRaw<{ id: string }>( + `SELECT id::text AS id FROM pages WHERE slug = 'people/alice-example' AND source_id = 'default'`, + ); + pageId = pid[0].id; +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +/** Force the index back to the broken pre-v102 3-column shape. */ +async function regressTo3Col() { + await engine.executeRaw(`DELETE FROM timeline_entries`); + await engine.executeRaw(`DROP INDEX IF EXISTS idx_timeline_dedup`); + await engine.executeRaw( + `CREATE UNIQUE INDEX idx_timeline_dedup ON timeline_entries(page_id, date, summary)`, + ); +} + +/** The other drift shape: the index was dropped entirely, letting true + * 4-tuple duplicates accumulate that would block a naive CREATE UNIQUE INDEX. */ +async function regressToAbsentWithDupes() { + await engine.executeRaw(`DELETE FROM timeline_entries`); + await engine.executeRaw(`DROP INDEX IF EXISTS idx_timeline_dedup`); + await engine.executeRaw( + `INSERT INTO timeline_entries (page_id, date, summary, source, detail) + VALUES ($1, '2026-04-03', 'met alice', 'meeting', ''), + ($1, '2026-04-03', 'met alice', 'meeting', ''), + ($1, '2026-04-03', 'met alice', 'cli:extract', '')`, + [pageId], + ); +} + +describe('#2038 idx_timeline_dedup drift repair', () => { + test('detects the 3-column drift', async () => { + await regressTo3Col(); + const status = await checkTimelineDedupIndex(engine); + expect(status.tablePresent).toBe(true); + expect(status.indexPresent).toBe(true); + expect(status.columns).toEqual(['page_id', 'date', 'summary']); + expect(status.needsRepair).toBe(true); + }); + + test('rebuilds the 3-column index to 4 columns (no dupes to collapse)', async () => { + await regressTo3Col(); + await engine.executeRaw( + `INSERT INTO timeline_entries (page_id, date, summary, source, detail) + VALUES ($1, '2026-04-03', 'met alice', 'meeting', '')`, + [pageId], + ); + + const res = await repairTimelineDedupIndex(engine); + expect(res.repaired).toBe(true); + expect(res.reason).toBe('rebuilt'); + expect(res.collapsedDuplicates).toBe(0); + + const after = await checkTimelineDedupIndex(engine); + expect(after.columns).toEqual(['page_id', 'date', 'summary', 'source']); + expect(after.needsRepair).toBe(false); + }); + + test('dedupes true 4-tuple duplicates before building the unique index', async () => { + await regressToAbsentWithDupes(); // index absent + a real (meeting) dup + + const before = await checkTimelineDedupIndex(engine); + expect(before.indexPresent).toBe(false); + expect(before.needsRepair).toBe(true); + + const res = await repairTimelineDedupIndex(engine); + expect(res.repaired).toBe(true); + expect(res.collapsedDuplicates).toBe(1); // one of the two 'meeting' rows removed + + const after = await checkTimelineDedupIndex(engine); + expect(after.columns).toEqual(['page_id', 'date', 'summary', 'source']); + const rows = await engine.executeRaw<{ n: string }>( + `SELECT COUNT(*)::text AS n FROM timeline_entries`, + ); + expect(parseInt(rows[0].n, 10)).toBe(2); // meeting (deduped) + cli:extract + }); + + test('idempotent — a second repair is a no-op', async () => { + await regressTo3Col(); + await repairTimelineDedupIndex(engine); + const second = await repairTimelineDedupIndex(engine); + expect(second.repaired).toBe(false); + expect(second.reason).toBe('already_correct'); + }); +}); diff --git a/test/write-through.test.ts b/test/write-through.test.ts index 67ed55117..238082a60 100644 --- a/test/write-through.test.ts +++ b/test/write-through.test.ts @@ -120,6 +120,58 @@ describe('writePageThrough', () => { expect(res).toEqual({ written: false, skipped: 'page_not_found_after_write' }); }); + test('[REGRESSION #2018] default page (null local_path) in a multi-source brain → skipped, no leak into a sibling source repo', async () => { + // A sibling federated source with its OWN working tree. + const siblingDir = path.join(tmpRoot, 'housefax'); + fs.mkdirSync(siblingDir, { recursive: true }); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, config) VALUES ('housefax', 'Housefax', $1, '{}'::jsonb)`, + [siblingDir], + ); + // The leak trigger: global sync.repo_path points at the sibling's tree + // while the default source (re-seeded by resetPgliteState) has no + // local_path of its own. + await engine.setConfig('sync.repo_path', siblingDir); + + const slug = 'internal/cross-cutting-note'; + await seedPage(slug); // sourceId 'default' + + const res = await writePageThrough(engine, slug, { sourceId: 'default' }); + + expect(res).toEqual({ written: false, skipped: 'source_repo_belongs_to_other_source' }); + // The sibling source's repo stays clean — the whole point of #2018. + expect(walkFiles(siblingDir).some((f) => f.endsWith('.md'))).toBe(false); + }); + + test('[#2018] page assigned to a source with its own local_path writes to that tree root, not the global path', async () => { + const alphaDir = path.join(tmpRoot, 'alpha-repo'); + fs.mkdirSync(alphaDir, { recursive: true }); + const globalDir = path.join(tmpRoot, 'global-repo'); + fs.mkdirSync(globalDir, { recursive: true }); + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, config) VALUES ('alpha', 'Alpha', $1, '{}'::jsonb)`, + [alphaDir], + ); + // Must NOT be used — the assigned source has its own tree. + await engine.setConfig('sync.repo_path', globalDir); + + const slug = 'notes/alpha-thing'; + await importFromContent(engine, slug, `---\ntitle: T\ntype: note\n---\n\n# Body\n`, { + noEmbed: true, + sourceId: 'alpha', + sourcePath: `${slug}.md`, + }); + + const res = await writePageThrough(engine, slug, { sourceId: 'alpha' }); + + expect(res.written).toBe(true); + // File at the source's tree ROOT, never nested under `.sources//`. + expect(res.path).toBe(path.join(alphaDir, `${slug}.md`)); + expect(fs.existsSync(path.join(alphaDir, `${slug}.md`))).toBe(true); + // The global repo path is untouched. + expect(walkFiles(globalDir).some((f) => f.endsWith('.md'))).toBe(false); + }); + test('[REGRESSION] mkdir ENOTDIR (parent is a file) → error, no partial .md, no .tmp', async () => { await engine.setConfig('sync.repo_path', brainDir); // Block the `wiki/` directory by putting a FILE named "wiki" under the repo,