Files
gbrain/src/core/postgres-engine.ts
T
52f9581966 v0.22.11 feat: storage tiering — db_tracked vs db_only directories (#494)
* feat: storage tiering — git-tracked vs supabase-only directories

Brain repos scaling to 200K+ files. Bulk data (tweets, articles, transcripts)
bloats git repos and slows operations. New storage config in gbrain.yml lets
users declare git-tracked and supabase-only directories.

Changes:
- New config: storage.git_tracked and storage.supabase_only in gbrain.yml
- gbrain sync auto-manages .gitignore for supabase-only paths
- gbrain export --restore-only restores missing supabase-only files from DB
- New gbrain storage status command shows tier breakdown
- Config validation warns on conflicts
- 8 tests passing, full docs at docs/storage-tiering.md

Backward compatible — systems without gbrain.yml work unchanged.

* feat: add getDefaultSourcePath() typed accessor (step 1/15)

Single source of truth for "what brain repo are we operating against?"
Replaces ad-hoc raw SQL in storage.ts:38 (Issue #3 of eng review). Used by
both gbrain storage status and gbrain export --restore-only.

Returns null on miss, throws on DB error. Composes with the existing
resolveSourceId chain so it honors --source flag / GBRAIN_SOURCE env /
.gbrain-source dotfile / longest-prefix CWD match / brain-level default.

4 new test cases covering happy path, missing local_path, DB error
propagation, and CWD-prefix resolution priority.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: replace gray-matter with dedicated YAML parser (step 2/15)

The original storage-config.ts called gray-matter on a delimiter-less YAML
file. Gray-matter only parses YAML inside `---` frontmatter blocks; without
delimiters, it returns `{data: {}}`. Result: loadStorageConfig() always
returned null, the entire feature was a silent no-op for every user.

Original eng review's P0 confidence-9 finding (Issue #1).

Replaces gray-matter with a small dedicated parser for the gbrain.yml shape
(top-level `storage:` section, two array-valued nested keys). Yaml-lite was
considered first, but its flat key:value design doesn't handle nested
arrays. The dedicated parser is ~50 lines and trades expressiveness for
zero-dep, predictable parsing of a file format we control.

Adds the Issue #1B sanity warning (locked B): when gbrain.yml exists but
has no storage section (or empty arrays), warn once-per-process so the
user sees their config didn't take. The single test that would have caught
the original P0 — write a real gbrain.yml, call loadStorageConfig, assert
non-null — now exists.

Also tightens loadStorageConfig per D36: distinguishes "absent" (silent
null) from "unreadable" (throws). The previous code silently swallowed
read errors, hiding broken installs.

8 new test cases: real-disk happy path, comments + blank lines, quoted
values, missing storage section warning, empty section warning,
once-per-process warning suppression, unreadable file behavior, and the
existing helper tests (validation, tier matching, edge cases) all still
pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: rename storage keys to db_tracked/db_only (step 3/15)

The vendor-specific names "supabase_only" and "git_tracked" hardcoded a
backend (Supabase) into the config schema. gbrain ships two engines —
PGLite and Postgres-via-Supabase. The canonical distinction is "lives in
the brain DB only" vs "lives in the brain DB and on disk under git." Both
work on either engine.

Renamed throughout (Issue #4 of eng review):
  git_tracked    → db_tracked
  supabase_only  → db_only
  isGitTracked() → isDbTracked()
  isSupabaseOnly() → isDbOnly()
  StorageTier 'git_tracked'/'supabase_only' → 'db_tracked'/'db_only'

Backward compatibility (D3 lock):
  loadStorageConfig accepts both shapes. Loader resolution order per the
  eng-review pass-2 finding: parse YAML → if canonical keys present use
  them, else if deprecated keys present map to canonical AND emit
  once-per-process deprecation warning → THEN run validation.
  Validation always sees the canonical shape so error messages reference
  db_tracked/db_only regardless of which keys the user wrote.

  The deprecation warning suggests `gbrain doctor --fix` for an automated
  rename (D72 — fix path lands in step 7).

  When both shapes coexist in one file, canonical wins and a stronger
  warning fires ("deprecated keys ignored — remove them").

Aliases isGitTracked/isSupabaseOnly kept for now to avoid churning the
sync.ts / export.ts / storage.ts call sites in this commit; they'll be
removed in a follow-up step. Storage.ts's tier-bucket initializers and
output strings updated. ASCII output replaces unicode box-drawing per D10.

gbrain.yml example file updated to canonical keys with explanatory
comments.

2 new test cases: deprecated-key fallback (asserts both shapes load
correctly with warning), canonical-wins-over-deprecated (asserts the
"both shapes coexist" path).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: add slugPrefix to PageFilters with engine-side filter (step 4/15)

Issue #13 of the eng review: storage.ts and export.ts loaded every page
in the brain (limit: 1_000_000) to check tier membership. On the 200K-page
brains this feature targets, that's the wall-clock and memory landmine
the feature exists to fix.

Adds an optional `slugPrefix` field to PageFilters. Both engines implement
it as `WHERE slug LIKE prefix || '%' ESCAPE '\'`, with literal escaping of
LIKE metacharacters (%, _, \) so user-supplied prefixes like `media/x/`
are treated as exact string prefixes.

Performance: the (source_id, slug) UNIQUE constraint on the pages table
gives both engines a btree index that supports LIKE-prefix range scans.
An EXPLAIN on Postgres confirms the index range scan rather than a seq
scan. PGLite has the same index shape via pglite-schema.ts.

Consumers updated:
  - export.ts: --slug-prefix flag now goes engine-side (no in-memory
    .filter(...)). The --restore-only path queries each db_only directory
    with slugPrefix in a loop instead of one full-table scan, with seen-set
    deduplication and disk-existence check inline.
  - storage.ts: keeps the full-scan path because storage-status needs the
    "unspecified" bucket count, which can't be computed without enumerating
    every page. Comment notes that step 5 (single-walk filesystem scan)
    will reduce per-page disk syscall cost.

2 new test cases on PGLiteEngine: slugPrefix happy path (3 tier dirs,
asserts only matching slugs return) and metacharacter escape regression
(asserts safe/ doesn't match unrelated slugs).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf: single-walk filesystem scan via walkBrainRepo() (step 5/15)

Issue #14 of the eng review: storage.ts called existsSync + statSync
per-page in a synchronous loop. On a 200K-page brain that's 400K syscalls
serialized. Wall-clock landmine.

Adds src/core/disk-walk.ts with walkBrainRepo(repoPath) — one recursive
readdirSync walk, builds a Map<slug, {size, mtimeMs}>. Storage.ts looks
up each DB page in the map (O(1)) instead of stat-checking on demand.
Slug derivation matches the pages-table convention: people/alice.md on
disk becomes people/alice as the map key.

Skipped during walk:
  - dot-directories (.git, .gbrain, .vscode, etc) — not part of the brain
    namespace
  - node_modules — guards against accidentally walking into imported repos
  - non-.md files (sidecar JSON, binaries) — tracked by the brain through
    the files table, not by slug

Reusable: future commands (gbrain doctor's storage_tiering check, the
optional autopilot tier-fix path) get the same walk for free.

9 new test cases: empty dir, nonexistent dir, top-level files, nested
dirs, dot-dir skipping, node_modules skipping, non-.md filtering, size
capture, mtimeMs capture.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: path-segment matching for tier directories (step 6/15)

Issue #5 + D6 of the eng review: tier matching used slug.startsWith(dir),
which falsely matches 'media/xerox/foo' against 'media/x' if a user wrote
the directory without a trailing slash.

The new matcher requires the configured directory to end with `/` and
treats it as a canonical path-segment ancestor:

  media/x/   matches  media/x/tweet-1       ✓
  media/x/   doesn't  media/xerox/foo       ✗
  media/x    refused  media/x/tweet-1       (matcher requires trailing /)

Non-canonical input (no trailing slash) is refused outright. Step 7's
auto-normalizing validator converts user-written 'media/x' → 'media/x/'
on load, so the matcher never sees non-canonical input from real configs.
The behavior tested here is the strict matcher's contract.

Regression test pins the media/xerox collision case explicitly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: auto-normalize trailing-slash, throw on tier overlap (step 7/15)

D7+D8 of the eng review: validation was warnings-only. Users miss warnings.
Now:

  - Cosmetic: missing trailing slash auto-corrected, one-time info note
    showing what changed ("normalized 2 storage paths: 'people' →
    'people/', 'media/x' → 'media/x/'"). Once-per-process to keep noise low.

  - Semantic: same directory in both tiers throws StorageConfigError.
    Ambiguous routing — does media/ win as db_tracked or db_only? — is a
    real bug the user must fix. Caller propagates to the CLI for a clean
    exit-1 with actionable message.

loadStorageConfig now applies normalize+validate after merging deprecated
keys, so the path-segment matcher (step 6) only ever sees canonical
trailing-slash directories.

The pure validateStorageConfig kept for callers who want the warnings list
without the auto-fix side effects (gbrain doctor's reporting path).

2 new test cases: auto-normalize round-trip with warning text assertion,
overlap throws StorageConfigError.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: wire manageGitignore into runSync, only on success (step 8/15)

Issue #2 of the eng review: manageGitignore was defined and never
invoked. Docs claimed "auto-managed by gbrain" — false. Users hit a
.gitignore that never updated and committed db_only directories anyway.

Wire-up: runSync now calls manageGitignore after each successful
performSync return, in both watch and one-shot modes.

Eng review pass-2 finding #1: skip on dry_run AND blocked_by_failures
status. A sync that aborted partway has stale state; mutating .gitignore
based on a partially-loaded config invites drift. Failure-skip test
added (uses .gitignore-as-a-directory to simulate write failure;
asserts warning fired and disk wasn't corrupted).

Hardened manageGitignore itself with three additional behaviors:

  - GBRAIN_NO_GITIGNORE=1 escape hatch (D23) for shared-repo setups
    where a maintainer wants gbrain to leave .gitignore alone.

  - Submodule detection (D49). When repoPath/.git is a regular file
    (gitdir: ... pointer), the repo is a git submodule. Submodule
    .gitignore changes don't survive parent submodule updates, so we
    skip with an actionable warning ("add db_only directories to your
    parent repo's .gitignore manually").

  - Graceful failure (D9). Read errors, write errors, and
    StorageConfigError (overlap from step 7) all log a warning and
    return — sync's primary job (moving data) shouldn't die because of
    a side-effect on .gitignore.

manageGitignore is now exported (previously private) so the
storage-sync test file can hit it directly without spinning up sync.

9 new test cases: no-op without gbrain.yml, no-op with empty db_only,
happy-path append, idempotency (run twice, single entry), preservation
of user-written rules, GBRAIN_NO_GITIGNORE skip, submodule skip,
.git-directory normal path, write-failure graceful warning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix: D5 resolution chain for --restore-only and storage status (step 9/15)

D5 of the eng review: gbrain export --restore-only without --repo
silently fell through to the regular export path, dumping every page in
the database to the wrong directory. Hard regression risk.

Now exits 1 with an actionable message when --restore-only has no
--repo AND no configured default source. Resolution order:
  1. Explicit --repo flag
  2. Typed sources.getDefault() (reuses step 1's accessor)
  3. Hard error — never fall through to cwd

storage.ts:38 also bypassed BrainEngine with raw SQL and a bare
try/catch (Issue #3 + Issue #9). Replaced with the same typed
getDefaultSourcePath() — single source of truth, errors propagate
cleanly to the user, no silent cwd fallback.

Regular export (no --restore-only) keeps its current behavior per D26:
exports include everything, --repo is optional.

4 new test cases on PGLite in-memory:
  - hard-errors with no --repo + no default
  - explicit --repo wins
  - falls back to sources default local_path
  - non-restore export does not require --repo

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor: split storage.ts into pure data + JSON + human formatters (step 10/15)

Issue #10 of the eng review: getStorageStatus and runStorageStatus mixed
data gathering, JSON serialization, and human-readable output in one
function. Hard to test, hard to reuse, mismatched the orphans.ts pattern
that CLAUDE.md cites as the precedent.

Now three pure functions + a thin dispatcher:

  getStorageStatus(engine, repoPath) — async, returns StorageStatusResult.
    Side effects: engine.listPages + one walkBrainRepo (Issue #14).
    Exported so MCP exposure (D14) and gbrain doctor (D13) can consume the
    same data without re-running the loop.

  formatStorageStatusJson(result) — pure, returns indented JSON. Stable
    contract on the StorageStatusResult shape, suitable for orchestrators.

  formatStorageStatusHuman(result) — pure, returns ASCII text (D10 — no
    unicode box-drawing). Composable into other commands later.

  runStorageStatus(engine, args) — thin dispatcher: parses --repo /
    --json, calls getStorageStatus, picks a formatter, prints.

8 new test cases on the formatters: JSON parse round-trip, null-config
fallback, missing-files capped at 10 with rollup, ASCII-only assertion
(D10 regression guard), warnings inline, configuration listing, disk-
usage block omitted when zero bytes.

The StorageStatusResult interface is now exported as a public type, so
gbrain doctor's storage_tiering check can build its own findings from
the same shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* types: distinct PageCountsByTier and DiskUsageByTier (step 11/15)

Issue #11 of the eng review: pagesByTier (page counts) and
diskUsageByTier (byte totals) shared the same structural type
(Record<StorageTier, number>). Both are tier-keyed numeric maps but
carry semantically different units. A future bug that swaps them at a
call site (e.g., displaying disk bytes where the count belongs) wouldn't
trip the compiler.

Replaced with distinct nominal types via a brand field. Structurally
identical at runtime (no overhead) but compile-time disjoint —
TypeScript catches accidental cross-assignment.

  PageCountsByTier   { db_tracked, db_only, unspecified } : numbers (count)
  DiskUsageByTier    { db_tracked, db_only, unspecified } : numbers (bytes)

Both initialized in getStorageStatus, both threaded into
StorageStatusResult, both consumed by formatStorageStatusHuman /
formatStorageStatusJson without further changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat: PGLite soft-warn + full lifecycle test (step 12/15)

D4: storage tiering on PGLite is a partial feature. The "DB" the pages
live in IS the local file gbrain uses for everything else, so "db_only"
has no real offload effect. The .gitignore management still helps
(keeps bulk content out of git history), so we warn and proceed —
not refuse.

Two warning sites (once-per-process each via module-local flags):
  - storage status: warns at runStorageStatus entry
  - sync: warns inside manageGitignore when engineKind='pglite' and
    config has db_only entries

Both phrased actionably ("To get full tiering, migrate to Postgres
with `gbrain migrate --to supabase`").

manageGitignore signature now takes an optional `engineKind` param.
runSync passes engine.kind. Stand-alone callers (tests, future
gbrain doctor --fix path) can omit it.

New test: test/storage-pglite.test.ts — D8 + D4 lifecycle. 6 cases:
engine.kind assertion, getStorageStatus loading gbrain.yml + reporting
tier counts, manageGitignore PGLite-warn (once per process), Postgres
no-warn, slugPrefix on PGLite, end-to-end (config + putPage + status
+ gitignore).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: add trailing-newline CI guard (step 14/15)

Issue #7 of the eng review: all four new files in the original
storage-tiering branch lacked POSIX trailing newlines. Linters complain,
git diffs phantom-flag every future edit. We've been adding newlines as
each file landed; this commit catches the regression class.

scripts/check-trailing-newline.sh:
  - sibling to check-jsonb-pattern.sh / check-progress-to-stdout.sh per
    CLAUDE.md's CI guard pattern
  - portable to bash 3.2 (macOS default; no mapfile, no associative arrays)
  - covers src/**, test/**, gbrain.yml, top-level *.md
  - reports each missing file by path and exits 1

Wired into `bun run test` between progress-to-stdout and typecheck.

Also fixed docs/storage-tiering.md (pre-existing missing newline from
the original branch — caught by the new guard on first run).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: v0.23.0 — VERSION, CHANGELOG, README, CLAUDE.md, storage-tiering.md (step 15/15)

VERSION → 0.23.0 (minor bump for new feature surface).

CHANGELOG entry in Garry voice with the canonical format:
  - Two-line bold headline ("Storage tiering, finally working...")
  - Lead paragraph naming what was broken before and what users get now
  - "Numbers that matter" before/after table for the 6 things that
    actually changed
  - "What this means for your brain" closer
  - "To take advantage of v0.23.0" self-repair block (per CLAUDE.md
    convention) — 6 numbered steps users can follow
  - Itemized changes split into critical fixes / new+renamed surface /
    architecture cleanup / tests + CI guards

CLAUDE.md "Key files" gains four new entries: storage-config.ts,
disk-walk.ts, the v0.23.0 storage.ts shape, and gbrain.yml itself.

README.md gains a new "Storage tiering" section between Skillify and
Getting Data In with the canonical example + commands + link to the
full guide.

docs/storage-tiering.md rewritten end-to-end with canonical key names
(db_tracked / db_only), v0.23.0 hardening details (idempotency,
submodule detection, GBRAIN_NO_GITIGNORE, dry-run gating), the
resolution chain for --restore-only, the auto-normalize +
throw-on-overlap validator, and the PGLite engine note.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: e2e Postgres lifecycle for storage tiering (step 16/16)

Per the v0.23.0 plan: full lifecycle E2E against real Postgres.

  - engine.kind === 'postgres' assertion
  - Full lifecycle: write 4 pages (1 db_tracked, 2 db_only, 1 unspecified)
    → getStorageStatus reports correct tier counts → human formatter
    renders → manageGitignore writes managed block → idempotency check
    → getDefaultSourcePath() resolves the configured local_path.
  - Container restart simulation: 2 db_only pages in DB, files missing
    on disk → status.missingFiles.length === 2 → slugPrefix engine
    filter on Postgres returns exactly the tier slugs.
  - slugPrefix index-based range scan regression: 50 media/x/* + 50
    people/p-* pages → slugPrefix='media/x/' returns exactly 50.
  - getDefaultSourcePath returns null when default source has no
    local_path (the hard-error path that replaces the original silent
    cwd fallback).
  - manageGitignore on Postgres engine does NOT emit the PGLite
    soft-warn (cross-engine assertion).

Skips gracefully when DATABASE_URL is unset, per CLAUDE.md E2E pattern.
Run via: DATABASE_URL=... bun test test/e2e/storage-tiering.test.ts

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: rebump version 0.23.0 → 0.22.9

Reverts the minor bump back to a patch-style version on the v0.22 line.
Storage tiering ships within the v0.22.x train alongside the recent
fix waves. Updates VERSION, package.json, CHANGELOG header + body refs,
CLAUDE.md Key files annotations, README.md section heading, and the
docs/storage-tiering.md backward-compat note.

* chore: bump version 0.22.9 → 0.22.11

Sibling workspaces claimed v0.22.10 in the queue. This branch advances
to v0.22.11 to keep the version monotonic on master.

Updates VERSION, package.json, CHANGELOG header + body refs, CLAUDE.md
Key files annotations, README.md section heading, and the
docs/storage-tiering.md backward-compat note.

* fix: address Codex pre-landing review findings (4 fixes)

Codex found 4 real issues during pre-landing review of v0.22.11 diff:

[P0] export --restore-only fell through to full export when
storageConfig was null (no gbrain.yml present). On older or
misconfigured brains, the recovery command would silently dump the
entire database. src/commands/export.ts now refuses with an actionable
error before any page query fires — matches the D5 lock spirit
("never silently fall through").

[P1] manageGitignore wire-up only fired when --repo was passed
explicitly. performSync resolves the repo from sync.repo_path or
sources.local_path, so the common `gbrain sync` path (after
setup, no flag) never updated .gitignore. src/commands/sync.ts now
uses the same source-resolver chain as the rest of /ship: opts.repoPath
→ getDefaultSourcePath → null. Fires in both watch and one-shot modes.

[P2] getDefaultSourcePath only consulted sources.local_path, missing
the legacy global sync.repo_path config key that pre-v0.18 brains use.
Added a fallback to engine.getConfig('sync.repo_path') when the
sources row has NULL local_path. Pre-v0.18 brains now work without
forcing a `gbrain sources add . --path .` migration.

[P2] sync --all multi-source loop never called manageGitignore even
though src.local_path was already known. Each source now gets its own
gitignore update on successful sync.

Tests:
  - test/storage-export.test.ts: replaced the old "falls through to
    full export" test with one that asserts the new refusal path
    (storage-tiering config required for --restore-only).
  - test/source-resolver.test.ts: added a fallback test exercising the
    legacy sync.repo_path code path for pre-v0.18 brains.
  - All 78 storage-tiering tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate llms.txt + llms-full.txt for v0.22.11

Per CLAUDE.md: "Run `bun run build:llms` after adding a new doc."
The README's new Storage tiering section + the rewritten
docs/storage-tiering.md changed the inlined bundle. test/build-llms.test.ts
catches the drift and was failing on master pre-regen.

* fix: typecheck error in disk-walk.ts (CI #73350475897)

tsc --noEmit failed in CI because ReturnType<typeof readdirSync> with
withFileTypes:true picks an overload union that includes
Dirent<Buffer<ArrayBufferLike>>. Strict tsc treats entry.name as Buffer,
so .startsWith / .endsWith / string comparisons all blew up.

Annotate the variable as Dirent[] (string-based) and cast through unknown,
matching the pattern sync.ts already uses for its own filesystem walk.
Same runtime behavior; clean typecheck.

Tests still 9/9.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
EOF

---------

Co-authored-by: root <root@localhost>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 22:21:07 -07:00

1722 lines
71 KiB
TypeScript

import postgres from 'postgres';
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection } from './engine.ts';
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
import { runMigrations } from './migrate.ts';
import { SCHEMA_SQL } from './schema-embedded.ts';
import { verifySchema } from './schema-verify.ts';
import type {
Page, PageInput, PageFilters, PageType,
Chunk, ChunkInput, StaleChunkRow,
SearchResult, SearchOpts,
Link, GraphNode, GraphPath,
TimelineEntry, TimelineInput, TimelineOpts,
RawData,
PageVersion,
BrainStats, BrainHealth,
IngestLogEntry, IngestLogInput,
EngineConfig,
} from './types.ts';
import { GBrainError } from './types.ts';
import * as db from './db.ts';
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts';
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
// CONNECTION_ERROR_PATTERNS / isConnectionError were used by the per-call
// executeRaw retry that #406 originally shipped. Eng-review D3 dropped that
// retry as unsound (regex idempotence-boundary doesn't hold for writable
// CTEs or side-effecting SELECTs). Recovery now happens at the supervisor
// level (3-strikes-then-reconnect). The unit tests in
// test/connection-resilience.test.ts retain a self-contained copy of the
// helper so the regression-against-future-reintroduction guard still works.
// See TODOS.md item: "err.code-based connection-error matching" for the
// follow-up that will reintroduce a typed retry mechanism.
export class PostgresEngine implements BrainEngine {
readonly kind = 'postgres' as const;
private _sql: ReturnType<typeof postgres> | null = null;
/** Saved config for reconnection. */
private _savedConfig: (EngineConfig & { poolSize?: number }) | null = null;
/** Whether a reconnect is in progress (prevents concurrent reconnects). */
private _reconnecting = false;
// Instance connection (for workers) or fall back to module global (backward compat)
get sql(): ReturnType<typeof postgres> {
if (this._sql) return this._sql;
return db.getConnection();
}
// Lifecycle
async connect(config: EngineConfig & { poolSize?: number }): Promise<void> {
this._savedConfig = config;
if (config.poolSize) {
// Instance-level connection for worker isolation. resolvePoolSize lets
// GBRAIN_POOL_SIZE cap below the caller's requested size when set — the
// env var is a user escape hatch, so it wins.
const url = config.database_url;
if (!url) throw new GBrainError('No database URL', 'database_url is missing', 'Provide --url');
const size = Math.min(config.poolSize, db.resolvePoolSize(config.poolSize));
// Honor PgBouncer transaction-mode detection on worker-instance pools too.
// Without this, `gbrain jobs work` against a Supabase pooler URL hits
// "prepared statement does not exist" under load just like the module
// singleton did before v0.15.4.
const prepare = db.resolvePrepare(url);
// Session timeouts (statement_timeout + idle_in_transaction_session_timeout)
// keep orphan pgbouncer backends from holding locks for hours when the
// postgres.js client disconnects mid-transaction. See resolveSessionTimeouts
// in db.ts for context + env var overrides.
const timeouts = db.resolveSessionTimeouts();
const opts: Record<string, unknown> = {
max: size,
idle_timeout: 20,
connect_timeout: 10,
types: { bigint: postgres.BigInt },
};
if (Object.keys(timeouts).length > 0) {
opts.connection = timeouts;
}
if (typeof prepare === 'boolean') {
opts.prepare = prepare;
}
this._sql = postgres(url, opts);
await this._sql`SELECT 1`;
await db.setSessionDefaults(this._sql);
} else {
// Module-level singleton (backward compat for CLI main engine)
await db.connect(config);
}
}
async disconnect(): Promise<void> {
if (this._sql) {
await this._sql.end();
this._sql = null;
} else {
await db.disconnect();
}
}
async initSchema(): Promise<void> {
const conn = this.sql;
// Advisory lock prevents concurrent initSchema() calls from deadlocking
// on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock).
//
// Honest limitation: pg_advisory_lock(42) is session-scoped to this pooled
// connection. runMigrations() below uses engine.transaction() and
// withReservedConnection() which may hop to a different backend in the
// pool. Cross-process serialization of initSchema is best-effort, not a
// correctness guarantee. Pre-existing concern; the bootstrap doesn't
// change it.
await conn`SELECT pg_advisory_lock(42)`;
try {
// Pre-schema bootstrap: add forward-referenced state the embedded schema
// blob requires but that older brains don't have yet. Without this, a
// pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)`
// (issues #366/#375/#378/#396), or a pre-v0.13 brain hits
// `CREATE INDEX idx_links_source ON links(link_source)` (#266/#357), and
// SCHEMA_SQL crashes before runMigrations gets a chance to apply the
// missing column. Bootstrap is structurally idempotent and a no-op on
// fresh installs and modern brains.
await this.applyForwardReferenceBootstrap();
await conn.unsafe(SCHEMA_SQL);
// Run any pending migrations automatically
const { applied } = await runMigrations(this);
if (applied > 0) {
console.log(` ${applied} migration(s) applied`);
}
// Post-migration schema verification: catches columns that migrations
// defined but PgBouncer transaction-mode silently failed to create.
// Self-heals missing columns via ALTER TABLE ADD COLUMN IF NOT EXISTS.
const verify = await verifySchema(this);
if (verify.healed.length > 0) {
console.log(` Schema verify: self-healed ${verify.healed.length} missing column(s)`);
}
} finally {
await conn`SELECT pg_advisory_unlock(42)`;
}
}
/**
* Bootstrap state that SCHEMA_SQL forward-references but that older brains
* don't have yet. Mirror of `PGLiteEngine#applyForwardReferenceBootstrap`
* in shape and intent. Currently covers:
*
* - `sources` table + default seed (FK target of pages.source_id) — v0.18
* - `pages.source_id` column (indexed by `idx_pages_source_id`) — v0.18
* - `links.link_source` column (indexed by `idx_links_source`) — v0.13
* - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19
* - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19
*
* Keep this in sync with the PGLite version; covered by
* `test/schema-bootstrap-coverage.test.ts` (PGLite side) and
* `test/e2e/postgres-bootstrap.test.ts` (Postgres side).
*/
private async applyForwardReferenceBootstrap(): Promise<void> {
const conn = this.sql;
// Single round-trip probe for every forward-reference target.
// current_schema() resolves to whatever search_path the connection uses,
// which matches schema-embedded.ts's `public.` references.
const probeRows = await conn<{
pages_exists: boolean;
source_id_exists: boolean;
links_exists: boolean;
link_source_exists: boolean;
origin_page_id_exists: boolean;
chunks_exists: boolean;
symbol_name_exists: boolean;
language_exists: boolean;
}[]>`
SELECT
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema = current_schema() AND table_name = 'pages') AS pages_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_id') AS source_id_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema = current_schema() AND table_name = 'links') AS links_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'link_source') AS link_source_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'origin_page_id') AS origin_page_id_exists,
EXISTS (SELECT 1 FROM information_schema.tables
WHERE table_schema = current_schema() AND table_name = 'content_chunks') AS chunks_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'symbol_name') AS symbol_name_exists,
EXISTS (SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'language') AS language_exists
`;
const probe = probeRows[0]!;
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
const needsLinksBootstrap = probe.links_exists
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
const needsChunksBootstrap = probe.chunks_exists
&& (!probe.symbol_name_exists || !probe.language_exists);
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
if (needsPagesBootstrap) {
// Mirror schema-embedded.ts's `sources` shape so the subsequent
// SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op.
await conn.unsafe(`
CREATE TABLE IF NOT EXISTS sources (
id TEXT PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
local_path TEXT,
last_commit TEXT,
last_sync_at TIMESTAMPTZ,
config JSONB NOT NULL DEFAULT '{}'::jsonb,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
INSERT INTO sources (id, name, config)
VALUES ('default', 'default', '{"federated": true}'::jsonb)
ON CONFLICT (id) DO NOTHING;
ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_id TEXT
NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE;
`);
}
if (needsLinksBootstrap) {
// v11 (links_provenance_columns) handles the CHECK constraint, the
// UNIQUE swap, and the backfill. The bootstrap only adds enough state
// for SCHEMA_SQL's `CREATE INDEX idx_links_source/origin` not to crash.
// v11 runs later via runMigrations and is idempotent.
await conn.unsafe(`
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT;
ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER
REFERENCES pages(id) ON DELETE SET NULL;
`);
}
if (needsChunksBootstrap) {
// v26 (content_chunks_code_metadata) adds the full code-chunk metadata
// surface. The bootstrap only adds the two columns the schema blob's
// partial indexes reference (idx_chunks_symbol_name, idx_chunks_language).
// v26 runs later via runMigrations and adds the rest idempotently.
await conn.unsafe(`
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT;
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
`);
}
}
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
const conn = this._sql || db.getConnection();
return conn.begin(async (tx) => {
// Create a scoped engine with tx as its connection, no shared state mutation
const txEngine = Object.create(this) as PostgresEngine;
Object.defineProperty(txEngine, 'sql', { get: () => tx });
Object.defineProperty(txEngine, '_sql', { value: tx as unknown as ReturnType<typeof postgres>, writable: false });
return fn(txEngine);
}) as Promise<T>;
}
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
const pool = this._sql || db.getConnection();
const reserved = await pool.reserve();
try {
const conn: ReservedConnection = {
async executeRaw<R = Record<string, unknown>>(query: string, params?: unknown[]): Promise<R[]> {
const rows = params === undefined
? await reserved.unsafe(query)
: await reserved.unsafe(query, params as Parameters<typeof reserved.unsafe>[1]);
return rows as unknown as R[];
},
};
return await fn(conn);
} finally {
reserved.release();
}
}
// Pages CRUD
async getPage(slug: string): Promise<Page | null> {
const sql = this.sql;
const rows = await sql`
SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
FROM pages WHERE slug = ${slug}
`;
if (rows.length === 0) return null;
return rowToPage(rows[0]);
}
async putPage(slug: string, page: PageInput): Promise<Page> {
slug = validateSlug(slug);
const sql = this.sql;
const hash = page.content_hash || contentHash(page);
const frontmatter = page.frontmatter || {};
// v0.18.0 Step 2: source_id relies on schema DEFAULT 'default'. ON
// CONFLICT target becomes (source_id, slug) since global UNIQUE(slug)
// was dropped in migration v17. See pglite-engine.ts for matching
// notes; multi-source sync (Step 5) will surface an explicit sourceId.
const pageKind = page.page_kind || 'markdown';
const rows = await sql`
INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
VALUES (${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters<typeof sql.json>[0])}, ${hash}, now())
ON CONFLICT (source_id, slug) DO UPDATE SET
type = EXCLUDED.type,
page_kind = EXCLUDED.page_kind,
title = EXCLUDED.title,
compiled_truth = EXCLUDED.compiled_truth,
timeline = EXCLUDED.timeline,
frontmatter = EXCLUDED.frontmatter,
content_hash = EXCLUDED.content_hash,
updated_at = now()
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
`;
return rowToPage(rows[0]);
}
async deletePage(slug: string): Promise<void> {
const sql = this.sql;
await sql`DELETE FROM pages WHERE slug = ${slug}`;
}
async listPages(filters?: PageFilters): Promise<Page[]> {
const sql = this.sql;
const limit = filters?.limit || 100;
const offset = filters?.offset || 0;
const updatedAfter = filters?.updated_after;
// postgres.js sql.unsafe is awkward for conditional WHERE; use raw query branching.
// The 4 dimensions (type, tag, updated_after, none) cross-product into 8 cases;
// we use postgres.js's tagged-template chaining via sql`` fragments instead.
// Build conditions with sql fragments. postgres.js supports fragment composition.
const typeCondition = filters?.type ? sql`AND p.type = ${filters.type}` : sql``;
const tagJoin = filters?.tag ? sql`JOIN tags t ON t.page_id = p.id` : sql``;
const tagCondition = filters?.tag ? sql`AND t.tag = ${filters.tag}` : sql``;
const updatedCondition = updatedAfter ? sql`AND p.updated_at > ${updatedAfter}::timestamptz` : sql``;
// slugPrefix uses the (source_id, slug) UNIQUE btree index for range scans.
// Escape LIKE metacharacters so the user prefix is treated as a literal.
const slugPrefix = filters?.slugPrefix;
const slugCondition = slugPrefix
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
: sql``;
const rows = await sql`
SELECT p.* FROM pages p
${tagJoin}
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition}
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
`;
return rows.map(rowToPage);
}
async getAllSlugs(): Promise<Set<string>> {
const sql = this.sql;
const rows = await sql`SELECT slug FROM pages`;
return new Set(rows.map((r) => r.slug as string));
}
async resolveSlugs(partial: string): Promise<string[]> {
const sql = this.sql;
// Try exact match first
const exact = await sql`SELECT slug FROM pages WHERE slug = ${partial}`;
if (exact.length > 0) return [exact[0].slug];
// Fuzzy match via pg_trgm
const fuzzy = await sql`
SELECT slug, similarity(title, ${partial}) AS sim
FROM pages
WHERE title % ${partial} OR slug ILIKE ${'%' + partial + '%'}
ORDER BY sim DESC
LIMIT 5
`;
return fuzzy.map((r) => r.slug as string);
}
// Search
// v0.20.0 Cathedral II Layer 3 (1b): chunk-grain FTS internally,
// dedup-to-best-chunk-per-page on the way out. External shape
// preserves the v0.19.0 contract so backlinks / enrichment-service /
// list_pages etc. see zero breaking changes. A2 two-pass (Layer 7)
// consumes searchKeywordChunks for the raw chunk-grain primitive.
async searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
const sql = this.sql;
const limit = clampSearchLimit(opts?.limit);
const offset = opts?.offset || 0;
const type = opts?.type;
const excludeSlugs = opts?.exclude_slugs;
const language = opts?.language;
const symbolKind = opts?.symbolKind;
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
}
const detailLow = opts?.detail === 'low';
// Fetch headroom for dedup: if we only fetch `limit` chunks, a cluster of
// co-occurring terms in one page can eat the entire result set and we'd
// ship < limit pages. 3x gives dedup enough to pick top N distinct pages.
const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3);
// Source-aware ranking (v0.22): boost curated content (originals/,
// concepts/, writing/) and dampen bulk content (chat/, daily/, media/x/)
// by multiplying the chunk-grain ts_rank with a source-factor CASE.
// Detail-gated — disabled for `detail='high'` (temporal queries) so
// chat surfaces normally for date-framed lookups. Hard-exclude prefixes
// (test/, archive/, attachments/, .raw/ by default) filter at the
// chunk-rank stage so they never enter the candidate set.
const boostMap = resolveBoostMap();
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
const params: unknown[] = [query];
let typeClause = '';
if (type) {
params.push(type);
typeClause = `AND p.type = $${params.length}`;
}
let excludeSlugsClause = '';
if (excludeSlugs?.length) {
params.push(excludeSlugs);
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
}
let languageClause = '';
if (language) {
params.push(language);
languageClause = `AND cc.language = $${params.length}`;
}
let symbolKindClause = '';
if (symbolKind) {
params.push(symbolKind);
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
}
params.push(innerLimit);
const innerLimitParam = `$${params.length}`;
params.push(limit);
const limitParam = `$${params.length}`;
params.push(offset);
const offsetParam = `$${params.length}`;
const rawQuery = `
WITH ranked_chunks AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
${typeClause}
${excludeSlugsClause}
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
${languageClause}
${symbolKindClause}
${hardExcludeClause}
ORDER BY score DESC
LIMIT ${innerLimitParam}
),
best_per_page AS (
SELECT DISTINCT ON (slug) *
FROM ranked_chunks
ORDER BY slug, score DESC
)
SELECT slug, page_id, title, type, source_id,
chunk_id, chunk_index, chunk_text, chunk_source, score,
false AS stale
FROM best_per_page
ORDER BY score DESC
LIMIT ${limitParam}
OFFSET ${offsetParam}
`;
// Search-only timeout. SET LOCAL inside sql.begin() scopes the GUC
// to the transaction so it can never leak onto a pooled connection.
const rows = await sql.begin(async sql => {
await sql`SET LOCAL statement_timeout = '8s'`;
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
});
return rows.map(rowToSearchResult);
}
/**
* v0.20.0 Cathedral II Layer 3 (1b) chunk-grain keyword search.
* Ranks chunks via content_chunks.search_vector WITHOUT the
* dedup-to-page pass searchKeyword applies. Used by A2 two-pass
* retrieval (Layer 7) as the anchor-discovery primitive.
*
* Most callers should prefer searchKeyword (external page-grain
* contract). This is intentionally a narrow internal knob.
*/
async searchKeywordChunks(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
const sql = this.sql;
const limit = clampSearchLimit(opts?.limit);
const offset = opts?.offset || 0;
const type = opts?.type;
const excludeSlugs = opts?.exclude_slugs;
const detailLow = opts?.detail === 'low';
const language = opts?.language;
const symbolKind = opts?.symbolKind;
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
}
// Source-aware ranking applies here too — searchKeywordChunks is the
// chunk-grain anchor primitive that two-pass retrieval (Layer 7) uses,
// so curated-vs-bulk dampening should affect the anchor pool. Same
// detail-gate, same hard-exclude behavior as searchKeyword.
const boostMap = resolveBoostMap();
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
const params: unknown[] = [query];
let typeClause = '';
if (type) {
params.push(type);
typeClause = `AND p.type = $${params.length}`;
}
let excludeSlugsClause = '';
if (excludeSlugs?.length) {
params.push(excludeSlugs);
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
}
let languageClause = '';
if (language) {
params.push(language);
languageClause = `AND cc.language = $${params.length}`;
}
let symbolKindClause = '';
if (symbolKind) {
params.push(symbolKind);
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
}
params.push(limit);
const limitParam = `$${params.length}`;
params.push(offset);
const offsetParam = `$${params.length}`;
const rawQuery = `
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
false AS stale
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
${typeClause}
${excludeSlugsClause}
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
${languageClause}
${symbolKindClause}
${hardExcludeClause}
ORDER BY score DESC
LIMIT ${limitParam}
OFFSET ${offsetParam}
`;
const rows = await sql.begin(async sql => {
await sql`SET LOCAL statement_timeout = '8s'`;
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
});
return rows.map(rowToSearchResult);
}
async searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]> {
const sql = this.sql;
const limit = clampSearchLimit(opts?.limit);
const offset = opts?.offset || 0;
const type = opts?.type;
const excludeSlugs = opts?.exclude_slugs;
const detailLow = opts?.detail === 'low';
const language = opts?.language;
const symbolKind = opts?.symbolKind;
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
}
const vecStr = '[' + Array.from(embedding).join(',') + ']';
// Two-stage CTE (v0.22): inner CTE keeps a pure-distance ORDER BY so
// the HNSW index stays usable. Folding source-boost into the inner
// ORDER BY would force a sequential scan over every chunk (seconds vs
// ~10ms with HNSW). Outer SELECT re-ranks the candidate pool by
// raw_score * source_factor.
//
// innerLimit scales with offset to preserve the pagination contract:
// a fixed cap of 100 would silently empty offset > 100.
const boostMap = resolveBoostMap();
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail);
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
const innerLimit = offset + Math.max(limit * 5, 100);
const params: unknown[] = [vecStr];
let typeClause = '';
if (type) {
params.push(type);
typeClause = `AND p.type = $${params.length}`;
}
let excludeSlugsClause = '';
if (excludeSlugs?.length) {
params.push(excludeSlugs);
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
}
let languageClause = '';
if (language) {
params.push(language);
languageClause = `AND cc.language = $${params.length}`;
}
let symbolKindClause = '';
if (symbolKind) {
params.push(symbolKind);
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
}
params.push(innerLimit);
const innerLimitParam = `$${params.length}`;
params.push(limit);
const limitParam = `$${params.length}`;
params.push(offset);
const offsetParam = `$${params.length}`;
const rawQuery = `
WITH hnsw_candidates AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
1 - (cc.embedding <=> $1::vector) AS raw_score
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NOT NULL
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
${typeClause}
${excludeSlugsClause}
${languageClause}
${symbolKindClause}
${hardExcludeClause}
ORDER BY cc.embedding <=> $1::vector
LIMIT ${innerLimitParam}
)
SELECT
slug, page_id, title, type, source_id,
chunk_id, chunk_index, chunk_text, chunk_source,
raw_score * ${sourceFactorCaseOnSlug} AS score,
false AS stale
FROM hnsw_candidates
ORDER BY score DESC
LIMIT ${limitParam}
OFFSET ${offsetParam}
`;
const rows = await sql.begin(async sql => {
await sql`SET LOCAL statement_timeout = '8s'`;
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
});
return rows.map(rowToSearchResult);
}
async getEmbeddingsByChunkIds(ids: number[]): Promise<Map<number, Float32Array>> {
if (ids.length === 0) return new Map();
const sql = this.sql;
const rows = await sql`
SELECT id, embedding FROM content_chunks
WHERE id = ANY(${ids}::int[]) AND embedding IS NOT NULL
`;
const result = new Map<number, Float32Array>();
for (const row of rows) {
const embedding = tryParseEmbedding(row.embedding);
if (embedding) result.set(row.id as number, embedding);
}
return result;
}
// Chunks
async upsertChunks(slug: string, chunks: ChunkInput[]): Promise<void> {
const sql = this.sql;
// Get page_id
const pages = await sql`SELECT id FROM pages WHERE slug = ${slug}`;
if (pages.length === 0) throw new Error(`Page not found: ${slug}`);
const pageId = pages[0].id;
// Remove chunks that no longer exist (chunk_index beyond new count)
const newIndices = chunks.map(c => c.chunk_index);
if (newIndices.length > 0) {
await sql`DELETE FROM content_chunks WHERE page_id = ${pageId} AND chunk_index != ALL(${newIndices})`;
} else {
await sql`DELETE FROM content_chunks WHERE page_id = ${pageId}`;
return;
}
// Batch upsert: build a single multi-row INSERT ON CONFLICT statement.
// v0.19.0: includes language/symbol_name/symbol_type/start_line/end_line
// so code chunks carry tree-sitter metadata into the DB. Markdown chunks
// pass NULL for all five.
// v0.20.0 Cathedral II Layer 6: adds parent_symbol_path / doc_comment /
// symbol_name_qualified so nested-chunk emission (A3) can round-trip
// scope metadata through upserts.
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at, language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified)';
const rows: string[] = [];
const params: unknown[] = [];
let paramIdx = 1;
for (const chunk of chunks) {
const embeddingStr = chunk.embedding
? '[' + Array.from(chunk.embedding).join(',') + ']'
: null;
const parentPath = chunk.parent_symbol_path && chunk.parent_symbol_path.length > 0
? chunk.parent_symbol_path
: null;
if (embeddingStr) {
rows.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::vector, $${paramIdx++}, $${paramIdx++}, now(), $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::text[], $${paramIdx++}, $${paramIdx++})`);
params.push(
pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source,
embeddingStr, chunk.model || 'text-embedding-3-large', chunk.token_count || null,
chunk.language || null, chunk.symbol_name || null, chunk.symbol_type || null,
chunk.start_line ?? null, chunk.end_line ?? null,
parentPath, chunk.doc_comment || null, chunk.symbol_name_qualified || null,
);
} else {
rows.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::text[], $${paramIdx++}, $${paramIdx++})`);
params.push(
pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source,
chunk.model || 'text-embedding-3-large', chunk.token_count || null,
chunk.language || null, chunk.symbol_name || null, chunk.symbol_type || null,
chunk.start_line ?? null, chunk.end_line ?? null,
parentPath, chunk.doc_comment || null, chunk.symbol_name_qualified || null,
);
}
}
// Single statement upsert: preserves existing embeddings via COALESCE when new value is NULL.
// CONSISTENCY: when chunk_text changes and no new embedding is supplied, BOTH embedding AND
// embedded_at must reset to NULL so `embed --stale` correctly picks up the row for re-embedding.
// Without this, embedded_at lies (says "embedded" while embedding=NULL), and any staleness
// predicate on embedded_at would silently skip the row. This is why the egress fix predicates
// on `embedding IS NULL` rather than `embedded_at IS NULL` — and it's why we now keep both
// columns honest at write time.
await sql.unsafe(
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
chunk_text = EXCLUDED.chunk_text,
chunk_source = EXCLUDED.chunk_source,
embedding = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding ELSE COALESCE(EXCLUDED.embedding, content_chunks.embedding) END,
model = COALESCE(EXCLUDED.model, content_chunks.model),
token_count = EXCLUDED.token_count,
embedded_at = CASE
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
ELSE COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at)
END,
language = EXCLUDED.language,
symbol_name = EXCLUDED.symbol_name,
symbol_type = EXCLUDED.symbol_type,
start_line = EXCLUDED.start_line,
end_line = EXCLUDED.end_line,
parent_symbol_path = EXCLUDED.parent_symbol_path,
doc_comment = EXCLUDED.doc_comment,
symbol_name_qualified = EXCLUDED.symbol_name_qualified`,
params as Parameters<typeof sql.unsafe>[1],
);
}
async getChunks(slug: string): Promise<Chunk[]> {
const sql = this.sql;
const rows = await sql`
SELECT cc.* FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = ${slug}
ORDER BY cc.chunk_index
`;
return rows.map((r) => rowToChunk(r as Record<string, unknown>));
}
async countStaleChunks(): Promise<number> {
const sql = this.sql;
const [row] = await sql`
SELECT count(*)::int AS count
FROM content_chunks
WHERE embedding IS NULL
`;
return Number((row as { count?: number } | undefined)?.count ?? 0);
}
async listStaleChunks(): Promise<StaleChunkRow[]> {
const sql = this.sql;
const rows = await sql`
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
cc.model, cc.token_count
FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE cc.embedding IS NULL
ORDER BY p.id, cc.chunk_index
LIMIT 100000
`;
return rows as unknown as StaleChunkRow[];
}
async deleteChunks(slug: string): Promise<void> {
const sql = this.sql;
await sql`
DELETE FROM content_chunks
WHERE page_id = (SELECT id FROM pages WHERE slug = ${slug})
`;
}
// Links
async addLink(
from: string,
to: string,
context?: string,
linkType?: string,
linkSource?: string,
originSlug?: string,
originField?: string,
): Promise<void> {
const sql = this.sql;
// Pre-check existence so we can throw a clear error (ON CONFLICT DO UPDATE
// returns 0 rows when source SELECT is empty, indistinguishable from missing page).
const exists = await sql`
SELECT 1 FROM pages WHERE slug = ${from}
INTERSECT
SELECT 1 FROM pages WHERE slug = ${to}
`;
if (exists.length === 0) {
throw new Error(`addLink failed: page "${from}" or "${to}" not found`);
}
// Default link_source to 'markdown' for back-compat with pre-v0.13 callers.
// origin_page_id resolves from originSlug via the pages join (NULL if no slug).
const src = linkSource ?? 'markdown';
await sql`
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
SELECT f.id, t.id, ${linkType || ''}, ${context || ''}, ${src},
(SELECT id FROM pages WHERE slug = ${originSlug ?? null}),
${originField ?? null}
FROM pages f, pages t
WHERE f.slug = ${from} AND t.slug = ${to}
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO UPDATE SET
context = EXCLUDED.context,
origin_field = EXCLUDED.origin_field
`;
}
async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
if (links.length === 0) return 0;
const sql = this.sql;
// unnest() pattern: 7 array-typed bound parameters regardless of batch size.
// Avoids the 65535-parameter cap and the postgres-js sql(rows, ...) helper's
// identifier-escape gotcha when used inside a (VALUES) subquery.
//
// v0.13: added link_source, origin_slug, origin_field. Defaults:
// link_source → 'markdown' (back-compat with pre-v0.13 callers)
// origin_slug → NULL (resolves to origin_page_id IS NULL via LEFT JOIN)
// origin_field → NULL
const fromSlugs = links.map(l => l.from_slug);
const toSlugs = links.map(l => l.to_slug);
const linkTypes = links.map(l => l.link_type || '');
const contexts = links.map(l => l.context || '');
const linkSources = links.map(l => l.link_source || 'markdown');
const originSlugs = links.map(l => l.origin_slug || null);
const originFields = links.map(l => l.origin_field || null);
const fromSourceIds = links.map(l => l.from_source_id || 'default');
const toSourceIds = links.map(l => l.to_source_id || 'default');
const originSourceIds = links.map(l => l.origin_source_id || 'default');
const result = await sql`
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
FROM unnest(
${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[],
${contexts}::text[], ${linkSources}::text[], ${originSlugs}::text[],
${originFields}::text[], ${fromSourceIds}::text[], ${toSourceIds}::text[],
${originSourceIds}::text[]
) AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id)
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
LEFT JOIN pages o ON o.slug = v.origin_slug AND o.source_id = v.origin_source_id
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO NOTHING
RETURNING 1
`;
return result.length;
}
async removeLink(from: string, to: string, linkType?: string, linkSource?: string): Promise<void> {
const sql = this.sql;
// Build up filters dynamically. linkType + linkSource are independent
// optional constraints; all four combinations are valid.
if (linkType !== undefined && linkSource !== undefined) {
await sql`
DELETE FROM links
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
AND link_type = ${linkType}
AND link_source IS NOT DISTINCT FROM ${linkSource}
`;
} else if (linkType !== undefined) {
await sql`
DELETE FROM links
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
AND link_type = ${linkType}
`;
} else if (linkSource !== undefined) {
await sql`
DELETE FROM links
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
AND link_source IS NOT DISTINCT FROM ${linkSource}
`;
} else {
await sql`
DELETE FROM links
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
`;
}
}
async getLinks(slug: string): Promise<Link[]> {
const sql = this.sql;
const rows = await sql`
SELECT f.slug as from_slug, t.slug as to_slug,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
LEFT JOIN pages o ON o.id = l.origin_page_id
WHERE f.slug = ${slug}
`;
return rows as unknown as Link[];
}
async getBacklinks(slug: string): Promise<Link[]> {
const sql = this.sql;
const rows = await sql`
SELECT f.slug as from_slug, t.slug as to_slug,
l.link_type, l.context, l.link_source,
o.slug as origin_slug, l.origin_field
FROM links l
JOIN pages f ON f.id = l.from_page_id
JOIN pages t ON t.id = l.to_page_id
LEFT JOIN pages o ON o.id = l.origin_page_id
WHERE t.slug = ${slug}
`;
return rows as unknown as Link[];
}
async findByTitleFuzzy(
name: string,
dirPrefix?: string,
minSimilarity: number = 0.55,
): Promise<{ slug: string; similarity: number } | null> {
const sql = this.sql;
// Use the `similarity()` function directly with an explicit threshold
// comparison. DO NOT use `SET LOCAL pg_trgm.similarity_threshold` +
// the `%` operator here — postgres.js auto-commits each sql`` call
// so `SET LOCAL` is a no-op across statement boundaries. Inline
// comparison is the only way to get predictable threshold behavior
// without wrapping the caller in a transaction.
//
// Tie-breaker: sort by slug after similarity so re-runs return the
// same winner when multiple pages score equally (prevents churn
// in put_page auto-link reconciliation).
const prefixPattern = dirPrefix ? `${dirPrefix}/%` : '%';
const rows = await sql`
SELECT slug, similarity(title, ${name}) AS sim
FROM pages
WHERE similarity(title, ${name}) >= ${minSimilarity}
AND slug LIKE ${prefixPattern}
ORDER BY sim DESC, slug ASC
LIMIT 1
`;
if (rows.length === 0) return null;
const row = rows[0] as { slug: string; sim: number };
return { slug: row.slug, similarity: row.sim };
}
async traverseGraph(slug: string, depth: number = 5): Promise<GraphNode[]> {
const sql = this.sql;
// Cycle prevention: visited array tracks page IDs already in the path.
const rows = await sql`
WITH RECURSIVE graph AS (
SELECT p.id, p.slug, p.title, p.type, 0 as depth, ARRAY[p.id] as visited
FROM pages p WHERE p.slug = ${slug}
UNION ALL
SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1, g.visited || p2.id
FROM graph g
JOIN links l ON l.from_page_id = g.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE g.depth < ${depth}
AND NOT (p2.id = ANY(g.visited))
)
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
coalesce(
-- jsonb_agg(DISTINCT ...) collapses duplicate (to_slug, link_type)
-- edges that originate from different provenance (markdown body
-- vs frontmatter vs auto-extracted). The underlying links table
-- preserves every row with its origin_page_id / link_source —
-- the dedup is presentation-only for the legacy traverseGraph
-- aggregation. traversePaths has its own in-memory dedup at a
-- different layer. See plan Bug 6/10.
(SELECT jsonb_agg(DISTINCT jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
FROM links l2
JOIN pages p3 ON p3.id = l2.to_page_id
WHERE l2.from_page_id = g.id),
'[]'::jsonb
) as links
FROM graph g
ORDER BY g.depth, g.slug
`;
return rows.map((r: Record<string, unknown>) => ({
slug: r.slug as string,
title: r.title as string,
type: r.type as PageType,
depth: r.depth as number,
links: (typeof r.links === 'string' ? JSON.parse(r.links) : r.links) as { to_slug: string; link_type: string }[],
}));
}
async traversePaths(
slug: string,
opts?: { depth?: number; linkType?: string; direction?: 'in' | 'out' | 'both' },
): Promise<GraphPath[]> {
const sql = this.sql;
const depth = opts?.depth ?? 5;
const direction = opts?.direction ?? 'out';
const linkType = opts?.linkType ?? null;
const linkTypeMatches = linkType !== null;
let rows;
if (direction === 'out') {
rows = await sql`
WITH RECURSIVE walk AS (
SELECT p.id, p.slug, 0::int as depth, ARRAY[p.id] as visited
FROM pages p WHERE p.slug = ${slug}
UNION ALL
SELECT p2.id, p2.slug, w.depth + 1, w.visited || p2.id
FROM walk w
JOIN links l ON l.from_page_id = w.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE w.depth < ${depth}
AND NOT (p2.id = ANY(w.visited))
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
)
SELECT w.slug as from_slug, p2.slug as to_slug,
l.link_type, l.context, w.depth + 1 as depth
FROM walk w
JOIN links l ON l.from_page_id = w.id
JOIN pages p2 ON p2.id = l.to_page_id
WHERE w.depth < ${depth}
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
ORDER BY depth, from_slug, to_slug
`;
} else if (direction === 'in') {
rows = await sql`
WITH RECURSIVE walk AS (
SELECT p.id, p.slug, 0::int as depth, ARRAY[p.id] as visited
FROM pages p WHERE p.slug = ${slug}
UNION ALL
SELECT p2.id, p2.slug, w.depth + 1, w.visited || p2.id
FROM walk w
JOIN links l ON l.to_page_id = w.id
JOIN pages p2 ON p2.id = l.from_page_id
WHERE w.depth < ${depth}
AND NOT (p2.id = ANY(w.visited))
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
)
SELECT p2.slug as from_slug, w.slug as to_slug,
l.link_type, l.context, w.depth + 1 as depth
FROM walk w
JOIN links l ON l.to_page_id = w.id
JOIN pages p2 ON p2.id = l.from_page_id
WHERE w.depth < ${depth}
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
ORDER BY depth, from_slug, to_slug
`;
} else {
rows = await sql`
WITH RECURSIVE walk AS (
SELECT p.id, 0::int as depth, ARRAY[p.id] as visited
FROM pages p WHERE p.slug = ${slug}
UNION ALL
SELECT p2.id, w.depth + 1, w.visited || p2.id
FROM walk w
JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id)
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END
WHERE w.depth < ${depth}
AND NOT (p2.id = ANY(w.visited))
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
)
SELECT pf.slug as from_slug, pt.slug as to_slug,
l.link_type, l.context, w.depth + 1 as depth
FROM walk w
JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id)
JOIN pages pf ON pf.id = l.from_page_id
JOIN pages pt ON pt.id = l.to_page_id
WHERE w.depth < ${depth}
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
ORDER BY depth, from_slug, to_slug
`;
}
// Dedup edges (same edge can appear via multiple visited paths).
const seen = new Set<string>();
const result: GraphPath[] = [];
for (const r of rows as Record<string, unknown>[]) {
const key = `${r.from_slug}|${r.to_slug}|${r.link_type}|${r.depth}`;
if (seen.has(key)) continue;
seen.add(key);
result.push({
from_slug: r.from_slug as string,
to_slug: r.to_slug as string,
link_type: r.link_type as string,
context: (r.context as string) || '',
depth: Number(r.depth),
});
}
return result;
}
async getBacklinkCounts(slugs: string[]): Promise<Map<string, number>> {
const result = new Map<string, number>();
if (slugs.length === 0) return result;
for (const s of slugs) result.set(s, 0);
const sql = this.sql;
const rows = await sql`
SELECT p.slug as slug, COUNT(l.id)::int as cnt
FROM pages p
LEFT JOIN links l ON l.to_page_id = p.id
WHERE p.slug = ANY(${slugs}::text[])
GROUP BY p.slug
`;
for (const r of rows as unknown as { slug: string; cnt: number }[]) {
result.set(r.slug, Number(r.cnt));
}
return result;
}
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
const sql = this.sql;
const rows = await sql`
SELECT
p.slug,
COALESCE(p.title, p.slug) AS title,
p.frontmatter->>'domain' AS domain
FROM pages p
WHERE NOT EXISTS (
SELECT 1 FROM links l WHERE l.to_page_id = p.id
)
ORDER BY p.slug
`;
return rows as unknown as Array<{ slug: string; title: string; domain: string | null }>;
}
// Tags
async addTag(slug: string, tag: string): Promise<void> {
const sql = this.sql;
// Verify page exists before attempting insert (ON CONFLICT DO NOTHING
// swallows the "already tagged" case, but we still need to detect missing pages)
const page = await sql`SELECT id FROM pages WHERE slug = ${slug}`;
if (page.length === 0) throw new Error(`addTag failed: page "${slug}" not found`);
await sql`
INSERT INTO tags (page_id, tag)
VALUES (${page[0].id}, ${tag})
ON CONFLICT (page_id, tag) DO NOTHING
`;
}
async removeTag(slug: string, tag: string): Promise<void> {
const sql = this.sql;
await sql`
DELETE FROM tags
WHERE page_id = (SELECT id FROM pages WHERE slug = ${slug})
AND tag = ${tag}
`;
}
async getTags(slug: string): Promise<string[]> {
const sql = this.sql;
const rows = await sql`
SELECT tag FROM tags
WHERE page_id = (SELECT id FROM pages WHERE slug = ${slug})
ORDER BY tag
`;
return rows.map((r) => r.tag as string);
}
// Timeline
async addTimelineEntry(
slug: string,
entry: TimelineInput,
opts?: { skipExistenceCheck?: boolean },
): Promise<void> {
const sql = this.sql;
if (!opts?.skipExistenceCheck) {
const exists = await sql`SELECT 1 FROM pages WHERE slug = ${slug}`;
if (exists.length === 0) {
throw new Error(`addTimelineEntry failed: page "${slug}" not found`);
}
}
// ON CONFLICT DO NOTHING via the (page_id, date, summary) unique index.
// Returning 0 rows means either page missing OR duplicate; skipExistenceCheck
// makes that ambiguity safe (caller asserts page exists).
await sql`
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
SELECT id, ${entry.date}::date, ${entry.source || ''}, ${entry.summary}, ${entry.detail || ''}
FROM pages WHERE slug = ${slug}
ON CONFLICT (page_id, date, summary) DO NOTHING
`;
}
async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number> {
if (entries.length === 0) return 0;
const sql = this.sql;
const slugs = entries.map(e => e.slug);
const dates = entries.map(e => e.date);
const sources = entries.map(e => e.source || '');
const summaries = entries.map(e => e.summary);
const details = entries.map(e => e.detail || '');
const sourceIds = entries.map(e => e.source_id || 'default');
const result = await sql`
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
SELECT p.id, v.date::date, v.source, v.summary, v.detail
FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[], ${sourceIds}::text[])
AS v(slug, date, source, summary, detail, source_id)
JOIN pages p ON p.slug = v.slug AND p.source_id = v.source_id
ON CONFLICT (page_id, date, summary) DO NOTHING
RETURNING 1
`;
return result.length;
}
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
const sql = this.sql;
const limit = opts?.limit || 100;
let rows;
if (opts?.after && opts?.before) {
rows = await sql`
SELECT te.* FROM timeline_entries te
JOIN pages p ON p.id = te.page_id
WHERE p.slug = ${slug} AND te.date >= ${opts.after}::date AND te.date <= ${opts.before}::date
ORDER BY te.date DESC LIMIT ${limit}
`;
} else if (opts?.after) {
rows = await sql`
SELECT te.* FROM timeline_entries te
JOIN pages p ON p.id = te.page_id
WHERE p.slug = ${slug} AND te.date >= ${opts.after}::date
ORDER BY te.date DESC LIMIT ${limit}
`;
} else {
rows = await sql`
SELECT te.* FROM timeline_entries te
JOIN pages p ON p.id = te.page_id
WHERE p.slug = ${slug}
ORDER BY te.date DESC LIMIT ${limit}
`;
}
return rows as unknown as TimelineEntry[];
}
// Raw data
async putRawData(slug: string, source: string, data: object): Promise<void> {
const sql = this.sql;
const result = await sql`
INSERT INTO raw_data (page_id, source, data)
SELECT id, ${source}, ${sql.json(data as Parameters<typeof sql.json>[0])}
FROM pages WHERE slug = ${slug}
ON CONFLICT (page_id, source) DO UPDATE SET
data = EXCLUDED.data,
fetched_at = now()
RETURNING id
`;
if (result.length === 0) throw new Error(`putRawData failed: page "${slug}" not found`);
}
async getRawData(slug: string, source?: string): Promise<RawData[]> {
const sql = this.sql;
let rows;
if (source) {
rows = await sql`
SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
JOIN pages p ON p.id = rd.page_id
WHERE p.slug = ${slug} AND rd.source = ${source}
`;
} else {
rows = await sql`
SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
JOIN pages p ON p.id = rd.page_id
WHERE p.slug = ${slug}
`;
}
return rows as unknown as RawData[];
}
// Versions
async createVersion(slug: string): Promise<PageVersion> {
const sql = this.sql;
const rows = await sql`
INSERT INTO page_versions (page_id, compiled_truth, frontmatter)
SELECT id, compiled_truth, frontmatter
FROM pages WHERE slug = ${slug}
RETURNING *
`;
if (rows.length === 0) throw new Error(`createVersion failed: page "${slug}" not found`);
return rows[0] as unknown as PageVersion;
}
async getVersions(slug: string): Promise<PageVersion[]> {
const sql = this.sql;
const rows = await sql`
SELECT pv.* FROM page_versions pv
JOIN pages p ON p.id = pv.page_id
WHERE p.slug = ${slug}
ORDER BY pv.snapshot_at DESC
`;
return rows as unknown as PageVersion[];
}
async revertToVersion(slug: string, versionId: number): Promise<void> {
const sql = this.sql;
await sql`
UPDATE pages SET
compiled_truth = pv.compiled_truth,
frontmatter = pv.frontmatter,
updated_at = now()
FROM page_versions pv
WHERE pages.slug = ${slug} AND pv.id = ${versionId} AND pv.page_id = pages.id
`;
}
// Stats + health
async getStats(): Promise<BrainStats> {
const sql = this.sql;
const [stats] = await sql`
SELECT
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks) as chunk_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count,
(SELECT count(*) FROM links) as link_count,
(SELECT count(DISTINCT tag) FROM tags) as tag_count,
(SELECT count(*) FROM timeline_entries) as timeline_entry_count
`;
const types = await sql`
SELECT type, count(*)::int as count FROM pages GROUP BY type ORDER BY count DESC
`;
const pages_by_type: Record<string, number> = {};
for (const t of types) {
pages_by_type[t.type as string] = t.count as number;
}
return {
page_count: Number(stats.page_count),
chunk_count: Number(stats.chunk_count),
embedded_count: Number(stats.embedded_count),
link_count: Number(stats.link_count),
tag_count: Number(stats.tag_count),
timeline_entry_count: Number(stats.timeline_entry_count),
pages_by_type,
};
}
async getHealth(): Promise<BrainHealth> {
const sql = this.sql;
// Bug 11 doc-drift fix — orphan_pages means "islanded" (no inbound AND
// no outbound links), aligning both engines with the user-facing
// definition. The type comment previously said "no inbound" but the
// SQL required both — docs now match code so users can trust the
// number. A hub page that links out to many but has no back-references
// is working as intended, not an orphan.
const [h] = await sql`
WITH entity_pages AS (
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
)
SELECT
(SELECT count(*) FROM pages) as page_count,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
(SELECT count(*) FROM pages p
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
) as stale_pages,
(SELECT count(*) FROM pages p
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
) as orphan_pages,
(SELECT count(*) FROM links l
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
) as dead_links,
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings,
(SELECT count(*) FROM links) as link_count,
(SELECT count(DISTINCT page_id) FROM timeline_entries) as pages_with_timeline,
(SELECT count(*) FROM entity_pages e
WHERE EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = e.id))::float /
GREATEST((SELECT count(*) FROM entity_pages), 1)::float as link_coverage,
(SELECT count(*) FROM entity_pages e
WHERE EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = e.id))::float /
GREATEST((SELECT count(*) FROM entity_pages), 1)::float as timeline_coverage
`;
const connected = await sql`
SELECT p.slug,
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
FROM pages p
WHERE p.type IN ('person', 'company')
ORDER BY link_count DESC
LIMIT 5
`;
const pageCount = Number(h.page_count);
const embedCoverage = Number(h.embed_coverage);
const orphanPages = Number(h.orphan_pages);
const deadLinks = Number(h.dead_links);
const linkCount = Number(h.link_count);
const pagesWithTimeline = Number(h.pages_with_timeline);
// brain_score: 0-100 weighted average
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
const timelineCoverageWhole = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
// Per-component points. Sum equals brainScore by construction.
const embedCoverageScore = pageCount === 0 ? 0 : Math.round(embedCoverage * 35);
const linkDensityScore = pageCount === 0 ? 0 : Math.round(linkDensity * 25);
const timelineCoverageScore = pageCount === 0 ? 0 : Math.round(timelineCoverageWhole * 15);
const noOrphansScore = pageCount === 0 ? 0 : Math.round(noOrphans * 15);
const noDeadLinksScore = pageCount === 0 ? 0 : Math.round(noDeadLinks * 10);
const brainScore = embedCoverageScore + linkDensityScore + timelineCoverageScore + noOrphansScore + noDeadLinksScore;
return {
page_count: pageCount,
embed_coverage: embedCoverage,
stale_pages: Number(h.stale_pages),
orphan_pages: orphanPages,
missing_embeddings: Number(h.missing_embeddings),
brain_score: brainScore,
dead_links: deadLinks,
link_coverage: Number(h.link_coverage),
timeline_coverage: Number(h.timeline_coverage),
most_connected: (connected as unknown as { slug: string; link_count: number }[]).map(c => ({
slug: c.slug,
link_count: Number(c.link_count),
})),
embed_coverage_score: embedCoverageScore,
link_density_score: linkDensityScore,
timeline_coverage_score: timelineCoverageScore,
no_orphans_score: noOrphansScore,
no_dead_links_score: noDeadLinksScore,
};
}
// Ingest log
async logIngest(entry: IngestLogInput): Promise<void> {
const sql = this.sql;
await sql`
INSERT INTO ingest_log (source_type, source_ref, pages_updated, summary)
VALUES (${entry.source_type}, ${entry.source_ref}, ${sql.json(entry.pages_updated)}, ${entry.summary})
`;
}
async getIngestLog(opts?: { limit?: number }): Promise<IngestLogEntry[]> {
const sql = this.sql;
const limit = opts?.limit || 50;
const rows = await sql`
SELECT * FROM ingest_log ORDER BY created_at DESC LIMIT ${limit}
`;
return rows as unknown as IngestLogEntry[];
}
// Sync
async updateSlug(oldSlug: string, newSlug: string): Promise<void> {
newSlug = validateSlug(newSlug);
const sql = this.sql;
await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug}`;
}
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
// Stub in v0.2. Links table uses integer page_id FKs, which are already
// correct after updateSlug (page_id doesn't change, only slug does).
// Textual [[wiki-links]] in compiled_truth are NOT rewritten here.
// The maintain skill's dead link detector surfaces stale references.
}
// Config
async getConfig(key: string): Promise<string | null> {
const sql = this.sql;
const rows = await sql`SELECT value FROM config WHERE key = ${key}`;
return rows.length > 0 ? (rows[0].value as string) : null;
}
async setConfig(key: string, value: string): Promise<void> {
const sql = this.sql;
await sql`
INSERT INTO config (key, value) VALUES (${key}, ${value})
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value
`;
}
// Migration support
async runMigration(_version: number, sqlStr: string): Promise<void> {
const conn = this.sql;
await conn.unsafe(sqlStr);
}
async getChunksWithEmbeddings(slug: string): Promise<Chunk[]> {
const conn = this.sql;
const rows = await conn`
SELECT cc.* FROM content_chunks cc
JOIN pages p ON p.id = cc.page_id
WHERE p.slug = ${slug}
ORDER BY cc.chunk_index
`;
return rows.map((r) => rowToChunk(r as Record<string, unknown>, true));
}
/**
* Reconnect the engine by tearing down the current pool and creating a fresh one.
* No-ops if no saved config (module-singleton mode) or if already reconnecting.
*/
async reconnect(): Promise<void> {
if (!this._savedConfig || this._reconnecting) return;
this._reconnecting = true;
try {
// Tear down old pool (best-effort — it may already be dead)
try { await this.disconnect(); } catch { /* swallow */ }
// Create fresh pool
await this.connect(this._savedConfig);
} finally {
this._reconnecting = false;
}
}
async executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]> {
const conn = this.sql;
return conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]) as unknown as T[];
// Pre-#406 behavior: throw on any error including connection death.
// Per-call auto-retry is not safe here because executeRaw is also used
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
// ALTER TABLE in migrations) where retrying after a connection-mid-statement
// death can phantom-write a row that already committed on the server.
// Recovery instead happens at the supervisor level: the watchdog detects
// 3 consecutive health-check failures and calls engine.reconnect() to
// swap in a fresh pool. See db.ts setSessionDefaults / supervisor.ts.
}
// ============================================================
// v0.20.0 Cathedral II: code edges (Layer 1 stubs — filled by Layer 5)
// ============================================================
// Declared here so the interface contract is satisfied and consumers can
// import against them. Implementations throw until the edge extractor +
// per-lang tree-sitter queries land in Layer 5/6.
// ============================================================
async addCodeEdges(edges: import('./types.ts').CodeEdgeInput[]): Promise<number> {
if (edges.length === 0) return 0;
const sql = this.sql;
let inserted = 0;
const resolved = edges.filter(e => e.to_chunk_id != null);
const unresolved = edges.filter(e => e.to_chunk_id == null);
if (resolved.length > 0) {
const fromIds = resolved.map(e => e.from_chunk_id);
const toIds = resolved.map(e => e.to_chunk_id as number);
const fromQual = resolved.map(e => e.from_symbol_qualified);
const toQual = resolved.map(e => e.to_symbol_qualified);
const edgeTypes = resolved.map(e => e.edge_type);
const metas = resolved.map(e => JSON.stringify(e.edge_metadata ?? {}));
const sources = resolved.map(e => e.source_id ?? null);
const res = await sql`
INSERT INTO code_edges_chunk (from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
SELECT * FROM unnest(
${fromIds}::int[], ${toIds}::int[],
${fromQual}::text[], ${toQual}::text[],
${edgeTypes}::text[], ${metas}::jsonb[],
${sources}::text[]
)
ON CONFLICT (from_chunk_id, to_chunk_id, edge_type) DO NOTHING
`;
inserted += (res as unknown as { count: number }).count ?? 0;
}
if (unresolved.length > 0) {
const fromIds = unresolved.map(e => e.from_chunk_id);
const fromQual = unresolved.map(e => e.from_symbol_qualified);
const toQual = unresolved.map(e => e.to_symbol_qualified);
const edgeTypes = unresolved.map(e => e.edge_type);
const metas = unresolved.map(e => JSON.stringify(e.edge_metadata ?? {}));
const sources = unresolved.map(e => e.source_id ?? null);
const res = await sql`
INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
SELECT * FROM unnest(
${fromIds}::int[],
${fromQual}::text[], ${toQual}::text[],
${edgeTypes}::text[], ${metas}::jsonb[],
${sources}::text[]
)
ON CONFLICT (from_chunk_id, to_symbol_qualified, edge_type) DO NOTHING
`;
inserted += (res as unknown as { count: number }).count ?? 0;
}
return inserted;
}
async deleteCodeEdgesForChunks(chunkIds: number[]): Promise<void> {
if (chunkIds.length === 0) return;
const sql = this.sql;
await sql`DELETE FROM code_edges_chunk WHERE from_chunk_id = ANY(${chunkIds}::int[]) OR to_chunk_id = ANY(${chunkIds}::int[])`;
await sql`DELETE FROM code_edges_symbol WHERE from_chunk_id = ANY(${chunkIds}::int[])`;
}
async getCallersOf(
qualifiedName: string,
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
): Promise<import('./types.ts').CodeEdgeResult[]> {
const sql = this.sql;
const limit = Math.min(opts?.limit ?? 100, 500);
const scopedSource: string | null =
!opts?.allSources && opts?.sourceId ? opts.sourceId : null;
const rows = await sql`
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
edge_type, edge_metadata, source_id, true as resolved
FROM code_edges_chunk
WHERE to_symbol_qualified = ${qualifiedName}
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
UNION ALL
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
edge_type, edge_metadata, source_id, false as resolved
FROM code_edges_symbol
WHERE to_symbol_qualified = ${qualifiedName}
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
LIMIT ${limit}
`;
return rows.map(r => pgRowToCodeEdge(r as Record<string, unknown>));
}
async getCalleesOf(
qualifiedName: string,
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
): Promise<import('./types.ts').CodeEdgeResult[]> {
const sql = this.sql;
const limit = Math.min(opts?.limit ?? 100, 500);
const scopedSource: string | null =
!opts?.allSources && opts?.sourceId ? opts.sourceId : null;
const rows = await sql`
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
edge_type, edge_metadata, source_id, true as resolved
FROM code_edges_chunk
WHERE from_symbol_qualified = ${qualifiedName}
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
UNION ALL
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
edge_type, edge_metadata, source_id, false as resolved
FROM code_edges_symbol
WHERE from_symbol_qualified = ${qualifiedName}
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
LIMIT ${limit}
`;
return rows.map(r => pgRowToCodeEdge(r as Record<string, unknown>));
}
async getEdgesByChunk(
chunkId: number,
opts?: { direction?: 'in' | 'out' | 'both'; edgeType?: string; limit?: number },
): Promise<import('./types.ts').CodeEdgeResult[]> {
const sql = this.sql;
const direction = opts?.direction ?? 'both';
const limit = Math.min(opts?.limit ?? 50, 200);
const typeFilter = opts?.edgeType;
const chunkRows = await sql`
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
edge_type, edge_metadata, source_id, true as resolved
FROM code_edges_chunk
WHERE
${direction === 'in' ? sql`to_chunk_id = ${chunkId}`
: direction === 'out' ? sql`from_chunk_id = ${chunkId}`
: sql`(from_chunk_id = ${chunkId} OR to_chunk_id = ${chunkId})`}
${typeFilter ? sql`AND edge_type = ${typeFilter}` : sql``}
LIMIT ${limit}
`;
let symbolRows: unknown[] = [];
if (direction !== 'in') {
const sRows = await sql`
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
edge_type, edge_metadata, source_id, false as resolved
FROM code_edges_symbol
WHERE from_chunk_id = ${chunkId}
${typeFilter ? sql`AND edge_type = ${typeFilter}` : sql``}
LIMIT ${limit}
`;
symbolRows = [...sRows];
}
return [...chunkRows, ...symbolRows].map(r => pgRowToCodeEdge(r as Record<string, unknown>));
}
}
function pgRowToCodeEdge(row: Record<string, unknown>): import('./types.ts').CodeEdgeResult {
return {
id: row.id as number,
from_chunk_id: row.from_chunk_id as number,
to_chunk_id: row.to_chunk_id == null ? null : (row.to_chunk_id as number),
from_symbol_qualified: (row.from_symbol_qualified as string) ?? '',
to_symbol_qualified: (row.to_symbol_qualified as string) ?? '',
edge_type: (row.edge_type as string) ?? '',
edge_metadata: (row.edge_metadata as Record<string, unknown>) ?? {},
source_id: row.source_id == null ? null : (row.source_id as string),
resolved: Boolean(row.resolved),
};
}