mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat: storage tiering — git-tracked vs supabase-only directories Brain repos scaling to 200K+ files. Bulk data (tweets, articles, transcripts) bloats git repos and slows operations. New storage config in gbrain.yml lets users declare git-tracked and supabase-only directories. Changes: - New config: storage.git_tracked and storage.supabase_only in gbrain.yml - gbrain sync auto-manages .gitignore for supabase-only paths - gbrain export --restore-only restores missing supabase-only files from DB - New gbrain storage status command shows tier breakdown - Config validation warns on conflicts - 8 tests passing, full docs at docs/storage-tiering.md Backward compatible — systems without gbrain.yml work unchanged. * feat: add getDefaultSourcePath() typed accessor (step 1/15) Single source of truth for "what brain repo are we operating against?" Replaces ad-hoc raw SQL in storage.ts:38 (Issue #3 of eng review). Used by both gbrain storage status and gbrain export --restore-only. Returns null on miss, throws on DB error. Composes with the existing resolveSourceId chain so it honors --source flag / GBRAIN_SOURCE env / .gbrain-source dotfile / longest-prefix CWD match / brain-level default. 4 new test cases covering happy path, missing local_path, DB error propagation, and CWD-prefix resolution priority. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: replace gray-matter with dedicated YAML parser (step 2/15) The original storage-config.ts called gray-matter on a delimiter-less YAML file. Gray-matter only parses YAML inside `---` frontmatter blocks; without delimiters, it returns `{data: {}}`. Result: loadStorageConfig() always returned null, the entire feature was a silent no-op for every user. Original eng review's P0 confidence-9 finding (Issue #1). Replaces gray-matter with a small dedicated parser for the gbrain.yml shape (top-level `storage:` section, two array-valued nested keys). Yaml-lite was considered first, but its flat key:value design doesn't handle nested arrays. The dedicated parser is ~50 lines and trades expressiveness for zero-dep, predictable parsing of a file format we control. Adds the Issue #1B sanity warning (locked B): when gbrain.yml exists but has no storage section (or empty arrays), warn once-per-process so the user sees their config didn't take. The single test that would have caught the original P0 — write a real gbrain.yml, call loadStorageConfig, assert non-null — now exists. Also tightens loadStorageConfig per D36: distinguishes "absent" (silent null) from "unreadable" (throws). The previous code silently swallowed read errors, hiding broken installs. 8 new test cases: real-disk happy path, comments + blank lines, quoted values, missing storage section warning, empty section warning, once-per-process warning suppression, unreadable file behavior, and the existing helper tests (validation, tier matching, edge cases) all still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: rename storage keys to db_tracked/db_only (step 3/15) The vendor-specific names "supabase_only" and "git_tracked" hardcoded a backend (Supabase) into the config schema. gbrain ships two engines — PGLite and Postgres-via-Supabase. The canonical distinction is "lives in the brain DB only" vs "lives in the brain DB and on disk under git." Both work on either engine. Renamed throughout (Issue #4 of eng review): git_tracked → db_tracked supabase_only → db_only isGitTracked() → isDbTracked() isSupabaseOnly() → isDbOnly() StorageTier 'git_tracked'/'supabase_only' → 'db_tracked'/'db_only' Backward compatibility (D3 lock): loadStorageConfig accepts both shapes. Loader resolution order per the eng-review pass-2 finding: parse YAML → if canonical keys present use them, else if deprecated keys present map to canonical AND emit once-per-process deprecation warning → THEN run validation. Validation always sees the canonical shape so error messages reference db_tracked/db_only regardless of which keys the user wrote. The deprecation warning suggests `gbrain doctor --fix` for an automated rename (D72 — fix path lands in step 7). When both shapes coexist in one file, canonical wins and a stronger warning fires ("deprecated keys ignored — remove them"). Aliases isGitTracked/isSupabaseOnly kept for now to avoid churning the sync.ts / export.ts / storage.ts call sites in this commit; they'll be removed in a follow-up step. Storage.ts's tier-bucket initializers and output strings updated. ASCII output replaces unicode box-drawing per D10. gbrain.yml example file updated to canonical keys with explanatory comments. 2 new test cases: deprecated-key fallback (asserts both shapes load correctly with warning), canonical-wins-over-deprecated (asserts the "both shapes coexist" path). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add slugPrefix to PageFilters with engine-side filter (step 4/15) Issue #13 of the eng review: storage.ts and export.ts loaded every page in the brain (limit: 1_000_000) to check tier membership. On the 200K-page brains this feature targets, that's the wall-clock and memory landmine the feature exists to fix. Adds an optional `slugPrefix` field to PageFilters. Both engines implement it as `WHERE slug LIKE prefix || '%' ESCAPE '\'`, with literal escaping of LIKE metacharacters (%, _, \) so user-supplied prefixes like `media/x/` are treated as exact string prefixes. Performance: the (source_id, slug) UNIQUE constraint on the pages table gives both engines a btree index that supports LIKE-prefix range scans. An EXPLAIN on Postgres confirms the index range scan rather than a seq scan. PGLite has the same index shape via pglite-schema.ts. Consumers updated: - export.ts: --slug-prefix flag now goes engine-side (no in-memory .filter(...)). The --restore-only path queries each db_only directory with slugPrefix in a loop instead of one full-table scan, with seen-set deduplication and disk-existence check inline. - storage.ts: keeps the full-scan path because storage-status needs the "unspecified" bucket count, which can't be computed without enumerating every page. Comment notes that step 5 (single-walk filesystem scan) will reduce per-page disk syscall cost. 2 new test cases on PGLiteEngine: slugPrefix happy path (3 tier dirs, asserts only matching slugs return) and metacharacter escape regression (asserts safe/ doesn't match unrelated slugs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf: single-walk filesystem scan via walkBrainRepo() (step 5/15) Issue #14 of the eng review: storage.ts called existsSync + statSync per-page in a synchronous loop. On a 200K-page brain that's 400K syscalls serialized. Wall-clock landmine. Adds src/core/disk-walk.ts with walkBrainRepo(repoPath) — one recursive readdirSync walk, builds a Map<slug, {size, mtimeMs}>. Storage.ts looks up each DB page in the map (O(1)) instead of stat-checking on demand. Slug derivation matches the pages-table convention: people/alice.md on disk becomes people/alice as the map key. Skipped during walk: - dot-directories (.git, .gbrain, .vscode, etc) — not part of the brain namespace - node_modules — guards against accidentally walking into imported repos - non-.md files (sidecar JSON, binaries) — tracked by the brain through the files table, not by slug Reusable: future commands (gbrain doctor's storage_tiering check, the optional autopilot tier-fix path) get the same walk for free. 9 new test cases: empty dir, nonexistent dir, top-level files, nested dirs, dot-dir skipping, node_modules skipping, non-.md filtering, size capture, mtimeMs capture. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: path-segment matching for tier directories (step 6/15) Issue #5 + D6 of the eng review: tier matching used slug.startsWith(dir), which falsely matches 'media/xerox/foo' against 'media/x' if a user wrote the directory without a trailing slash. The new matcher requires the configured directory to end with `/` and treats it as a canonical path-segment ancestor: media/x/ matches media/x/tweet-1 ✓ media/x/ doesn't media/xerox/foo ✗ media/x refused media/x/tweet-1 (matcher requires trailing /) Non-canonical input (no trailing slash) is refused outright. Step 7's auto-normalizing validator converts user-written 'media/x' → 'media/x/' on load, so the matcher never sees non-canonical input from real configs. The behavior tested here is the strict matcher's contract. Regression test pins the media/xerox collision case explicitly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: auto-normalize trailing-slash, throw on tier overlap (step 7/15) D7+D8 of the eng review: validation was warnings-only. Users miss warnings. Now: - Cosmetic: missing trailing slash auto-corrected, one-time info note showing what changed ("normalized 2 storage paths: 'people' → 'people/', 'media/x' → 'media/x/'"). Once-per-process to keep noise low. - Semantic: same directory in both tiers throws StorageConfigError. Ambiguous routing — does media/ win as db_tracked or db_only? — is a real bug the user must fix. Caller propagates to the CLI for a clean exit-1 with actionable message. loadStorageConfig now applies normalize+validate after merging deprecated keys, so the path-segment matcher (step 6) only ever sees canonical trailing-slash directories. The pure validateStorageConfig kept for callers who want the warnings list without the auto-fix side effects (gbrain doctor's reporting path). 2 new test cases: auto-normalize round-trip with warning text assertion, overlap throws StorageConfigError. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: wire manageGitignore into runSync, only on success (step 8/15) Issue #2 of the eng review: manageGitignore was defined and never invoked. Docs claimed "auto-managed by gbrain" — false. Users hit a .gitignore that never updated and committed db_only directories anyway. Wire-up: runSync now calls manageGitignore after each successful performSync return, in both watch and one-shot modes. Eng review pass-2 finding #1: skip on dry_run AND blocked_by_failures status. A sync that aborted partway has stale state; mutating .gitignore based on a partially-loaded config invites drift. Failure-skip test added (uses .gitignore-as-a-directory to simulate write failure; asserts warning fired and disk wasn't corrupted). Hardened manageGitignore itself with three additional behaviors: - GBRAIN_NO_GITIGNORE=1 escape hatch (D23) for shared-repo setups where a maintainer wants gbrain to leave .gitignore alone. - Submodule detection (D49). When repoPath/.git is a regular file (gitdir: ... pointer), the repo is a git submodule. Submodule .gitignore changes don't survive parent submodule updates, so we skip with an actionable warning ("add db_only directories to your parent repo's .gitignore manually"). - Graceful failure (D9). Read errors, write errors, and StorageConfigError (overlap from step 7) all log a warning and return — sync's primary job (moving data) shouldn't die because of a side-effect on .gitignore. manageGitignore is now exported (previously private) so the storage-sync test file can hit it directly without spinning up sync. 9 new test cases: no-op without gbrain.yml, no-op with empty db_only, happy-path append, idempotency (run twice, single entry), preservation of user-written rules, GBRAIN_NO_GITIGNORE skip, submodule skip, .git-directory normal path, write-failure graceful warning. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: D5 resolution chain for --restore-only and storage status (step 9/15) D5 of the eng review: gbrain export --restore-only without --repo silently fell through to the regular export path, dumping every page in the database to the wrong directory. Hard regression risk. Now exits 1 with an actionable message when --restore-only has no --repo AND no configured default source. Resolution order: 1. Explicit --repo flag 2. Typed sources.getDefault() (reuses step 1's accessor) 3. Hard error — never fall through to cwd storage.ts:38 also bypassed BrainEngine with raw SQL and a bare try/catch (Issue #3 + Issue #9). Replaced with the same typed getDefaultSourcePath() — single source of truth, errors propagate cleanly to the user, no silent cwd fallback. Regular export (no --restore-only) keeps its current behavior per D26: exports include everything, --repo is optional. 4 new test cases on PGLite in-memory: - hard-errors with no --repo + no default - explicit --repo wins - falls back to sources default local_path - non-restore export does not require --repo Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor: split storage.ts into pure data + JSON + human formatters (step 10/15) Issue #10 of the eng review: getStorageStatus and runStorageStatus mixed data gathering, JSON serialization, and human-readable output in one function. Hard to test, hard to reuse, mismatched the orphans.ts pattern that CLAUDE.md cites as the precedent. Now three pure functions + a thin dispatcher: getStorageStatus(engine, repoPath) — async, returns StorageStatusResult. Side effects: engine.listPages + one walkBrainRepo (Issue #14). Exported so MCP exposure (D14) and gbrain doctor (D13) can consume the same data without re-running the loop. formatStorageStatusJson(result) — pure, returns indented JSON. Stable contract on the StorageStatusResult shape, suitable for orchestrators. formatStorageStatusHuman(result) — pure, returns ASCII text (D10 — no unicode box-drawing). Composable into other commands later. runStorageStatus(engine, args) — thin dispatcher: parses --repo / --json, calls getStorageStatus, picks a formatter, prints. 8 new test cases on the formatters: JSON parse round-trip, null-config fallback, missing-files capped at 10 with rollup, ASCII-only assertion (D10 regression guard), warnings inline, configuration listing, disk- usage block omitted when zero bytes. The StorageStatusResult interface is now exported as a public type, so gbrain doctor's storage_tiering check can build its own findings from the same shape. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * types: distinct PageCountsByTier and DiskUsageByTier (step 11/15) Issue #11 of the eng review: pagesByTier (page counts) and diskUsageByTier (byte totals) shared the same structural type (Record<StorageTier, number>). Both are tier-keyed numeric maps but carry semantically different units. A future bug that swaps them at a call site (e.g., displaying disk bytes where the count belongs) wouldn't trip the compiler. Replaced with distinct nominal types via a brand field. Structurally identical at runtime (no overhead) but compile-time disjoint — TypeScript catches accidental cross-assignment. PageCountsByTier { db_tracked, db_only, unspecified } : numbers (count) DiskUsageByTier { db_tracked, db_only, unspecified } : numbers (bytes) Both initialized in getStorageStatus, both threaded into StorageStatusResult, both consumed by formatStorageStatusHuman / formatStorageStatusJson without further changes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: PGLite soft-warn + full lifecycle test (step 12/15) D4: storage tiering on PGLite is a partial feature. The "DB" the pages live in IS the local file gbrain uses for everything else, so "db_only" has no real offload effect. The .gitignore management still helps (keeps bulk content out of git history), so we warn and proceed — not refuse. Two warning sites (once-per-process each via module-local flags): - storage status: warns at runStorageStatus entry - sync: warns inside manageGitignore when engineKind='pglite' and config has db_only entries Both phrased actionably ("To get full tiering, migrate to Postgres with `gbrain migrate --to supabase`"). manageGitignore signature now takes an optional `engineKind` param. runSync passes engine.kind. Stand-alone callers (tests, future gbrain doctor --fix path) can omit it. New test: test/storage-pglite.test.ts — D8 + D4 lifecycle. 6 cases: engine.kind assertion, getStorageStatus loading gbrain.yml + reporting tier counts, manageGitignore PGLite-warn (once per process), Postgres no-warn, slugPrefix on PGLite, end-to-end (config + putPage + status + gitignore). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: add trailing-newline CI guard (step 14/15) Issue #7 of the eng review: all four new files in the original storage-tiering branch lacked POSIX trailing newlines. Linters complain, git diffs phantom-flag every future edit. We've been adding newlines as each file landed; this commit catches the regression class. scripts/check-trailing-newline.sh: - sibling to check-jsonb-pattern.sh / check-progress-to-stdout.sh per CLAUDE.md's CI guard pattern - portable to bash 3.2 (macOS default; no mapfile, no associative arrays) - covers src/**, test/**, gbrain.yml, top-level *.md - reports each missing file by path and exits 1 Wired into `bun run test` between progress-to-stdout and typecheck. Also fixed docs/storage-tiering.md (pre-existing missing newline from the original branch — caught by the new guard on first run). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: v0.23.0 — VERSION, CHANGELOG, README, CLAUDE.md, storage-tiering.md (step 15/15) VERSION → 0.23.0 (minor bump for new feature surface). CHANGELOG entry in Garry voice with the canonical format: - Two-line bold headline ("Storage tiering, finally working...") - Lead paragraph naming what was broken before and what users get now - "Numbers that matter" before/after table for the 6 things that actually changed - "What this means for your brain" closer - "To take advantage of v0.23.0" self-repair block (per CLAUDE.md convention) — 6 numbered steps users can follow - Itemized changes split into critical fixes / new+renamed surface / architecture cleanup / tests + CI guards CLAUDE.md "Key files" gains four new entries: storage-config.ts, disk-walk.ts, the v0.23.0 storage.ts shape, and gbrain.yml itself. README.md gains a new "Storage tiering" section between Skillify and Getting Data In with the canonical example + commands + link to the full guide. docs/storage-tiering.md rewritten end-to-end with canonical key names (db_tracked / db_only), v0.23.0 hardening details (idempotency, submodule detection, GBRAIN_NO_GITIGNORE, dry-run gating), the resolution chain for --restore-only, the auto-normalize + throw-on-overlap validator, and the PGLite engine note. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: e2e Postgres lifecycle for storage tiering (step 16/16) Per the v0.23.0 plan: full lifecycle E2E against real Postgres. - engine.kind === 'postgres' assertion - Full lifecycle: write 4 pages (1 db_tracked, 2 db_only, 1 unspecified) → getStorageStatus reports correct tier counts → human formatter renders → manageGitignore writes managed block → idempotency check → getDefaultSourcePath() resolves the configured local_path. - Container restart simulation: 2 db_only pages in DB, files missing on disk → status.missingFiles.length === 2 → slugPrefix engine filter on Postgres returns exactly the tier slugs. - slugPrefix index-based range scan regression: 50 media/x/* + 50 people/p-* pages → slugPrefix='media/x/' returns exactly 50. - getDefaultSourcePath returns null when default source has no local_path (the hard-error path that replaces the original silent cwd fallback). - manageGitignore on Postgres engine does NOT emit the PGLite soft-warn (cross-engine assertion). Skips gracefully when DATABASE_URL is unset, per CLAUDE.md E2E pattern. Run via: DATABASE_URL=... bun test test/e2e/storage-tiering.test.ts Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump version 0.23.0 → 0.22.9 Reverts the minor bump back to a patch-style version on the v0.22 line. Storage tiering ships within the v0.22.x train alongside the recent fix waves. Updates VERSION, package.json, CHANGELOG header + body refs, CLAUDE.md Key files annotations, README.md section heading, and the docs/storage-tiering.md backward-compat note. * chore: bump version 0.22.9 → 0.22.11 Sibling workspaces claimed v0.22.10 in the queue. This branch advances to v0.22.11 to keep the version monotonic on master. Updates VERSION, package.json, CHANGELOG header + body refs, CLAUDE.md Key files annotations, README.md section heading, and the docs/storage-tiering.md backward-compat note. * fix: address Codex pre-landing review findings (4 fixes) Codex found 4 real issues during pre-landing review of v0.22.11 diff: [P0] export --restore-only fell through to full export when storageConfig was null (no gbrain.yml present). On older or misconfigured brains, the recovery command would silently dump the entire database. src/commands/export.ts now refuses with an actionable error before any page query fires — matches the D5 lock spirit ("never silently fall through"). [P1] manageGitignore wire-up only fired when --repo was passed explicitly. performSync resolves the repo from sync.repo_path or sources.local_path, so the common `gbrain sync` path (after setup, no flag) never updated .gitignore. src/commands/sync.ts now uses the same source-resolver chain as the rest of /ship: opts.repoPath → getDefaultSourcePath → null. Fires in both watch and one-shot modes. [P2] getDefaultSourcePath only consulted sources.local_path, missing the legacy global sync.repo_path config key that pre-v0.18 brains use. Added a fallback to engine.getConfig('sync.repo_path') when the sources row has NULL local_path. Pre-v0.18 brains now work without forcing a `gbrain sources add . --path .` migration. [P2] sync --all multi-source loop never called manageGitignore even though src.local_path was already known. Each source now gets its own gitignore update on successful sync. Tests: - test/storage-export.test.ts: replaced the old "falls through to full export" test with one that asserts the new refusal path (storage-tiering config required for --restore-only). - test/source-resolver.test.ts: added a fallback test exercising the legacy sync.repo_path code path for pre-v0.18 brains. - All 78 storage-tiering tests still pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: regenerate llms.txt + llms-full.txt for v0.22.11 Per CLAUDE.md: "Run `bun run build:llms` after adding a new doc." The README's new Storage tiering section + the rewritten docs/storage-tiering.md changed the inlined bundle. test/build-llms.test.ts catches the drift and was failing on master pre-regen. * fix: typecheck error in disk-walk.ts (CI #73350475897) tsc --noEmit failed in CI because ReturnType<typeof readdirSync> with withFileTypes:true picks an overload union that includes Dirent<Buffer<ArrayBufferLike>>. Strict tsc treats entry.name as Buffer, so .startsWith / .endsWith / string comparisons all blew up. Annotate the variable as Dirent[] (string-based) and cast through unknown, matching the pattern sync.ts already uses for its own filesystem walk. Same runtime behavior; clean typecheck. Tests still 9/9. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> EOF --------- Co-authored-by: root <root@localhost> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1722 lines
71 KiB
TypeScript
1722 lines
71 KiB
TypeScript
import postgres from 'postgres';
|
|
import type { BrainEngine, LinkBatchInput, TimelineBatchInput, ReservedConnection } from './engine.ts';
|
|
import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
|
|
import { runMigrations } from './migrate.ts';
|
|
import { SCHEMA_SQL } from './schema-embedded.ts';
|
|
import { verifySchema } from './schema-verify.ts';
|
|
import type {
|
|
Page, PageInput, PageFilters, PageType,
|
|
Chunk, ChunkInput, StaleChunkRow,
|
|
SearchResult, SearchOpts,
|
|
Link, GraphNode, GraphPath,
|
|
TimelineEntry, TimelineInput, TimelineOpts,
|
|
RawData,
|
|
PageVersion,
|
|
BrainStats, BrainHealth,
|
|
IngestLogEntry, IngestLogInput,
|
|
EngineConfig,
|
|
} from './types.ts';
|
|
import { GBrainError } from './types.ts';
|
|
import * as db from './db.ts';
|
|
import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts';
|
|
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
|
|
import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts';
|
|
|
|
// CONNECTION_ERROR_PATTERNS / isConnectionError were used by the per-call
|
|
// executeRaw retry that #406 originally shipped. Eng-review D3 dropped that
|
|
// retry as unsound (regex idempotence-boundary doesn't hold for writable
|
|
// CTEs or side-effecting SELECTs). Recovery now happens at the supervisor
|
|
// level (3-strikes-then-reconnect). The unit tests in
|
|
// test/connection-resilience.test.ts retain a self-contained copy of the
|
|
// helper so the regression-against-future-reintroduction guard still works.
|
|
// See TODOS.md item: "err.code-based connection-error matching" for the
|
|
// follow-up that will reintroduce a typed retry mechanism.
|
|
|
|
export class PostgresEngine implements BrainEngine {
|
|
readonly kind = 'postgres' as const;
|
|
private _sql: ReturnType<typeof postgres> | null = null;
|
|
/** Saved config for reconnection. */
|
|
private _savedConfig: (EngineConfig & { poolSize?: number }) | null = null;
|
|
/** Whether a reconnect is in progress (prevents concurrent reconnects). */
|
|
private _reconnecting = false;
|
|
|
|
// Instance connection (for workers) or fall back to module global (backward compat)
|
|
get sql(): ReturnType<typeof postgres> {
|
|
if (this._sql) return this._sql;
|
|
return db.getConnection();
|
|
}
|
|
|
|
// Lifecycle
|
|
async connect(config: EngineConfig & { poolSize?: number }): Promise<void> {
|
|
this._savedConfig = config;
|
|
if (config.poolSize) {
|
|
// Instance-level connection for worker isolation. resolvePoolSize lets
|
|
// GBRAIN_POOL_SIZE cap below the caller's requested size when set — the
|
|
// env var is a user escape hatch, so it wins.
|
|
const url = config.database_url;
|
|
if (!url) throw new GBrainError('No database URL', 'database_url is missing', 'Provide --url');
|
|
const size = Math.min(config.poolSize, db.resolvePoolSize(config.poolSize));
|
|
// Honor PgBouncer transaction-mode detection on worker-instance pools too.
|
|
// Without this, `gbrain jobs work` against a Supabase pooler URL hits
|
|
// "prepared statement does not exist" under load just like the module
|
|
// singleton did before v0.15.4.
|
|
const prepare = db.resolvePrepare(url);
|
|
// Session timeouts (statement_timeout + idle_in_transaction_session_timeout)
|
|
// keep orphan pgbouncer backends from holding locks for hours when the
|
|
// postgres.js client disconnects mid-transaction. See resolveSessionTimeouts
|
|
// in db.ts for context + env var overrides.
|
|
const timeouts = db.resolveSessionTimeouts();
|
|
const opts: Record<string, unknown> = {
|
|
max: size,
|
|
idle_timeout: 20,
|
|
connect_timeout: 10,
|
|
types: { bigint: postgres.BigInt },
|
|
};
|
|
if (Object.keys(timeouts).length > 0) {
|
|
opts.connection = timeouts;
|
|
}
|
|
if (typeof prepare === 'boolean') {
|
|
opts.prepare = prepare;
|
|
}
|
|
this._sql = postgres(url, opts);
|
|
await this._sql`SELECT 1`;
|
|
await db.setSessionDefaults(this._sql);
|
|
} else {
|
|
// Module-level singleton (backward compat for CLI main engine)
|
|
await db.connect(config);
|
|
}
|
|
}
|
|
|
|
async disconnect(): Promise<void> {
|
|
if (this._sql) {
|
|
await this._sql.end();
|
|
this._sql = null;
|
|
} else {
|
|
await db.disconnect();
|
|
}
|
|
}
|
|
|
|
async initSchema(): Promise<void> {
|
|
const conn = this.sql;
|
|
// Advisory lock prevents concurrent initSchema() calls from deadlocking
|
|
// on DDL statements (DROP TRIGGER + CREATE TRIGGER acquire AccessExclusiveLock).
|
|
//
|
|
// Honest limitation: pg_advisory_lock(42) is session-scoped to this pooled
|
|
// connection. runMigrations() below uses engine.transaction() and
|
|
// withReservedConnection() which may hop to a different backend in the
|
|
// pool. Cross-process serialization of initSchema is best-effort, not a
|
|
// correctness guarantee. Pre-existing concern; the bootstrap doesn't
|
|
// change it.
|
|
await conn`SELECT pg_advisory_lock(42)`;
|
|
try {
|
|
// Pre-schema bootstrap: add forward-referenced state the embedded schema
|
|
// blob requires but that older brains don't have yet. Without this, a
|
|
// pre-v0.18 brain hits `CREATE INDEX idx_pages_source_id ON pages(source_id)`
|
|
// (issues #366/#375/#378/#396), or a pre-v0.13 brain hits
|
|
// `CREATE INDEX idx_links_source ON links(link_source)` (#266/#357), and
|
|
// SCHEMA_SQL crashes before runMigrations gets a chance to apply the
|
|
// missing column. Bootstrap is structurally idempotent and a no-op on
|
|
// fresh installs and modern brains.
|
|
await this.applyForwardReferenceBootstrap();
|
|
|
|
await conn.unsafe(SCHEMA_SQL);
|
|
|
|
// Run any pending migrations automatically
|
|
const { applied } = await runMigrations(this);
|
|
if (applied > 0) {
|
|
console.log(` ${applied} migration(s) applied`);
|
|
}
|
|
|
|
// Post-migration schema verification: catches columns that migrations
|
|
// defined but PgBouncer transaction-mode silently failed to create.
|
|
// Self-heals missing columns via ALTER TABLE ADD COLUMN IF NOT EXISTS.
|
|
const verify = await verifySchema(this);
|
|
if (verify.healed.length > 0) {
|
|
console.log(` Schema verify: self-healed ${verify.healed.length} missing column(s)`);
|
|
}
|
|
} finally {
|
|
await conn`SELECT pg_advisory_unlock(42)`;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Bootstrap state that SCHEMA_SQL forward-references but that older brains
|
|
* don't have yet. Mirror of `PGLiteEngine#applyForwardReferenceBootstrap`
|
|
* in shape and intent. Currently covers:
|
|
*
|
|
* - `sources` table + default seed (FK target of pages.source_id) — v0.18
|
|
* - `pages.source_id` column (indexed by `idx_pages_source_id`) — v0.18
|
|
* - `links.link_source` column (indexed by `idx_links_source`) — v0.13
|
|
* - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13
|
|
* - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19
|
|
* - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19
|
|
*
|
|
* Keep this in sync with the PGLite version; covered by
|
|
* `test/schema-bootstrap-coverage.test.ts` (PGLite side) and
|
|
* `test/e2e/postgres-bootstrap.test.ts` (Postgres side).
|
|
*/
|
|
private async applyForwardReferenceBootstrap(): Promise<void> {
|
|
const conn = this.sql;
|
|
|
|
// Single round-trip probe for every forward-reference target.
|
|
// current_schema() resolves to whatever search_path the connection uses,
|
|
// which matches schema-embedded.ts's `public.` references.
|
|
const probeRows = await conn<{
|
|
pages_exists: boolean;
|
|
source_id_exists: boolean;
|
|
links_exists: boolean;
|
|
link_source_exists: boolean;
|
|
origin_page_id_exists: boolean;
|
|
chunks_exists: boolean;
|
|
symbol_name_exists: boolean;
|
|
language_exists: boolean;
|
|
}[]>`
|
|
SELECT
|
|
EXISTS (SELECT 1 FROM information_schema.tables
|
|
WHERE table_schema = current_schema() AND table_name = 'pages') AS pages_exists,
|
|
EXISTS (SELECT 1 FROM information_schema.columns
|
|
WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_id') AS source_id_exists,
|
|
EXISTS (SELECT 1 FROM information_schema.tables
|
|
WHERE table_schema = current_schema() AND table_name = 'links') AS links_exists,
|
|
EXISTS (SELECT 1 FROM information_schema.columns
|
|
WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'link_source') AS link_source_exists,
|
|
EXISTS (SELECT 1 FROM information_schema.columns
|
|
WHERE table_schema = current_schema() AND table_name = 'links' AND column_name = 'origin_page_id') AS origin_page_id_exists,
|
|
EXISTS (SELECT 1 FROM information_schema.tables
|
|
WHERE table_schema = current_schema() AND table_name = 'content_chunks') AS chunks_exists,
|
|
EXISTS (SELECT 1 FROM information_schema.columns
|
|
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'symbol_name') AS symbol_name_exists,
|
|
EXISTS (SELECT 1 FROM information_schema.columns
|
|
WHERE table_schema = current_schema() AND table_name = 'content_chunks' AND column_name = 'language') AS language_exists
|
|
`;
|
|
const probe = probeRows[0]!;
|
|
|
|
const needsPagesBootstrap = probe.pages_exists && !probe.source_id_exists;
|
|
const needsLinksBootstrap = probe.links_exists
|
|
&& (!probe.link_source_exists || !probe.origin_page_id_exists);
|
|
const needsChunksBootstrap = probe.chunks_exists
|
|
&& (!probe.symbol_name_exists || !probe.language_exists);
|
|
|
|
if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return;
|
|
|
|
console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap');
|
|
|
|
if (needsPagesBootstrap) {
|
|
// Mirror schema-embedded.ts's `sources` shape so the subsequent
|
|
// SCHEMA_SQL CREATE TABLE IF NOT EXISTS is a true no-op.
|
|
await conn.unsafe(`
|
|
CREATE TABLE IF NOT EXISTS sources (
|
|
id TEXT PRIMARY KEY,
|
|
name TEXT NOT NULL UNIQUE,
|
|
local_path TEXT,
|
|
last_commit TEXT,
|
|
last_sync_at TIMESTAMPTZ,
|
|
config JSONB NOT NULL DEFAULT '{}'::jsonb,
|
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
|
);
|
|
INSERT INTO sources (id, name, config)
|
|
VALUES ('default', 'default', '{"federated": true}'::jsonb)
|
|
ON CONFLICT (id) DO NOTHING;
|
|
ALTER TABLE pages ADD COLUMN IF NOT EXISTS source_id TEXT
|
|
NOT NULL DEFAULT 'default' REFERENCES sources(id) ON DELETE CASCADE;
|
|
`);
|
|
}
|
|
|
|
if (needsLinksBootstrap) {
|
|
// v11 (links_provenance_columns) handles the CHECK constraint, the
|
|
// UNIQUE swap, and the backfill. The bootstrap only adds enough state
|
|
// for SCHEMA_SQL's `CREATE INDEX idx_links_source/origin` not to crash.
|
|
// v11 runs later via runMigrations and is idempotent.
|
|
await conn.unsafe(`
|
|
ALTER TABLE links ADD COLUMN IF NOT EXISTS link_source TEXT;
|
|
ALTER TABLE links ADD COLUMN IF NOT EXISTS origin_page_id INTEGER
|
|
REFERENCES pages(id) ON DELETE SET NULL;
|
|
`);
|
|
}
|
|
|
|
if (needsChunksBootstrap) {
|
|
// v26 (content_chunks_code_metadata) adds the full code-chunk metadata
|
|
// surface. The bootstrap only adds the two columns the schema blob's
|
|
// partial indexes reference (idx_chunks_symbol_name, idx_chunks_language).
|
|
// v26 runs later via runMigrations and adds the rest idempotently.
|
|
await conn.unsafe(`
|
|
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS language TEXT;
|
|
ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT;
|
|
`);
|
|
}
|
|
}
|
|
|
|
async transaction<T>(fn: (engine: BrainEngine) => Promise<T>): Promise<T> {
|
|
const conn = this._sql || db.getConnection();
|
|
return conn.begin(async (tx) => {
|
|
// Create a scoped engine with tx as its connection, no shared state mutation
|
|
const txEngine = Object.create(this) as PostgresEngine;
|
|
Object.defineProperty(txEngine, 'sql', { get: () => tx });
|
|
Object.defineProperty(txEngine, '_sql', { value: tx as unknown as ReturnType<typeof postgres>, writable: false });
|
|
return fn(txEngine);
|
|
}) as Promise<T>;
|
|
}
|
|
|
|
async withReservedConnection<T>(fn: (conn: ReservedConnection) => Promise<T>): Promise<T> {
|
|
const pool = this._sql || db.getConnection();
|
|
const reserved = await pool.reserve();
|
|
try {
|
|
const conn: ReservedConnection = {
|
|
async executeRaw<R = Record<string, unknown>>(query: string, params?: unknown[]): Promise<R[]> {
|
|
const rows = params === undefined
|
|
? await reserved.unsafe(query)
|
|
: await reserved.unsafe(query, params as Parameters<typeof reserved.unsafe>[1]);
|
|
return rows as unknown as R[];
|
|
},
|
|
};
|
|
return await fn(conn);
|
|
} finally {
|
|
reserved.release();
|
|
}
|
|
}
|
|
|
|
// Pages CRUD
|
|
async getPage(slug: string): Promise<Page | null> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
|
|
FROM pages WHERE slug = ${slug}
|
|
`;
|
|
if (rows.length === 0) return null;
|
|
return rowToPage(rows[0]);
|
|
}
|
|
|
|
async putPage(slug: string, page: PageInput): Promise<Page> {
|
|
slug = validateSlug(slug);
|
|
const sql = this.sql;
|
|
const hash = page.content_hash || contentHash(page);
|
|
const frontmatter = page.frontmatter || {};
|
|
|
|
// v0.18.0 Step 2: source_id relies on schema DEFAULT 'default'. ON
|
|
// CONFLICT target becomes (source_id, slug) since global UNIQUE(slug)
|
|
// was dropped in migration v17. See pglite-engine.ts for matching
|
|
// notes; multi-source sync (Step 5) will surface an explicit sourceId.
|
|
const pageKind = page.page_kind || 'markdown';
|
|
const rows = await sql`
|
|
INSERT INTO pages (slug, type, page_kind, title, compiled_truth, timeline, frontmatter, content_hash, updated_at)
|
|
VALUES (${slug}, ${page.type}, ${pageKind}, ${page.title}, ${page.compiled_truth}, ${page.timeline || ''}, ${sql.json(frontmatter as Parameters<typeof sql.json>[0])}, ${hash}, now())
|
|
ON CONFLICT (source_id, slug) DO UPDATE SET
|
|
type = EXCLUDED.type,
|
|
page_kind = EXCLUDED.page_kind,
|
|
title = EXCLUDED.title,
|
|
compiled_truth = EXCLUDED.compiled_truth,
|
|
timeline = EXCLUDED.timeline,
|
|
frontmatter = EXCLUDED.frontmatter,
|
|
content_hash = EXCLUDED.content_hash,
|
|
updated_at = now()
|
|
RETURNING id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at
|
|
`;
|
|
return rowToPage(rows[0]);
|
|
}
|
|
|
|
async deletePage(slug: string): Promise<void> {
|
|
const sql = this.sql;
|
|
await sql`DELETE FROM pages WHERE slug = ${slug}`;
|
|
}
|
|
|
|
async listPages(filters?: PageFilters): Promise<Page[]> {
|
|
const sql = this.sql;
|
|
const limit = filters?.limit || 100;
|
|
const offset = filters?.offset || 0;
|
|
const updatedAfter = filters?.updated_after;
|
|
|
|
// postgres.js sql.unsafe is awkward for conditional WHERE; use raw query branching.
|
|
// The 4 dimensions (type, tag, updated_after, none) cross-product into 8 cases;
|
|
// we use postgres.js's tagged-template chaining via sql`` fragments instead.
|
|
|
|
// Build conditions with sql fragments. postgres.js supports fragment composition.
|
|
const typeCondition = filters?.type ? sql`AND p.type = ${filters.type}` : sql``;
|
|
const tagJoin = filters?.tag ? sql`JOIN tags t ON t.page_id = p.id` : sql``;
|
|
const tagCondition = filters?.tag ? sql`AND t.tag = ${filters.tag}` : sql``;
|
|
const updatedCondition = updatedAfter ? sql`AND p.updated_at > ${updatedAfter}::timestamptz` : sql``;
|
|
// slugPrefix uses the (source_id, slug) UNIQUE btree index for range scans.
|
|
// Escape LIKE metacharacters so the user prefix is treated as a literal.
|
|
const slugPrefix = filters?.slugPrefix;
|
|
const slugCondition = slugPrefix
|
|
? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'`
|
|
: sql``;
|
|
|
|
const rows = await sql`
|
|
SELECT p.* FROM pages p
|
|
${tagJoin}
|
|
WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition}
|
|
ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset}
|
|
`;
|
|
|
|
return rows.map(rowToPage);
|
|
}
|
|
|
|
async getAllSlugs(): Promise<Set<string>> {
|
|
const sql = this.sql;
|
|
const rows = await sql`SELECT slug FROM pages`;
|
|
return new Set(rows.map((r) => r.slug as string));
|
|
}
|
|
|
|
async resolveSlugs(partial: string): Promise<string[]> {
|
|
const sql = this.sql;
|
|
|
|
// Try exact match first
|
|
const exact = await sql`SELECT slug FROM pages WHERE slug = ${partial}`;
|
|
if (exact.length > 0) return [exact[0].slug];
|
|
|
|
// Fuzzy match via pg_trgm
|
|
const fuzzy = await sql`
|
|
SELECT slug, similarity(title, ${partial}) AS sim
|
|
FROM pages
|
|
WHERE title % ${partial} OR slug ILIKE ${'%' + partial + '%'}
|
|
ORDER BY sim DESC
|
|
LIMIT 5
|
|
`;
|
|
return fuzzy.map((r) => r.slug as string);
|
|
}
|
|
|
|
// Search
|
|
// v0.20.0 Cathedral II Layer 3 (1b): chunk-grain FTS internally,
|
|
// dedup-to-best-chunk-per-page on the way out. External shape
|
|
// preserves the v0.19.0 contract so backlinks / enrichment-service /
|
|
// list_pages etc. see zero breaking changes. A2 two-pass (Layer 7)
|
|
// consumes searchKeywordChunks for the raw chunk-grain primitive.
|
|
async searchKeyword(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
|
const sql = this.sql;
|
|
const limit = clampSearchLimit(opts?.limit);
|
|
const offset = opts?.offset || 0;
|
|
const type = opts?.type;
|
|
const excludeSlugs = opts?.exclude_slugs;
|
|
const language = opts?.language;
|
|
const symbolKind = opts?.symbolKind;
|
|
|
|
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
|
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
|
}
|
|
|
|
const detailLow = opts?.detail === 'low';
|
|
// Fetch headroom for dedup: if we only fetch `limit` chunks, a cluster of
|
|
// co-occurring terms in one page can eat the entire result set and we'd
|
|
// ship < limit pages. 3x gives dedup enough to pick top N distinct pages.
|
|
const innerLimit = Math.min(limit * 3, MAX_SEARCH_LIMIT * 3);
|
|
|
|
// Source-aware ranking (v0.22): boost curated content (originals/,
|
|
// concepts/, writing/) and dampen bulk content (chat/, daily/, media/x/)
|
|
// by multiplying the chunk-grain ts_rank with a source-factor CASE.
|
|
// Detail-gated — disabled for `detail='high'` (temporal queries) so
|
|
// chat surfaces normally for date-framed lookups. Hard-exclude prefixes
|
|
// (test/, archive/, attachments/, .raw/ by default) filter at the
|
|
// chunk-rank stage so they never enter the candidate set.
|
|
const boostMap = resolveBoostMap();
|
|
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
|
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
|
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
|
|
|
const params: unknown[] = [query];
|
|
let typeClause = '';
|
|
if (type) {
|
|
params.push(type);
|
|
typeClause = `AND p.type = $${params.length}`;
|
|
}
|
|
let excludeSlugsClause = '';
|
|
if (excludeSlugs?.length) {
|
|
params.push(excludeSlugs);
|
|
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
|
}
|
|
let languageClause = '';
|
|
if (language) {
|
|
params.push(language);
|
|
languageClause = `AND cc.language = $${params.length}`;
|
|
}
|
|
let symbolKindClause = '';
|
|
if (symbolKind) {
|
|
params.push(symbolKind);
|
|
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
|
}
|
|
params.push(innerLimit);
|
|
const innerLimitParam = `$${params.length}`;
|
|
params.push(limit);
|
|
const limitParam = `$${params.length}`;
|
|
params.push(offset);
|
|
const offsetParam = `$${params.length}`;
|
|
|
|
const rawQuery = `
|
|
WITH ranked_chunks AS (
|
|
SELECT
|
|
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
|
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
|
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
|
|
FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
|
${typeClause}
|
|
${excludeSlugsClause}
|
|
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
|
${languageClause}
|
|
${symbolKindClause}
|
|
${hardExcludeClause}
|
|
ORDER BY score DESC
|
|
LIMIT ${innerLimitParam}
|
|
),
|
|
best_per_page AS (
|
|
SELECT DISTINCT ON (slug) *
|
|
FROM ranked_chunks
|
|
ORDER BY slug, score DESC
|
|
)
|
|
SELECT slug, page_id, title, type, source_id,
|
|
chunk_id, chunk_index, chunk_text, chunk_source, score,
|
|
false AS stale
|
|
FROM best_per_page
|
|
ORDER BY score DESC
|
|
LIMIT ${limitParam}
|
|
OFFSET ${offsetParam}
|
|
`;
|
|
|
|
// Search-only timeout. SET LOCAL inside sql.begin() scopes the GUC
|
|
// to the transaction so it can never leak onto a pooled connection.
|
|
const rows = await sql.begin(async sql => {
|
|
await sql`SET LOCAL statement_timeout = '8s'`;
|
|
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
|
});
|
|
return rows.map(rowToSearchResult);
|
|
}
|
|
|
|
/**
|
|
* v0.20.0 Cathedral II Layer 3 (1b) chunk-grain keyword search.
|
|
* Ranks chunks via content_chunks.search_vector WITHOUT the
|
|
* dedup-to-page pass searchKeyword applies. Used by A2 two-pass
|
|
* retrieval (Layer 7) as the anchor-discovery primitive.
|
|
*
|
|
* Most callers should prefer searchKeyword (external page-grain
|
|
* contract). This is intentionally a narrow internal knob.
|
|
*/
|
|
async searchKeywordChunks(query: string, opts?: SearchOpts): Promise<SearchResult[]> {
|
|
const sql = this.sql;
|
|
const limit = clampSearchLimit(opts?.limit);
|
|
const offset = opts?.offset || 0;
|
|
const type = opts?.type;
|
|
const excludeSlugs = opts?.exclude_slugs;
|
|
const detailLow = opts?.detail === 'low';
|
|
const language = opts?.language;
|
|
const symbolKind = opts?.symbolKind;
|
|
|
|
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
|
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
|
}
|
|
|
|
// Source-aware ranking applies here too — searchKeywordChunks is the
|
|
// chunk-grain anchor primitive that two-pass retrieval (Layer 7) uses,
|
|
// so curated-vs-bulk dampening should affect the anchor pool. Same
|
|
// detail-gate, same hard-exclude behavior as searchKeyword.
|
|
const boostMap = resolveBoostMap();
|
|
const sourceFactorCase = buildSourceFactorCase('p.slug', boostMap, opts?.detail);
|
|
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
|
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
|
|
|
const params: unknown[] = [query];
|
|
let typeClause = '';
|
|
if (type) {
|
|
params.push(type);
|
|
typeClause = `AND p.type = $${params.length}`;
|
|
}
|
|
let excludeSlugsClause = '';
|
|
if (excludeSlugs?.length) {
|
|
params.push(excludeSlugs);
|
|
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
|
}
|
|
let languageClause = '';
|
|
if (language) {
|
|
params.push(language);
|
|
languageClause = `AND cc.language = $${params.length}`;
|
|
}
|
|
let symbolKindClause = '';
|
|
if (symbolKind) {
|
|
params.push(symbolKind);
|
|
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
|
}
|
|
params.push(limit);
|
|
const limitParam = `$${params.length}`;
|
|
params.push(offset);
|
|
const offsetParam = `$${params.length}`;
|
|
|
|
const rawQuery = `
|
|
SELECT
|
|
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
|
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
|
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
|
|
false AS stale
|
|
FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE cc.search_vector @@ websearch_to_tsquery('english', $1)
|
|
${typeClause}
|
|
${excludeSlugsClause}
|
|
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
|
${languageClause}
|
|
${symbolKindClause}
|
|
${hardExcludeClause}
|
|
ORDER BY score DESC
|
|
LIMIT ${limitParam}
|
|
OFFSET ${offsetParam}
|
|
`;
|
|
|
|
const rows = await sql.begin(async sql => {
|
|
await sql`SET LOCAL statement_timeout = '8s'`;
|
|
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
|
});
|
|
return rows.map(rowToSearchResult);
|
|
}
|
|
|
|
async searchVector(embedding: Float32Array, opts?: SearchOpts): Promise<SearchResult[]> {
|
|
const sql = this.sql;
|
|
const limit = clampSearchLimit(opts?.limit);
|
|
const offset = opts?.offset || 0;
|
|
const type = opts?.type;
|
|
const excludeSlugs = opts?.exclude_slugs;
|
|
const detailLow = opts?.detail === 'low';
|
|
const language = opts?.language;
|
|
const symbolKind = opts?.symbolKind;
|
|
|
|
if (opts?.limit && opts.limit > MAX_SEARCH_LIMIT) {
|
|
console.warn(`[gbrain] Warning: search limit clamped from ${opts.limit} to ${MAX_SEARCH_LIMIT}`);
|
|
}
|
|
|
|
const vecStr = '[' + Array.from(embedding).join(',') + ']';
|
|
|
|
// Two-stage CTE (v0.22): inner CTE keeps a pure-distance ORDER BY so
|
|
// the HNSW index stays usable. Folding source-boost into the inner
|
|
// ORDER BY would force a sequential scan over every chunk (seconds vs
|
|
// ~10ms with HNSW). Outer SELECT re-ranks the candidate pool by
|
|
// raw_score * source_factor.
|
|
//
|
|
// innerLimit scales with offset to preserve the pagination contract:
|
|
// a fixed cap of 100 would silently empty offset > 100.
|
|
const boostMap = resolveBoostMap();
|
|
const sourceFactorCaseOnSlug = buildSourceFactorCase('slug', boostMap, opts?.detail);
|
|
const hardExcludePrefixes = resolveHardExcludes(opts?.exclude_slug_prefixes, opts?.include_slug_prefixes);
|
|
const hardExcludeClause = buildHardExcludeClause('p.slug', hardExcludePrefixes);
|
|
const innerLimit = offset + Math.max(limit * 5, 100);
|
|
|
|
const params: unknown[] = [vecStr];
|
|
let typeClause = '';
|
|
if (type) {
|
|
params.push(type);
|
|
typeClause = `AND p.type = $${params.length}`;
|
|
}
|
|
let excludeSlugsClause = '';
|
|
if (excludeSlugs?.length) {
|
|
params.push(excludeSlugs);
|
|
excludeSlugsClause = `AND p.slug != ALL($${params.length}::text[])`;
|
|
}
|
|
let languageClause = '';
|
|
if (language) {
|
|
params.push(language);
|
|
languageClause = `AND cc.language = $${params.length}`;
|
|
}
|
|
let symbolKindClause = '';
|
|
if (symbolKind) {
|
|
params.push(symbolKind);
|
|
symbolKindClause = `AND cc.symbol_type = $${params.length}`;
|
|
}
|
|
params.push(innerLimit);
|
|
const innerLimitParam = `$${params.length}`;
|
|
params.push(limit);
|
|
const limitParam = `$${params.length}`;
|
|
params.push(offset);
|
|
const offsetParam = `$${params.length}`;
|
|
|
|
const rawQuery = `
|
|
WITH hnsw_candidates AS (
|
|
SELECT
|
|
p.slug, p.id as page_id, p.title, p.type, p.source_id,
|
|
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
|
1 - (cc.embedding <=> $1::vector) AS raw_score
|
|
FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE cc.embedding IS NOT NULL
|
|
${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''}
|
|
${typeClause}
|
|
${excludeSlugsClause}
|
|
${languageClause}
|
|
${symbolKindClause}
|
|
${hardExcludeClause}
|
|
ORDER BY cc.embedding <=> $1::vector
|
|
LIMIT ${innerLimitParam}
|
|
)
|
|
SELECT
|
|
slug, page_id, title, type, source_id,
|
|
chunk_id, chunk_index, chunk_text, chunk_source,
|
|
raw_score * ${sourceFactorCaseOnSlug} AS score,
|
|
false AS stale
|
|
FROM hnsw_candidates
|
|
ORDER BY score DESC
|
|
LIMIT ${limitParam}
|
|
OFFSET ${offsetParam}
|
|
`;
|
|
|
|
const rows = await sql.begin(async sql => {
|
|
await sql`SET LOCAL statement_timeout = '8s'`;
|
|
return await sql.unsafe(rawQuery, params as Parameters<typeof sql.unsafe>[1]);
|
|
});
|
|
return rows.map(rowToSearchResult);
|
|
}
|
|
|
|
async getEmbeddingsByChunkIds(ids: number[]): Promise<Map<number, Float32Array>> {
|
|
if (ids.length === 0) return new Map();
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT id, embedding FROM content_chunks
|
|
WHERE id = ANY(${ids}::int[]) AND embedding IS NOT NULL
|
|
`;
|
|
const result = new Map<number, Float32Array>();
|
|
for (const row of rows) {
|
|
const embedding = tryParseEmbedding(row.embedding);
|
|
if (embedding) result.set(row.id as number, embedding);
|
|
}
|
|
return result;
|
|
}
|
|
|
|
// Chunks
|
|
async upsertChunks(slug: string, chunks: ChunkInput[]): Promise<void> {
|
|
const sql = this.sql;
|
|
|
|
// Get page_id
|
|
const pages = await sql`SELECT id FROM pages WHERE slug = ${slug}`;
|
|
if (pages.length === 0) throw new Error(`Page not found: ${slug}`);
|
|
const pageId = pages[0].id;
|
|
|
|
// Remove chunks that no longer exist (chunk_index beyond new count)
|
|
const newIndices = chunks.map(c => c.chunk_index);
|
|
if (newIndices.length > 0) {
|
|
await sql`DELETE FROM content_chunks WHERE page_id = ${pageId} AND chunk_index != ALL(${newIndices})`;
|
|
} else {
|
|
await sql`DELETE FROM content_chunks WHERE page_id = ${pageId}`;
|
|
return;
|
|
}
|
|
|
|
// Batch upsert: build a single multi-row INSERT ON CONFLICT statement.
|
|
// v0.19.0: includes language/symbol_name/symbol_type/start_line/end_line
|
|
// so code chunks carry tree-sitter metadata into the DB. Markdown chunks
|
|
// pass NULL for all five.
|
|
// v0.20.0 Cathedral II Layer 6: adds parent_symbol_path / doc_comment /
|
|
// symbol_name_qualified so nested-chunk emission (A3) can round-trip
|
|
// scope metadata through upserts.
|
|
const cols = '(page_id, chunk_index, chunk_text, chunk_source, embedding, model, token_count, embedded_at, language, symbol_name, symbol_type, start_line, end_line, parent_symbol_path, doc_comment, symbol_name_qualified)';
|
|
const rows: string[] = [];
|
|
const params: unknown[] = [];
|
|
let paramIdx = 1;
|
|
|
|
for (const chunk of chunks) {
|
|
const embeddingStr = chunk.embedding
|
|
? '[' + Array.from(chunk.embedding).join(',') + ']'
|
|
: null;
|
|
const parentPath = chunk.parent_symbol_path && chunk.parent_symbol_path.length > 0
|
|
? chunk.parent_symbol_path
|
|
: null;
|
|
|
|
if (embeddingStr) {
|
|
rows.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::vector, $${paramIdx++}, $${paramIdx++}, now(), $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::text[], $${paramIdx++}, $${paramIdx++})`);
|
|
params.push(
|
|
pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source,
|
|
embeddingStr, chunk.model || 'text-embedding-3-large', chunk.token_count || null,
|
|
chunk.language || null, chunk.symbol_name || null, chunk.symbol_type || null,
|
|
chunk.start_line ?? null, chunk.end_line ?? null,
|
|
parentPath, chunk.doc_comment || null, chunk.symbol_name_qualified || null,
|
|
);
|
|
} else {
|
|
rows.push(`($${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, NULL, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}, $${paramIdx++}::text[], $${paramIdx++}, $${paramIdx++})`);
|
|
params.push(
|
|
pageId, chunk.chunk_index, chunk.chunk_text, chunk.chunk_source,
|
|
chunk.model || 'text-embedding-3-large', chunk.token_count || null,
|
|
chunk.language || null, chunk.symbol_name || null, chunk.symbol_type || null,
|
|
chunk.start_line ?? null, chunk.end_line ?? null,
|
|
parentPath, chunk.doc_comment || null, chunk.symbol_name_qualified || null,
|
|
);
|
|
}
|
|
}
|
|
|
|
// Single statement upsert: preserves existing embeddings via COALESCE when new value is NULL.
|
|
// CONSISTENCY: when chunk_text changes and no new embedding is supplied, BOTH embedding AND
|
|
// embedded_at must reset to NULL so `embed --stale` correctly picks up the row for re-embedding.
|
|
// Without this, embedded_at lies (says "embedded" while embedding=NULL), and any staleness
|
|
// predicate on embedded_at would silently skip the row. This is why the egress fix predicates
|
|
// on `embedding IS NULL` rather than `embedded_at IS NULL` — and it's why we now keep both
|
|
// columns honest at write time.
|
|
await sql.unsafe(
|
|
`INSERT INTO content_chunks ${cols} VALUES ${rows.join(', ')}
|
|
ON CONFLICT (page_id, chunk_index) DO UPDATE SET
|
|
chunk_text = EXCLUDED.chunk_text,
|
|
chunk_source = EXCLUDED.chunk_source,
|
|
embedding = CASE WHEN EXCLUDED.chunk_text != content_chunks.chunk_text THEN EXCLUDED.embedding ELSE COALESCE(EXCLUDED.embedding, content_chunks.embedding) END,
|
|
model = COALESCE(EXCLUDED.model, content_chunks.model),
|
|
token_count = EXCLUDED.token_count,
|
|
embedded_at = CASE
|
|
WHEN EXCLUDED.chunk_text != content_chunks.chunk_text AND EXCLUDED.embedding IS NULL THEN NULL
|
|
ELSE COALESCE(EXCLUDED.embedded_at, content_chunks.embedded_at)
|
|
END,
|
|
language = EXCLUDED.language,
|
|
symbol_name = EXCLUDED.symbol_name,
|
|
symbol_type = EXCLUDED.symbol_type,
|
|
start_line = EXCLUDED.start_line,
|
|
end_line = EXCLUDED.end_line,
|
|
parent_symbol_path = EXCLUDED.parent_symbol_path,
|
|
doc_comment = EXCLUDED.doc_comment,
|
|
symbol_name_qualified = EXCLUDED.symbol_name_qualified`,
|
|
params as Parameters<typeof sql.unsafe>[1],
|
|
);
|
|
}
|
|
|
|
async getChunks(slug: string): Promise<Chunk[]> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT cc.* FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE p.slug = ${slug}
|
|
ORDER BY cc.chunk_index
|
|
`;
|
|
return rows.map((r) => rowToChunk(r as Record<string, unknown>));
|
|
}
|
|
|
|
async countStaleChunks(): Promise<number> {
|
|
const sql = this.sql;
|
|
const [row] = await sql`
|
|
SELECT count(*)::int AS count
|
|
FROM content_chunks
|
|
WHERE embedding IS NULL
|
|
`;
|
|
return Number((row as { count?: number } | undefined)?.count ?? 0);
|
|
}
|
|
|
|
async listStaleChunks(): Promise<StaleChunkRow[]> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT p.slug, cc.chunk_index, cc.chunk_text, cc.chunk_source,
|
|
cc.model, cc.token_count
|
|
FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE cc.embedding IS NULL
|
|
ORDER BY p.id, cc.chunk_index
|
|
LIMIT 100000
|
|
`;
|
|
return rows as unknown as StaleChunkRow[];
|
|
}
|
|
|
|
async deleteChunks(slug: string): Promise<void> {
|
|
const sql = this.sql;
|
|
await sql`
|
|
DELETE FROM content_chunks
|
|
WHERE page_id = (SELECT id FROM pages WHERE slug = ${slug})
|
|
`;
|
|
}
|
|
|
|
// Links
|
|
async addLink(
|
|
from: string,
|
|
to: string,
|
|
context?: string,
|
|
linkType?: string,
|
|
linkSource?: string,
|
|
originSlug?: string,
|
|
originField?: string,
|
|
): Promise<void> {
|
|
const sql = this.sql;
|
|
// Pre-check existence so we can throw a clear error (ON CONFLICT DO UPDATE
|
|
// returns 0 rows when source SELECT is empty, indistinguishable from missing page).
|
|
const exists = await sql`
|
|
SELECT 1 FROM pages WHERE slug = ${from}
|
|
INTERSECT
|
|
SELECT 1 FROM pages WHERE slug = ${to}
|
|
`;
|
|
if (exists.length === 0) {
|
|
throw new Error(`addLink failed: page "${from}" or "${to}" not found`);
|
|
}
|
|
// Default link_source to 'markdown' for back-compat with pre-v0.13 callers.
|
|
// origin_page_id resolves from originSlug via the pages join (NULL if no slug).
|
|
const src = linkSource ?? 'markdown';
|
|
await sql`
|
|
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
|
|
SELECT f.id, t.id, ${linkType || ''}, ${context || ''}, ${src},
|
|
(SELECT id FROM pages WHERE slug = ${originSlug ?? null}),
|
|
${originField ?? null}
|
|
FROM pages f, pages t
|
|
WHERE f.slug = ${from} AND t.slug = ${to}
|
|
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO UPDATE SET
|
|
context = EXCLUDED.context,
|
|
origin_field = EXCLUDED.origin_field
|
|
`;
|
|
}
|
|
|
|
async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
|
|
if (links.length === 0) return 0;
|
|
const sql = this.sql;
|
|
// unnest() pattern: 7 array-typed bound parameters regardless of batch size.
|
|
// Avoids the 65535-parameter cap and the postgres-js sql(rows, ...) helper's
|
|
// identifier-escape gotcha when used inside a (VALUES) subquery.
|
|
//
|
|
// v0.13: added link_source, origin_slug, origin_field. Defaults:
|
|
// link_source → 'markdown' (back-compat with pre-v0.13 callers)
|
|
// origin_slug → NULL (resolves to origin_page_id IS NULL via LEFT JOIN)
|
|
// origin_field → NULL
|
|
const fromSlugs = links.map(l => l.from_slug);
|
|
const toSlugs = links.map(l => l.to_slug);
|
|
const linkTypes = links.map(l => l.link_type || '');
|
|
const contexts = links.map(l => l.context || '');
|
|
const linkSources = links.map(l => l.link_source || 'markdown');
|
|
const originSlugs = links.map(l => l.origin_slug || null);
|
|
const originFields = links.map(l => l.origin_field || null);
|
|
const fromSourceIds = links.map(l => l.from_source_id || 'default');
|
|
const toSourceIds = links.map(l => l.to_source_id || 'default');
|
|
const originSourceIds = links.map(l => l.origin_source_id || 'default');
|
|
const result = await sql`
|
|
INSERT INTO links (from_page_id, to_page_id, link_type, context, link_source, origin_page_id, origin_field)
|
|
SELECT f.id, t.id, v.link_type, v.context, v.link_source, o.id, v.origin_field
|
|
FROM unnest(
|
|
${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[],
|
|
${contexts}::text[], ${linkSources}::text[], ${originSlugs}::text[],
|
|
${originFields}::text[], ${fromSourceIds}::text[], ${toSourceIds}::text[],
|
|
${originSourceIds}::text[]
|
|
) AS v(from_slug, to_slug, link_type, context, link_source, origin_slug, origin_field, from_source_id, to_source_id, origin_source_id)
|
|
JOIN pages f ON f.slug = v.from_slug AND f.source_id = v.from_source_id
|
|
JOIN pages t ON t.slug = v.to_slug AND t.source_id = v.to_source_id
|
|
LEFT JOIN pages o ON o.slug = v.origin_slug AND o.source_id = v.origin_source_id
|
|
ON CONFLICT (from_page_id, to_page_id, link_type, link_source, origin_page_id) DO NOTHING
|
|
RETURNING 1
|
|
`;
|
|
return result.length;
|
|
}
|
|
|
|
async removeLink(from: string, to: string, linkType?: string, linkSource?: string): Promise<void> {
|
|
const sql = this.sql;
|
|
// Build up filters dynamically. linkType + linkSource are independent
|
|
// optional constraints; all four combinations are valid.
|
|
if (linkType !== undefined && linkSource !== undefined) {
|
|
await sql`
|
|
DELETE FROM links
|
|
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
|
|
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
|
|
AND link_type = ${linkType}
|
|
AND link_source IS NOT DISTINCT FROM ${linkSource}
|
|
`;
|
|
} else if (linkType !== undefined) {
|
|
await sql`
|
|
DELETE FROM links
|
|
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
|
|
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
|
|
AND link_type = ${linkType}
|
|
`;
|
|
} else if (linkSource !== undefined) {
|
|
await sql`
|
|
DELETE FROM links
|
|
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
|
|
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
|
|
AND link_source IS NOT DISTINCT FROM ${linkSource}
|
|
`;
|
|
} else {
|
|
await sql`
|
|
DELETE FROM links
|
|
WHERE from_page_id = (SELECT id FROM pages WHERE slug = ${from})
|
|
AND to_page_id = (SELECT id FROM pages WHERE slug = ${to})
|
|
`;
|
|
}
|
|
}
|
|
|
|
async getLinks(slug: string): Promise<Link[]> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT f.slug as from_slug, t.slug as to_slug,
|
|
l.link_type, l.context, l.link_source,
|
|
o.slug as origin_slug, l.origin_field
|
|
FROM links l
|
|
JOIN pages f ON f.id = l.from_page_id
|
|
JOIN pages t ON t.id = l.to_page_id
|
|
LEFT JOIN pages o ON o.id = l.origin_page_id
|
|
WHERE f.slug = ${slug}
|
|
`;
|
|
return rows as unknown as Link[];
|
|
}
|
|
|
|
async getBacklinks(slug: string): Promise<Link[]> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT f.slug as from_slug, t.slug as to_slug,
|
|
l.link_type, l.context, l.link_source,
|
|
o.slug as origin_slug, l.origin_field
|
|
FROM links l
|
|
JOIN pages f ON f.id = l.from_page_id
|
|
JOIN pages t ON t.id = l.to_page_id
|
|
LEFT JOIN pages o ON o.id = l.origin_page_id
|
|
WHERE t.slug = ${slug}
|
|
`;
|
|
return rows as unknown as Link[];
|
|
}
|
|
|
|
async findByTitleFuzzy(
|
|
name: string,
|
|
dirPrefix?: string,
|
|
minSimilarity: number = 0.55,
|
|
): Promise<{ slug: string; similarity: number } | null> {
|
|
const sql = this.sql;
|
|
// Use the `similarity()` function directly with an explicit threshold
|
|
// comparison. DO NOT use `SET LOCAL pg_trgm.similarity_threshold` +
|
|
// the `%` operator here — postgres.js auto-commits each sql`` call
|
|
// so `SET LOCAL` is a no-op across statement boundaries. Inline
|
|
// comparison is the only way to get predictable threshold behavior
|
|
// without wrapping the caller in a transaction.
|
|
//
|
|
// Tie-breaker: sort by slug after similarity so re-runs return the
|
|
// same winner when multiple pages score equally (prevents churn
|
|
// in put_page auto-link reconciliation).
|
|
const prefixPattern = dirPrefix ? `${dirPrefix}/%` : '%';
|
|
const rows = await sql`
|
|
SELECT slug, similarity(title, ${name}) AS sim
|
|
FROM pages
|
|
WHERE similarity(title, ${name}) >= ${minSimilarity}
|
|
AND slug LIKE ${prefixPattern}
|
|
ORDER BY sim DESC, slug ASC
|
|
LIMIT 1
|
|
`;
|
|
if (rows.length === 0) return null;
|
|
const row = rows[0] as { slug: string; sim: number };
|
|
return { slug: row.slug, similarity: row.sim };
|
|
}
|
|
|
|
async traverseGraph(slug: string, depth: number = 5): Promise<GraphNode[]> {
|
|
const sql = this.sql;
|
|
// Cycle prevention: visited array tracks page IDs already in the path.
|
|
const rows = await sql`
|
|
WITH RECURSIVE graph AS (
|
|
SELECT p.id, p.slug, p.title, p.type, 0 as depth, ARRAY[p.id] as visited
|
|
FROM pages p WHERE p.slug = ${slug}
|
|
|
|
UNION ALL
|
|
|
|
SELECT p2.id, p2.slug, p2.title, p2.type, g.depth + 1, g.visited || p2.id
|
|
FROM graph g
|
|
JOIN links l ON l.from_page_id = g.id
|
|
JOIN pages p2 ON p2.id = l.to_page_id
|
|
WHERE g.depth < ${depth}
|
|
AND NOT (p2.id = ANY(g.visited))
|
|
)
|
|
SELECT DISTINCT g.slug, g.title, g.type, g.depth,
|
|
coalesce(
|
|
-- jsonb_agg(DISTINCT ...) collapses duplicate (to_slug, link_type)
|
|
-- edges that originate from different provenance (markdown body
|
|
-- vs frontmatter vs auto-extracted). The underlying links table
|
|
-- preserves every row with its origin_page_id / link_source —
|
|
-- the dedup is presentation-only for the legacy traverseGraph
|
|
-- aggregation. traversePaths has its own in-memory dedup at a
|
|
-- different layer. See plan Bug 6/10.
|
|
(SELECT jsonb_agg(DISTINCT jsonb_build_object('to_slug', p3.slug, 'link_type', l2.link_type))
|
|
FROM links l2
|
|
JOIN pages p3 ON p3.id = l2.to_page_id
|
|
WHERE l2.from_page_id = g.id),
|
|
'[]'::jsonb
|
|
) as links
|
|
FROM graph g
|
|
ORDER BY g.depth, g.slug
|
|
`;
|
|
|
|
return rows.map((r: Record<string, unknown>) => ({
|
|
slug: r.slug as string,
|
|
title: r.title as string,
|
|
type: r.type as PageType,
|
|
depth: r.depth as number,
|
|
links: (typeof r.links === 'string' ? JSON.parse(r.links) : r.links) as { to_slug: string; link_type: string }[],
|
|
}));
|
|
}
|
|
|
|
async traversePaths(
|
|
slug: string,
|
|
opts?: { depth?: number; linkType?: string; direction?: 'in' | 'out' | 'both' },
|
|
): Promise<GraphPath[]> {
|
|
const sql = this.sql;
|
|
const depth = opts?.depth ?? 5;
|
|
const direction = opts?.direction ?? 'out';
|
|
const linkType = opts?.linkType ?? null;
|
|
const linkTypeMatches = linkType !== null;
|
|
|
|
let rows;
|
|
if (direction === 'out') {
|
|
rows = await sql`
|
|
WITH RECURSIVE walk AS (
|
|
SELECT p.id, p.slug, 0::int as depth, ARRAY[p.id] as visited
|
|
FROM pages p WHERE p.slug = ${slug}
|
|
UNION ALL
|
|
SELECT p2.id, p2.slug, w.depth + 1, w.visited || p2.id
|
|
FROM walk w
|
|
JOIN links l ON l.from_page_id = w.id
|
|
JOIN pages p2 ON p2.id = l.to_page_id
|
|
WHERE w.depth < ${depth}
|
|
AND NOT (p2.id = ANY(w.visited))
|
|
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
|
|
)
|
|
SELECT w.slug as from_slug, p2.slug as to_slug,
|
|
l.link_type, l.context, w.depth + 1 as depth
|
|
FROM walk w
|
|
JOIN links l ON l.from_page_id = w.id
|
|
JOIN pages p2 ON p2.id = l.to_page_id
|
|
WHERE w.depth < ${depth}
|
|
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
|
|
ORDER BY depth, from_slug, to_slug
|
|
`;
|
|
} else if (direction === 'in') {
|
|
rows = await sql`
|
|
WITH RECURSIVE walk AS (
|
|
SELECT p.id, p.slug, 0::int as depth, ARRAY[p.id] as visited
|
|
FROM pages p WHERE p.slug = ${slug}
|
|
UNION ALL
|
|
SELECT p2.id, p2.slug, w.depth + 1, w.visited || p2.id
|
|
FROM walk w
|
|
JOIN links l ON l.to_page_id = w.id
|
|
JOIN pages p2 ON p2.id = l.from_page_id
|
|
WHERE w.depth < ${depth}
|
|
AND NOT (p2.id = ANY(w.visited))
|
|
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
|
|
)
|
|
SELECT p2.slug as from_slug, w.slug as to_slug,
|
|
l.link_type, l.context, w.depth + 1 as depth
|
|
FROM walk w
|
|
JOIN links l ON l.to_page_id = w.id
|
|
JOIN pages p2 ON p2.id = l.from_page_id
|
|
WHERE w.depth < ${depth}
|
|
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
|
|
ORDER BY depth, from_slug, to_slug
|
|
`;
|
|
} else {
|
|
rows = await sql`
|
|
WITH RECURSIVE walk AS (
|
|
SELECT p.id, 0::int as depth, ARRAY[p.id] as visited
|
|
FROM pages p WHERE p.slug = ${slug}
|
|
UNION ALL
|
|
SELECT p2.id, w.depth + 1, w.visited || p2.id
|
|
FROM walk w
|
|
JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id)
|
|
JOIN pages p2 ON p2.id = CASE WHEN l.from_page_id = w.id THEN l.to_page_id ELSE l.from_page_id END
|
|
WHERE w.depth < ${depth}
|
|
AND NOT (p2.id = ANY(w.visited))
|
|
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
|
|
)
|
|
SELECT pf.slug as from_slug, pt.slug as to_slug,
|
|
l.link_type, l.context, w.depth + 1 as depth
|
|
FROM walk w
|
|
JOIN links l ON (l.from_page_id = w.id OR l.to_page_id = w.id)
|
|
JOIN pages pf ON pf.id = l.from_page_id
|
|
JOIN pages pt ON pt.id = l.to_page_id
|
|
WHERE w.depth < ${depth}
|
|
AND (${!linkTypeMatches} OR l.link_type = ${linkType ?? ''})
|
|
ORDER BY depth, from_slug, to_slug
|
|
`;
|
|
}
|
|
|
|
// Dedup edges (same edge can appear via multiple visited paths).
|
|
const seen = new Set<string>();
|
|
const result: GraphPath[] = [];
|
|
for (const r of rows as Record<string, unknown>[]) {
|
|
const key = `${r.from_slug}|${r.to_slug}|${r.link_type}|${r.depth}`;
|
|
if (seen.has(key)) continue;
|
|
seen.add(key);
|
|
result.push({
|
|
from_slug: r.from_slug as string,
|
|
to_slug: r.to_slug as string,
|
|
link_type: r.link_type as string,
|
|
context: (r.context as string) || '',
|
|
depth: Number(r.depth),
|
|
});
|
|
}
|
|
return result;
|
|
}
|
|
|
|
async getBacklinkCounts(slugs: string[]): Promise<Map<string, number>> {
|
|
const result = new Map<string, number>();
|
|
if (slugs.length === 0) return result;
|
|
for (const s of slugs) result.set(s, 0);
|
|
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT p.slug as slug, COUNT(l.id)::int as cnt
|
|
FROM pages p
|
|
LEFT JOIN links l ON l.to_page_id = p.id
|
|
WHERE p.slug = ANY(${slugs}::text[])
|
|
GROUP BY p.slug
|
|
`;
|
|
for (const r of rows as unknown as { slug: string; cnt: number }[]) {
|
|
result.set(r.slug, Number(r.cnt));
|
|
}
|
|
return result;
|
|
}
|
|
|
|
async findOrphanPages(): Promise<Array<{ slug: string; title: string; domain: string | null }>> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT
|
|
p.slug,
|
|
COALESCE(p.title, p.slug) AS title,
|
|
p.frontmatter->>'domain' AS domain
|
|
FROM pages p
|
|
WHERE NOT EXISTS (
|
|
SELECT 1 FROM links l WHERE l.to_page_id = p.id
|
|
)
|
|
ORDER BY p.slug
|
|
`;
|
|
return rows as unknown as Array<{ slug: string; title: string; domain: string | null }>;
|
|
}
|
|
|
|
// Tags
|
|
async addTag(slug: string, tag: string): Promise<void> {
|
|
const sql = this.sql;
|
|
// Verify page exists before attempting insert (ON CONFLICT DO NOTHING
|
|
// swallows the "already tagged" case, but we still need to detect missing pages)
|
|
const page = await sql`SELECT id FROM pages WHERE slug = ${slug}`;
|
|
if (page.length === 0) throw new Error(`addTag failed: page "${slug}" not found`);
|
|
await sql`
|
|
INSERT INTO tags (page_id, tag)
|
|
VALUES (${page[0].id}, ${tag})
|
|
ON CONFLICT (page_id, tag) DO NOTHING
|
|
`;
|
|
}
|
|
|
|
async removeTag(slug: string, tag: string): Promise<void> {
|
|
const sql = this.sql;
|
|
await sql`
|
|
DELETE FROM tags
|
|
WHERE page_id = (SELECT id FROM pages WHERE slug = ${slug})
|
|
AND tag = ${tag}
|
|
`;
|
|
}
|
|
|
|
async getTags(slug: string): Promise<string[]> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT tag FROM tags
|
|
WHERE page_id = (SELECT id FROM pages WHERE slug = ${slug})
|
|
ORDER BY tag
|
|
`;
|
|
return rows.map((r) => r.tag as string);
|
|
}
|
|
|
|
// Timeline
|
|
async addTimelineEntry(
|
|
slug: string,
|
|
entry: TimelineInput,
|
|
opts?: { skipExistenceCheck?: boolean },
|
|
): Promise<void> {
|
|
const sql = this.sql;
|
|
if (!opts?.skipExistenceCheck) {
|
|
const exists = await sql`SELECT 1 FROM pages WHERE slug = ${slug}`;
|
|
if (exists.length === 0) {
|
|
throw new Error(`addTimelineEntry failed: page "${slug}" not found`);
|
|
}
|
|
}
|
|
// ON CONFLICT DO NOTHING via the (page_id, date, summary) unique index.
|
|
// Returning 0 rows means either page missing OR duplicate; skipExistenceCheck
|
|
// makes that ambiguity safe (caller asserts page exists).
|
|
await sql`
|
|
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
|
SELECT id, ${entry.date}::date, ${entry.source || ''}, ${entry.summary}, ${entry.detail || ''}
|
|
FROM pages WHERE slug = ${slug}
|
|
ON CONFLICT (page_id, date, summary) DO NOTHING
|
|
`;
|
|
}
|
|
|
|
async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number> {
|
|
if (entries.length === 0) return 0;
|
|
const sql = this.sql;
|
|
const slugs = entries.map(e => e.slug);
|
|
const dates = entries.map(e => e.date);
|
|
const sources = entries.map(e => e.source || '');
|
|
const summaries = entries.map(e => e.summary);
|
|
const details = entries.map(e => e.detail || '');
|
|
const sourceIds = entries.map(e => e.source_id || 'default');
|
|
const result = await sql`
|
|
INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
|
SELECT p.id, v.date::date, v.source, v.summary, v.detail
|
|
FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[], ${sourceIds}::text[])
|
|
AS v(slug, date, source, summary, detail, source_id)
|
|
JOIN pages p ON p.slug = v.slug AND p.source_id = v.source_id
|
|
ON CONFLICT (page_id, date, summary) DO NOTHING
|
|
RETURNING 1
|
|
`;
|
|
return result.length;
|
|
}
|
|
|
|
async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
|
|
const sql = this.sql;
|
|
const limit = opts?.limit || 100;
|
|
|
|
let rows;
|
|
if (opts?.after && opts?.before) {
|
|
rows = await sql`
|
|
SELECT te.* FROM timeline_entries te
|
|
JOIN pages p ON p.id = te.page_id
|
|
WHERE p.slug = ${slug} AND te.date >= ${opts.after}::date AND te.date <= ${opts.before}::date
|
|
ORDER BY te.date DESC LIMIT ${limit}
|
|
`;
|
|
} else if (opts?.after) {
|
|
rows = await sql`
|
|
SELECT te.* FROM timeline_entries te
|
|
JOIN pages p ON p.id = te.page_id
|
|
WHERE p.slug = ${slug} AND te.date >= ${opts.after}::date
|
|
ORDER BY te.date DESC LIMIT ${limit}
|
|
`;
|
|
} else {
|
|
rows = await sql`
|
|
SELECT te.* FROM timeline_entries te
|
|
JOIN pages p ON p.id = te.page_id
|
|
WHERE p.slug = ${slug}
|
|
ORDER BY te.date DESC LIMIT ${limit}
|
|
`;
|
|
}
|
|
|
|
return rows as unknown as TimelineEntry[];
|
|
}
|
|
|
|
// Raw data
|
|
async putRawData(slug: string, source: string, data: object): Promise<void> {
|
|
const sql = this.sql;
|
|
const result = await sql`
|
|
INSERT INTO raw_data (page_id, source, data)
|
|
SELECT id, ${source}, ${sql.json(data as Parameters<typeof sql.json>[0])}
|
|
FROM pages WHERE slug = ${slug}
|
|
ON CONFLICT (page_id, source) DO UPDATE SET
|
|
data = EXCLUDED.data,
|
|
fetched_at = now()
|
|
RETURNING id
|
|
`;
|
|
if (result.length === 0) throw new Error(`putRawData failed: page "${slug}" not found`);
|
|
}
|
|
|
|
async getRawData(slug: string, source?: string): Promise<RawData[]> {
|
|
const sql = this.sql;
|
|
let rows;
|
|
if (source) {
|
|
rows = await sql`
|
|
SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
|
JOIN pages p ON p.id = rd.page_id
|
|
WHERE p.slug = ${slug} AND rd.source = ${source}
|
|
`;
|
|
} else {
|
|
rows = await sql`
|
|
SELECT rd.source, rd.data, rd.fetched_at FROM raw_data rd
|
|
JOIN pages p ON p.id = rd.page_id
|
|
WHERE p.slug = ${slug}
|
|
`;
|
|
}
|
|
return rows as unknown as RawData[];
|
|
}
|
|
|
|
// Versions
|
|
async createVersion(slug: string): Promise<PageVersion> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
INSERT INTO page_versions (page_id, compiled_truth, frontmatter)
|
|
SELECT id, compiled_truth, frontmatter
|
|
FROM pages WHERE slug = ${slug}
|
|
RETURNING *
|
|
`;
|
|
if (rows.length === 0) throw new Error(`createVersion failed: page "${slug}" not found`);
|
|
return rows[0] as unknown as PageVersion;
|
|
}
|
|
|
|
async getVersions(slug: string): Promise<PageVersion[]> {
|
|
const sql = this.sql;
|
|
const rows = await sql`
|
|
SELECT pv.* FROM page_versions pv
|
|
JOIN pages p ON p.id = pv.page_id
|
|
WHERE p.slug = ${slug}
|
|
ORDER BY pv.snapshot_at DESC
|
|
`;
|
|
return rows as unknown as PageVersion[];
|
|
}
|
|
|
|
async revertToVersion(slug: string, versionId: number): Promise<void> {
|
|
const sql = this.sql;
|
|
await sql`
|
|
UPDATE pages SET
|
|
compiled_truth = pv.compiled_truth,
|
|
frontmatter = pv.frontmatter,
|
|
updated_at = now()
|
|
FROM page_versions pv
|
|
WHERE pages.slug = ${slug} AND pv.id = ${versionId} AND pv.page_id = pages.id
|
|
`;
|
|
}
|
|
|
|
// Stats + health
|
|
async getStats(): Promise<BrainStats> {
|
|
const sql = this.sql;
|
|
const [stats] = await sql`
|
|
SELECT
|
|
(SELECT count(*) FROM pages) as page_count,
|
|
(SELECT count(*) FROM content_chunks) as chunk_count,
|
|
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count,
|
|
(SELECT count(*) FROM links) as link_count,
|
|
(SELECT count(DISTINCT tag) FROM tags) as tag_count,
|
|
(SELECT count(*) FROM timeline_entries) as timeline_entry_count
|
|
`;
|
|
|
|
const types = await sql`
|
|
SELECT type, count(*)::int as count FROM pages GROUP BY type ORDER BY count DESC
|
|
`;
|
|
const pages_by_type: Record<string, number> = {};
|
|
for (const t of types) {
|
|
pages_by_type[t.type as string] = t.count as number;
|
|
}
|
|
|
|
return {
|
|
page_count: Number(stats.page_count),
|
|
chunk_count: Number(stats.chunk_count),
|
|
embedded_count: Number(stats.embedded_count),
|
|
link_count: Number(stats.link_count),
|
|
tag_count: Number(stats.tag_count),
|
|
timeline_entry_count: Number(stats.timeline_entry_count),
|
|
pages_by_type,
|
|
};
|
|
}
|
|
|
|
async getHealth(): Promise<BrainHealth> {
|
|
const sql = this.sql;
|
|
// Bug 11 doc-drift fix — orphan_pages means "islanded" (no inbound AND
|
|
// no outbound links), aligning both engines with the user-facing
|
|
// definition. The type comment previously said "no inbound" but the
|
|
// SQL required both — docs now match code so users can trust the
|
|
// number. A hub page that links out to many but has no back-references
|
|
// is working as intended, not an orphan.
|
|
const [h] = await sql`
|
|
WITH entity_pages AS (
|
|
SELECT id, slug FROM pages WHERE type IN ('person', 'company')
|
|
)
|
|
SELECT
|
|
(SELECT count(*) FROM pages) as page_count,
|
|
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL)::float /
|
|
GREATEST((SELECT count(*) FROM content_chunks), 1)::float as embed_coverage,
|
|
(SELECT count(*) FROM pages p
|
|
WHERE p.updated_at < (SELECT MAX(te.created_at) FROM timeline_entries te WHERE te.page_id = p.id)
|
|
) as stale_pages,
|
|
(SELECT count(*) FROM pages p
|
|
WHERE NOT EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = p.id)
|
|
AND NOT EXISTS (SELECT 1 FROM links l WHERE l.from_page_id = p.id)
|
|
) as orphan_pages,
|
|
(SELECT count(*) FROM links l
|
|
WHERE NOT EXISTS (SELECT 1 FROM pages p WHERE p.id = l.to_page_id)
|
|
) as dead_links,
|
|
(SELECT count(*) FROM content_chunks WHERE embedded_at IS NULL) as missing_embeddings,
|
|
(SELECT count(*) FROM links) as link_count,
|
|
(SELECT count(DISTINCT page_id) FROM timeline_entries) as pages_with_timeline,
|
|
(SELECT count(*) FROM entity_pages e
|
|
WHERE EXISTS (SELECT 1 FROM links l WHERE l.to_page_id = e.id))::float /
|
|
GREATEST((SELECT count(*) FROM entity_pages), 1)::float as link_coverage,
|
|
(SELECT count(*) FROM entity_pages e
|
|
WHERE EXISTS (SELECT 1 FROM timeline_entries te WHERE te.page_id = e.id))::float /
|
|
GREATEST((SELECT count(*) FROM entity_pages), 1)::float as timeline_coverage
|
|
`;
|
|
|
|
const connected = await sql`
|
|
SELECT p.slug,
|
|
(SELECT count(*) FROM links l WHERE l.from_page_id = p.id OR l.to_page_id = p.id)::int as link_count
|
|
FROM pages p
|
|
WHERE p.type IN ('person', 'company')
|
|
ORDER BY link_count DESC
|
|
LIMIT 5
|
|
`;
|
|
|
|
const pageCount = Number(h.page_count);
|
|
const embedCoverage = Number(h.embed_coverage);
|
|
const orphanPages = Number(h.orphan_pages);
|
|
const deadLinks = Number(h.dead_links);
|
|
const linkCount = Number(h.link_count);
|
|
const pagesWithTimeline = Number(h.pages_with_timeline);
|
|
|
|
// brain_score: 0-100 weighted average
|
|
const linkDensity = pageCount > 0 ? Math.min(linkCount / pageCount, 1) : 0;
|
|
const timelineCoverageWhole = pageCount > 0 ? Math.min(pagesWithTimeline / pageCount, 1) : 0;
|
|
const noOrphans = pageCount > 0 ? 1 - (orphanPages / pageCount) : 1;
|
|
const noDeadLinks = pageCount > 0 ? 1 - Math.min(deadLinks / pageCount, 1) : 1;
|
|
// Per-component points. Sum equals brainScore by construction.
|
|
const embedCoverageScore = pageCount === 0 ? 0 : Math.round(embedCoverage * 35);
|
|
const linkDensityScore = pageCount === 0 ? 0 : Math.round(linkDensity * 25);
|
|
const timelineCoverageScore = pageCount === 0 ? 0 : Math.round(timelineCoverageWhole * 15);
|
|
const noOrphansScore = pageCount === 0 ? 0 : Math.round(noOrphans * 15);
|
|
const noDeadLinksScore = pageCount === 0 ? 0 : Math.round(noDeadLinks * 10);
|
|
const brainScore = embedCoverageScore + linkDensityScore + timelineCoverageScore + noOrphansScore + noDeadLinksScore;
|
|
|
|
return {
|
|
page_count: pageCount,
|
|
embed_coverage: embedCoverage,
|
|
stale_pages: Number(h.stale_pages),
|
|
orphan_pages: orphanPages,
|
|
missing_embeddings: Number(h.missing_embeddings),
|
|
brain_score: brainScore,
|
|
dead_links: deadLinks,
|
|
link_coverage: Number(h.link_coverage),
|
|
timeline_coverage: Number(h.timeline_coverage),
|
|
most_connected: (connected as unknown as { slug: string; link_count: number }[]).map(c => ({
|
|
slug: c.slug,
|
|
link_count: Number(c.link_count),
|
|
})),
|
|
embed_coverage_score: embedCoverageScore,
|
|
link_density_score: linkDensityScore,
|
|
timeline_coverage_score: timelineCoverageScore,
|
|
no_orphans_score: noOrphansScore,
|
|
no_dead_links_score: noDeadLinksScore,
|
|
};
|
|
}
|
|
|
|
// Ingest log
|
|
async logIngest(entry: IngestLogInput): Promise<void> {
|
|
const sql = this.sql;
|
|
await sql`
|
|
INSERT INTO ingest_log (source_type, source_ref, pages_updated, summary)
|
|
VALUES (${entry.source_type}, ${entry.source_ref}, ${sql.json(entry.pages_updated)}, ${entry.summary})
|
|
`;
|
|
}
|
|
|
|
async getIngestLog(opts?: { limit?: number }): Promise<IngestLogEntry[]> {
|
|
const sql = this.sql;
|
|
const limit = opts?.limit || 50;
|
|
const rows = await sql`
|
|
SELECT * FROM ingest_log ORDER BY created_at DESC LIMIT ${limit}
|
|
`;
|
|
return rows as unknown as IngestLogEntry[];
|
|
}
|
|
|
|
// Sync
|
|
async updateSlug(oldSlug: string, newSlug: string): Promise<void> {
|
|
newSlug = validateSlug(newSlug);
|
|
const sql = this.sql;
|
|
await sql`UPDATE pages SET slug = ${newSlug}, updated_at = now() WHERE slug = ${oldSlug}`;
|
|
}
|
|
|
|
async rewriteLinks(_oldSlug: string, _newSlug: string): Promise<void> {
|
|
// Stub in v0.2. Links table uses integer page_id FKs, which are already
|
|
// correct after updateSlug (page_id doesn't change, only slug does).
|
|
// Textual [[wiki-links]] in compiled_truth are NOT rewritten here.
|
|
// The maintain skill's dead link detector surfaces stale references.
|
|
}
|
|
|
|
// Config
|
|
async getConfig(key: string): Promise<string | null> {
|
|
const sql = this.sql;
|
|
const rows = await sql`SELECT value FROM config WHERE key = ${key}`;
|
|
return rows.length > 0 ? (rows[0].value as string) : null;
|
|
}
|
|
|
|
async setConfig(key: string, value: string): Promise<void> {
|
|
const sql = this.sql;
|
|
await sql`
|
|
INSERT INTO config (key, value) VALUES (${key}, ${value})
|
|
ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value
|
|
`;
|
|
}
|
|
|
|
// Migration support
|
|
async runMigration(_version: number, sqlStr: string): Promise<void> {
|
|
const conn = this.sql;
|
|
await conn.unsafe(sqlStr);
|
|
}
|
|
|
|
async getChunksWithEmbeddings(slug: string): Promise<Chunk[]> {
|
|
const conn = this.sql;
|
|
const rows = await conn`
|
|
SELECT cc.* FROM content_chunks cc
|
|
JOIN pages p ON p.id = cc.page_id
|
|
WHERE p.slug = ${slug}
|
|
ORDER BY cc.chunk_index
|
|
`;
|
|
return rows.map((r) => rowToChunk(r as Record<string, unknown>, true));
|
|
}
|
|
|
|
/**
|
|
* Reconnect the engine by tearing down the current pool and creating a fresh one.
|
|
* No-ops if no saved config (module-singleton mode) or if already reconnecting.
|
|
*/
|
|
async reconnect(): Promise<void> {
|
|
if (!this._savedConfig || this._reconnecting) return;
|
|
this._reconnecting = true;
|
|
try {
|
|
// Tear down old pool (best-effort — it may already be dead)
|
|
try { await this.disconnect(); } catch { /* swallow */ }
|
|
// Create fresh pool
|
|
await this.connect(this._savedConfig);
|
|
} finally {
|
|
this._reconnecting = false;
|
|
}
|
|
}
|
|
|
|
async executeRaw<T = Record<string, unknown>>(sql: string, params?: unknown[]): Promise<T[]> {
|
|
const conn = this.sql;
|
|
return conn.unsafe(sql, params as Parameters<typeof conn.unsafe>[1]) as unknown as T[];
|
|
// Pre-#406 behavior: throw on any error including connection death.
|
|
// Per-call auto-retry is not safe here because executeRaw is also used
|
|
// for non-transactional mutations (DELETE/UPDATE/INSERT in sources.ts,
|
|
// ALTER TABLE in migrations) where retrying after a connection-mid-statement
|
|
// death can phantom-write a row that already committed on the server.
|
|
// Recovery instead happens at the supervisor level: the watchdog detects
|
|
// 3 consecutive health-check failures and calls engine.reconnect() to
|
|
// swap in a fresh pool. See db.ts setSessionDefaults / supervisor.ts.
|
|
}
|
|
|
|
// ============================================================
|
|
// v0.20.0 Cathedral II: code edges (Layer 1 stubs — filled by Layer 5)
|
|
// ============================================================
|
|
// Declared here so the interface contract is satisfied and consumers can
|
|
// import against them. Implementations throw until the edge extractor +
|
|
// per-lang tree-sitter queries land in Layer 5/6.
|
|
// ============================================================
|
|
|
|
async addCodeEdges(edges: import('./types.ts').CodeEdgeInput[]): Promise<number> {
|
|
if (edges.length === 0) return 0;
|
|
const sql = this.sql;
|
|
let inserted = 0;
|
|
const resolved = edges.filter(e => e.to_chunk_id != null);
|
|
const unresolved = edges.filter(e => e.to_chunk_id == null);
|
|
|
|
if (resolved.length > 0) {
|
|
const fromIds = resolved.map(e => e.from_chunk_id);
|
|
const toIds = resolved.map(e => e.to_chunk_id as number);
|
|
const fromQual = resolved.map(e => e.from_symbol_qualified);
|
|
const toQual = resolved.map(e => e.to_symbol_qualified);
|
|
const edgeTypes = resolved.map(e => e.edge_type);
|
|
const metas = resolved.map(e => JSON.stringify(e.edge_metadata ?? {}));
|
|
const sources = resolved.map(e => e.source_id ?? null);
|
|
const res = await sql`
|
|
INSERT INTO code_edges_chunk (from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
|
|
SELECT * FROM unnest(
|
|
${fromIds}::int[], ${toIds}::int[],
|
|
${fromQual}::text[], ${toQual}::text[],
|
|
${edgeTypes}::text[], ${metas}::jsonb[],
|
|
${sources}::text[]
|
|
)
|
|
ON CONFLICT (from_chunk_id, to_chunk_id, edge_type) DO NOTHING
|
|
`;
|
|
inserted += (res as unknown as { count: number }).count ?? 0;
|
|
}
|
|
|
|
if (unresolved.length > 0) {
|
|
const fromIds = unresolved.map(e => e.from_chunk_id);
|
|
const fromQual = unresolved.map(e => e.from_symbol_qualified);
|
|
const toQual = unresolved.map(e => e.to_symbol_qualified);
|
|
const edgeTypes = unresolved.map(e => e.edge_type);
|
|
const metas = unresolved.map(e => JSON.stringify(e.edge_metadata ?? {}));
|
|
const sources = unresolved.map(e => e.source_id ?? null);
|
|
const res = await sql`
|
|
INSERT INTO code_edges_symbol (from_chunk_id, from_symbol_qualified, to_symbol_qualified, edge_type, edge_metadata, source_id)
|
|
SELECT * FROM unnest(
|
|
${fromIds}::int[],
|
|
${fromQual}::text[], ${toQual}::text[],
|
|
${edgeTypes}::text[], ${metas}::jsonb[],
|
|
${sources}::text[]
|
|
)
|
|
ON CONFLICT (from_chunk_id, to_symbol_qualified, edge_type) DO NOTHING
|
|
`;
|
|
inserted += (res as unknown as { count: number }).count ?? 0;
|
|
}
|
|
|
|
return inserted;
|
|
}
|
|
|
|
async deleteCodeEdgesForChunks(chunkIds: number[]): Promise<void> {
|
|
if (chunkIds.length === 0) return;
|
|
const sql = this.sql;
|
|
await sql`DELETE FROM code_edges_chunk WHERE from_chunk_id = ANY(${chunkIds}::int[]) OR to_chunk_id = ANY(${chunkIds}::int[])`;
|
|
await sql`DELETE FROM code_edges_symbol WHERE from_chunk_id = ANY(${chunkIds}::int[])`;
|
|
}
|
|
|
|
async getCallersOf(
|
|
qualifiedName: string,
|
|
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
|
|
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
|
const sql = this.sql;
|
|
const limit = Math.min(opts?.limit ?? 100, 500);
|
|
const scopedSource: string | null =
|
|
!opts?.allSources && opts?.sourceId ? opts.sourceId : null;
|
|
const rows = await sql`
|
|
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
|
edge_type, edge_metadata, source_id, true as resolved
|
|
FROM code_edges_chunk
|
|
WHERE to_symbol_qualified = ${qualifiedName}
|
|
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
|
|
UNION ALL
|
|
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
|
edge_type, edge_metadata, source_id, false as resolved
|
|
FROM code_edges_symbol
|
|
WHERE to_symbol_qualified = ${qualifiedName}
|
|
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
|
|
LIMIT ${limit}
|
|
`;
|
|
return rows.map(r => pgRowToCodeEdge(r as Record<string, unknown>));
|
|
}
|
|
|
|
async getCalleesOf(
|
|
qualifiedName: string,
|
|
opts?: { sourceId?: string; allSources?: boolean; limit?: number },
|
|
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
|
const sql = this.sql;
|
|
const limit = Math.min(opts?.limit ?? 100, 500);
|
|
const scopedSource: string | null =
|
|
!opts?.allSources && opts?.sourceId ? opts.sourceId : null;
|
|
const rows = await sql`
|
|
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
|
edge_type, edge_metadata, source_id, true as resolved
|
|
FROM code_edges_chunk
|
|
WHERE from_symbol_qualified = ${qualifiedName}
|
|
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
|
|
UNION ALL
|
|
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
|
edge_type, edge_metadata, source_id, false as resolved
|
|
FROM code_edges_symbol
|
|
WHERE from_symbol_qualified = ${qualifiedName}
|
|
${scopedSource ? sql`AND source_id = ${scopedSource}` : sql``}
|
|
LIMIT ${limit}
|
|
`;
|
|
return rows.map(r => pgRowToCodeEdge(r as Record<string, unknown>));
|
|
}
|
|
|
|
async getEdgesByChunk(
|
|
chunkId: number,
|
|
opts?: { direction?: 'in' | 'out' | 'both'; edgeType?: string; limit?: number },
|
|
): Promise<import('./types.ts').CodeEdgeResult[]> {
|
|
const sql = this.sql;
|
|
const direction = opts?.direction ?? 'both';
|
|
const limit = Math.min(opts?.limit ?? 50, 200);
|
|
const typeFilter = opts?.edgeType;
|
|
|
|
const chunkRows = await sql`
|
|
SELECT id, from_chunk_id, to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
|
edge_type, edge_metadata, source_id, true as resolved
|
|
FROM code_edges_chunk
|
|
WHERE
|
|
${direction === 'in' ? sql`to_chunk_id = ${chunkId}`
|
|
: direction === 'out' ? sql`from_chunk_id = ${chunkId}`
|
|
: sql`(from_chunk_id = ${chunkId} OR to_chunk_id = ${chunkId})`}
|
|
${typeFilter ? sql`AND edge_type = ${typeFilter}` : sql``}
|
|
LIMIT ${limit}
|
|
`;
|
|
let symbolRows: unknown[] = [];
|
|
if (direction !== 'in') {
|
|
const sRows = await sql`
|
|
SELECT id, from_chunk_id, NULL::int as to_chunk_id, from_symbol_qualified, to_symbol_qualified,
|
|
edge_type, edge_metadata, source_id, false as resolved
|
|
FROM code_edges_symbol
|
|
WHERE from_chunk_id = ${chunkId}
|
|
${typeFilter ? sql`AND edge_type = ${typeFilter}` : sql``}
|
|
LIMIT ${limit}
|
|
`;
|
|
symbolRows = [...sRows];
|
|
}
|
|
return [...chunkRows, ...symbolRows].map(r => pgRowToCodeEdge(r as Record<string, unknown>));
|
|
}
|
|
|
|
}
|
|
|
|
function pgRowToCodeEdge(row: Record<string, unknown>): import('./types.ts').CodeEdgeResult {
|
|
return {
|
|
id: row.id as number,
|
|
from_chunk_id: row.from_chunk_id as number,
|
|
to_chunk_id: row.to_chunk_id == null ? null : (row.to_chunk_id as number),
|
|
from_symbol_qualified: (row.from_symbol_qualified as string) ?? '',
|
|
to_symbol_qualified: (row.to_symbol_qualified as string) ?? '',
|
|
edge_type: (row.edge_type as string) ?? '',
|
|
edge_metadata: (row.edge_metadata as Record<string, unknown>) ?? {},
|
|
source_id: row.source_id == null ? null : (row.source_id as string),
|
|
resolved: Boolean(row.resolved),
|
|
};
|
|
}
|