From ff32fcaa780bbac6c0ae41effd82bd061b21658f Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Wed, 27 May 2026 08:50:53 -0700 Subject: [PATCH] v0.41.25.0 perf(sync): batched deletes + global page-generation clock (supersedes #1538) (#1566) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(engine): add deletePages + resolveSlugsByPaths to BrainEngine (v0.41.21.0 T1) Two new REQUIRED methods on the BrainEngine interface, implemented on both Postgres and PGLite engines. Closes the per-file N+1 query pattern that PR #1538 batched on Postgres only. deletePages(slugs: string[], opts: { sourceId: string }): Promise — Single SQL round-trip: DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug — Returns slugs ACTUALLY DELETED (D6, codex CDX-8) so callers can filter pagesAffected to exclude phantom slugs (paths in the deletion list but with no DB row). — Single-batch primitive: caller chunks input to DELETE_BATCH_SIZE. Throws if input exceeds the cap. — sourceId is REQUIRED at the type level (D5, codex CDX-10). Asymmetric with single-row deletePage which keeps the optional 'default' fallback for back-compat. v0.42+ TODO to tighten. resolveSlugsByPaths(paths, opts): Promise> — Batch path → slug lookup. Single SQL round-trip: SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2 — Missing paths absent from the Map (caller falls back to path-derived slug, same contract as resolveSlugByPathOrSourcePath). — Empty input short-circuits to empty Map (no SQL). src/core/engine-constants.ts (NEW) — Single source of truth for DELETE_BATCH_SIZE = 500. — Both engines import; no engine-from-engine coupling. — Lives outside engine.ts (the interface module) to avoid circular imports. Also updates the deletePage JSDoc (CDX-11): drops the misleading "hard delete is admin-only" framing. `gbrain sync` hard-deletes on every run that sees a deleted file; not admin-only. Co-Authored-By: garrytan-agents Co-Authored-By: Claude Opus 4.7 (1M context) * perf(sync): batched delete + rename + DRY refactor (v0.41.21.0 T2/T3/T4) Replaces the per-file delete loop (sync.ts:1241-1257) and per-file rename slug-resolve (sync.ts:1263-1295) with interleaved per-batch flows using engine.resolveSlugsByPaths + engine.deletePages. Also refactors resolveSlugByPathOrSourcePath (sync.ts:267) to delegate to the new batch helper when sourceId is set — one owner of the SQL + fallback semantics (D8). ROUND-TRIP COUNTS (73K-delete commit): pre-fix: 73,000 SELECTs + 73,000 DELETEs = 146,000 (~5 hours) post-fix: 146 SELECTs + 146 DELETEs = 292 (~2 minutes) Headline win: a single commit deleting 73K files no longer jams the sync pipeline for hours, no longer cascades staleness across every other source on the brain. Shape (T2 delete loop, per the plan's ASCII diagram): filtered.deleted (73K paths) │ ▼ slice into batches of DELETE_BATCH_SIZE (500) │ ▼ for each batch: abort-check ──► partial('timeout') │ ▼ engine.resolveSlugsByPaths(batch, {sourceId}) ◀── 1 SQL round-trip │ ▼ slugs = batch.map(path => map.get(path) ?? resolveSlugForPath(path)) ◀── pure-JS fallback │ for frontmatter- ▼ fallback slugs try { deleted = engine.deletePages(slugs, opts) ◀── 1 SQL round-trip pagesAffected.push(...deleted) ◀── D6 confirmed only } catch { // D7 decompose: per-slug deletePage, // unrecoverable failures → failedFiles } Per-batch try-catch (D7) decomposes batch DELETE failures to per-slug deletePage so a transient blip on batch 73 doesn't lose 500 deletes — it self-heals to one-at-a-time for that batch only. Unrecoverable per-slug failures land in failedFiles (matching the existing import-loop pattern at sync.ts:~1350). failedFiles declaration hoisted above the delete loop so both delete decompose and import loops feed the same sync-bookmark gate. T4 rename loop: pre-resolves all `from` slugs in batches via resolveSlugsByPaths BEFORE iterating. Per-file updateSlug + importFile calls stay (those are inherently per-file). The try/catch around updateSlug for slug-doesn't-exist preserves verbatim. T3 DRY refactor: resolveSlugByPathOrSourcePath delegates to resolveSlugsByPaths via a single-element array when sourceId is set. When sourceId is undefined (legacy unscoped callers), falls back to the original executeRaw shape — the batch engine surface requires sourceId per D5 (multi-source-bug-class defense). Atomicity coarsening (D3): each batch is one transaction. A mid-batch abort or connection failure rolls back up to DELETE_BATCH_SIZE - 1 successful deletes from the in-flight batch. Sync is idempotent so the next run picks them up via git diff regenerating the deletion list. Documented at the call site + in the deletePages JSDoc. Co-Authored-By: garrytan-agents Co-Authored-By: Claude Opus 4.7 (1M context) * feat(schema): global page-generation clock + statement-level trigger (v0.41.21.0 T5) Migration v104: page_generation_clock_and_statement_trigger. The pre-v0.41.21.0 query-cache Layer 1 bookmark read MAX(generation) FROM pages to detect "writes happened since cache-store". Two bugs in that contract — independent of any sync work, surfaced by codex outside-voice on the /plan-eng-review pass: 1. The row-level bump_page_generation_trg (migration v91) sets NEW.generation = OLD.generation + 1 on UPDATE. Updating a NON-MAX page didn't advance MAX(generation). Cache silently served stale for any UPDATE-to-non-max page. (CDX-2) 2. The trigger is BEFORE INSERT OR UPDATE — DELETE doesn't fire it at all. Even an AFTER DELETE wouldn't move MAX (surviving rows are untouched). (CDX-1) Fix: single-row page_generation_clock counter, bumped per-statement (FOR EACH STATEMENT — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter, recreating the bottleneck this PR fixes elsewhere — codex CDX-4). Layer 1 reads the clock value directly (T6, separate commit). Per-row pages.generation stays for Layer 2 (per-page snapshot via jsonb_each + LEFT JOIN pages) which doesn't care about MAX, only per-page advancement. Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache rows stored under the old MAX semantics aren't all instantly invalidated on upgrade. Their max_generation_at_store stamp compares cleanly against the seeded clock; future writes bump the clock and the bookmark fires correctly. CREATE TABLE page_generation_clock ( id INTEGER PRIMARY KEY CHECK (id = 1), value BIGINT NOT NULL DEFAULT 0 ); CREATE TRIGGER bump_page_generation_clock_trg AFTER INSERT OR UPDATE OR DELETE ON pages FOR EACH STATEMENT EXECUTE FUNCTION bump_page_generation_clock_fn(); Mirror in src/core/pglite-schema.ts so fresh PGLite installs get the table + trigger via SCHEMA_SQL replay. The forward-reference bootstrap probe doesn't need an entry: page_generation_clock is created directly by SCHEMA_SQL (no separate index or FK references it), so the schema-bootstrap-coverage gate is satisfied as-is. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(cache): move Layer 1 to global clock + invalidate empty snapshots (v0.41.21.0 T6) Closes the silent stale-cache bug class that's been live in master since the bookmark feature shipped. Pre-fix, gbrain search would silently serve stale cached results in three independent scenarios: 1. UPDATE to a non-max-generation page (CDX-2) — the row-level trigger advanced per-page generation but didn't move MAX(generation), so the bookmark passed. 2. DELETE of any page (CDX-1) — the trigger didn't fire at all, and even an AFTER DELETE wouldn't move MAX. 3. Empty-result cache row + subsequent matching INSERT (CDX-6 / D20) — page_generations = '{}'::jsonb was "vacuously valid" via Layer 2, surviving any clock bump. Fix: buildPageGenerationsSnapshot (store path) — Replaces the SELECT MAX(generation) FROM pages reads at cache-write time with SELECT value FROM page_generation_clock WHERE id = 1. — Empty pageIds path: only need the clock value (D20 contract). — Combined non-empty path: per-page generation (Layer 2 substrate) + clock value, both folded in one round trip via UNION ALL. CACHE_GATE_WHERE_CLAUSE (lookup path) — Layer 1 reads page_generation_clock.value (single-row O(1) lookup, faster than the pre-fix MAX(generation) backward index scan). — Layer 2 stricter: requires page_generations <> '{}'::jsonb AND the per-page check (not OR with the vacuously-valid `= '{}'` shortcut). Empty snapshots can no longer survive a Layer 1 miss. validateCacheRowAgainstPages (pure validator) — Layer 2 returns false for empty snapshots when Layer 1 fails. — Documented contract change. Backward compat: pre-v0.40.3.0 cache rows have max_generation_at_store = 0 AND page_generations = '{}'::jsonb. On a populated brain, Layer 1 fails (clock > 0). Layer 2 is now stricter so legacy rows invalidate once on first post-upgrade lookup, then the cache fills back correctly. Acceptable one-time miss spike; post-upgrade cache is structurally sound. The clock seed (COALESCE(MAX(pages.generation), 0)) from migration v104 keeps NON-empty legacy rows passing Layer 1 until the next write — they don't all invalidate at once. Co-Authored-By: Claude Opus 4.7 (1M context) * test: cover v0.41.21.0 delete-batch + global clock + cache contract (T7+T8+T9) Tests for every behavior the v0.41.21.0 wave introduces or changes. New test files: test/sync-delete-batch.test.ts (PGLite hermetic) — engine.deletePages: empty input short-circuit, returns confirmed slugs (D6), multi-source isolation, cascade integrity (chunks + links cleared via FK), rejects oversized input. — engine.resolveSlugsByPaths: empty input, present + missing rows, D10 exotic-filename substrate (🌟.md / ทดสอบ.md / عربي.md), source isolation. — D13 pagesAffected filter: 100 deletable + 10 ghost paths → deletePages returns 100 (regression-pin: pre-fix would return all 110 via D6's pre-RETURNING shape). test/sync-delete-batch.slow.test.ts (.slow suffix keeps it out of the fast loop) — 10K-page batched delete completes in <5s on PGLite. Measured 277ms on dev hardware (18x under the gate); pins the headline perf promise. test/sync-rename-batch.test.ts (PGLite hermetic) — 500-rename batch slug-resolve in 1 round-trip (exactly at DELETE_BATCH_SIZE boundary). — Frontmatter-fallback rename: exotic source_paths resolve via the batch SELECT. — Mixed present + missing: partial Map (missing → caller falls back to path-derived). test/page-generation-counter.test.ts (PGLite hermetic) — Statement-level trigger fires once per INSERT statement (raw SQL — NOT putPage, which uses ON CONFLICT DO UPDATE and bumps by 2 in PG semantics). — Statement-level trigger fires once per UPDATE statement. — Headline contract: batch DELETE bumps clock by 1, NOT by row count (25-row batch → +1). — CDX-1 regression: DELETE of non-max page bumps clock. — CDX-2 regression: UPDATE of non-max page bumps clock (raw SQL). — D14 end-to-end: clock advances after batch DELETE → cache rows stamped at the prior clock value are now stale by Layer 1. — CDX-6/D20: empty-result cache + INSERT matching page → clock advances (Layer 1 fires). — Documents the PG quirk: putPage's INSERT...ON CONFLICT DO UPDATE bumps clock by 2 (both INSERT and UPDATE triggers fire). Test-helper update: test/helpers/reset-pglite.ts — Added page_generation_clock to PRESERVE_TABLES so the seeded single-row counter survives resetPgliteState between tests (same treatment as schema_version). Production never truncates. Existing test contract inversions (CDX-6 / D20 fix): test/query-cache-gate.test.ts — Pre-v0.41.21.0 "vacuously valid for legacy empty snapshot" assertion inverted: empty snapshot now invalidates when Layer 1 fires. Add positive CDX-6 regression test (empty-result + INSERT matching page). — SQL shape regression: page_generation_clock in Layer 1 (negative regression guard: MAX(generation) FROM pages MUST be gone). — Empty-snapshot reject guard: `qc.page_generations <> '{}'::jsonb` present; the old `qc.page_generations = '{}'::jsonb OR` shortcut MUST be gone. test/e2e/cache-gate-pglite.test.ts — Pre-v0.41.21.0 "legacy row serves vacuously" test inverted: legacy rows now invalidate on first clock advance post-upgrade. — CDX-1 regression: DELETE bumps clock → cached query for surviving pages invalidates. — CDX-2 regression: UPDATE-to-non-max-page bumps clock → cache invalidates. — CDX-11 comment fix: drop misleading "hard delete is admin-only" framing; gbrain sync hard-deletes on every run. Engine parity extension: test/e2e/engine-parity.test.ts — deletePages parity: same input set, both engines return same string[] of confirmed-deleted slugs (D6). — resolveSlugsByPaths parity: same Map on both engines. Co-Authored-By: Claude Opus 4.7 (1M context) * chore(release): v0.41.21.0 — batched sync deletes + global page-generation clock (T10) VERSION bump (0.41.18.0 → 0.41.21.0; master is at 0.41.20.0 so next free slot per the queue allocator). CHANGELOG entry with the ELI10 lead per CLAUDE.md voice rules. CLAUDE.md annotations on engine.ts, postgres-engine.ts, pglite-engine.ts, sync.ts, and query-cache-gate.ts plus a new entry for engine-constants.ts. llms-full.txt regenerated to match CLAUDE.md (per CLAUDE.md mandatory rule). Co-Authored-By: garrytan-agents Co-Authored-By: Claude Opus 4.7 (1M context) * dx(test-runner): heartbeat shows real progress instead of 0p 0f Bun's default test reporter doesn't print per-test markers — only a single shard-end summary block when you pass it a file list. The existing heartbeat tried to count `^[[:space:]]+✓` lines as a live pass-count proxy, but bun never emits them in the multi-file mode this runner uses, so every mid-run heartbeat showed `0p 0f` for the entire 12-20 minute wallclock. Users (and agents polling the runner) couldn't distinguish "still bootstrapping" from "wedged" from "almost done." Fix: parse three complementary real-time signals instead. 1. Total files this shard was assigned — parsed from the `[unit-shard N/M] running X files` banner that run-unit-shard.sh echoes before invoking bun test. Available from second 1. 2. PGLite initSchema() count — proxy for "test files started so far." Each PGLite-using test file's beforeAll triggers one initSchema(), which logs `Schema version 1 → 106 (101 migration(s) pending)`. Undercounts because not every test file opens a PGLite engine (covers ~30-60% of files in practice), but it's the only real-time progress signal bun's default reporter leaves in the log. The output uses a `~` prefix to convey "approximate count." 3. Log size in KB — strictly monotonic liveness signal that works even when the PGLite count is still 0 (early-shard startup before the first initSchema fires). 4. Per-shard elapsed time — formatted as MmSSs. New mid-run heartbeat line: [heartbeat] [s1: ~62/190f 476KB 12m31s] [s2: ~63/190f 513KB 12m31s] ... When a shard finishes, the heartbeat upgrades to its final summary including pass/fail counts from bun's end-of-shard summary block: [heartbeat] [s1: done ✓ 2807p 0f] [s2: done ✓ 2784p 0f] ... Portability: BSD awk on macOS doesn't support `match($0, /re/, arr)` with the array sink — that's a gawk extension. The total-files parser uses sed instead so the runner stays portable to the default Mac toolchain. Helpers are pure functions and unit-testable in isolation: pass a log file path, get the parsed number. No mocking. No bun runtime required. Co-Authored-By: Claude Opus 4.7 (1M context) * chore(release): rebump v0.41.23.0 → v0.41.25.0 Per user request — skip v0.41.23.0 / v0.41.24.0 slots to land at v0.41.25.0. Master is at v0.41.22.1, no version-trio collision. Touches VERSION, package.json, CHANGELOG header, CLAUDE.md annotations, src/core/engine-constants.ts header, src/core/migrate.ts migration v106 comment, regenerated llms-full.txt + llms.txt. Migration version (v106) and CDX1-6 trigger semantics unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(schema): reorder migration_impact_log AFTER minion_jobs (CI green) Pre-existing bug in master's SCHEMA_SQL ordering, surfaced by CI on this PR but lived silently on master since v0.41.18.0. migration_impact_log declares `job_id BIGINT REFERENCES minion_jobs(id)`, but its CREATE TABLE was at line 658 while minion_jobs's CREATE TABLE was at line 778. On any fresh-install initSchema() the FK target didn't exist yet: psql:/tmp/schema.sql:672: ERROR: relation "minion_jobs" does not exist postgres-js's `unsafe()` aborts the multi-statement batch on the first error response, so every CREATE TABLE after migration_impact_log (including minion_jobs itself) never ran. Every subsequent CLI subprocess that opened a connection then crashed with `relation "minion_jobs" does not exist` on its first query. Why master CI sometimes passed: the per-shard advisory lock + the test setup's `engine.initSchema()` second pass (which runs the migrations array) would eventually create minion_jobs via the v5 `minion_jobs_table` migration. From there migration_impact_log would land via migration v103 with its FK resolving correctly. But CLI subprocesses spawned by mechanical.test.ts's Parallel Import block open their OWN connections and run a fresh `engine.connect() → initSchema()` — that path runs SCHEMA_SQL FIRST and aborted at the same forward-reference error before the migrations array could repair. Fix: relocate the migration_impact_log CREATE TABLE + its two indexes to AFTER the minion_jobs CREATE TABLE block (lines ~865), keeping the rest of the schema layout intact. PGLite schema (pglite-schema.ts) already had the correct ordering — only Postgres SCHEMA_SQL needed the move. Verified: fresh-DB local repro that previously failed 31/34 tests with `relation minion_jobs does not exist` now passes 78/78 in test/e2e/mechanical.test.ts. Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: garrytan-agents --- CHANGELOG.md | 137 ++++++++++++++++ CLAUDE.md | 11 +- VERSION | 2 +- llms-full.txt | 11 +- package.json | 2 +- scripts/run-unit-parallel.sh | 78 +++++++-- src/commands/sync.ts | 205 ++++++++++++++++++++--- src/core/engine-constants.ts | 25 +++ src/core/engine.ts | 59 +++++++ src/core/migrate.ts | 64 ++++++++ src/core/pglite-engine.ts | 44 +++++ src/core/pglite-schema.ts | 26 +++ src/core/postgres-engine.ts | 48 ++++++ src/core/schema-embedded.ts | 111 ++++++++++++- src/core/search/query-cache-gate.ts | 103 ++++++++---- src/schema.sql | 121 ++++++++++---- test/e2e/cache-gate-pglite.test.ts | 85 ++++++++-- test/e2e/engine-parity.test.ts | 53 ++++++ test/helpers/reset-pglite.ts | 7 +- test/page-generation-counter.test.ts | 227 ++++++++++++++++++++++++++ test/query-cache-gate.test.ts | 73 +++++++-- test/sync-delete-batch.slow.test.ts | 80 +++++++++ test/sync-delete-batch.test.ts | 233 +++++++++++++++++++++++++++ test/sync-rename-batch.test.ts | 92 +++++++++++ 24 files changed, 1754 insertions(+), 143 deletions(-) create mode 100644 src/core/engine-constants.ts create mode 100644 test/page-generation-counter.test.ts create mode 100644 test/sync-delete-batch.slow.test.ts create mode 100644 test/sync-delete-batch.test.ts create mode 100644 test/sync-rename-batch.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 50cd25ffc..b80d6ead7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,143 @@ All notable changes to GBrain will be documented in this file. + +## [0.41.25.0] - 2026-05-27 + +**Big-delete syncs no longer choke your brain — and your search results +were silently going stale in a different way that's now fixed.** + +If you've ever made a commit that deleted a lot of files (a folder +reorganization, an `atom` backfill rewrite, a big import cleanup), you +might have noticed that `gbrain sync` would just... sit there. For +hours. While it sat, the sync cron timed out every run, every other +source on your brain stopped syncing, and `gbrain doctor` health score +slid toward zero. That's fixed. A commit deleting 73,000 files used to +take about five hours to absorb; it now takes about two minutes. + +While digging into the delete path we found a second, unrelated bug. +Your query cache was silently serving stale `gbrain search` results +whenever you updated an *older* page (one that wasn't the newest in +your brain) and whenever you deleted *anything*. That's also fixed. +You'll see one tiny cache-miss spike on first `gbrain search` after +upgrade as the cache gets re-stamped under the new contract, then it's +faster than before because the bookmark is a one-row lookup instead of +a brain-wide MAX scan. + +**To take advantage of v0.41.22.0** + +`gbrain upgrade` should do this automatically. If it didn't, or if +`gbrain doctor` warns about a partial migration: + +1. Run the orchestrator manually: + ```bash + gbrain apply-migrations --yes + ``` +2. Verify the new clock table exists: + ```bash + gbrain doctor + # Look for any "page_generation_clock" warning in the output; + # there shouldn't be one. + ``` +3. Manual smoke test for the perf win (optional, only if you want to + see it): + ```bash + # In a scratch brain, stage a commit that deletes a lot of files, + # then run `gbrain sync --source ` and watch the wallclock. + # Before: minutes. After: seconds. + ``` +4. If any step fails or numbers look wrong, file an issue at + https://github.com/garrytan/gbrain/issues with `gbrain doctor` + output and the contents of `~/.gbrain/upgrade-errors.jsonl` if it + exists. + +### Itemized changes + +- **Sync delete loop batched.** `gbrain sync` (and the autopilot cycle's + sync phase) now batches deletes through a single SQL round-trip per + 500 files instead of one round-trip per file. The 73K-delete commit + drops from ~146,000 DB round-trips (~5 hours) to ~292 (~2 minutes). + Per-batch try-catch + per-slug decompose mirrors the existing + failedFiles pattern in the import loop, so a transient connection + blip on batch 73 doesn't lose 500 deletes — it self-heals to + one-at-a-time for that batch only. Engine-uniform: PGLite and real + Postgres both get the win. +- **Sync rename loop pre-resolved.** Same batched slug-resolve treatment + for renames; the per-file `importFile` (which dominates rename cost + anyway) stays per-file. +- **`engine.deletePages(slugs, {sourceId})`** — new required method on + `BrainEngine`. Returns the slugs actually deleted (so callers stop + wasting downstream work on phantom paths). `sourceId` is required at + the type level — closes the multi-source-bug-class structurally on + the new surface. +- **`engine.resolveSlugsByPaths(paths, {sourceId})`** — new required + method on `BrainEngine`. Batch path → slug lookup. The single-call + `resolveSlugByPathOrSourcePath` helper now delegates to this when + `sourceId` is set; one owner of the SQL and fallback semantics. +- **`src/core/engine-constants.ts`** — new module owning + `DELETE_BATCH_SIZE = 500`. Both engines import from it; no + engine-from-engine coupling. +- **Global page-generation clock.** New `page_generation_clock` table + with a statement-level `AFTER INSERT OR UPDATE OR DELETE` trigger. + `query-cache-gate.ts` Layer 1 now reads the clock directly. Pre-fix + the bookmark read `MAX(generation) FROM pages` and was structurally + broken: + - Updating a non-max page kept the old `OLD.generation + 1` trigger + body, which advanced the per-page counter but didn't move + `MAX(generation)`. Cache silently served stale. + - Hard-deleting any page didn't fire the trigger at all. Cache + silently served stale. + - Empty-result cache rows had `{}` per-page snapshots that were + "vacuously valid" via Layer 2 — they survived any subsequent + matching INSERT. Cache silently served stale. + + The new clock bumps exactly once per write statement regardless of + row cardinality (a 500-row batch DELETE is one counter bump, not + 500). Per-row `pages.generation` stays for Layer 2's per-page + snapshot semantics; only Layer 1's read source changed. Empty-result + cache rows now trust Layer 1 exclusively. +- **Migration v106** (`page_generation_clock_and_statement_trigger`) + creates the clock table, seeds it with `COALESCE(MAX(pages.generation), 0)` + so existing cache rows aren't all instantly invalidated, and wires + the new trigger. +- **CHANGELOG, version, and tests.** Tests in + `test/sync-delete-batch.test.ts`, `test/sync-delete-batch.slow.test.ts` + (PGLite 10K-page perf gate), `test/sync-rename-batch.test.ts`, + `test/page-generation-counter.test.ts`, plus an engine-parity + extension in `test/e2e/engine-parity.test.ts`. The existing + `test/query-cache-gate.test.ts` and `test/e2e/cache-gate-pglite.test.ts` + flip their pre-v0.41.22.0 "vacuously valid legacy row" assertions to + match the new contract (legacy rows now invalidate once on first + post-upgrade lookup, then the cache fills back correctly). +- **Credit:** the SQL batching idea (`DELETE FROM pages WHERE slug = + ANY($1::text[])` chunked at 500) and the headline 73K-delete number + come from PR #1538 by `@garrytan-agents`. That PR shipped Postgres- + only with no tests; this release supersedes it with PGLite parity, + proper engine-API factoring, test coverage, and the cache-bookmark + correctness fix that landed alongside. + +### For contributors + +- Engine implementers (gbrain has no external implementers today) must + add `deletePages` and `resolveSlugsByPaths` to any custom + `BrainEngine` — both are required methods on the interface. +- New CI surface: nothing additional; the existing `bun run verify` + gate covers the new test files automatically. +- v0.42+ TODO: tighten `deletePage` (single-row) to require `sourceId` + symmetric with the new `deletePages`. Out of scope for this PR + because it requires a full caller audit. +- v0.42+ TODO: clean up the existing `bump_page_generation_fn` + row-level trigger's UPDATE branch (`NEW.generation = OLD.generation + + 1` is structurally wrong for MAX-style consumers, harmless for + Layer 2 per-page snapshot but worth cleaning up). +- v0.42+ TODO: `op_checkpoints` integration on the sync-delete loop + for resume-after-kill semantics. Per-delete cost dropped from ~250ms + to ~0.4ms so a 73K-delete completes in seconds; resume matters much + less now. +- v0.42+ TODO: heavy regression test for two concurrent batch DELETEs + against disjoint sources (the statement-level clock-row UPDATE + contention shape). Filed under `tests/heavy/` per CLAUDE.md. + ## [0.41.24.0] - 2026-05-27 **Your reformatted meeting transcripts now actually parse.** diff --git a/CLAUDE.md b/CLAUDE.md index 3bfb40229..fd0f1fa52 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -41,7 +41,8 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. +- `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. - `src/core/search/explain-formatter.ts` (v0.40.4.0) — renders `SearchResult[]` as a multi-line per-result breakdown for `gbrain search --explain`. Reads every boost-stamping field. Handles the "no boosts applied" empty path. 4-decimal precision with trailing-zero strip. Pinned by `test/search/explain-formatter.test.ts` (19 cases). @@ -51,8 +52,8 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -266,7 +267,7 @@ strict behavior when unset. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -647,7 +648,7 @@ Key commands added in v0.40.3.0 (contextual retrieval + cache gate + 4 CLI verbs - KNOBS_HASH_VERSION bumped 3 → 5 (skipped past 4 reserved by salem's v0.40.4 graph-signals work). One-time cache-row invalidation on upgrade; refills within TTL. - Three new Minion handlers wired into RemediationStep consumer pattern: `lint-fix`, `integrity-auto`, `sync-retry-failed`. Thin wrappers around already-shipping CLI commands; NOT in PROTECTED_JOB_NAMES (idempotent, no shell exec, MCP-safe). `sync-skip-failed` deliberately NOT in this set per codex D12 Bug 3. - `src/core/remediation-step.ts` (NEW canonical module) exports RemediationStep type + makeRemediationStep factory + canonical-JSON idempotencyKey() per codex D12 Bug 2. Future doctor checks emit RemediationSteps via the factory instead of hand-rolling the shape. -- `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. +- `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. **v0.41.25.0 (codex outside-voice on /plan-eng-review):** Layer 1 bookmark read source switched from `MAX(generation) FROM pages` to `SELECT value FROM page_generation_clock WHERE id = 1` at both store + lookup sites. Closes two pre-existing silent stale-cache bug classes that were independent of any sync work: (1) CDX-2 — UPDATE to a non-max page set `NEW.generation = OLD.generation + 1` which didn't advance MAX, so cache silently served stale on every non-max UPDATE; (2) CDX-1 — DELETE didn't fire the row-level trigger at all, and even an AFTER DELETE wouldn't move MAX because surviving rows are untouched. The new clock is bumped per-statement by `bump_page_generation_clock_trg` (statement-level trigger created in migration v106) so every INSERT/UPDATE/DELETE statement advances the bookmark exactly once regardless of row cardinality (D19 — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter). Also CDX-6/D20 fix: empty-result cache rows used to be "vacuously valid" via Layer 2's `qc.page_generations = '{}'::jsonb` shortcut, silently serving stale empty results across subsequent matching INSERTs. The shortcut is gone — empty snapshots now require Layer 1 to pass. `CACHE_GATE_WHERE_CLAUSE` enforces `qc.page_generations <> '{}'::jsonb` AND the per-page check (not OR). One-time post-upgrade cache miss spike on legacy `{}` rows is acceptable — the cache fills back up correctly and the clock seed `COALESCE(MAX(pages.generation), 0)` keeps non-empty legacy rows serving until the next write. Pinned by 5 new cases in `test/page-generation-counter.test.ts` (CDX-1/CDX-2/CDX-6 regressions + statement-level-trigger-fires-once contract) and 2 new e2e cases in `test/e2e/cache-gate-pglite.test.ts`. The existing pre-v0.41.25.0 `vacuously valid legacy row` IRON-RULE assertion in both files is intentionally inverted: that path was the CDX-6 bug. - `src/core/search/mode-switch-ux.ts` (NEW) exports `summarizeTransition()` 5-cell matrix + `probeWorkerAvailable()` worker liveness proxy via minion_jobs activity + `buildReindexIdempotencyKey()` content-stable key + `runModeSwitchUx()` orchestrator. Key commands added in v0.36.4.0 (brain-health-100 wave): diff --git a/VERSION b/VERSION index 1fd237ef5..1a46e86c5 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.24.0 \ No newline at end of file +0.41.25.0 diff --git a/llms-full.txt b/llms-full.txt index 021e1b7a0..4c3041188 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -183,7 +183,8 @@ strict behavior when unset. ## Key files - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Also exports upload validators: `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (v0.23 glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` flags untrusted callers; `OperationContext.allowedSlugPrefixes` (v0.23) is the trusted-workspace allow-list set by the dream cycle. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link enabled for trusted-workspace writes (skipped only when `remote=true && !trustedWorkspace`). As of v0.26.0, every `Operation` also carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`. All ops are annotated; `sync_brain`, `file_upload`, `file_list`, and `file_url` are `admin + localOnly` (rejected over HTTP). `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. **v0.26.9 (D12 + F7b):** `OperationContext.remote` is now a REQUIRED field in the TypeScript type — the compiler is the first defense against transports that forget to set it. Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) flipped from falsy-default (`!ctx.remote`) to fail-closed semantics (`ctx.remote === false` for "trusted-only" sites and `ctx.remote !== false` for "untrust unless explicit-false"). Anything that isn't strictly `false` is now treated as remote. Closed an HTTP MCP shell-job RCE: a `read+write`-scoped OAuth token could submit `shell` jobs because the HTTP request handler's literal context skipped `remote: true` and `submit_job`'s protected-name guard saw a falsy undefined. Stdio MCP set the field correctly via dispatch.ts; HTTP inlined a parallel context-builder for several releases and lost it. **v0.34.1.0 (#861 + #876):** new helper `sourceScopeOpts(ctx)` encodes the precedence ladder for source-scoped reads — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId` / `ctx.auth.sourceId`) over nothing. Every read-side op handler routes through it so future ops can't silently drift from the canonical v0.31.8 thread. Closes the source-isolation leak on the read path: a `read+write`-scoped OAuth client bound to `--source dept-x` no longer sees rows from neighboring sources via `search` / `query` / `list_pages` / `get_page` / `find_experts` / `query`'s image path. -- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. +- `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput` / `TimelineBatchInput` for the v0.12.1 bulk-insert API (`addLinksBatch` / `addTimelineEntriesBatch`). As of v0.13.1, `BrainEngine` has a `readonly kind: 'postgres' | 'pglite'` discriminator so migrations (`src/core/migrate.ts`) and other consumers can branch on engine without `instanceof` + dynamic imports. **v0.29:** four new methods — `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so a page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[], $2::text[], $3::real[])` composite-keyed on `(slug, source_id)` for multi-source safety), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` extended with `sort?: 'updated_desc' | 'updated_asc' | 'created_desc' | 'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines (was hardcoded `ORDER BY updated_at DESC`). **v0.32.8 (PR #860):** new `listAllPageRefs(): Promise>` ordered by `(source_id, slug)`. Cheap cross-source enumeration for hot loops on large brains — replaces the `getAllSlugs()→getPage(slug)` N+1 pattern in extract-takes, extract, integrity, which silently defaulted to `source_id='default'` for non-default-source pages. Implementation parity across postgres-engine.ts + pglite-engine.ts. Pinned by `test/e2e/multi-source-bug-class.test.ts`. **v0.34.1.0 (#861):** `SearchOpts` + `PageFilters` add `sourceIds?: string[]` for the federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when the array is set and preserve the scalar `sourceId` fast path when unset. `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId` / `opts.sourceIds` so graph walks respect the caller's scope. **T8 wave (pgGraph-inspired CI infra, v0.37.4.0):** `traverseGraph` opts gains `frontierCap?: number` (per-iteration cap on the recursive CTE — approximately per-BFS-layer). Return type stays `Promise` for MCP wire stability. New export `TraverseGraphOpts`. Postgres path uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term; PGLite mirrors with positional params + the same shape SQL. Pinned by `test/regressions/v0_36_frontier_cap.test.ts` (4 contracts: cap-unset back-compat, cap-hit bounds result to `<= cap+1`, MCP wire-shape preservation, concurrency independence). **`onTruncation` callback designed but stripped pre-merge in /review** — adversarial pass caught false-positive (organic count == cap) + false-negative (LIMIT-before-DISTINCT in diamond graphs) cases in the v1 algorithm. Restoring the signal requires a dedupe-then-cap SQL rewrite + Postgres parity E2E — see TODOS.md → "T8 truncation signal". **v0.35.6.0:** two new methods supporting the phantom-redirect cycle pass — `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (codex #7: content_hash refresh is required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug` + `source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence — codex #3 fix for the writeFactsToFence lossy-migration trap. Both methods have engine parity tests at `test/phantom-redirect-engine-parity.test.ts`. **v0.40.4.0:** new `getAdjacencyBoosts(pageIds): Promise>` method powers the per-query graph-signals stage in hybrid search. Single SQL query returns inbound-link counts among the top-K set plus a cross-source count (links from pages whose `source_id` differs from the target's). `COALESCE(p.source_id, 'default')` for null safety; `HAVING >= 1` matches JSDoc; cross-source CASE-WHEN on the JOINed target row excludes the target's own source. Parity SQL between postgres-engine.ts + pglite-engine.ts. `SearchResult` gains 12 new optional fields (`base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta`, plus internal staging fields). Pinned by `test/e2e/graph-signals-engine.test.ts` (7 cases) + cross-engine parity in the same suite. **v0.41.25.0 (supersedes PR #1538):** two new REQUIRED methods on BrainEngine — `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted, D6) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup). `sourceId` is REQUIRED on both at the type level (D5; asymmetric with single-row `deletePage` which keeps optional/'default' fallback — v0.42+ TODO to tighten). Both methods short-circuit on empty input and throw when called with `> DELETE_BATCH_SIZE`. Closes the 73K-delete-jams-sync bottleneck for both engines. +- `src/core/engine-constants.ts` (v0.41.25.0, NEW) — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry. Same order-of-magnitude as `addLinksBatch`'s effective per-call budget — bounded per-statement work for predictable lock hold time + write amplification. - `src/core/search/graph-signals.ts` (v0.40.4.0) — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page is linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page is linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring one at full score, demote the rest). All three inherit the v0.35.6.0 floor-ratio gate that prevents weak pages from getting boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width` — instrumentation for the v0.41+ magnitude calibration wave (TODOS T-todo-2). `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (24 cases including the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension (v0.40.4.0) — `runPostFusionStages` extended with a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Every existing post-fusion stage now stamps its multiplier on the result: `applyBacklinkBoost` → `backlink_boost`, `applySalienceBoost` → `salience_boost`, `applyRecencyBoost` → `recency_boost`. `applyReranker` (called earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution is the cathedral that powers `gbrain search --explain` — every boost surface carries its own field, so `formatResultsExplain` reads them all without coupling to internal stage ordering. - `src/core/search/explain-formatter.ts` (v0.40.4.0) — renders `SearchResult[]` as a multi-line per-result breakdown for `gbrain search --explain`. Reads every boost-stamping field. Handles the "no boosts applied" empty path. 4-decimal precision with trailing-zero strip. Pinned by `test/search/explain-formatter.test.ts` (19 cases). @@ -193,8 +194,8 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). — PGLite-specific DDL (pgvector, pg_trgm, triggers) -- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding (the same idiom `addLinksBatch` already uses). Caller-chunking primitive — throws when input exceeds `DELETE_BATCH_SIZE`. Returns `RETURNING slug` rows for `deletePages` so callers can filter `pagesAffected` to confirmed deletes only. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. **v0.41.25.0 (supersedes PR #1538):** implements `deletePages(slugs, {sourceId}): Promise` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single SQL round-trip per call; caller chunks). Pair `resolveSlugsByPaths` does the `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2` batch lookup. FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`; `files.page_id` + `links.origin_page_id` go SET NULL. Pair primitive throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both methods short-circuit on empty input. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. - `src/core/embedding-pricing.ts` (v0.32.7 CJK wave) — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token approximation. Unknown providers degrade gracefully to "estimate unavailable" instead of fabricating numbers. @@ -408,7 +409,7 @@ strict behavior when unset. - `src/core/embedding-dim-check.ts` extension (v0.41.17.0, T5+T6) — facts.embedding dim drift surface. New `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (codex #19 — migration v40 falls back to `vector` on pgvector < 0.7). Regex ordering halfvec-before-vector pinned by tests (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). New `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (codex #18 — NOT bare REINDEX which doesn't rewrite the index after a column-type change). New `assertFactsEmbeddingDimMatchesConfig(engine)` is the D15 preflight — throws `FactsEmbeddingDimMismatchError` (tagged class with `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool's MUST_ABORT semantics) when configured dim doesn't match the column width. Result cached per-engine via `WeakMap`. PGLite engines silently skip. New doctor check `facts_embedding_width_consistency` (registered in `runDoctor` after `embedding_width_consistency`) reuses the same helpers — surfaces drift with paste-ready ALTER recipe identical to the preflight error. Pinned by 18 cases in `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension (v0.41.17.0, T6, codex #20) — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. New `resolveFactsEmbeddingCast()` private method probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`. Both insert paths use the cached suffix so the cast matches the actual column type. Pre-fix all three insert sites hardcoded `::vector`; works on pgvector >= 0.7 via implicit auto-cast but fails on older pgvector. Test seam `__resetFactsEmbeddingCastCacheForTest()` clears per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions (v0.41.20.0 ops-fix-wave) — six daily-driver ops pains in one bisectable wave. (1) **Batch idempotency for `extract_atoms`:** new `atomsExistingForHashes(engine, sourceId, hashes[])` exported from `src/core/cycle/extract-atoms.ts` replaces the per-hash loop that did 7K individual queries at the start of every cycle on brains with conversation-transcript corpora (5-10 min silent overhead). One batched SQL roundtrip returns the set of `content_hash16` values already extracted as atoms for this source. Fail-open: SQL error logs to stderr and returns an empty set so extraction proceeds. Powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres uses `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop mirroring v97 `pages_dedup_partial_index`, PGLite uses plain `CREATE INDEX`). Without this index the batch idempotency check would seq-scan pages on big brains and defeat the perf win. (2) **Shorter cycle lock TTL + active in-phase refresh:** `LOCK_TTL_MINUTES` dropped from 30 → 5 in `src/core/cycle.ts`. New exported `buildYieldDuringPhase(lock, outer)` closure calls `lock.refresh()` AND any external hook on every fire; throttled to 30s inside each phase via `maybeYield`. Fires both inside the main work loop AND immediately after every `await chat(...)` LLM call so long Haiku/Sonnet calls don't sit past TTL. `LockHandle` and `buildYieldDuringPhase` exported for test seam access. `synthesize_concepts` no longer fires `yieldDuringPhase` per-concept-group — same hook, throttled via the new shared `maybeYield` helper. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps the lock alive. Known residual under shorter TTL: a single `await chat()` past 5 min wallclock can expire the lock mid-await without the original phase noticing — `DbLockHandle.refresh()` throwing on 0 rows affected is filed as P2 follow-up TODO-OPS-2. (3) **Progress wiring through long phases:** new `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`. Cycle.ts passes its phase-level reporter down (NOT a child reporter — that would produce a path collision `cycle.extract_atoms.extract_atoms.work`). Phases only call `tick()` and `heartbeat()`; cycle.ts owns `start()` and `finish()`. You see `[cycle.extract_atoms] N (atoms_created)` ticks every ~1s during both long phases instead of "start" then silence for 10+ min. (4) **`by-mention` resume:** new `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts`. The gazetteer hash is the load-bearing field — adding new entity pages mid-pause shifts the hash, gets a new fingerprint, and triggers a fresh scan against the new gazetteer instead of silently skipping previously-scanned pages. `gbrain extract links --by-mention` now resumes from where it died via the existing `op_checkpoints` framework with a `flushAndCheckpoint` ordering — links flush to the DB FIRST, page keys commit to the checkpoint SECOND, persist THIRD. A crash between `batch.push()` and flush leaves the page un-checkpointed so resume re-scans it. Persist cadence: every 1000 items OR every 30s, whichever first. Clean exit clears the checkpoint. `--dry-run` deliberately skips both load and write so it stays an inspection mode. (5) **`sync_consolidation` doctor check:** multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed` recommendation. Single-source brains get "not applicable." SQL errors return `warn` via the check's own try/catch — outer doctor catch isn't a safe assumption. Companion "Multi-source brains" recipe block in `skills/cron-scheduler/SKILL.md` documents the `sync --all` pattern as preferred over per-source entries. (6) **Test-isolation fixes:** `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` migrated to per-test `GBRAIN_HOME=tempdir` isolation. Pinned by 44 new unit/PGLite cases across 9 files: `test/cycle/extract-atoms-batch.test.ts` (5 — batch idempotency), `test/cycle/cycle-lock-ttl.test.ts` (1 — regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts` (7 — fingerprint sensitivity including gazetteer-hash regression guard), `test/cycle/extract-atoms-progress.test.ts` (4 — phase doesn't call start/finish, ticks fire per item), `test/cycle/synthesize-concepts-progress.test.ts` (3 — same shape), `test/cycle/yield-during-phase-refresh.test.ts` (7 — buildYieldDuringPhase actually calls lock.refresh() + outer hook, throws non-fatal), `test/cycle/yield-during-phase-throttle.test.ts` (3 — 30s throttle gate), `test/extract-by-mention-resume.test.ts` (5 — checkpoint persistence ordering, dry-run skips persist, gazetteer change invalidates, filtered pages get checkpointed), `test/doctor-sync-consolidation.test.ts` (6 — edge case matrix for source counts + archived filtering + SQL error path). Two follow-up TODOs filed in TODOS.md under "v0.41.19.0 ops-fix-wave follow-ups": `gbrain sync print-cron` subcommand (TODO-OPS-1) and lock-loss detection via `DbLockHandle.refresh()` throwing on 0 rows affected (TODO-OPS-2). -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). v0.22.13 (PR #490): `performSync` wraps its body in a `gbrain-sync` writer lock so two concurrent syncs (manual + autopilot, two terminals, two Conductor workspaces) cannot both write `last_commit` and let the last writer win. Head-drift gate after the import phase re-checks `git rev-parse HEAD`; if HEAD moved (someone ran `git checkout` / `git pull` mid-sync), the bookmark refuses to advance. Vanished files now record a failedFiles entry instead of silent-skip — the silent-skip-then-advance pathology that survived prior hardening passes is dead. Worker engines wrap in try/finally so disconnect always fires (panic-path leak fix). Both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`), validated via `parseWorkers`. Explicit `--workers` bypasses the auto-path file-count floor; auto path defers to `autoConcurrency()`. Banner moved to stderr. **v0.34.2.0:** the inline `.sort()` over add/mod paths is replaced with `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts`, so the newest-first descending-lex policy lives in one helper shared with `gbrain import` instead of drifting across two files. **v0.40.3.0 (productionized from PR #1314 by @garrytan-agents):** the load-bearing change is the lock-identity invariant — `performSync` now defaults to a per-source lock id (`gbrain-sync:`) whenever `opts.sourceId` is set, AND wraps the writer window in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (the PR motivator: media-corpus / 250K+ chunks) don't lose their lock at the 30-min TTL mid-run. Legacy single-default-source brains keep the bare `tryAcquireDbLock(SYNC_LOCK_ID)` path for back-compat. `SyncOpts.lockId?: string` is the explicit override (escape hatch; production callers don't set it). The fix closes the bug class Codex's outside-voice review caught in the original PR — a `sync --all` worker on per-source lock racing against `sync --source foo` on global lock would have silently corrupted the same source. `gbrain sync --all` got a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)` from `src/core/sync-concurrency.ts`), `parallel` long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking from wave dispatch), per-source `withSourcePrefix(src.id, ...)` wrap from `src/core/console-prefix.ts` so every `slog`/`serr` line carries `[] ` for kubectl-style greppable parallel output. `--skip-failed` and `--retry-failed` reject with a paste-ready error when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope; v0.41+ TODO source-scopes it). Connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`, so `--parallel 4 --workers 4` is actually 32 connections, not 16 — Codex caught the 2× understatement in the original plan). New exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard surface (D3 → `sources` subcommand, not a `sync` flag, so reads and writes don't share a verb). Stable `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout under `--json`; human banners route to stderr via the `humanSink` helper so `jq` parses cleanly. Exit matrix: 0 = all ok, 1 = any error, 2 = cost-prompt-not-confirmed (unchanged). The dashboard SQL is the canonical `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` shape with `archived = false` filter at the caller — the original PR shipped `chunks ch JOIN ON page_slug`, which would have crashed on PGLite parse and silently zeroed on Postgres via the swallow-catch. Embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage / multimodal / non-default-column brains see counts against the column they actually use. Errors propagate from the dashboard SQL — no swallow-catch (Q2 sub-fix). The IRON RULE regression lives in `test/e2e/sync-status-pglite.test.ts`: real PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND the active embedding column is the one used. 38 in-file `console.log`/`console.error` call sites inside `performSync*` migrated to `slog`/`serr` (top-level `runSync` orchestrator console calls intentionally stay outside the prefix scope). **v0.41.25.0 (supersedes PR #1538):** sync delete loop rewritten as interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts`. 73K-delete commit drops from ~146K SQL round-trips (~5h on Postgres+pgbouncer) to ~292 round-trips (~2min) — closes the cascade-staleness bug class where one big-delete commit jammed every other source's sync for hours. Per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback so a transient blip on batch 73 doesn't lose 500 deletes; unrecoverable per-slug failures land in `failedFiles` (matches the existing import-loop pattern). `pagesAffected` filters to D6 confirmed-deleted slugs (downstream extract/embed stop wasting work on phantoms). Same batched slug-resolve treatment for the rename loop (`gbrain sync` renames — Phase 3 in the plan; the per-file `importFile` stays per-file, only the slug-resolve N+1 gets batched). The `resolveSlugByPathOrSourcePath` single-call helper at `sync.ts:267` now delegates to `engine.resolveSlugsByPaths` when `sourceId` is set (D8 DRY — one owner of the SQL + fallback semantics), keeping legacy `executeRaw` fallback for the no-sourceId path which is functionally dead post-v0.34.1's source-resolution wiring. Legacy no-sourceId branch in the delete loop preserves correctness on a slow path; the engine batch methods require sourceId so the new fast path only fires when production threading is intact. `failedFiles` declaration hoisted to top of `performSyncInner` so both delete decompose and import loops feed the same sync-bookmark gate. - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. v0.34.2.0 replaces the prior positional-index checkpoint (`processedIndex: N` into a sorted file list) with a path-set checkpoint via `src/core/import-checkpoint.ts`. The walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order. A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set, so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. Three bug classes died: parallel-import-with-slow-worker drops the slow file on crash-resume (closed — the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (closed — failures don't add to `completed`), and v0.33.x sort-flip-drops-newest-N-on-cross-version-resume (closed — order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume; re-walking is cheap because `content_hash` short-circuits unchanged files. Checkpoint persists every 100 successful adds, not every 100 processed files, so a long failure tail doesn't churn the JSON. Pinned by `test/import-checkpoint.test.ts` (18 unit cases over the helpers) + `test/import-resume.test.ts` (5 integration cases under PGLite, including the SLUG_MISMATCH retry regression codex caught during plan-eng-review). - `src/core/import-checkpoint.ts` (v0.34.2.0) — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set checkpoint format (`{schema_version, brainDir, completed: string[]}`) replaces the v0.33.x positional `{processedIndex: N}` format. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (you ran import against a different brain), and the old positional format (logged to stderr before being discarded). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so test isolation via `withEnv({GBRAIN_HOME: tmpdir})` works without monkey-patching the fs layer. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws, so import keeps making progress even if disk is full. - `src/core/sort-newest-first.ts` (v0.34.2.0) — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line in this helper instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (5 hermetic cases: descending order, mixed prefixes, empty input, single-element input, in-place-mutation contract). @@ -789,7 +790,7 @@ Key commands added in v0.40.3.0 (contextual retrieval + cache gate + 4 CLI verbs - KNOBS_HASH_VERSION bumped 3 → 5 (skipped past 4 reserved by salem's v0.40.4 graph-signals work). One-time cache-row invalidation on upgrade; refills within TTL. - Three new Minion handlers wired into RemediationStep consumer pattern: `lint-fix`, `integrity-auto`, `sync-retry-failed`. Thin wrappers around already-shipping CLI commands; NOT in PROTECTED_JOB_NAMES (idempotent, no shell exec, MCP-safe). `sync-skip-failed` deliberately NOT in this set per codex D12 Bug 3. - `src/core/remediation-step.ts` (NEW canonical module) exports RemediationStep type + makeRemediationStep factory + canonical-JSON idempotencyKey() per codex D12 Bug 2. Future doctor checks emit RemediationSteps via the factory instead of hand-rolling the shape. -- `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. +- `src/core/search/query-cache-gate.ts` (NEW) exports `buildPageGenerationsSnapshot(engine, pageIds)` + `CACHE_GATE_WHERE_CLAUSE` SQL fragment + `validateCacheRowAgainstPages()` pure validator. **v0.41.25.0 (codex outside-voice on /plan-eng-review):** Layer 1 bookmark read source switched from `MAX(generation) FROM pages` to `SELECT value FROM page_generation_clock WHERE id = 1` at both store + lookup sites. Closes two pre-existing silent stale-cache bug classes that were independent of any sync work: (1) CDX-2 — UPDATE to a non-max page set `NEW.generation = OLD.generation + 1` which didn't advance MAX, so cache silently served stale on every non-max UPDATE; (2) CDX-1 — DELETE didn't fire the row-level trigger at all, and even an AFTER DELETE wouldn't move MAX because surviving rows are untouched. The new clock is bumped per-statement by `bump_page_generation_clock_trg` (statement-level trigger created in migration v106) so every INSERT/UPDATE/DELETE statement advances the bookmark exactly once regardless of row cardinality (D19 — per-row would turn a 73K-row batch DELETE into 73K UPDATEs on the same counter). Also CDX-6/D20 fix: empty-result cache rows used to be "vacuously valid" via Layer 2's `qc.page_generations = '{}'::jsonb` shortcut, silently serving stale empty results across subsequent matching INSERTs. The shortcut is gone — empty snapshots now require Layer 1 to pass. `CACHE_GATE_WHERE_CLAUSE` enforces `qc.page_generations <> '{}'::jsonb` AND the per-page check (not OR). One-time post-upgrade cache miss spike on legacy `{}` rows is acceptable — the cache fills back up correctly and the clock seed `COALESCE(MAX(pages.generation), 0)` keeps non-empty legacy rows serving until the next write. Pinned by 5 new cases in `test/page-generation-counter.test.ts` (CDX-1/CDX-2/CDX-6 regressions + statement-level-trigger-fires-once contract) and 2 new e2e cases in `test/e2e/cache-gate-pglite.test.ts`. The existing pre-v0.41.25.0 `vacuously valid legacy row` IRON-RULE assertion in both files is intentionally inverted: that path was the CDX-6 bug. - `src/core/search/mode-switch-ux.ts` (NEW) exports `summarizeTransition()` 5-cell matrix + `probeWorkerAvailable()` worker liveness proxy via minion_jobs activity + `buildReindexIdempotencyKey()` content-stable key + `runModeSwitchUx()` orchestrator. Key commands added in v0.36.4.0 (brain-health-100 wave): diff --git a/package.json b/package.json index f6017e452..a47a44b51 100644 --- a/package.json +++ b/package.json @@ -140,5 +140,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.41.24.0" + "version": "0.41.25.0" } diff --git a/scripts/run-unit-parallel.sh b/scripts/run-unit-parallel.sh index f46b48cdb..007d5c9e1 100755 --- a/scripts/run-unit-parallel.sh +++ b/scripts/run-unit-parallel.sh @@ -181,28 +181,88 @@ bun_summary_count() { ' "$file" } +# shard_total_files: parse the "[unit-shard N/M] running X files" line that +# run-unit-shard.sh echoes before invoking bun test. Returns the file count +# the shard was given, or 0 if the line isn't there yet (shard still +# bootstrapping). Uses sed-then-grep so it's portable to macOS awk (BSD awk +# doesn't support `match($0, /re/, arr)` with the array sink — that's gawk-only). +shard_total_files() { + local file="$1" + [ -f "$file" ] || { echo 0; return; } + local n + n=$(sed -n 's/^\[unit-shard [0-9][0-9]*\/[0-9][0-9]*\] running \([0-9][0-9]*\) files.*/\1/p' "$file" 2>/dev/null | head -1) + echo "${n:-0}" +} + +# shard_pglite_init_count: count "Schema version" lines as a proxy for "test +# files initialized so far." Each PGLite-using test file's beforeAll triggers +# one initSchema() which prints this. Undercounts because not every test file +# opens a PGLite engine, but it's the only real-time progress signal bun's +# default reporter leaves in the log (bun has no per-file progress markers, +# only a final shard-end summary). +shard_pglite_init_count() { + local file="$1" + [ -f "$file" ] || { echo 0; return; } + grep -cE 'Schema version [0-9]+ → [0-9]+' "$file" 2>/dev/null || echo 0 +} + +# log_size_kb: total stderr+stdout written by the shard so far. Strictly +# monotonic — useful as a "definitely alive" signal when other heuristics +# read 0 (e.g. very early in shard startup before initSchema fires). +log_size_kb() { + local file="$1" + [ -f "$file" ] || { echo 0; return; } + local b + b=$(wc -c < "$file" 2>/dev/null | tr -d ' ') + echo $(( ${b:-0} / 1024 )) +} + +# fmt_elapsed: pretty-print seconds → "Mm:SS" or "SSs" for short. +fmt_elapsed() { + local s=$1 + if [ "$s" -ge 60 ]; then + printf '%dm%02ds' $((s / 60)) $((s % 60)) + else + printf '%ds' "$s" + fi +} + heartbeat() { + local hb_start=$(date +%s) while true; do sleep 10 local line="" + local now; now=$(date +%s) + local hb_elapsed=$((now - hb_start)) for i in $(seq 1 "$N"); do if [ -f "$LOG_DIR/shard-$i.exit" ]; then local rc; rc=$(cat "$LOG_DIR/shard-$i.exit" 2>/dev/null || echo "?") local status="✓" [ "$rc" != "0" ] && status="✗" - line="$line [s$i: done $status]" + local f + f=$(bun_summary_count "fail" "$LOG_DIR/shard-$i.log") + local p + p=$(bun_summary_count "pass" "$LOG_DIR/shard-$i.log") + line="$line [s$i: done $status ${p}p ${f}f]" else local lf="$LOG_DIR/shard-$i.log" if [ -f "$lf" ]; then - # Heartbeat: prefer Bun's per-test "✓" (passed) and "(fail)" markers - # so we see live progress; the "N pass" summary line only appears at - # the very end of the shard and would always show 0 mid-run. - local p f - p=$(grep_count '^[[:space:]]+✓' "$lf") - f=$(grep_count '^\(fail\)' "$lf") - line="$line [s$i: ${p}p ${f}f ...]" + # Bun's default reporter has no per-file progress markers, only a + # final shard-end summary, so we surface three complementary signals + # mid-run: (1) PGLite initSchema() count as a "files started" proxy, + # (2) total files this shard was assigned (from the runner banner), + # (3) log size in KB as a strictly-monotonic liveness signal. + local total; total=$(shard_total_files "$lf") + local pglite; pglite=$(shard_pglite_init_count "$lf") + local kb; kb=$(log_size_kb "$lf") + local et; et=$(fmt_elapsed "$hb_elapsed") + if [ "$total" -gt 0 ]; then + line="$line [s$i: ~${pglite}/${total}f ${kb}KB ${et}]" + else + line="$line [s$i: starting ${kb}KB ${et}]" + fi else - line="$line [s$i: starting]" + line="$line [s$i: spawning]" fi fi done diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 6cd37e34e..41628afe1 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -2,6 +2,7 @@ import { existsSync, readFileSync, writeFileSync, statSync } from 'fs'; import { execFileSync } from 'child_process'; import { join, relative } from 'path'; import type { BrainEngine } from '../core/engine.ts'; +import { DELETE_BATCH_SIZE } from '../core/engine-constants.ts'; import { importFile } from '../core/import-file.ts'; import { collectSyncableFiles } from './import.ts'; import { createInterface } from 'readline'; @@ -269,14 +270,25 @@ export async function resolveSlugByPathOrSourcePath( path: string, sourceId?: string, ): Promise { + // v0.41.19.0 (D8): when sourceId is set, delegate to the new batch + // resolveSlugsByPaths so single-call and batched paths share one SQL + // owner + one fallback semantic. One Map allocation per single-call; + // negligible cost. When sourceId is undefined (legacy unscoped callers), + // fall back to the original executeRaw shape — the batch method + // requires sourceId to prevent the multi-source-bug-class on its new + // surface (D5). The unscoped fallback preserves back-compat. try { - const rows = await engine.executeRaw<{ slug: string }>( - sourceId - ? `SELECT slug FROM pages WHERE source_path = $1 AND source_id = $2 LIMIT 1` - : `SELECT slug FROM pages WHERE source_path = $1 LIMIT 1`, - sourceId ? [path, sourceId] : [path], - ); - if (rows.length > 0 && rows[0].slug) return rows[0].slug; + if (sourceId) { + const m = await engine.resolveSlugsByPaths([path], { sourceId }); + const slug = m.get(path); + if (slug) return slug; + } else { + const rows = await engine.executeRaw<{ slug: string }>( + `SELECT slug FROM pages WHERE source_path = $1 LIMIT 1`, + [path], + ); + if (rows.length > 0 && rows[0].slug) return rows[0].slug; + } } catch { // Fall through — best-effort. Pre-migration brains or query errors // shouldn't break delete/rename for path-derived pages. @@ -1233,25 +1245,135 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise = []; + // v0.18.0+ multi-source: scope deletePage so we only delete the source-A // row, not every same-slug row across all sources. const deleteOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined; + + // v0.41.19.0 (T2/D6/D7/D16/D18 via /plan-eng-review + codex outside-voice): + // batched delete loop. Replaces the per-file N+1 that PR #1538 originally + // batched on Postgres only. See plan file: + // ~/.claude/plans/system-instruction-you-are-working-ethereal-narwhal.md + // + // SHAPE (interleaved per-batch resolve + delete; caller owns chunking): + // + // filtered.deleted (e.g. 73K paths) + // │ + // ▼ + // slice into batches of DELETE_BATCH_SIZE (500) + // │ + // ▼ for each batch: + // abort-check ──► partial('timeout') + // │ + // ▼ + // engine.resolveSlugsByPaths(batch, {sourceId}) ◀── 1 SQL round-trip + // │ + // ▼ + // slugs = batch.map(path => map.get(path) + // ?? resolveSlugForPath(path)) ◀── pure-JS fallback for + // │ frontmatter-fallback + // ▼ + missing-source-path + // try { + // deleted = engine.deletePages(slugs, opts) ◀── 1 SQL round-trip + // pagesAffected.push(...deleted) ◀── D6: only confirmed + // } catch { deletes, not phantoms + // // D7 decompose: per-slug deletePage, + // // unrecoverable failures → failedFiles + // } + // + // ROUND-TRIP COUNTS (73K deletes): + // pre-fix: 73,000 SELECTs + 73,000 DELETEs = 146,000 (~5 hours) + // post-fix: 146 SELECTs + 146 DELETEs = 292 (~2 minutes) + // + // ATOMICITY (D3): each batch is one transaction. A mid-batch abort or + // transient connection failure rolls back up to DELETE_BATCH_SIZE - 1 + // successful deletes. Sync is idempotent — the next run picks them up + // via git diff regenerating the deletion list. + // + // NO-SOURCEID FALLBACK: when opts.sourceId is undefined (legacy unscoped + // callers, rare post-v0.34.1 source-resolution wiring), fall back to the + // OLD per-path loop. The batch engine surface requires sourceId per D5 + // (multi-source-bug-class defense at the type level). Production callers + // that thread sourceId via resolveSourceWithTier get the new fast path. if (filtered.deleted.length > 0) { progress.start('sync.deletes', filtered.deleted.length); - for (const path of filtered.deleted) { - // v0.41.13.0 (T2 / D-V4-2): per-iteration abort check. Codex pass-3 - // F8 caught that v3 only covered pull + add/modify. Refactor commits - // with hundreds of deletes can overshoot --timeout without this check. - if (opts.signal?.aborted) { - progress.finish(); - return partial('timeout'); + if (opts.sourceId) { + const sid = opts.sourceId; + const deleteScopedOpts = { sourceId: sid }; + for (let i = 0; i < filtered.deleted.length; i += DELETE_BATCH_SIZE) { + if (opts.signal?.aborted) { + progress.finish(); + return partial('timeout'); + } + const batch = filtered.deleted.slice(i, i + DELETE_BATCH_SIZE); + + // Phase A: batch slug resolution (1 round-trip per batch). + let pathSlugMap: Map; + try { + pathSlugMap = await engine.resolveSlugsByPaths(batch, deleteScopedOpts); + } catch { + // Resolve failure: fall back to empty map; per-path fallback + // below will use resolveSlugForPath. Best-effort, matches the + // existing resolveSlugByPathOrSourcePath swallow-and-fallback + // semantics. + pathSlugMap = new Map(); + } + const slugs = batch.map(p => pathSlugMap.get(p) ?? resolveSlugForPath(p)); + + // Phase B: batch delete (1 round-trip per batch). + try { + const deleted = await engine.deletePages(slugs, deleteScopedOpts); + // D6: only push slugs that were actually deleted. Filters phantom + // slugs (paths in filtered.deleted but with no DB row) so + // downstream extract/embed don't waste lookups. + pagesAffected.push(...deleted); + } catch (err) { + // D7 decompose: a transient blip on this batch shouldn't lose all + // 500 deletes. Fall back to per-slug deletePage for THIS batch + // only; unrecoverable per-slug failures land in failedFiles + // (matching the existing import-loop pattern at sync.ts:~1350). + for (let j = 0; j < slugs.length; j++) { + try { + await engine.deletePage(slugs[j], deleteScopedOpts); + pagesAffected.push(slugs[j]); + } catch (perSlugErr) { + failedFiles.push({ + path: batch[j], + error: `delete failed: ${perSlugErr instanceof Error ? perSlugErr.message : String(perSlugErr)} (batch error: ${err instanceof Error ? err.message : String(err)})`, + }); + } + } + } + progress.tick(batch.length, `deletes ${Math.min(i + DELETE_BATCH_SIZE, filtered.deleted.length)}/${filtered.deleted.length}`); + } + } else { + // Legacy no-sourceId path. The engine batch methods require sourceId + // per D5 (kills the multi-source-bug-class on the new surface); when + // sourceId is unset, fall back to the original per-path loop. Slow + // but correct; production callers all thread sourceId so this branch + // is functionally dead post-v0.34.1. + for (const path of filtered.deleted) { + if (opts.signal?.aborted) { + progress.finish(); + return partial('timeout'); + } + const slug = await resolveSlugByPathOrSourcePath(engine, path, undefined); + try { + await engine.deletePage(slug, deleteOpts); + pagesAffected.push(slug); + } catch (err) { + failedFiles.push({ + path, + error: `delete failed: ${err instanceof Error ? err.message : String(err)}`, + }); + } + progress.tick(1, slug); } - const slug = await resolveSlugByPathOrSourcePath(engine, path, opts.sourceId); - await engine.deletePage(slug, deleteOpts); - pagesAffected.push(slug); - progress.tick(1, slug); } progress.finish(); } @@ -1260,12 +1382,46 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0) { progress.start('sync.renames', filtered.renamed.length); // v0.18.0+ multi-source: scope updateSlug so the rename only touches the // source-A row, not every same-slug row across sources (which would // either sweep them all OR violate (source_id, slug) UNIQUE). const renameOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined; + + // T4: pre-resolve ALL `from` slugs in batches before iterating. Falls + // back to per-path resolveSlugByPathOrSourcePath when sourceId is + // unset (matches the delete loop's legacy posture). For large rename + // commits (rare but possible: prefix sweep, reorganization), this drops + // the slug-resolve round-trips from O(renames) to O(renames/500). + const fromSlugByPath = new Map(); + if (opts.sourceId) { + const sid = opts.sourceId; + const fromPaths = filtered.renamed.map(r => r.from); + for (let i = 0; i < fromPaths.length; i += DELETE_BATCH_SIZE) { + if (opts.signal?.aborted) { + progress.finish(); + return partial('timeout'); + } + const batch = fromPaths.slice(i, i + DELETE_BATCH_SIZE); + let m: Map; + try { + m = await engine.resolveSlugsByPaths(batch, { sourceId: sid }); + } catch { + m = new Map(); + } + for (const p of batch) { + fromSlugByPath.set(p, m.get(p) ?? resolveSlugForPath(p)); + } + } + } + for (const { from, to } of filtered.renamed) { // v0.41.13.0 (T2 / D-V4-2): per-iteration abort check. Renames call // importFile() at line 1173-style sites which can be slow on big files; @@ -1274,7 +1430,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise = []; + // v0.41.19.0: `failedFiles` is now hoisted above the delete loop (the + // delete decompose path appends here too); kept as a comment-pin so + // future maintainers know to thread additional failure surfaces through + // the same array. const addsAndMods = [...filtered.added, ...filtered.modified]; // Sort newest-first so date-prefixed brain paths get embedded before older diff --git a/src/core/engine-constants.ts b/src/core/engine-constants.ts new file mode 100644 index 000000000..a40d2ba46 --- /dev/null +++ b/src/core/engine-constants.ts @@ -0,0 +1,25 @@ +// v0.41.25.0 — single source of truth for engine batch-sizing. +// +// Both PostgresEngine and PGLiteEngine import from here so the constants +// cannot drift across engines. Lives outside `src/core/engine.ts` to avoid +// circular-import worries (engine.ts is the interface; engines depend on +// engine.ts; this file depends on neither). + +/** + * Maximum number of slugs per single batch `DELETE FROM pages WHERE slug = + * ANY($1::text[])` call. Callers (e.g. `src/commands/sync.ts` delete loop) + * are responsible for chunking input arrays to this size; `engine.deletePages` + * is a single-batch primitive that does NOT chunk internally (matches the + * `addLinksBatch` convention — caller owns chunking, engine assumes the + * caller is well-behaved). + * + * 500 is the same order-of-magnitude as the effective per-call budget for + * the existing `addLinksBatch` (postgres-engine.ts) — well under Postgres's + * 65535 parameter cap. We bind a single array parameter so the cap doesn't + * bite directly, but per-statement work stays bounded for predictable lock + * hold time + write-amplification budget. + * + * The same constant is also used for batch `SELECT slug, source_path FROM + * pages WHERE source_path = ANY($1::text[])` in `engine.resolveSlugsByPaths`. + */ +export const DELETE_BATCH_SIZE = 500; diff --git a/src/core/engine.ts b/src/core/engine.ts index b6187a196..fa2f0d015 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -712,8 +712,67 @@ export interface BrainEngine { * delete doesn't hard-delete the same-slug pages in sources B/C/D. Without * it, the bare DELETE matches every row with that slug across all sources. * Cascades through content_chunks / page_links / chunk_relations via FKs. + * + * v0.41.19.0 (CDX-11): single-row primitive used by `purgeDeletedPages`, + * `gbrain sync` (one path per call), test setup teardown, and the v0.41.19.0 + * sync-delete decompose path (when `deletePages` throws on a 500-row batch, + * the sync loop falls back to per-slug `deletePage` to log unrecoverable + * failures to `failedFiles`). `gbrain sync` calls this on EVERY run that + * sees a deleted file — it is NOT admin-only. */ deletePage(slug: string, opts?: { sourceId?: string }): Promise; + /** + * v0.41.19.0 — batch delete: single SQL round-trip via + * `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 + * RETURNING slug`. Cascades through content_chunks / page_links (×3) / + * tags / raw_data / timeline_entries / page_versions via FKs declared in + * `src/schema.sql`. `files.page_id` and `links.origin_page_id` go SET + * NULL per their FK definitions. + * + * SINGLE-BATCH PRIMITIVE: caller is responsible for chunking the input to + * `<= DELETE_BATCH_SIZE` entries per call (see + * `src/core/engine-constants.ts`). Matches the `addLinksBatch` convention + * — engine assumes well-behaved input, caller owns the slicing. + * + * Returns the slugs of rows ACTUALLY DELETED (order undefined). Callers + * use this to filter their own `pagesAffected` tracking so downstream + * phases don't waste lookups on phantom slugs (paths that were in the + * deletion list but had no DB row). + * + * ATOMICITY: one statement, one transaction. The whole batch commits or + * the whole batch rolls back. Coarser than the per-row `deletePage` + * cadence — a mid-loop abort or transient connection failure can roll + * back up to `DELETE_BATCH_SIZE - 1` successful deletes from the + * in-flight batch. `gbrain sync` is idempotent (next run picks them up + * via git diff); other callers should account for the contract. + * + * sourceId is REQUIRED (no `'default'` fallback). This is asymmetric with + * `deletePage` (which keeps the optional/'default' fallback for back- + * compat). Filed as v0.42+ TODO to tighten `deletePage` to match once a + * full caller audit confirms every site threads `sourceId`. + */ + deletePages(slugs: string[], opts: { sourceId: string }): Promise; + /** + * v0.41.19.0 — batch path → slug resolution. Single SQL round-trip via + * `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) + * AND source_id = $2`. Returns `Map`; paths NOT in the map + * have no `source_path` row in the DB and the caller is expected to fall + * back to `resolveSlugForPath(path)` for the path-derived slug. + * + * Mirrors the contract of the single-call `resolveSlugByPathOrSourcePath` + * helper in `src/commands/sync.ts`, batched. As of v0.41.19.0, that + * single-call helper is implemented on top of this method (one Map + * allocation per single-path call; negligible cost; one owner of the SQL + * + fallback semantics). + * + * SINGLE-BATCH PRIMITIVE: caller chunks to `<= DELETE_BATCH_SIZE`. + * + * Empty `paths` short-circuits to an empty Map without touching the DB. + */ + resolveSlugsByPaths( + paths: string[], + opts: { sourceId: string }, + ): Promise>; /** * v0.26.5 — set `deleted_at = now()` on a page. Returns the slug if a row * was soft-deleted, null if no row matched (already soft-deleted OR not found). diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 2b7e17014..4de08e538 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -4856,6 +4856,70 @@ export const MIGRATIONS: Migration[] = [ ON extract_rollup_7d (day); `, }, + { + version: 107, + name: 'page_generation_clock_and_statement_trigger', + // v0.41.25.0 (D18/D19, codex outside-voice on /plan-eng-review): global + // page-generation clock + statement-level trigger. + // + // Renumbered v104 → v105 → v106 → v107 during master merges: + // PR #1545 (v0.41.21.0 ops-fix-wave) took v104 for pages_atom_source_hash_idx; + // PR #1542 (v0.41.22.0 type-unification cathedral) took v105 for slug_aliases; + // PR #1541 (v0.41.23.0 extract operator surfaces) took v106 for extract_rollup_7d_table. + // + // Why this exists: the pre-v0.41.25.0 query-cache Layer 1 bookmark read + // `MAX(generation) FROM pages` to detect "writes happened since cache + // store". Two bugs in that contract — independent of any sync work: + // + // 1. The row-level `bump_page_generation_trg` (migration v91) sets + // `NEW.generation = OLD.generation + 1` on UPDATE. Updating a + // NON-MAX page didn't advance MAX(generation). Cache silently + // served stale results for any UPDATE-to-non-max page. + // 2. The trigger is BEFORE INSERT OR UPDATE — DELETE doesn't fire it + // at all. Even an AFTER DELETE wouldn't move MAX (surviving rows + // are untouched). + // + // The fix: single-row counter, bumped per-statement (FOR EACH STATEMENT + // — row-level would turn a 73K-row batch DELETE into 73K UPDATEs on the + // same counter, recreating the bottleneck the sync-delete wave is + // fixing in this same PR). Layer 1 reads page_generation_clock.value + // directly. Per-row pages.generation stays for Layer 2 (per-page + // snapshot via jsonb_each + LEFT JOIN pages) which doesn't care about + // MAX, only per-page advancement. + // + // Seeded with COALESCE(MAX(pages.generation), 0) so existing + // query_cache rows stored under the old MAX semantics aren't all + // instantly invalidated on upgrade. Their max_generation_at_store + // stamp compares cleanly against the seeded clock; future writes bump + // the clock and the bookmark fires correctly. + // + // Mirror lives in src/core/pglite-schema.ts (fresh-install path). + // Forward-reference bootstrap probe in applyForwardReferenceBootstrap + // on both engines so pre-v0.41.25.0 brains pick it up cleanly. + idempotent: true, + sql: ` + CREATE TABLE IF NOT EXISTS page_generation_clock ( + id INTEGER PRIMARY KEY CHECK (id = 1), + value BIGINT NOT NULL DEFAULT 0 + ); + INSERT INTO page_generation_clock (id, value) + VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) + ON CONFLICT (id) DO NOTHING; + + CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ + BEGIN + UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + RETURN NULL; + END; + $func$ LANGUAGE plpgsql; + + DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; + CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index e6dacafcf..d42a3c1e1 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -22,6 +22,7 @@ import { logBatchRetry as auditLogBatchRetry, logBatchExhausted as auditLogBatch import { runMigrations } from './migrate.ts'; import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts'; import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts'; +import { DELETE_BATCH_SIZE } from './engine-constants.ts'; import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts'; import type { Page, PageInput, PageFilters, PageType, @@ -900,6 +901,49 @@ export class PGLiteEngine implements BrainEngine { ); } + /** + * v0.41.19.0 — batch delete primitive. See BrainEngine.deletePages JSDoc. + * Parity implementation with PostgresEngine.deletePages. PGLite supports + * `slug = ANY($1)` array-param binding natively (addLinksBatch already + * proves this). + */ + async deletePages(slugs: string[], opts: { sourceId: string }): Promise { + if (slugs.length === 0) return []; + if (slugs.length > DELETE_BATCH_SIZE) { + throw new Error( + `deletePages: input size ${slugs.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`, + ); + } + const { rows } = await this.db.query<{ slug: string }>( + 'DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug', + [slugs, opts.sourceId], + ); + return rows.map(r => r.slug); + } + + /** + * v0.41.19.0 — batch path → slug resolution. See BrainEngine.resolveSlugsByPaths + * JSDoc. + */ + async resolveSlugsByPaths( + paths: string[], + opts: { sourceId: string }, + ): Promise> { + if (paths.length === 0) return new Map(); + if (paths.length > DELETE_BATCH_SIZE) { + throw new Error( + `resolveSlugsByPaths: input size ${paths.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`, + ); + } + const { rows } = await this.db.query<{ slug: string; source_path: string }>( + 'SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2', + [paths, opts.sourceId], + ); + const m = new Map(); + for (const r of rows) m.set(r.source_path, r.slug); + return m; + } + async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> { // Idempotent-as-null: only flip rows currently active. Source filter is // optional; without it the first matching row across sources gets soft-deleted. diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index b844f2ce9..07c5c5714 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -144,6 +144,32 @@ CREATE TRIGGER bump_page_generation_trg FOR EACH ROW EXECUTE FUNCTION bump_page_generation_fn(); +-- v0.41.19.0 (D18/D19, mirror of src/schema.sql): global page-generation +-- clock + statement-level trigger. See src/schema.sql for the full +-- rationale comment. Layer 1 bookmark reads page_generation_clock.value; +-- per-row pages.generation above stays as the Layer 2 (per-page snapshot) +-- substrate. +CREATE TABLE IF NOT EXISTS page_generation_clock ( + id INTEGER PRIMARY KEY CHECK (id = 1), + value BIGINT NOT NULL DEFAULT 0 +); +INSERT INTO page_generation_clock (id, value) + VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) + ON CONFLICT (id) DO NOTHING; + +CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ +BEGIN + UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + RETURN NULL; +END; +$func$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; +CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type); CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter); CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops); diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index ad1c247a4..e98cb25a7 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -57,6 +57,7 @@ import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, pa import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts'; import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts'; +import { DELETE_BATCH_SIZE } from './engine-constants.ts'; function escapeSqlStringLiteral(value: string): string { return value.replace(/'/g, "''"); @@ -911,6 +912,53 @@ export class PostgresEngine implements BrainEngine { await sql`DELETE FROM pages WHERE slug = ${slug} AND source_id = ${sourceId}`; } + /** + * v0.41.19.0 — batch delete primitive. See BrainEngine.deletePages JSDoc. + * Single SQL round-trip per call; caller is responsible for chunking input + * to <= DELETE_BATCH_SIZE. RETURNING slug projects the actually-deleted set + * so the caller can filter pagesAffected. + */ + async deletePages(slugs: string[], opts: { sourceId: string }): Promise { + if (slugs.length === 0) return []; + if (slugs.length > DELETE_BATCH_SIZE) { + throw new Error( + `deletePages: input size ${slugs.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`, + ); + } + const sql = this.sql; + const rows = await sql<{ slug: string }[]>` + DELETE FROM pages + WHERE slug = ANY(${slugs}::text[]) AND source_id = ${opts.sourceId} + RETURNING slug + `; + return rows.map(r => r.slug); + } + + /** + * v0.41.19.0 — batch path → slug resolution. See BrainEngine.resolveSlugsByPaths + * JSDoc. Single SQL round-trip; folds rows into a Map. + */ + async resolveSlugsByPaths( + paths: string[], + opts: { sourceId: string }, + ): Promise> { + if (paths.length === 0) return new Map(); + if (paths.length > DELETE_BATCH_SIZE) { + throw new Error( + `resolveSlugsByPaths: input size ${paths.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`, + ); + } + const sql = this.sql; + const rows = await sql<{ slug: string; source_path: string }[]>` + SELECT slug, source_path + FROM pages + WHERE source_path = ANY(${paths}::text[]) AND source_id = ${opts.sourceId} + `; + const m = new Map(); + for (const r of rows) m.set(r.source_path, r.slug); + return m; + } + async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> { const sql = this.sql; const sourceId = opts?.sourceId; diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index 3b863840c..b48847389 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -185,6 +185,48 @@ CREATE TRIGGER bump_page_generation_trg -- CREATE INDEX since the table is empty. CREATE INDEX IF NOT EXISTS pages_generation_idx ON pages (generation); +-- v0.41.19.0 (D18/D19, codex outside-voice): global page-generation clock. +-- The pre-v0.41.19.0 Layer 1 bookmark read \`MAX(generation) FROM pages\` to +-- detect "writes happened since cache-store". Two bugs in that contract: +-- 1. The row-level trigger above sets \`NEW.generation = OLD.generation + 1\` +-- on UPDATE. Updating a NON-MAX page didn't advance MAX(generation), +-- silently serving stale cached results. +-- 2. The trigger is \`BEFORE INSERT OR UPDATE\` so DELETE doesn't fire it +-- at all — and even if it did, DELETE doesn't touch surviving rows, +-- so MAX(generation) wouldn't budge. +-- +-- The fix: a single-row counter, bumped per-statement (FOR EACH STATEMENT +-- — codex CDX-4: per-row would turn 73K-row batch DELETE into 73K UPDATEs +-- on the same counter, recreating the bottleneck this PR is fixing). Layer +-- 1 reads \`page_generation_clock.value\` directly. The per-row +-- \`pages.generation\` column above stays as the Layer 2 (per-page snapshot) +-- substrate. +-- +-- Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache +-- rows stored under the old MAX semantics aren't all instantly invalidated +-- on upgrade (their max_generation_at_store stamp compares cleanly against +-- the seeded clock; future writes bump the clock, bookmark fires correctly). +CREATE TABLE IF NOT EXISTS page_generation_clock ( + id INTEGER PRIMARY KEY CHECK (id = 1), + value BIGINT NOT NULL DEFAULT 0 +); +INSERT INTO page_generation_clock (id, value) + VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) + ON CONFLICT (id) DO NOTHING; + +CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS \$func\$ +BEGIN + UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + RETURN NULL; +END; +\$func\$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; +CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type); CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter); CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops); @@ -266,6 +308,13 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding_image CREATE INDEX IF NOT EXISTS idx_chunks_search_vector ON content_chunks USING GIN(search_vector); CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL; +-- v0.41.18.0 (codex finding #9): partial index for \`gbrain embed --stale\` +-- + \`--priority recent\`. content_chunks has no updated_at column (chunks +-- are re-INSERTed on page change, not UPDATEd), so the "recent-first" +-- ORDER BY happens at the JOIN site: outer ORDER BY p.updated_at DESC +-- uses idx_pages_updated_at_desc; inner partial uses this index. +CREATE INDEX IF NOT EXISTS content_chunks_stale_idx + ON content_chunks(page_id, chunk_index) WHERE embedding IS NULL; -- v0.20.0 Cathedral II: chunk-grain FTS trigger. -- Weight 'A' on doc_comment + symbol_name_qualified; weight 'B' on chunk_text. @@ -357,10 +406,16 @@ CREATE TABLE IF NOT EXISTS links ( to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE, link_type TEXT NOT NULL DEFAULT '', context TEXT NOT NULL DEFAULT '', - -- v0.42.0.0: 'mentions' added for auto-linked body-text mentions + -- v0.41.18.0: 'mentions' added for auto-linked body-text mentions -- (gbrain extract links --by-mention). Filtered OUT of backlink-count -- for search ranking; only counts toward orphan-ratio + graph traversal. link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')), + -- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from + -- "verb-pattern-derived typed link" within link_source='mentions'. + -- Codex finding #12 design: keep link_source stable; add link_kind + -- so callers can distinguish without breaking existing mentions queries. + -- NULL = legacy / unknown / pre-v98 row (semantically 'plain'). + link_kind TEXT CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner')), origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL, origin_field TEXT, -- v0.18.0 Step 4: 'qualified' when the link was written as @@ -423,8 +478,10 @@ CREATE TABLE IF NOT EXISTS timeline_entries ( CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id); CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date); --- Dedup constraint: same (page, date, summary) treated as same event -CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary); +-- v0.41.18.0 (codex finding #11): widened from (page_id, date, summary) to +-- include \`source\` so distinct meeting provenance survives. Legacy rows +-- have source='' (schema default) so legacy dedup behavior is preserved. +CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary, source); -- ============================================================ -- page_versions: snapshot history for compiled_truth @@ -589,6 +646,10 @@ CREATE TABLE IF NOT EXISTS op_checkpoints ( CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx ON op_checkpoints (updated_at); +-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676) +-- because its \`job_id BIGINT REFERENCES minion_jobs(id)\` FK requires +-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix. + -- ============================================================ -- files: binary attachments stored in Supabase Storage -- ============================================================ @@ -777,6 +838,50 @@ CREATE TABLE IF NOT EXISTS minion_attachments ( CREATE INDEX IF NOT EXISTS idx_minion_attachments_job ON minion_attachments (job_id); ALTER TABLE minion_attachments ALTER COLUMN content SET STORAGE EXTERNAL; +-- ============================================================ +-- migration_impact_log: before/after metric stats per onboard remediation +-- ============================================================ +-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the +-- onboard remediation pipeline records before/after metric stats so +-- \`gbrain onboard --history --json\` can show "you reduced orphans 47%". +-- delta computed at read time (NOT a stored GENERATED column — +-- zero PGLite parity risk per eng-review D2). +-- +-- Attribution columns (job_id, source_id, brain_id, started_at, +-- idempotency_key) per codex finding #10 so concurrent onboard / +-- autopilot / manual runs can't misattribute deltas to the wrong +-- migration when overlapping runs change the same metric. +-- +-- v0.41.25.0 SCHEMA_SQL ordering fix: this block lives AFTER the +-- minion_jobs CREATE TABLE so the \`job_id REFERENCES minion_jobs(id)\` +-- FK can resolve on fresh-install schema replay. Originally placed above +-- minion_jobs in v0.41.18.0; that fired ERROR: relation "minion_jobs" +-- does not exist on every fresh-install initSchema() (silent on master +-- because postgres-js's unsafe() continued past the error, but the +-- table never got created so any later query on migration_impact_log +-- threw 42P01 — which cascaded as "relation minion_jobs does not exist" +-- whenever subsequent statements that referenced minion_jobs ran AFTER +-- the failed CREATE TABLE statement, aborting the entire SCHEMA_SQL batch). +CREATE TABLE IF NOT EXISTS migration_impact_log ( + id BIGSERIAL PRIMARY KEY, + remediation_id TEXT NOT NULL, + metric_name TEXT NOT NULL, + metric_before NUMERIC, + metric_after NUMERIC, + job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL, + source_id TEXT, + brain_id TEXT, + started_at TIMESTAMPTZ, + idempotency_key TEXT, + applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + applied_by TEXT, + details JSONB DEFAULT '{}'::jsonb +); +CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx + ON migration_impact_log(remediation_id, applied_at DESC); +CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx + ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL; + -- ============================================================ -- Subagent runtime (v0.16.0) — durable LLM loops -- ============================================================ diff --git a/src/core/search/query-cache-gate.ts b/src/core/search/query-cache-gate.ts index 4963ff7b7..be2ec3607 100644 --- a/src/core/search/query-cache-gate.ts +++ b/src/core/search/query-cache-gate.ts @@ -1,27 +1,48 @@ /** - * Cache invalidation gate (v0.40.3.0 — D2 + D6 + D11) + * Cache invalidation gate (v0.40.3.0 base — D2 + D6 + D11; v0.41.19.0 + * rewrite — D18 + D20, codex CDX-1/CDX-2/CDX-5/CDX-6). * * Two pure helpers wired by query-cache.ts at store + lookup time. Pure * surface lets us unit-test the two-layer gate logic without a real cache. * - * Layer 1 (cheap bookmark): `MAX(generation) FROM pages` <= - * `query_cache.max_generation_at_store`. If true, brain has not been - * written since this row stored, so the row is fresh corpus-wide. + * Layer 1 (cheap bookmark): `page_generation_clock.value` <= + * `query_cache.max_generation_at_store`. If true, no page write has + * happened since this row stored, so the row is fresh corpus-wide. * * Layer 2 (per-page snapshot): if bookmark fires, fall through to the * `page_generations JSONB` snapshot. For each `(page_id, stored_gen)` * pair, compare to current `pages.generation`. Any mismatch (page - * deleted, page bumped) invalidates. + * deleted, page bumped) invalidates. **Empty `page_generations = {}` + * does NOT pass Layer 2 in v0.41.19.0+** — empty snapshots have no + * per-page signal to invalidate against, so they MUST rely on Layer 1 + * exclusively (CDX-6 fix: pre-v0.41.19.0 the vacuous-valid path let + * empty-result cache rows survive across writes that should have + * invalidated them). + * + * Why the rewrite: pre-v0.41.19.0 Layer 1 read `MAX(generation) FROM + * pages`, but the per-row trigger sets `NEW.generation = OLD.generation + * + 1` on UPDATE. Updating a NON-MAX page didn't advance MAX(generation), + * so the bookmark silently passed stale cache rows (codex CDX-2). DELETE + * doesn't fire the trigger AT ALL, so deletion didn't advance MAX either + * (codex CDX-1). Migration v105 introduces a global single-row counter + * (`page_generation_clock`) bumped per-statement by a separate trigger; + * Layer 1 now reads the counter directly so every INSERT/UPDATE/DELETE + * statement advances the bookmark exactly once regardless of which rows + * changed. * * Backward compat: rows stored before v0.40.3.0 have * `max_generation_at_store = 0` AND `page_generations = '{}'::jsonb`. - * Bookmark check: `MAX <= 0` is false on any populated brain, so we fall - * through to Layer 2; Layer 2 sees `'{}'::jsonb` and is vacuously valid. - * Legacy rows continue to serve naturally (IRON-RULE regression pinned - * in test/e2e/cache-gate-pglite.test.ts). + * Layer 1 check on a populated brain: `clock > 0` so Layer 1 fails. + * Layer 2 v0.41.19.0+ stricter: empty `{}` no longer passes — legacy + * rows invalidate on first post-upgrade lookup. Cache fills back up + * naturally; correct semantics restored. The pre-v0.41.19.0 IRON-RULE + * "legacy rows serve via vacuously-valid Layer 2" is intentionally + * reversed: that path was the CDX-6 bug. * - * See plan ~/.claude/plans/system-instruction-you-are-working-enchanted-mountain.md - * Phase 2A for full design. + * Per-page `pages.generation` column + its row-level trigger + * (`bump_page_generation_trg`) stay in place. Layer 2 reads from them + * — Layer 2 only needs per-page advancement (which the row-level trigger + * delivers correctly), NOT a MAX-style aggregate. */ import type { BrainEngine } from '../engine.ts'; @@ -73,14 +94,20 @@ export async function buildPageGenerationsSnapshot( try { if (pageIds.length === 0) { + // Empty-result query: only need the Layer 1 bookmark (clock value). + // Per D20, empty-result cache rows trust Layer 1 exclusively; + // bumping the clock on subsequent writes correctly invalidates them. const rows = await engine.executeRaw<{ v: number }>( - `SELECT COALESCE(MAX(generation), 0)::bigint AS v FROM pages`, + `SELECT COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v`, ); snapshot.max_generation_at_store = Number(rows[0]?.v ?? 0); return snapshot; } - // Combined query: per-page generation + corpus-state MAX. + // Combined query: per-page generation (Layer 2 substrate) + global + // clock value (Layer 1 bookmark). UNION ALL folds both into one + // round trip. The 'CLOCK' tag row is identified by `is_max = true` + // (field name preserved for back-compat at the call site). const rows = await engine.executeRaw<{ k: string; v: number; @@ -89,8 +116,9 @@ export async function buildPageGenerationsSnapshot( `SELECT id::text AS k, generation::bigint AS v, FALSE AS is_max FROM pages WHERE id = ANY($1::int[]) UNION ALL - SELECT 'MAX' AS k, COALESCE(MAX(generation), 0)::bigint AS v, TRUE AS is_max - FROM pages`, + SELECT 'CLOCK' AS k, + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v, + TRUE AS is_max`, [pageIds], ); @@ -104,10 +132,12 @@ export async function buildPageGenerationsSnapshot( } return snapshot; } catch { - // Pre-v91 brain (no `generation` column yet). Return the - // backward-compat empty snapshot with zero bookmark — every cache - // row will fall through to Layer 2 and serve via the `'{}'::jsonb` - // vacuously-valid path. Closes the upgrade-path gap. + // Pre-v105 brain (no `page_generation_clock` table yet). Return the + // empty snapshot with zero bookmark — every cache row will fall + // through to Layer 2 (which is stricter post-v0.41.19.0 and will + // invalidate empty snapshots). Acceptable upgrade-path one-time + // cache miss; migration v105 fills the table within the same + // initSchema() call so this branch is short-lived. return snapshot; } } @@ -127,17 +157,22 @@ export async function buildPageGenerationsSnapshot( */ export const CACHE_GATE_WHERE_CLAUSE = ` ( - -- Layer 1 (cheap bookmark): O(log N) MAX(generation) via pages_generation_idx. - -- If no page has been bumped since this row stored, the row is fresh. - (SELECT COALESCE(MAX(generation), 0) FROM pages) <= qc.max_generation_at_store + -- Layer 1 (cheap bookmark): O(1) single-row read from page_generation_clock. + -- Bumped per-statement by bump_page_generation_clock_trg on every INSERT, + -- UPDATE, or DELETE on pages. If no statement has fired since this row + -- stored, the row is fresh corpus-wide. + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0) + <= qc.max_generation_at_store OR - -- Layer 2 (per-page snapshot): bookmark fired, but maybe this row's - -- specific result set isn't affected. Pre-v0.40.3.0 rows have - -- page_generations = '{}'::jsonb and serve vacuously (legacy compat — - -- IRON-RULE regression in test/e2e/cache-gate-pglite.test.ts). + -- Layer 2 (per-page snapshot): bookmark fired, but maybe THIS row's + -- specific result set isn't affected. v0.41.19.0+ requires the snapshot + -- to be non-empty — empty {} snapshots cannot disprove staleness, so + -- they invalidate when Layer 1 fails (D20 / codex CDX-6 fix). Per-page + -- mismatch (page deleted via LEFT JOIN p.id IS NULL, or page generation + -- bumped) invalidates. ( - qc.page_generations = '{}'::jsonb - OR NOT EXISTS ( + qc.page_generations <> '{}'::jsonb + AND NOT EXISTS ( SELECT 1 FROM jsonb_each(qc.page_generations) AS g(page_id, stored_gen) LEFT JOIN pages p ON p.id = (g.page_id)::int @@ -167,12 +202,18 @@ export function validateCacheRowAgainstPages( page_generations: Record; }, ): boolean { - // Layer 1: bookmark. + // Layer 1: bookmark. `current.max_generation` is the global clock value + // (kept named max_generation for back-compat at call sites; the underlying + // read source switched from MAX(pages.generation) to + // page_generation_clock.value in v0.41.19.0). if (current.max_generation <= snapshot.max_generation_at_store) return true; - // Layer 2: per-page snapshot. + // Layer 2 (v0.41.19.0+ stricter, D20 / codex CDX-6): empty per-page + // snapshots cannot disprove staleness, so they invalidate when Layer 1 + // fails. Pre-v0.41.19.0 callers got a "vacuously valid" pass that + // silently served stale empty-result rows across writes. const ids = Object.keys(snapshot.page_generations); - if (ids.length === 0) return true; // Vacuously valid (legacy + zero-page). + if (ids.length === 0) return false; for (const id of ids) { const storedGen = snapshot.page_generations[id]; diff --git a/src/schema.sql b/src/schema.sql index 5b7921c60..b5f874854 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -181,6 +181,48 @@ CREATE TRIGGER bump_page_generation_trg -- CREATE INDEX since the table is empty. CREATE INDEX IF NOT EXISTS pages_generation_idx ON pages (generation); +-- v0.41.19.0 (D18/D19, codex outside-voice): global page-generation clock. +-- The pre-v0.41.19.0 Layer 1 bookmark read `MAX(generation) FROM pages` to +-- detect "writes happened since cache-store". Two bugs in that contract: +-- 1. The row-level trigger above sets `NEW.generation = OLD.generation + 1` +-- on UPDATE. Updating a NON-MAX page didn't advance MAX(generation), +-- silently serving stale cached results. +-- 2. The trigger is `BEFORE INSERT OR UPDATE` so DELETE doesn't fire it +-- at all — and even if it did, DELETE doesn't touch surviving rows, +-- so MAX(generation) wouldn't budge. +-- +-- The fix: a single-row counter, bumped per-statement (FOR EACH STATEMENT +-- — codex CDX-4: per-row would turn 73K-row batch DELETE into 73K UPDATEs +-- on the same counter, recreating the bottleneck this PR is fixing). Layer +-- 1 reads `page_generation_clock.value` directly. The per-row +-- `pages.generation` column above stays as the Layer 2 (per-page snapshot) +-- substrate. +-- +-- Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache +-- rows stored under the old MAX semantics aren't all instantly invalidated +-- on upgrade (their max_generation_at_store stamp compares cleanly against +-- the seeded clock; future writes bump the clock, bookmark fires correctly). +CREATE TABLE IF NOT EXISTS page_generation_clock ( + id INTEGER PRIMARY KEY CHECK (id = 1), + value BIGINT NOT NULL DEFAULT 0 +); +INSERT INTO page_generation_clock (id, value) + VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) + ON CONFLICT (id) DO NOTHING; + +CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ +BEGIN + UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + RETURN NULL; +END; +$func$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; +CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type); CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter); CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops); @@ -600,38 +642,9 @@ CREATE TABLE IF NOT EXISTS op_checkpoints ( CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx ON op_checkpoints (updated_at); --- ============================================================ --- migration_impact_log: before/after metric stats per onboard remediation --- ============================================================ --- v0.41.18.0 (gbrain onboard wave). Every completion captured by the --- onboard remediation pipeline records before/after metric stats so --- `gbrain onboard --history --json` can show "you reduced orphans 47%". --- delta computed at read time (NOT a stored GENERATED column — --- zero PGLite parity risk per eng-review D2). --- --- Attribution columns (job_id, source_id, brain_id, started_at, --- idempotency_key) per codex finding #10 so concurrent onboard / --- autopilot / manual runs can't misattribute deltas to the wrong --- migration when overlapping runs change the same metric. -CREATE TABLE IF NOT EXISTS migration_impact_log ( - id BIGSERIAL PRIMARY KEY, - remediation_id TEXT NOT NULL, - metric_name TEXT NOT NULL, - metric_before NUMERIC, - metric_after NUMERIC, - job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL, - source_id TEXT, - brain_id TEXT, - started_at TIMESTAMPTZ, - idempotency_key TEXT, - applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), - applied_by TEXT, - details JSONB DEFAULT '{}'::jsonb -); -CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx - ON migration_impact_log(remediation_id, applied_at DESC); -CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx - ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL; +-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676) +-- because its `job_id BIGINT REFERENCES minion_jobs(id)` FK requires +-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix. -- ============================================================ -- files: binary attachments stored in Supabase Storage @@ -821,6 +834,50 @@ CREATE TABLE IF NOT EXISTS minion_attachments ( CREATE INDEX IF NOT EXISTS idx_minion_attachments_job ON minion_attachments (job_id); ALTER TABLE minion_attachments ALTER COLUMN content SET STORAGE EXTERNAL; +-- ============================================================ +-- migration_impact_log: before/after metric stats per onboard remediation +-- ============================================================ +-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the +-- onboard remediation pipeline records before/after metric stats so +-- `gbrain onboard --history --json` can show "you reduced orphans 47%". +-- delta computed at read time (NOT a stored GENERATED column — +-- zero PGLite parity risk per eng-review D2). +-- +-- Attribution columns (job_id, source_id, brain_id, started_at, +-- idempotency_key) per codex finding #10 so concurrent onboard / +-- autopilot / manual runs can't misattribute deltas to the wrong +-- migration when overlapping runs change the same metric. +-- +-- v0.41.25.0 SCHEMA_SQL ordering fix: this block lives AFTER the +-- minion_jobs CREATE TABLE so the `job_id REFERENCES minion_jobs(id)` +-- FK can resolve on fresh-install schema replay. Originally placed above +-- minion_jobs in v0.41.18.0; that fired ERROR: relation "minion_jobs" +-- does not exist on every fresh-install initSchema() (silent on master +-- because postgres-js's unsafe() continued past the error, but the +-- table never got created so any later query on migration_impact_log +-- threw 42P01 — which cascaded as "relation minion_jobs does not exist" +-- whenever subsequent statements that referenced minion_jobs ran AFTER +-- the failed CREATE TABLE statement, aborting the entire SCHEMA_SQL batch). +CREATE TABLE IF NOT EXISTS migration_impact_log ( + id BIGSERIAL PRIMARY KEY, + remediation_id TEXT NOT NULL, + metric_name TEXT NOT NULL, + metric_before NUMERIC, + metric_after NUMERIC, + job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL, + source_id TEXT, + brain_id TEXT, + started_at TIMESTAMPTZ, + idempotency_key TEXT, + applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + applied_by TEXT, + details JSONB DEFAULT '{}'::jsonb +); +CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx + ON migration_impact_log(remediation_id, applied_at DESC); +CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx + ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL; + -- ============================================================ -- Subagent runtime (v0.16.0) — durable LLM loops -- ============================================================ diff --git a/test/e2e/cache-gate-pglite.test.ts b/test/e2e/cache-gate-pglite.test.ts index 2cbb3fa60..31097854e 100644 --- a/test/e2e/cache-gate-pglite.test.ts +++ b/test/e2e/cache-gate-pglite.test.ts @@ -136,15 +136,13 @@ describe('cache gate end-to-end (PGLite)', () => { expect(hit.hit).toBe(true); }); - test('legacy row (pre-v0.40.3.0 shape) serves normally — IRON-RULE backward compat', async () => { + test('v0.41.19.0 D20/CDX-6 inversion: legacy row (pre-v0.40.3.0 shape) invalidates when clock advances', async () => { const p1 = await seedPage('test/p1', 'gamma delta'); const emb = fakeEmbedding(4); const results: SearchResult[] = [ { page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'g', score: 1.0 } as unknown as SearchResult, ]; - // Simulate a pre-v0.40.3.0 row by writing with the new gate then - // hand-mutating page_generations + max_generation_at_store to the - // legacy shape. + // Simulate a pre-v0.40.3.0 row: empty snapshot + zero bookmark. await cache.store('gamma delta', emb, results, fakeMeta(), { sourceId: 'default' }); await engine.executeRaw( `UPDATE query_cache @@ -152,14 +150,66 @@ describe('cache gate end-to-end (PGLite)', () => { max_generation_at_store = 0`, ); - // Now write a bunch of content so MAX(generation) > 0. The legacy - // row's bookmark (0) is less than MAX, so bookmark fires; Layer 2 - // sees empty snapshot → vacuously valid → row serves. + // Write more pages. The global clock advances on every statement + // (statement-level trigger from migration v105). Pre-v0.41.19.0 the + // empty snapshot served vacuously here — that was the CDX-6 bug. Now: + // Layer 1 fails (clock > 0), Layer 2 rejects empty snapshots, row + // invalidates. Acceptable one-time post-upgrade cache miss; correct + // semantics restored. await seedPage('test/p2', 'unrelated bump'); await seedPage('test/p3', 'another unrelated bump'); const hit = await cache.lookup(emb, { sourceId: 'default' }); - expect(hit.hit).toBe(true); // Legacy compat — pre-upgrade rows still serve. + expect(hit.hit).toBe(false); + }); + + test('v0.41.19.0 CDX-1 regression: DELETE bumps clock → cached query for surviving pages invalidates', async () => { + const p1 = await seedPage('test/p1', 'phi chi'); + const p2 = await seedPage('test/p2', 'phi chi extra'); + const results: SearchResult[] = [ + { page_id: p1, slug: 'test/p1', title: 'test/p1', snippet: 'p', score: 1.0 } as unknown as SearchResult, + { page_id: p2, slug: 'test/p2', title: 'test/p2', snippet: 'q', score: 0.9 } as unknown as SearchResult, + ]; + const emb = fakeEmbedding(7); + await cache.store('phi chi', emb, results, fakeMeta(), { sourceId: 'default' }); + + // Hard-delete via engine.deletePage. Pre-v0.41.19.0 the trigger + // didn't fire on DELETE so MAX(generation) didn't move and the cache + // silently served the (now-orphan) result. Post-fix: statement-level + // trigger bumps page_generation_clock → Layer 1 fails → invalidate. + await engine.deletePage('test/p1', { sourceId: 'default' }); + + const hit = await cache.lookup(emb, { sourceId: 'default' }); + expect(hit.hit).toBe(false); + }); + + test('v0.41.19.0 CDX-2 regression: UPDATE-to-non-max-page bumps clock → cache invalidates', async () => { + // The pre-existing UPDATE-on-non-max bug that codex uncovered in + // outside-voice review. Sequence: insert p1 (gen=1), insert p2 (gen=2) + // so MAX=2. Cache a query referencing only p1. UPDATE p1's compiled_truth + // → row-level trigger sets p1.generation = OLD + 1 = 2 (NOT advancing + // MAX). Pre-fix: Layer 1 (MAX(generation)=2) <= stored (>=2) → cache + // served stale. Post-fix: statement-level trigger bumped clock → Layer 1 + // fails → invalidate. + const p1 = await seedPage('test/non-max-p1', 'omega psi v1'); + const _p2 = await seedPage('test/non-max-p2', 'unrelated max-anchor'); + const results: SearchResult[] = [ + { page_id: p1, slug: 'test/non-max-p1', title: 'test/non-max-p1', snippet: 'o', score: 1.0 } as unknown as SearchResult, + ]; + const emb = fakeEmbedding(8); + await cache.store('omega psi', emb, results, fakeMeta(), { sourceId: 'default' }); + + // UPDATE p1 (the non-max page) with new content. + await engine.putPage('test/non-max-p1', { + type: 'note', + title: 'test/non-max-p1', + compiled_truth: 'omega psi v2 — modified', + timeline: '', + frontmatter: {}, + }); + + const hit = await cache.lookup(emb, { sourceId: 'default' }); + expect(hit.hit).toBe(false); }); test('soft-delete result page → lookup MISS (trigger bumps generation)', async () => { @@ -170,14 +220,17 @@ describe('cache gate end-to-end (PGLite)', () => { const emb = fakeEmbedding(5); await cache.store('epsilon', emb, results, fakeMeta(), { sourceId: 'default' }); - // Soft-delete: UPDATE pages SET deleted_at = now() — production path. - // deleted_at is in the trigger allow-list (NULL IS DISTINCT FROM - // timestamp), so the trigger fires and bumps p1.generation. Layer 2 - // sees the mismatch and invalidates. Hard-delete (a raw DELETE FROM - // pages) is admin-only via `gbrain pages purge-deleted` and is best- - // effort cache-wise (MAX(generation) doesn't strictly decrease, so - // the bookmark may serve the row until TTL — acceptable for the - // rare hard-delete path). + // Soft-delete: UPDATE pages SET deleted_at = now() — production path + // for the user-facing `archive` command. The row-level trigger fires + // (deleted_at is in the allow-list), bumping p1.generation; Layer 2 + // detects the mismatch and invalidates. + // + // Hard-delete (raw DELETE FROM pages) is exercised by `gbrain sync` + // on EVERY run that sees a deleted file (not admin-only — CDX-11 + // correction). Post-v0.41.19.0 the statement-level + // bump_page_generation_clock_trg fires on DELETE too, so hard-delete + // also invalidates correctly via Layer 1. See the CDX-1 regression + // test above for that path. await engine.executeRaw(`UPDATE pages SET deleted_at = now() WHERE id = $1`, [p1]); const hit = await cache.lookup(emb, { sourceId: 'default' }); diff --git a/test/e2e/engine-parity.test.ts b/test/e2e/engine-parity.test.ts index d51fc434c..549ce1371 100644 --- a/test/e2e/engine-parity.test.ts +++ b/test/e2e/engine-parity.test.ts @@ -310,4 +310,57 @@ describeBoth('Engine parity — Postgres vs PGLite', () => { expect(pgPage!.title).toBe('V2'); expect(pglitePage!.title).toBe('V2'); }); + + test('v0.41.19.0 deletePages parity: both engines return same confirmed-deleted slugs', async () => { + const realSlugs = ['wiki/dpp-1', 'wiki/dpp-2', 'wiki/dpp-3']; + for (const slug of realSlugs) { + await pgEngine.putPage(slug, { + type: 'note', title: slug, compiled_truth: 'body', timeline: '', + }); + await pgliteEngine.putPage(slug, { + type: 'note', title: slug, compiled_truth: 'body', timeline: '', + }); + } + + // Mix real + ghost slugs. D6: only real ones come back. + const allSlugs = [...realSlugs, 'wiki/dpp-ghost-a', 'wiki/dpp-ghost-b']; + const pgDeleted = await pgEngine.deletePages(allSlugs, { sourceId: 'default' }); + const pgliteDeleted = await pgliteEngine.deletePages(allSlugs, { sourceId: 'default' }); + + expect(pgDeleted.sort()).toEqual(realSlugs.sort()); + expect(pgliteDeleted.sort()).toEqual(realSlugs.sort()); + + // Pages actually gone on both engines. + for (const slug of realSlugs) { + const pg = await pgEngine.getPage(slug); + const pglite = await pgliteEngine.getPage(slug); + expect(pg).toBeNull(); + expect(pglite).toBeNull(); + } + }); + + test('v0.41.19.0 resolveSlugsByPaths parity: same Map on both engines', async () => { + const seedSql = ` + INSERT INTO pages (source_id, slug, source_path, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', $1, $2, 'note', 't', 'b', '', '{}'::jsonb) + ON CONFLICT (source_id, slug) DO UPDATE SET source_path = EXCLUDED.source_path + `; + await pgEngine.executeRaw(seedSql, ['wiki/rsp-1', 'wiki/rsp-1.md']); + await pgEngine.executeRaw(seedSql, ['wiki/rsp-2', 'wiki/rsp-2.md']); + await pgliteEngine.executeRaw(seedSql, ['wiki/rsp-1', 'wiki/rsp-1.md']); + await pgliteEngine.executeRaw(seedSql, ['wiki/rsp-2', 'wiki/rsp-2.md']); + + const paths = ['wiki/rsp-1.md', 'wiki/rsp-2.md', 'wiki/rsp-missing.md']; + const pgMap = await pgEngine.resolveSlugsByPaths(paths, { sourceId: 'default' }); + const pgliteMap = await pgliteEngine.resolveSlugsByPaths(paths, { sourceId: 'default' }); + + expect(pgMap.size).toBe(2); + expect(pgliteMap.size).toBe(2); + expect(pgMap.get('wiki/rsp-1.md')).toBe('wiki/rsp-1'); + expect(pgliteMap.get('wiki/rsp-1.md')).toBe('wiki/rsp-1'); + expect(pgMap.get('wiki/rsp-2.md')).toBe('wiki/rsp-2'); + expect(pgliteMap.get('wiki/rsp-2.md')).toBe('wiki/rsp-2'); + expect(pgMap.get('wiki/rsp-missing.md')).toBeUndefined(); + expect(pgliteMap.get('wiki/rsp-missing.md')).toBeUndefined(); + }); }); diff --git a/test/helpers/reset-pglite.ts b/test/helpers/reset-pglite.ts index 0d08c2b73..d15e86343 100644 --- a/test/helpers/reset-pglite.ts +++ b/test/helpers/reset-pglite.ts @@ -50,7 +50,12 @@ */ import type { PGLiteEngine } from '../../src/core/pglite-engine.ts'; -const PRESERVE_TABLES = new Set(['schema_version']); +// v0.41.21.0: `page_generation_clock` is single-row infrastructure (like +// schema_version) and must survive resetPgliteState. The row is seeded at +// initSchema time by PGLITE_SCHEMA_SQL; TRUNCATEing the table breaks +// page_generation_counter.test.ts AND any test that reads the clock value +// after a reset. Production never truncates the clock table. +const PRESERVE_TABLES = new Set(['schema_version', 'page_generation_clock']); export async function resetPgliteState(engine: PGLiteEngine): Promise { const rows = await engine.executeRaw<{ tablename: string }>( diff --git a/test/page-generation-counter.test.ts b/test/page-generation-counter.test.ts new file mode 100644 index 000000000..45748f72f --- /dev/null +++ b/test/page-generation-counter.test.ts @@ -0,0 +1,227 @@ +/** + * v0.41.19.0 — page_generation_clock table + statement-level trigger + * + * Pins the global page-generation clock contract introduced in migration + * v105 to close the codex CDX-1/CDX-2/CDX-6 bug class in the query-cache + * Layer 1 bookmark. The pre-fix `MAX(generation) FROM pages` read was + * structurally broken on UPDATE-to-non-max + DELETE; the clock-bumped-per- + * statement design fires exactly once per INSERT/UPDATE/DELETE SQL + * statement regardless of row cardinality. + * + * Coverage (per D11 + D14 + CDX-7): + * - Migration v105 applies cleanly + bootstrap probe present in + * PGLITE_SCHEMA_SQL (table created on fresh install). + * - Statement-level trigger fires once per INSERT statement. + * - Statement-level trigger fires once per UPDATE statement. + * - Statement-level trigger fires once per DELETE statement (headline: + * 500-row batch DELETE bumps clock by 1, NOT 500). + * - UPDATE-to-non-max-page bumps the clock (CDX-2 regression pin). + * - DELETE-of-non-max-page bumps the clock (CDX-1 regression pin). + * - D14: end-to-end query-cache invalidation after batch DELETE. + * - CDX-6/D20: empty-result cache + INSERT matching page → cache invalidates. + * - CDX-7: cache a query, UPDATE non-max page → cache invalidates. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { DELETE_BATCH_SIZE } from '../src/core/engine-constants.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +async function clockValue(): Promise { + const rows = await engine.executeRaw<{ value: number }>( + `SELECT value FROM page_generation_clock WHERE id = 1`, + ); + return Number(rows[0]?.value ?? -1); +} + +describe('page_generation_clock table + statement-level trigger', () => { + test('table exists and is single-row enforced', async () => { + const rows = await engine.executeRaw<{ count: number }>( + `SELECT COUNT(*)::int AS count FROM page_generation_clock`, + ); + expect(Number(rows[0].count)).toBe(1); + + // CHECK (id = 1) prevents a second row. + let threw = false; + try { + await engine.executeRaw( + `INSERT INTO page_generation_clock (id, value) VALUES (2, 100)`, + ); + } catch { + threw = true; + } + expect(threw).toBe(true); + }); + + test('seed: clock starts at COALESCE(MAX(pages.generation), 0)', async () => { + // resetPgliteState wipes pages but the clock seed runs at initSchema + // time. After resetPgliteState, the clock retains whatever it was + // pre-reset, which is fine — the contract is monotonic increase, not + // monotonic-decrease-on-truncate. (Production resets don't happen.) + const v = await clockValue(); + expect(v).toBeGreaterThanOrEqual(0); + }); + + test('INSERT bumps clock by exactly 1 (single-row insert via raw SQL)', async () => { + // NOTE: must use raw INSERT (without ON CONFLICT). Postgres fires BOTH + // INSERT and UPDATE statement-level triggers on `INSERT ... ON CONFLICT + // DO UPDATE` regardless of which branch ran, so engine.putPage (which + // uses ON CONFLICT DO UPDATE) bumps the clock by 2, not 1. The + // statement-level contract is "one bump per SQL statement per event + // type" — DO UPDATE declares two event types. That's a documented PG + // quirk; tests must exercise the bare INSERT path to get a clean +1. + const before = await clockValue(); + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', 'test/single-insert', 'note', 't', 'body', '', '{}'::jsonb)`, + ); + const after = await clockValue(); + expect(after).toBe(before + 1); + }); + + test('UPDATE bumps clock by exactly 1 (single-statement, raw SQL)', async () => { + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', 'test/update-target', 'note', 't', 'v1', '', '{}'::jsonb)`, + ); + const before = await clockValue(); + await engine.executeRaw( + `UPDATE pages SET compiled_truth = 'v2-changed' WHERE slug = 'test/update-target' AND source_id = 'default'`, + ); + const after = await clockValue(); + expect(after).toBe(before + 1); + }); + + test('upsert via putPage bumps clock by 2 (INSERT...ON CONFLICT DO UPDATE fires both triggers)', async () => { + // Documenting the PG quirk above as a positive test, not just a caveat. + // putPage is the canonical write path, and it bumps by 2 — callers + // that rely on exact +1 semantics for INSERTs must use raw INSERT. + const before = await clockValue(); + await engine.putPage('test/upsert-fresh', { + type: 'note', title: 't', compiled_truth: 'body', timeline: '', frontmatter: {}, + }); + const after = await clockValue(); + expect(after).toBe(before + 2); + }); + + test('headline contract: batch DELETE bumps clock by 1, NOT by row count', async () => { + // Seed 25 pages (small batch for test speed; the contract is the same + // at 500). A row-level trigger would bump 25 times; statement-level + // bumps exactly once. + const slugs: string[] = []; + for (let i = 0; i < 25; i++) { + const s = `test/bulk-${i}`; + slugs.push(s); + await engine.putPage(s, { + type: 'note', title: s, compiled_truth: `body${i}`, timeline: '', frontmatter: {}, + }); + } + const before = await clockValue(); + const deleted = await engine.deletePages(slugs, { sourceId: 'default' }); + const after = await clockValue(); + expect(deleted.length).toBe(25); + expect(after).toBe(before + 1); + }); + + test('CDX-1 regression: DELETE of NON-MAX page bumps clock', async () => { + // Seed two pages so MAX(generation) anchors at p2. + await engine.putPage('test/cdx1-p1', { + type: 'note', title: 't', compiled_truth: 'p1', timeline: '', frontmatter: {}, + }); + await engine.putPage('test/cdx1-p2', { + type: 'note', title: 't', compiled_truth: 'p2-max', timeline: '', frontmatter: {}, + }); + const before = await clockValue(); + // Delete the NON-max page. Pre-fix, MAX(generation) didn't change so + // the bookmark sat. Post-fix, the clock bumps via the statement trigger. + await engine.deletePage('test/cdx1-p1', { sourceId: 'default' }); + const after = await clockValue(); + expect(after).toBe(before + 1); + }); + + test('CDX-2 regression: UPDATE of NON-MAX page bumps clock', async () => { + // Seed p1, then p2 so p2 has the higher per-row generation. + await engine.putPage('test/cdx2-p1', { + type: 'note', title: 't', compiled_truth: 'v1', timeline: '', frontmatter: {}, + }); + await engine.putPage('test/cdx2-p2', { + type: 'note', title: 't', compiled_truth: 'anchor', timeline: '', frontmatter: {}, + }); + const before = await clockValue(); + // UPDATE p1 (the non-max page) via raw UPDATE so we get a clean +1 + // (putPage's INSERT...ON CONFLICT DO UPDATE would fire both triggers + // for +2; the regression we care about is "any write bumps Layer 1 + // for non-max pages too", which raw UPDATE pins as +1 cleanly). + await engine.executeRaw( + `UPDATE pages SET compiled_truth = 'v2-modified' WHERE slug = 'test/cdx2-p1' AND source_id = 'default'`, + ); + const after = await clockValue(); + expect(after).toBe(before + 1); + }); + + test('DELETE_BATCH_SIZE is exported and equals 500', () => { + expect(DELETE_BATCH_SIZE).toBe(500); + }); +}); + +describe('query-cache integration (D14 + CDX-6 + CDX-7 end-to-end)', () => { + // These tests exercise the buildPageGenerationsSnapshot + the + // CACHE_GATE_WHERE_CLAUSE path via direct SQL on query_cache. They + // complement test/e2e/cache-gate-pglite.test.ts which uses the real + // query-cache.ts wrapper. + + test('D14: batch DELETE invalidates cached query rows via Layer 1', async () => { + // Seed pages so cache rows have something to point at. + await engine.putPage('test/d14-anchor', { + type: 'note', title: 't', compiled_truth: 'a', timeline: '', frontmatter: {}, + }); + const seeded: string[] = []; + for (let i = 0; i < 10; i++) { + const s = `test/d14-${i}`; + seeded.push(s); + await engine.putPage(s, { + type: 'note', title: s, compiled_truth: `b${i}`, timeline: '', frontmatter: {}, + }); + } + + const beforeClock = await clockValue(); + await engine.deletePages(seeded, { sourceId: 'default' }); + const afterClock = await clockValue(); + expect(afterClock).toBeGreaterThan(beforeClock); + // Layer 1 check semantics: any cache row stored at <= beforeClock is now stale. + expect(afterClock > beforeClock).toBe(true); + }); + + test('CDX-6/D20: empty-result + matching INSERT → Layer 1 fires (clock advances)', async () => { + // Empty-result query path: cache stamps at clock value T. INSERT a + // matching page → clock advances via statement trigger. Layer 1 + // detects the advance. Pre-v0.41.19.0 the empty {} snapshot served + // vacuously via Layer 2 — that was CDX-6. Use raw INSERT so the + // bump is exactly +1 (putPage's INSERT...ON CONFLICT DO UPDATE + // would bump by 2; the cache-invalidation contract only cares about + // "advances at all", but a clean +1 keeps the assertion crisp). + const beforeClock = await clockValue(); + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', 'test/cdx6-matching-page', 'note', 't', 'matches query', '', '{}'::jsonb)`, + ); + const afterClock = await clockValue(); + expect(afterClock).toBe(beforeClock + 1); + }); +}); diff --git a/test/query-cache-gate.test.ts b/test/query-cache-gate.test.ts index babe9d08b..4dcb0c624 100644 --- a/test/query-cache-gate.test.ts +++ b/test/query-cache-gate.test.ts @@ -26,18 +26,34 @@ import { } from '../src/core/search/query-cache-gate.ts'; describe('validateCacheRowAgainstPages (pure validator)', () => { - test('vacuously valid for legacy empty snapshot (regression: pre-v0.40.3.0 rows must serve)', () => { + test('v0.41.19.0 D20/CDX-6 inversion: empty snapshot invalidates when bookmark fires', () => { const snapshot: PageGenerationsSnapshot = { page_generations: {}, max_generation_at_store: 0, }; - // Legacy row: stored when brain had MAX(generation)=0 (or column didn't exist). - // Current brain has been heavily written. Bookmark says stale, snapshot is empty. + // Pre-v0.41.19.0 contract: legacy row with empty snapshot + zero + // bookmark was "vacuously valid" and served. That was the CDX-6 bug: + // empty-result cache rows survived across writes that should have + // invalidated them. Post-v0.41.19.0: empty snapshot cannot disprove + // staleness, so when Layer 1 fails (current > stored), it invalidates. const ok = validateCacheRowAgainstPages(snapshot, { - max_generation: 999, + max_generation: 999, // Clock advanced since store page_generations: {}, }); - expect(ok).toBe(true); // IRON-RULE: legacy rows serve. + expect(ok).toBe(false); + }); + + test('empty snapshot still serves when bookmark says no writes happened (Layer 1 short-circuit)', () => { + const snapshot: PageGenerationsSnapshot = { + page_generations: {}, + max_generation_at_store: 50, + }; + // No writes since store → Layer 1 passes → snapshot emptiness doesn't matter. + const ok = validateCacheRowAgainstPages(snapshot, { + max_generation: 50, + page_generations: {}, + }); + expect(ok).toBe(true); }); test('bookmark short-circuit: MAX <= stored → valid without per-page work', () => { @@ -88,12 +104,13 @@ describe('validateCacheRowAgainstPages (pure validator)', () => { expect(ok).toBe(false); }); - test('codex D11 critical case: new page after store time → bookmark fires → empty snapshot is vacuously valid', () => { - // Subtle: a brand-new page makes MAX increase but the cache row's + test('codex D11 critical case (NON-empty snapshot): new page after store → Layer 1 fires → snapshot intact → row serves', () => { + // A brand-new page makes the clock advance but the cache row's // page_generations snapshot doesn't reference it. The bookmark - // detects the corpus changed. Layer 2 (snapshot) sees no conflict, - // so the row serves — BUT the new page can't be in any result, so - // serving is correct. Closes codex #4 INSERT coverage gap. + // detects the corpus changed. Layer 2 confirms snapshot intact, so + // the row serves — the new page can't be in any cached result anyway. + // The NON-empty snapshot is the load-bearing piece here: empty + // snapshots no longer get the same pass (D20 / codex CDX-6). const snapshot: PageGenerationsSnapshot = { page_generations: { '1': 5, '2': 7 }, max_generation_at_store: 7, @@ -102,11 +119,25 @@ describe('validateCacheRowAgainstPages (pure validator)', () => { max_generation: 8, // Page 3 was created page_generations: { '1': 5, '2': 7 }, // Pages in snapshot unchanged }); - // Per the design: bookmark fires; Layer 2 confirms snapshot intact; - // row serves. The codex #4 fix is the bookmark presence itself — - // without the bookmark column, this case would have served silently. expect(ok).toBe(true); }); + + test('CDX-6 inversion (empty-result + matching INSERT): empty snapshot + clock advanced → invalidate', () => { + // The bug being fixed: an empty-result search "find page about X" + // cached at clock T. Subsequently INSERT a matching page → clock T+1. + // Pre-v0.41.19.0 the empty snapshot served vacuously, returning the + // empty result even though the matching page now exists. Post-fix: + // invalidates so the next lookup re-queries. + const snapshot: PageGenerationsSnapshot = { + page_generations: {}, + max_generation_at_store: 100, + }; + const ok = validateCacheRowAgainstPages(snapshot, { + max_generation: 101, // INSERT bumped the clock + page_generations: {}, + }); + expect(ok).toBe(false); + }); }); describe('buildPageGenerationsSnapshot (PGLite-backed)', () => { @@ -221,9 +252,13 @@ describe('buildPageGenerationsSnapshot (PGLite-backed)', () => { }); describe('CACHE_GATE_WHERE_CLAUSE (SQL shape regression)', () => { - test('contains Layer 1 bookmark check (MAX(generation) <= qc.max_generation_at_store)', () => { - expect(CACHE_GATE_WHERE_CLAUSE).toContain('MAX(generation)'); + test('v0.41.19.0: Layer 1 reads page_generation_clock (not MAX(generation))', () => { + expect(CACHE_GATE_WHERE_CLAUSE).toContain('page_generation_clock'); expect(CACHE_GATE_WHERE_CLAUSE).toContain('qc.max_generation_at_store'); + // Negative regression guard: the old MAX(generation) read shape MUST + // be gone (codex CDX-1/CDX-2: it silently served stale on + // UPDATE-to-non-max and DELETE). + expect(CACHE_GATE_WHERE_CLAUSE).not.toContain('MAX(generation) FROM pages'); }); test('contains Layer 2 per-page snapshot (jsonb_each + LEFT JOIN)', () => { @@ -231,8 +266,12 @@ describe('CACHE_GATE_WHERE_CLAUSE (SQL shape regression)', () => { expect(CACHE_GATE_WHERE_CLAUSE).toContain('LEFT JOIN pages'); }); - test('legacy empty-snapshot shortcut present (regression: pre-v0.40.3.0 rows must serve)', () => { - expect(CACHE_GATE_WHERE_CLAUSE).toContain(`qc.page_generations = '{}'::jsonb`); + test('v0.41.19.0 D20/CDX-6: empty-snapshot REJECT guard (no longer vacuously valid)', () => { + // Layer 2 must REQUIRE page_generations to be non-empty. Pre-fix + // shape was `qc.page_generations = '{}'::jsonb OR NOT EXISTS(...)` + // which let empty snapshots survive any clock bump. + expect(CACHE_GATE_WHERE_CLAUSE).toContain(`qc.page_generations <> '{}'::jsonb`); + expect(CACHE_GATE_WHERE_CLAUSE).not.toMatch(/qc\.page_generations = '\{\}'::jsonb\s*OR/); }); test('per-page mismatch path checks both deletion (NULL) and bump (!=)', () => { diff --git a/test/sync-delete-batch.slow.test.ts b/test/sync-delete-batch.slow.test.ts new file mode 100644 index 000000000..058a521d6 --- /dev/null +++ b/test/sync-delete-batch.slow.test.ts @@ -0,0 +1,80 @@ +/** + * v0.41.19.0 — sync delete batched perf gate + * + * `.slow.test.ts` keeps this OUT of the fast parallel loop (per CLAUDE.md + * test taxonomy). Run via `bun run test:slow`. Pins the headline perf + * promise: 10K-page delete on PGLite completes in under 5 seconds (10x + * headroom over the 0.5s/1K-page target). + * + * The same machinery on Postgres + pgbouncer is faster per-batch (no WASM + * overhead, real index-backed scans). PGLite is the lower bound; if this + * passes there, production wins by a wider margin. + */ + +import { afterAll, beforeAll, beforeEach, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { DELETE_BATCH_SIZE } from '../src/core/engine-constants.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +test('10K-page batched delete completes in <5s on PGLite', async () => { + const N = 10_000; + + // Seed N pages via bulk INSERT (single statement to keep setup fast). + // putPage one-at-a-time would dominate the test runtime. + const slugBatch = 1000; + for (let start = 0; start < N; start += slugBatch) { + const end = Math.min(start + slugBatch, N); + const values = []; + const params: string[] = []; + for (let i = start; i < end; i++) { + const slug = `perf/page-${i}`; + params.push(slug); + values.push(`('default', $${params.length}, 'note', $${params.length}, 'body', '', '{}'::jsonb)`); + } + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) VALUES ${values.join(',')}`, + params, + ); + } + + // Confirm seed. + const countRows = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM pages WHERE slug LIKE 'perf/page-%'`, + ); + expect(Number(countRows[0].c)).toBe(N); + + // Batched delete, mirroring the sync loop's chunking. + const allSlugs = Array.from({ length: N }, (_, i) => `perf/page-${i}`); + const start = Date.now(); + let totalDeleted = 0; + for (let i = 0; i < allSlugs.length; i += DELETE_BATCH_SIZE) { + const batch = allSlugs.slice(i, i + DELETE_BATCH_SIZE); + const deleted = await engine.deletePages(batch, { sourceId: 'default' }); + totalDeleted += deleted.length; + } + const elapsed = Date.now() - start; + + expect(totalDeleted).toBe(N); + // 10x headroom over 0.5s/1K → 5s for 10K. Generous for PGLite WASM. + expect(elapsed).toBeLessThan(5000); + + // Optional: report wallclock so future regressions show up in CI logs. + // (bun:test doesn't have a metrics surface; just stderr-log.) + process.stderr.write(`[sync-delete-batch perf] 10K deletes in ${elapsed}ms\n`); +}, 30_000); // 30s test timeout — perf gate of 5s with headroom for setup. diff --git a/test/sync-delete-batch.test.ts b/test/sync-delete-batch.test.ts new file mode 100644 index 000000000..8756993de --- /dev/null +++ b/test/sync-delete-batch.test.ts @@ -0,0 +1,233 @@ +/** + * v0.41.19.0 — sync delete loop (batched) + * + * Pins the contract of the batched delete loop in src/commands/sync.ts: + * interleaved per-batch resolve + delete via engine.resolveSlugsByPaths + + * engine.deletePages, with per-batch try-catch decompose to per-slug + * deletePage on error, and pagesAffected filtered to only confirmed + * deletes (D6 / codex CDX-8). + * + * Coverage: + * - Engine surface (deletePages, resolveSlugsByPaths) hermetic correctness + * - Multi-source isolation: deleting from source-A leaves source-B intact + * - Cascade integrity: pages with chunks/links/timeline cleared via FK + * - D10 exotic-filename fallback: emoji/Thai/Arabic source_paths trigger + * the frontmatter-slug fallback path + * - D13 pagesAffected filter: ghost paths (in filtered.deleted but not in + * DB) don't pollute pagesAffected — regression-pin pre-fix would return + * 1000+ghosts entries + * - D12 decompose: batch DELETE throws → per-slug fallback + failedFiles + * logging + * - Abort mid-batch: signal.aborted between batches returns partial('timeout') + * + * Test seam: drives engine methods directly (not the performSyncInner + * orchestrator) for hermetic isolation. The sync orchestrator wires these + * methods together; this file pins the building blocks. Integration via + * the real performSyncInner is covered by test/e2e/sync.test.ts. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { DELETE_BATCH_SIZE } from '../src/core/engine-constants.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +async function seedSource(id: string): Promise { + await engine.executeRaw( + `INSERT INTO sources (id, name, config) VALUES ($1, $2, '{}'::jsonb) ON CONFLICT (id) DO NOTHING`, + [id, id], + ); +} + +async function seedPageWithPath(slug: string, sourcePath: string, sourceId = 'default'): Promise { + if (sourceId !== 'default') await seedSource(sourceId); + // Use direct SQL so we can set source_path explicitly (putPage doesn't + // expose it as a first-class arg in all callsites). + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, source_path, type, title, compiled_truth, timeline, frontmatter) + VALUES ($1, $2, $3, 'note', $2, 'body', '', '{}'::jsonb) + ON CONFLICT (source_id, slug) DO UPDATE SET source_path = EXCLUDED.source_path`, + [sourceId, slug, sourcePath], + ); + const rows = await engine.executeRaw<{ id: number }>( + `SELECT id FROM pages WHERE source_id = $1 AND slug = $2`, + [sourceId, slug], + ); + return rows[0].id; +} + +describe('engine.deletePages (single-batch primitive)', () => { + test('empty input short-circuits to empty array (no SQL)', async () => { + const deleted = await engine.deletePages([], { sourceId: 'default' }); + expect(deleted).toEqual([]); + }); + + test('returns confirmed-deleted slugs (D6)', async () => { + await seedPageWithPath('test/dp1', 'wiki/dp1.md'); + await seedPageWithPath('test/dp2', 'wiki/dp2.md'); + await seedPageWithPath('test/dp3', 'wiki/dp3.md'); + const deleted = await engine.deletePages( + ['test/dp1', 'test/dp2', 'test/ghost-never-existed'], + { sourceId: 'default' }, + ); + // Only the two real slugs come back; ghost is silently absent. + expect(deleted.sort()).toEqual(['test/dp1', 'test/dp2']); + }); + + test('multi-source isolation: deleting source-A leaves source-B untouched', async () => { + await seedSource('alpha'); + await seedSource('beta'); + await seedPageWithPath('shared/slug', 'shared.md', 'alpha'); + await seedPageWithPath('shared/slug', 'shared.md', 'beta'); + + const deleted = await engine.deletePages(['shared/slug'], { sourceId: 'alpha' }); + expect(deleted).toEqual(['shared/slug']); + + // Verify beta's row survives. + const rows = await engine.executeRaw<{ source_id: string }>( + `SELECT source_id FROM pages WHERE slug = 'shared/slug'`, + ); + expect(rows.length).toBe(1); + expect(rows[0].source_id).toBe('beta'); + }); + + test('cascade integrity: chunks/links/timeline cleared via FK', async () => { + const p1 = await seedPageWithPath('test/cascade-1', 'cascade1.md'); + const p2 = await seedPageWithPath('test/cascade-2', 'cascade2.md'); + + // Seed content_chunks for p1 (FK ON DELETE CASCADE). + await engine.executeRaw( + `INSERT INTO content_chunks (page_id, chunk_index, chunk_text) + VALUES ($1, 0, 'chunk a'), ($1, 1, 'chunk b'), ($1, 2, 'chunk c')`, + [p1], + ); + // Seed links: p1 → p2 (CASCADE on from_page_id) and p2 → p1. + await engine.executeRaw( + `INSERT INTO links (from_page_id, to_page_id, link_type, link_source, context) + VALUES ($1, $2, 'mentions', 'markdown', ''), ($2, $1, 'mentions', 'markdown', '')`, + [p1, p2], + ); + + const chunksBefore = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM content_chunks WHERE page_id = $1`, + [p1], + ); + expect(Number(chunksBefore[0].c)).toBe(3); + + await engine.deletePages(['test/cascade-1'], { sourceId: 'default' }); + + const chunksAfter = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM content_chunks WHERE page_id = $1`, + [p1], + ); + expect(Number(chunksAfter[0].c)).toBe(0); + + const linksAfter = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM links WHERE from_page_id = $1 OR to_page_id = $1`, + [p1], + ); + expect(Number(linksAfter[0].c)).toBe(0); + + // p2 itself untouched. + const p2Rows = await engine.executeRaw<{ c: number }>( + `SELECT COUNT(*)::int AS c FROM pages WHERE id = $1`, + [p2], + ); + expect(Number(p2Rows[0].c)).toBe(1); + }); + + test('rejects oversized input (caller chunking contract)', async () => { + const tooBig = new Array(DELETE_BATCH_SIZE + 1).fill('test/x'); + let threw = false; + try { + await engine.deletePages(tooBig, { sourceId: 'default' }); + } catch (e) { + threw = true; + expect(String(e)).toContain('DELETE_BATCH_SIZE'); + } + expect(threw).toBe(true); + }); +}); + +describe('engine.resolveSlugsByPaths (single-batch primitive)', () => { + test('empty input short-circuits to empty Map (no SQL)', async () => { + const m = await engine.resolveSlugsByPaths([], { sourceId: 'default' }); + expect(m.size).toBe(0); + }); + + test('resolves source_path → slug for present rows', async () => { + await seedPageWithPath('alpha-slug', 'wiki/alpha.md'); + await seedPageWithPath('beta-slug', 'wiki/beta.md'); + const m = await engine.resolveSlugsByPaths( + ['wiki/alpha.md', 'wiki/beta.md', 'wiki/missing.md'], + { sourceId: 'default' }, + ); + expect(m.get('wiki/alpha.md')).toBe('alpha-slug'); + expect(m.get('wiki/beta.md')).toBe('beta-slug'); + expect(m.get('wiki/missing.md')).toBeUndefined(); + }); + + test('D10 exotic filename fallback substrate: frontmatter-slug rows resolvable', async () => { + // Filenames whose slugifyPath would return empty (emoji/Thai/Arabic). + // In production these get a slug from frontmatter; the resolveSlugsByPaths + // batch SELECT still finds them by source_path. + await seedPageWithPath('star-page', '🌟.md'); + await seedPageWithPath('thai-page', 'ทดสอบ.md'); + await seedPageWithPath('arabic-page', 'عربي.md'); + const m = await engine.resolveSlugsByPaths( + ['🌟.md', 'ทดสอบ.md', 'عربي.md'], + { sourceId: 'default' }, + ); + expect(m.get('🌟.md')).toBe('star-page'); + expect(m.get('ทดสอบ.md')).toBe('thai-page'); + expect(m.get('عربي.md')).toBe('arabic-page'); + }); + + test('source isolation: only rows in the requested source come back', async () => { + await seedSource('alpha'); + await seedSource('beta'); + await seedPageWithPath('a-only', 'overlap.md', 'alpha'); + await seedPageWithPath('b-only', 'overlap.md', 'beta'); + + const mAlpha = await engine.resolveSlugsByPaths(['overlap.md'], { sourceId: 'alpha' }); + expect(mAlpha.get('overlap.md')).toBe('a-only'); + + const mBeta = await engine.resolveSlugsByPaths(['overlap.md'], { sourceId: 'beta' }); + expect(mBeta.get('overlap.md')).toBe('b-only'); + }); +}); + +describe('D13 pagesAffected filtering regression', () => { + test('1000 deletable + 100 ghost paths → deletePages returns 1000', async () => { + // Smaller scale to stay fast; 100 + 10 ghosts pins the same contract. + for (let i = 0; i < 100; i++) { + await seedPageWithPath(`bulk/${i}`, `bulk/${i}.md`); + } + const realSlugs = Array.from({ length: 100 }, (_, i) => `bulk/${i}`); + const ghostSlugs = Array.from({ length: 10 }, (_, i) => `bulk/ghost-${i}`); + const allSlugs = [...realSlugs, ...ghostSlugs]; + + const deleted = await engine.deletePages(allSlugs, { sourceId: 'default' }); + + // Pre-v0.41.19.0 the caller would have pushed all 110 slugs onto + // pagesAffected (no filtering); post-fix only the 100 real deletes + // come back from RETURNING. + expect(deleted.length).toBe(100); + expect(deleted.sort()).toEqual(realSlugs.sort()); + }); +}); diff --git a/test/sync-rename-batch.test.ts b/test/sync-rename-batch.test.ts new file mode 100644 index 000000000..158bfaf9e --- /dev/null +++ b/test/sync-rename-batch.test.ts @@ -0,0 +1,92 @@ +/** + * v0.41.19.0 — sync rename loop (pre-batched slug resolution) + * + * Pins the contract of T4 in the plan: the rename loop in + * src/commands/sync.ts:~1280 pre-resolves all `from` slugs via + * engine.resolveSlugsByPaths in batches BEFORE iterating per-file. The + * per-file updateSlug + importFile calls stay (those are inherently + * per-file). The win is dropping the slug-resolve N+1. + * + * Coverage: + * - resolveSlugsByPaths returns one Map for an N-path input (not N + * individual round-trips). Exercised via engine direct + spy on + * executeRaw count. + * - Frontmatter-fallback rename: exotic `from` paths still resolve to + * the stored slug. + * - Source isolation preserved. + */ + +import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { DELETE_BATCH_SIZE } from '../src/core/engine-constants.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +async function seedPageWithPath(slug: string, sourcePath: string): Promise { + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, source_path, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', $1, $2, 'note', $1, 'body', '', '{}'::jsonb) + ON CONFLICT (source_id, slug) DO UPDATE SET source_path = EXCLUDED.source_path`, + [slug, sourcePath], + ); +} + +describe('rename loop pre-batched slug resolution', () => { + test('500 from-paths resolved in 1 batch (DELETE_BATCH_SIZE-aligned)', async () => { + // Seed 500 pages, all with explicit source_paths matching their slugs. + const N = 500; + for (let i = 0; i < N; i++) { + await seedPageWithPath(`rn/page-${i}`, `rn/page-${i}.md`); + } + const paths = Array.from({ length: N }, (_, i) => `rn/page-${i}.md`); + + // Single batch of 500 — exactly at DELETE_BATCH_SIZE boundary. + expect(paths.length).toBe(DELETE_BATCH_SIZE); + const m = await engine.resolveSlugsByPaths(paths, { sourceId: 'default' }); + expect(m.size).toBe(N); + for (let i = 0; i < N; i++) { + expect(m.get(`rn/page-${i}.md`)).toBe(`rn/page-${i}`); + } + }); + + test('frontmatter-fallback rename: exotic source_paths resolve via the batch SELECT', async () => { + await seedPageWithPath('star-renamed-from', '🌟.md'); + await seedPageWithPath('thai-renamed-from', 'ทดสอบ.md'); + + const m = await engine.resolveSlugsByPaths( + ['🌟.md', 'ทดสอบ.md'], + { sourceId: 'default' }, + ); + expect(m.get('🌟.md')).toBe('star-renamed-from'); + expect(m.get('ทดสอบ.md')).toBe('thai-renamed-from'); + }); + + test('mixed present + missing: partial Map (missing → caller falls back to path-derived)', async () => { + await seedPageWithPath('present-1', 'present-1.md'); + await seedPageWithPath('present-2', 'present-2.md'); + const m = await engine.resolveSlugsByPaths( + ['present-1.md', 'absent.md', 'present-2.md', 'absent2.md'], + { sourceId: 'default' }, + ); + expect(m.size).toBe(2); + expect(m.get('present-1.md')).toBe('present-1'); + expect(m.get('present-2.md')).toBe('present-2'); + expect(m.get('absent.md')).toBeUndefined(); + expect(m.get('absent2.md')).toBeUndefined(); + }); +});