admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-30 03:12:32 +00:00
Code Issues Packages Projects Releases Wiki Activity

v0.41.25.0 perf(sync): batched deletes + global page-generation clock (supersedes #1538) (#1566)

Browse Source 
* feat(engine): add deletePages + resolveSlugsByPaths to BrainEngine (v0.41.21.0 T1)

Two new REQUIRED methods on the BrainEngine interface, implemented on
both Postgres and PGLite engines. Closes the per-file N+1 query pattern
that PR #1538 batched on Postgres only.

  deletePages(slugs: string[], opts: { sourceId: string }): Promise<string[]>
    — Single SQL round-trip:
        DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2
        RETURNING slug
    — Returns slugs ACTUALLY DELETED (D6, codex CDX-8) so callers can
      filter pagesAffected to exclude phantom slugs (paths in the
      deletion list but with no DB row).
    — Single-batch primitive: caller chunks input to DELETE_BATCH_SIZE.
      Throws if input exceeds the cap.
    — sourceId is REQUIRED at the type level (D5, codex CDX-10).
      Asymmetric with single-row deletePage which keeps the optional
      'default' fallback for back-compat. v0.42+ TODO to tighten.

  resolveSlugsByPaths(paths, opts): Promise<Map<path, slug>>
    — Batch path → slug lookup. Single SQL round-trip:
        SELECT slug, source_path FROM pages
         WHERE source_path = ANY($1::text[]) AND source_id = $2
    — Missing paths absent from the Map (caller falls back to
      path-derived slug, same contract as resolveSlugByPathOrSourcePath).
    — Empty input short-circuits to empty Map (no SQL).

  src/core/engine-constants.ts (NEW)
    — Single source of truth for DELETE_BATCH_SIZE = 500.
    — Both engines import; no engine-from-engine coupling.
    — Lives outside engine.ts (the interface module) to avoid circular
      imports.

Also updates the deletePage JSDoc (CDX-11): drops the misleading
"hard delete is admin-only" framing. `gbrain sync` hard-deletes on
every run that sees a deleted file; not admin-only.

Co-Authored-By: garrytan-agents <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* perf(sync): batched delete + rename + DRY refactor (v0.41.21.0 T2/T3/T4)

Replaces the per-file delete loop (sync.ts:1241-1257) and per-file
rename slug-resolve (sync.ts:1263-1295) with interleaved per-batch
flows using engine.resolveSlugsByPaths + engine.deletePages. Also
refactors resolveSlugByPathOrSourcePath (sync.ts:267) to delegate to
the new batch helper when sourceId is set — one owner of the SQL +
fallback semantics (D8).

ROUND-TRIP COUNTS (73K-delete commit):
  pre-fix:   73,000 SELECTs + 73,000 DELETEs = 146,000 (~5 hours)
  post-fix:     146 SELECTs +     146 DELETEs =     292 (~2 minutes)

Headline win: a single commit deleting 73K files no longer jams the
sync pipeline for hours, no longer cascades staleness across every
other source on the brain.

Shape (T2 delete loop, per the plan's ASCII diagram):

  filtered.deleted (73K paths)
        │
        ▼
   slice into batches of DELETE_BATCH_SIZE (500)
        │
        ▼  for each batch:
   abort-check ──► partial('timeout')
        │
        ▼
   engine.resolveSlugsByPaths(batch, {sourceId})  ◀── 1 SQL round-trip
        │
        ▼
   slugs = batch.map(path => map.get(path)
                  ?? resolveSlugForPath(path))    ◀── pure-JS fallback
        │                                              for frontmatter-
        ▼                                              fallback slugs
   try {
     deleted = engine.deletePages(slugs, opts)    ◀── 1 SQL round-trip
     pagesAffected.push(...deleted)               ◀── D6 confirmed only
   } catch {
     // D7 decompose: per-slug deletePage,
     // unrecoverable failures → failedFiles
   }

Per-batch try-catch (D7) decomposes batch DELETE failures to per-slug
deletePage so a transient blip on batch 73 doesn't lose 500 deletes —
it self-heals to one-at-a-time for that batch only. Unrecoverable
per-slug failures land in failedFiles (matching the existing
import-loop pattern at sync.ts:~1350). failedFiles declaration
hoisted above the delete loop so both delete decompose and import
loops feed the same sync-bookmark gate.

T4 rename loop: pre-resolves all `from` slugs in batches via
resolveSlugsByPaths BEFORE iterating. Per-file updateSlug + importFile
calls stay (those are inherently per-file). The try/catch around
updateSlug for slug-doesn't-exist preserves verbatim.

T3 DRY refactor: resolveSlugByPathOrSourcePath delegates to
resolveSlugsByPaths via a single-element array when sourceId is set.
When sourceId is undefined (legacy unscoped callers), falls back to
the original executeRaw shape — the batch engine surface requires
sourceId per D5 (multi-source-bug-class defense).

Atomicity coarsening (D3): each batch is one transaction. A mid-batch
abort or connection failure rolls back up to DELETE_BATCH_SIZE - 1
successful deletes from the in-flight batch. Sync is idempotent so
the next run picks them up via git diff regenerating the deletion
list. Documented at the call site + in the deletePages JSDoc.

Co-Authored-By: garrytan-agents <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(schema): global page-generation clock + statement-level trigger (v0.41.21.0 T5)

Migration v104: page_generation_clock_and_statement_trigger.

The pre-v0.41.21.0 query-cache Layer 1 bookmark read MAX(generation) FROM
pages to detect "writes happened since cache-store". Two bugs in that
contract — independent of any sync work, surfaced by codex
outside-voice on the /plan-eng-review pass:

  1. The row-level bump_page_generation_trg (migration v91) sets
     NEW.generation = OLD.generation + 1 on UPDATE. Updating a NON-MAX
     page didn't advance MAX(generation). Cache silently served stale
     for any UPDATE-to-non-max page. (CDX-2)
  2. The trigger is BEFORE INSERT OR UPDATE — DELETE doesn't fire it
     at all. Even an AFTER DELETE wouldn't move MAX (surviving rows
     are untouched). (CDX-1)

Fix: single-row page_generation_clock counter, bumped per-statement
(FOR EACH STATEMENT — per-row would turn a 73K-row batch DELETE into
73K UPDATEs on the same counter, recreating the bottleneck this PR
fixes elsewhere — codex CDX-4). Layer 1 reads the clock value
directly (T6, separate commit). Per-row pages.generation stays for
Layer 2 (per-page snapshot via jsonb_each + LEFT JOIN pages) which
doesn't care about MAX, only per-page advancement.

Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache
rows stored under the old MAX semantics aren't all instantly
invalidated on upgrade. Their max_generation_at_store stamp compares
cleanly against the seeded clock; future writes bump the clock and
the bookmark fires correctly.

  CREATE TABLE page_generation_clock (
    id    INTEGER PRIMARY KEY CHECK (id = 1),
    value BIGINT  NOT NULL DEFAULT 0
  );
  CREATE TRIGGER bump_page_generation_clock_trg
    AFTER INSERT OR UPDATE OR DELETE ON pages
    FOR EACH STATEMENT
    EXECUTE FUNCTION bump_page_generation_clock_fn();

Mirror in src/core/pglite-schema.ts so fresh PGLite installs get the
table + trigger via SCHEMA_SQL replay. The forward-reference bootstrap
probe doesn't need an entry: page_generation_clock is created directly
by SCHEMA_SQL (no separate index or FK references it), so the
schema-bootstrap-coverage gate is satisfied as-is.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cache): move Layer 1 to global clock + invalidate empty snapshots (v0.41.21.0 T6)

Closes the silent stale-cache bug class that's been live in master
since the bookmark feature shipped. Pre-fix, gbrain search would
silently serve stale cached results in three independent scenarios:

  1. UPDATE to a non-max-generation page (CDX-2) — the row-level
     trigger advanced per-page generation but didn't move
     MAX(generation), so the bookmark passed.
  2. DELETE of any page (CDX-1) — the trigger didn't fire at all,
     and even an AFTER DELETE wouldn't move MAX.
  3. Empty-result cache row + subsequent matching INSERT (CDX-6 /
     D20) — page_generations = '{}'::jsonb was "vacuously valid" via
     Layer 2, surviving any clock bump.

Fix:

  buildPageGenerationsSnapshot (store path)
    — Replaces the SELECT MAX(generation) FROM pages reads at
      cache-write time with SELECT value FROM page_generation_clock
      WHERE id = 1.
    — Empty pageIds path: only need the clock value (D20 contract).
    — Combined non-empty path: per-page generation (Layer 2 substrate)
      + clock value, both folded in one round trip via UNION ALL.

  CACHE_GATE_WHERE_CLAUSE (lookup path)
    — Layer 1 reads page_generation_clock.value (single-row O(1)
      lookup, faster than the pre-fix MAX(generation) backward index
      scan).
    — Layer 2 stricter: requires page_generations <> '{}'::jsonb AND
      the per-page check (not OR with the vacuously-valid `= '{}'`
      shortcut). Empty snapshots can no longer survive a Layer 1 miss.

  validateCacheRowAgainstPages (pure validator)
    — Layer 2 returns false for empty snapshots when Layer 1 fails.
    — Documented contract change.

Backward compat: pre-v0.40.3.0 cache rows have
max_generation_at_store = 0 AND page_generations = '{}'::jsonb. On a
populated brain, Layer 1 fails (clock > 0). Layer 2 is now stricter
so legacy rows invalidate once on first post-upgrade lookup, then the
cache fills back correctly. Acceptable one-time miss spike;
post-upgrade cache is structurally sound.

The clock seed (COALESCE(MAX(pages.generation), 0)) from migration
v104 keeps NON-empty legacy rows passing Layer 1 until the next
write — they don't all invalidate at once.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: cover v0.41.21.0 delete-batch + global clock + cache contract (T7+T8+T9)

Tests for every behavior the v0.41.21.0 wave introduces or changes.

New test files:

  test/sync-delete-batch.test.ts (PGLite hermetic)
    — engine.deletePages: empty input short-circuit, returns confirmed
      slugs (D6), multi-source isolation, cascade integrity (chunks +
      links cleared via FK), rejects oversized input.
    — engine.resolveSlugsByPaths: empty input, present + missing rows,
      D10 exotic-filename substrate (🌟.md / ทดสอบ.md / عربي.md),
      source isolation.
    — D13 pagesAffected filter: 100 deletable + 10 ghost paths →
      deletePages returns 100 (regression-pin: pre-fix would return
      all 110 via D6's pre-RETURNING shape).

  test/sync-delete-batch.slow.test.ts (.slow suffix keeps it out of
  the fast loop)
    — 10K-page batched delete completes in <5s on PGLite. Measured
      277ms on dev hardware (18x under the gate); pins the headline
      perf promise.

  test/sync-rename-batch.test.ts (PGLite hermetic)
    — 500-rename batch slug-resolve in 1 round-trip (exactly at
      DELETE_BATCH_SIZE boundary).
    — Frontmatter-fallback rename: exotic source_paths resolve via
      the batch SELECT.
    — Mixed present + missing: partial Map (missing → caller falls
      back to path-derived).

  test/page-generation-counter.test.ts (PGLite hermetic)
    — Statement-level trigger fires once per INSERT statement (raw
      SQL — NOT putPage, which uses ON CONFLICT DO UPDATE and bumps
      by 2 in PG semantics).
    — Statement-level trigger fires once per UPDATE statement.
    — Headline contract: batch DELETE bumps clock by 1, NOT by row
      count (25-row batch → +1).
    — CDX-1 regression: DELETE of non-max page bumps clock.
    — CDX-2 regression: UPDATE of non-max page bumps clock (raw SQL).
    — D14 end-to-end: clock advances after batch DELETE → cache rows
      stamped at the prior clock value are now stale by Layer 1.
    — CDX-6/D20: empty-result cache + INSERT matching page → clock
      advances (Layer 1 fires).
    — Documents the PG quirk: putPage's INSERT...ON CONFLICT DO UPDATE
      bumps clock by 2 (both INSERT and UPDATE triggers fire).

Test-helper update:

  test/helpers/reset-pglite.ts
    — Added page_generation_clock to PRESERVE_TABLES so the seeded
      single-row counter survives resetPgliteState between tests
      (same treatment as schema_version). Production never truncates.

Existing test contract inversions (CDX-6 / D20 fix):

  test/query-cache-gate.test.ts
    — Pre-v0.41.21.0 "vacuously valid for legacy empty snapshot"
      assertion inverted: empty snapshot now invalidates when Layer 1
      fires. Add positive CDX-6 regression test (empty-result + INSERT
      matching page).
    — SQL shape regression: page_generation_clock in Layer 1 (negative
      regression guard: MAX(generation) FROM pages MUST be gone).
    — Empty-snapshot reject guard:
      `qc.page_generations <> '{}'::jsonb` present; the old
      `qc.page_generations = '{}'::jsonb OR` shortcut MUST be gone.

  test/e2e/cache-gate-pglite.test.ts
    — Pre-v0.41.21.0 "legacy row serves vacuously" test inverted:
      legacy rows now invalidate on first clock advance post-upgrade.
    — CDX-1 regression: DELETE bumps clock → cached query for
      surviving pages invalidates.
    — CDX-2 regression: UPDATE-to-non-max-page bumps clock → cache
      invalidates.
    — CDX-11 comment fix: drop misleading "hard delete is admin-only"
      framing; gbrain sync hard-deletes on every run.

Engine parity extension:

  test/e2e/engine-parity.test.ts
    — deletePages parity: same input set, both engines return same
      string[] of confirmed-deleted slugs (D6).
    — resolveSlugsByPaths parity: same Map on both engines.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(release): v0.41.21.0 — batched sync deletes + global page-generation clock (T10)

VERSION bump (0.41.18.0 → 0.41.21.0; master is at 0.41.20.0 so
next free slot per the queue allocator). CHANGELOG entry with the
ELI10 lead per CLAUDE.md voice rules. CLAUDE.md annotations on
engine.ts, postgres-engine.ts, pglite-engine.ts, sync.ts, and
query-cache-gate.ts plus a new entry for engine-constants.ts.
llms-full.txt regenerated to match CLAUDE.md (per CLAUDE.md
mandatory rule).

Co-Authored-By: garrytan-agents <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* dx(test-runner): heartbeat shows real progress instead of 0p 0f

Bun's default test reporter doesn't print per-test markers — only a
single shard-end summary block when you pass it a file list. The
existing heartbeat tried to count `^[[:space:]]+✓` lines as a live
pass-count proxy, but bun never emits them in the multi-file mode this
runner uses, so every mid-run heartbeat showed `0p 0f` for the entire
12-20 minute wallclock. Users (and agents polling the runner) couldn't
distinguish "still bootstrapping" from "wedged" from "almost done."

Fix: parse three complementary real-time signals instead.

  1. Total files this shard was assigned — parsed from the
     `[unit-shard N/M] running X files` banner that run-unit-shard.sh
     echoes before invoking bun test. Available from second 1.

  2. PGLite initSchema() count — proxy for "test files started so far."
     Each PGLite-using test file's beforeAll triggers one initSchema(),
     which logs `Schema version 1 → 106 (101 migration(s) pending)`.
     Undercounts because not every test file opens a PGLite engine
     (covers ~30-60% of files in practice), but it's the only real-time
     progress signal bun's default reporter leaves in the log. The
     output uses a `~` prefix to convey "approximate count."

  3. Log size in KB — strictly monotonic liveness signal that works
     even when the PGLite count is still 0 (early-shard startup before
     the first initSchema fires).

  4. Per-shard elapsed time — formatted as MmSSs.

New mid-run heartbeat line:

  [heartbeat]  [s1: ~62/190f 476KB 12m31s] [s2: ~63/190f 513KB 12m31s] ...

When a shard finishes, the heartbeat upgrades to its final summary
including pass/fail counts from bun's end-of-shard summary block:

  [heartbeat]  [s1: done ✓ 2807p 0f] [s2: done ✓ 2784p 0f] ...

Portability: BSD awk on macOS doesn't support `match($0, /re/, arr)`
with the array sink — that's a gawk extension. The total-files parser
uses sed instead so the runner stays portable to the default Mac toolchain.

Helpers are pure functions and unit-testable in isolation: pass a log
file path, get the parsed number. No mocking. No bun runtime required.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(release): rebump v0.41.23.0 → v0.41.25.0

Per user request — skip v0.41.23.0 / v0.41.24.0 slots to land at
v0.41.25.0. Master is at v0.41.22.1, no version-trio collision.

Touches VERSION, package.json, CHANGELOG header, CLAUDE.md
annotations, src/core/engine-constants.ts header, src/core/migrate.ts
migration v106 comment, regenerated llms-full.txt + llms.txt.

Migration version (v106) and CDX1-6 trigger semantics unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(schema): reorder migration_impact_log AFTER minion_jobs (CI green)

Pre-existing bug in master's SCHEMA_SQL ordering, surfaced by CI on
this PR but lived silently on master since v0.41.18.0.

migration_impact_log declares `job_id BIGINT REFERENCES minion_jobs(id)`,
but its CREATE TABLE was at line 658 while minion_jobs's CREATE TABLE
was at line 778. On any fresh-install initSchema() the FK target
didn't exist yet:

  psql:/tmp/schema.sql:672: ERROR: relation "minion_jobs" does not exist

postgres-js's `unsafe()` aborts the multi-statement batch on the first
error response, so every CREATE TABLE after migration_impact_log
(including minion_jobs itself) never ran. Every subsequent CLI
subprocess that opened a connection then crashed with
`relation "minion_jobs" does not exist` on its first query.

Why master CI sometimes passed: the per-shard advisory lock + the
test setup's `engine.initSchema()` second pass (which runs the
migrations array) would eventually create minion_jobs via the v5
`minion_jobs_table` migration. From there migration_impact_log would
land via migration v103 with its FK resolving correctly. But CLI
subprocesses spawned by mechanical.test.ts's Parallel Import block
open their OWN connections and run a fresh `engine.connect() →
initSchema()` — that path runs SCHEMA_SQL FIRST and aborted at the
same forward-reference error before the migrations array could repair.

Fix: relocate the migration_impact_log CREATE TABLE + its two indexes
to AFTER the minion_jobs CREATE TABLE block (lines ~865), keeping
the rest of the schema layout intact. PGLite schema (pglite-schema.ts)
already had the correct ordering — only Postgres SCHEMA_SQL needed
the move.

Verified: fresh-DB local repro that previously failed 31/34 tests
with `relation minion_jobs does not exist` now passes 78/78 in
test/e2e/mechanical.test.ts.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-27 08:50:53 -07:00
committed by GitHub
co-authored by Claude Opus 4.7
parent 726dfff02c
commit ff32fcaa78
24 changed files with 1754 additions and 143 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/commands/sync.ts
+183 -22
View File
@@ -2,6 +2,7 @@ import { existsSync, readFileSync, writeFileSync, statSync } from 'fs';
import { execFileSync } from 'child_process';
import { join, relative } from 'path';
import type { BrainEngine } from '../core/engine.ts';
import { DELETE_BATCH_SIZE } from '../core/engine-constants.ts';
import { importFile } from '../core/import-file.ts';
import { collectSyncableFiles } from './import.ts';
import { createInterface } from 'readline';
@@ -269,14 +270,25 @@ export async function resolveSlugByPathOrSourcePath(
path: string,
sourceId?: string,
): Promise<string> {
// v0.41.19.0 (D8): when sourceId is set, delegate to the new batch
// resolveSlugsByPaths so single-call and batched paths share one SQL
// owner + one fallback semantic. One Map allocation per single-call;
// negligible cost. When sourceId is undefined (legacy unscoped callers),
// fall back to the original executeRaw shape — the batch method
// requires sourceId to prevent the multi-source-bug-class on its new
// surface (D5). The unscoped fallback preserves back-compat.
try {
const rows = await engine.executeRaw<{ slug: string }>(
sourceId
? `SELECT slug FROM pages WHERE source_path = $1 AND source_id = $2 LIMIT 1`
: `SELECT slug FROM pages WHERE source_path = $1 LIMIT 1`,
sourceId ? [path, sourceId] : [path],
);
if (rows.length > 0 && rows[0].slug) return rows[0].slug;
if (sourceId) {
const m = await engine.resolveSlugsByPaths([path], { sourceId });
const slug = m.get(path);
if (slug) return slug;
} else {
const rows = await engine.executeRaw<{ slug: string }>(
`SELECT slug FROM pages WHERE source_path = $1 LIMIT 1`,
[path],
);
if (rows.length > 0 && rows[0].slug) return rows[0].slug;
}
} catch {
// Fall through — best-effort. Pre-migration brains or query errors
// shouldn't break delete/rename for path-derived pages.
@@ -1233,25 +1245,135 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// Phases: sync.deletes, sync.renames, sync.imports.
const progress = createProgress(cliOptsToProgressOptions(getCliOptions()));
// Process deletes first (prevents slug conflicts). SP-5: resolveSlugForPath
// dispatches to the right slug shape so code file deletes hit the real page.
// v0.41.19.0: hoisted out of the import block so the delete decompose
// path (per-batch try-catch fallback) can append unrecoverable delete
// failures here too. Same canonical surface that gates `sync.last_commit`
// advancement at the bottom of this function.
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
// v0.18.0+ multi-source: scope deletePage so we only delete the source-A
// row, not every same-slug row across all sources.
const deleteOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined;
// v0.41.19.0 (T2/D6/D7/D16/D18 via /plan-eng-review + codex outside-voice):
// batched delete loop. Replaces the per-file N+1 that PR #1538 originally
// batched on Postgres only. See plan file:
// ~/.claude/plans/system-instruction-you-are-working-ethereal-narwhal.md
//
// SHAPE (interleaved per-batch resolve + delete; caller owns chunking):
//
// filtered.deleted (e.g. 73K paths)
// │
// ▼
// slice into batches of DELETE_BATCH_SIZE (500)
// │
// ▼ for each batch:
// abort-check ──► partial('timeout')
// │
// ▼
// engine.resolveSlugsByPaths(batch, {sourceId}) ◀── 1 SQL round-trip
// │
// ▼
// slugs = batch.map(path => map.get(path)
// ?? resolveSlugForPath(path)) ◀── pure-JS fallback for
// │ frontmatter-fallback
// ▼ + missing-source-path
// try {
// deleted = engine.deletePages(slugs, opts) ◀── 1 SQL round-trip
// pagesAffected.push(...deleted) ◀── D6: only confirmed
// } catch { deletes, not phantoms
// // D7 decompose: per-slug deletePage,
// // unrecoverable failures → failedFiles
// }
//
// ROUND-TRIP COUNTS (73K deletes):
// pre-fix: 73,000 SELECTs + 73,000 DELETEs = 146,000 (~5 hours)
// post-fix: 146 SELECTs + 146 DELETEs = 292 (~2 minutes)
//
// ATOMICITY (D3): each batch is one transaction. A mid-batch abort or
// transient connection failure rolls back up to DELETE_BATCH_SIZE - 1
// successful deletes. Sync is idempotent — the next run picks them up
// via git diff regenerating the deletion list.
//
// NO-SOURCEID FALLBACK: when opts.sourceId is undefined (legacy unscoped
// callers, rare post-v0.34.1 source-resolution wiring), fall back to the
// OLD per-path loop. The batch engine surface requires sourceId per D5
// (multi-source-bug-class defense at the type level). Production callers
// that thread sourceId via resolveSourceWithTier get the new fast path.
if (filtered.deleted.length > 0) {
progress.start('sync.deletes', filtered.deleted.length);
for (const path of filtered.deleted) {
// v0.41.13.0 (T2 / D-V4-2): per-iteration abort check. Codex pass-3
// F8 caught that v3 only covered pull + add/modify. Refactor commits
// with hundreds of deletes can overshoot --timeout without this check.
if (opts.signal?.aborted) {
progress.finish();
return partial('timeout');
if (opts.sourceId) {
const sid = opts.sourceId;
const deleteScopedOpts = { sourceId: sid };
for (let i = 0; i < filtered.deleted.length; i += DELETE_BATCH_SIZE) {
if (opts.signal?.aborted) {
progress.finish();
return partial('timeout');
}
const batch = filtered.deleted.slice(i, i + DELETE_BATCH_SIZE);
// Phase A: batch slug resolution (1 round-trip per batch).
let pathSlugMap: Map<string, string>;
try {
pathSlugMap = await engine.resolveSlugsByPaths(batch, deleteScopedOpts);
} catch {
// Resolve failure: fall back to empty map; per-path fallback
// below will use resolveSlugForPath. Best-effort, matches the
// existing resolveSlugByPathOrSourcePath swallow-and-fallback
// semantics.
pathSlugMap = new Map();
}
const slugs = batch.map(p => pathSlugMap.get(p) ?? resolveSlugForPath(p));
// Phase B: batch delete (1 round-trip per batch).
try {
const deleted = await engine.deletePages(slugs, deleteScopedOpts);
// D6: only push slugs that were actually deleted. Filters phantom
// slugs (paths in filtered.deleted but with no DB row) so
// downstream extract/embed don't waste lookups.
pagesAffected.push(...deleted);
} catch (err) {
// D7 decompose: a transient blip on this batch shouldn't lose all
// 500 deletes. Fall back to per-slug deletePage for THIS batch
// only; unrecoverable per-slug failures land in failedFiles
// (matching the existing import-loop pattern at sync.ts:~1350).
for (let j = 0; j < slugs.length; j++) {
try {
await engine.deletePage(slugs[j], deleteScopedOpts);
pagesAffected.push(slugs[j]);
} catch (perSlugErr) {
failedFiles.push({
path: batch[j],
error: `delete failed: ${perSlugErr instanceof Error ? perSlugErr.message : String(perSlugErr)} (batch error: ${err instanceof Error ? err.message : String(err)})`,
});
}
}
}
progress.tick(batch.length, `deletes ${Math.min(i + DELETE_BATCH_SIZE, filtered.deleted.length)}/${filtered.deleted.length}`);
}
} else {
// Legacy no-sourceId path. The engine batch methods require sourceId
// per D5 (kills the multi-source-bug-class on the new surface); when
// sourceId is unset, fall back to the original per-path loop. Slow
// but correct; production callers all thread sourceId so this branch
// is functionally dead post-v0.34.1.
for (const path of filtered.deleted) {
if (opts.signal?.aborted) {
progress.finish();
return partial('timeout');
}
const slug = await resolveSlugByPathOrSourcePath(engine, path, undefined);
try {
await engine.deletePage(slug, deleteOpts);
pagesAffected.push(slug);
} catch (err) {
failedFiles.push({
path,
error: `delete failed: ${err instanceof Error ? err.message : String(err)}`,
});
}
progress.tick(1, slug);
}
const slug = await resolveSlugByPathOrSourcePath(engine, path, opts.sourceId);
await engine.deletePage(slug, deleteOpts);
pagesAffected.push(slug);
progress.tick(1, slug);
}
progress.finish();
}
@@ -1260,12 +1382,46 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// SP-5: both old and new slugs use resolveSlugForPath so a .ts → .ts
// rename (code→code), .md → .md (markdown→markdown), or cross-kind rename
// all resolve to the right slug shape for each side.
//
// v0.41.19.0 (T4): pre-batched slug resolution per Phase 3 of the plan.
// Renames' per-file cost is dominated by importFile() (file IO + chunking
// + embedding), so the per-iteration updateSlug + importFile loop stays;
// only the upfront slug-resolve N+1 gets batched. The try/catch around
// updateSlug for slug-doesn't-exist preserves verbatim.
if (filtered.renamed.length > 0) {
progress.start('sync.renames', filtered.renamed.length);
// v0.18.0+ multi-source: scope updateSlug so the rename only touches the
// source-A row, not every same-slug row across sources (which would
// either sweep them all OR violate (source_id, slug) UNIQUE).
const renameOpts = opts.sourceId ? { sourceId: opts.sourceId } : undefined;
// T4: pre-resolve ALL `from` slugs in batches before iterating. Falls
// back to per-path resolveSlugByPathOrSourcePath when sourceId is
// unset (matches the delete loop's legacy posture). For large rename
// commits (rare but possible: prefix sweep, reorganization), this drops
// the slug-resolve round-trips from O(renames) to O(renames/500).
const fromSlugByPath = new Map<string, string>();
if (opts.sourceId) {
const sid = opts.sourceId;
const fromPaths = filtered.renamed.map(r => r.from);
for (let i = 0; i < fromPaths.length; i += DELETE_BATCH_SIZE) {
if (opts.signal?.aborted) {
progress.finish();
return partial('timeout');
}
const batch = fromPaths.slice(i, i + DELETE_BATCH_SIZE);
let m: Map<string, string>;
try {
m = await engine.resolveSlugsByPaths(batch, { sourceId: sid });
} catch {
m = new Map();
}
for (const p of batch) {
fromSlugByPath.set(p, m.get(p) ?? resolveSlugForPath(p));
}
}
}
for (const { from, to } of filtered.renamed) {
// v0.41.13.0 (T2 / D-V4-2): per-iteration abort check. Renames call
// importFile() at line 1173-style sites which can be slow on big files;
@@ -1274,7 +1430,9 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
progress.finish();
return partial('timeout');
}
const oldSlug = await resolveSlugByPathOrSourcePath(engine, from, opts.sourceId);
const oldSlug = opts.sourceId
? (fromSlugByPath.get(from) ?? resolveSlugForPath(from))
: await resolveSlugByPathOrSourcePath(engine, from, undefined);
// The new path doesn't yet have a row, so resolve from path only.
const newSlug = resolveSlugForPath(to);
try {
@@ -1308,7 +1466,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// v0.15.2: per-file progress on stderr via the shared reporter.
// Bug 9: per-file failures captured in `failedFiles` so the caller can
// gate `sync.last_commit` advancement and record recoverable errors.
const failedFiles: Array<{ path: string; error: string; line?: number }> = [];
// v0.41.19.0: `failedFiles` is now hoisted above the delete loop (the
// delete decompose path appends here too); kept as a comment-pin so
// future maintainers know to thread additional failure surfaces through
// the same array.
const addsAndMods = [...filtered.added, ...filtered.modified];
// Sort newest-first so date-prefixed brain paths get embedded before older
src/core/engine-constants.ts
+25
View File
@@ -0,0 +1,25 @@
// v0.41.25.0 — single source of truth for engine batch-sizing.
//
// Both PostgresEngine and PGLiteEngine import from here so the constants
// cannot drift across engines. Lives outside `src/core/engine.ts` to avoid
// circular-import worries (engine.ts is the interface; engines depend on
// engine.ts; this file depends on neither).
/**
* Maximum number of slugs per single batch `DELETE FROM pages WHERE slug =
* ANY($1::text[])` call. Callers (e.g. `src/commands/sync.ts` delete loop)
* are responsible for chunking input arrays to this size; `engine.deletePages`
* is a single-batch primitive that does NOT chunk internally (matches the
* `addLinksBatch` convention — caller owns chunking, engine assumes the
* caller is well-behaved).
*
* 500 is the same order-of-magnitude as the effective per-call budget for
* the existing `addLinksBatch` (postgres-engine.ts) — well under Postgres's
* 65535 parameter cap. We bind a single array parameter so the cap doesn't
* bite directly, but per-statement work stays bounded for predictable lock
* hold time + write-amplification budget.
*
* The same constant is also used for batch `SELECT slug, source_path FROM
* pages WHERE source_path = ANY($1::text[])` in `engine.resolveSlugsByPaths`.
*/
export const DELETE_BATCH_SIZE = 500;
src/core/engine.ts
+59
View File
@@ -712,8 +712,67 @@ export interface BrainEngine {
* delete doesn't hard-delete the same-slug pages in sources B/C/D. Without
* it, the bare DELETE matches every row with that slug across all sources.
* Cascades through content_chunks / page_links / chunk_relations via FKs.
*
* v0.41.19.0 (CDX-11): single-row primitive used by `purgeDeletedPages`,
* `gbrain sync` (one path per call), test setup teardown, and the v0.41.19.0
* sync-delete decompose path (when `deletePages` throws on a 500-row batch,
* the sync loop falls back to per-slug `deletePage` to log unrecoverable
* failures to `failedFiles`). `gbrain sync` calls this on EVERY run that
* sees a deleted file — it is NOT admin-only.
*/
deletePage(slug: string, opts?: { sourceId?: string }): Promise<void>;
/**
* v0.41.19.0 — batch delete: single SQL round-trip via
* `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2
* RETURNING slug`. Cascades through content_chunks / page_links (×3) /
* tags / raw_data / timeline_entries / page_versions via FKs declared in
* `src/schema.sql`. `files.page_id` and `links.origin_page_id` go SET
* NULL per their FK definitions.
*
* SINGLE-BATCH PRIMITIVE: caller is responsible for chunking the input to
* `<= DELETE_BATCH_SIZE` entries per call (see
* `src/core/engine-constants.ts`). Matches the `addLinksBatch` convention
* — engine assumes well-behaved input, caller owns the slicing.
*
* Returns the slugs of rows ACTUALLY DELETED (order undefined). Callers
* use this to filter their own `pagesAffected` tracking so downstream
* phases don't waste lookups on phantom slugs (paths that were in the
* deletion list but had no DB row).
*
* ATOMICITY: one statement, one transaction. The whole batch commits or
* the whole batch rolls back. Coarser than the per-row `deletePage`
* cadence — a mid-loop abort or transient connection failure can roll
* back up to `DELETE_BATCH_SIZE - 1` successful deletes from the
* in-flight batch. `gbrain sync` is idempotent (next run picks them up
* via git diff); other callers should account for the contract.
*
* sourceId is REQUIRED (no `'default'` fallback). This is asymmetric with
* `deletePage` (which keeps the optional/'default' fallback for back-
* compat). Filed as v0.42+ TODO to tighten `deletePage` to match once a
* full caller audit confirms every site threads `sourceId`.
*/
deletePages(slugs: string[], opts: { sourceId: string }): Promise<string[]>;
/**
* v0.41.19.0 — batch path → slug resolution. Single SQL round-trip via
* `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[])
* AND source_id = $2`. Returns `Map<path, slug>`; paths NOT in the map
* have no `source_path` row in the DB and the caller is expected to fall
* back to `resolveSlugForPath(path)` for the path-derived slug.
*
* Mirrors the contract of the single-call `resolveSlugByPathOrSourcePath`
* helper in `src/commands/sync.ts`, batched. As of v0.41.19.0, that
* single-call helper is implemented on top of this method (one Map
* allocation per single-path call; negligible cost; one owner of the SQL
* + fallback semantics).
*
* SINGLE-BATCH PRIMITIVE: caller chunks to `<= DELETE_BATCH_SIZE`.
*
* Empty `paths` short-circuits to an empty Map without touching the DB.
*/
resolveSlugsByPaths(
paths: string[],
opts: { sourceId: string },
): Promise<Map<string, string>>;
/**
* v0.26.5 — set `deleted_at = now()` on a page. Returns the slug if a row
* was soft-deleted, null if no row matched (already soft-deleted OR not found).
src/core/migrate.ts
+64
View File
@@ -4856,6 +4856,70 @@ export const MIGRATIONS: Migration[] = [
ON extract_rollup_7d (day);
`,
},
{
version: 107,
name: 'page_generation_clock_and_statement_trigger',
// v0.41.25.0 (D18/D19, codex outside-voice on /plan-eng-review): global
// page-generation clock + statement-level trigger.
//
// Renumbered v104 → v105 → v106 → v107 during master merges:
// PR #1545 (v0.41.21.0 ops-fix-wave) took v104 for pages_atom_source_hash_idx;
// PR #1542 (v0.41.22.0 type-unification cathedral) took v105 for slug_aliases;
// PR #1541 (v0.41.23.0 extract operator surfaces) took v106 for extract_rollup_7d_table.
//
// Why this exists: the pre-v0.41.25.0 query-cache Layer 1 bookmark read
// `MAX(generation) FROM pages` to detect "writes happened since cache
// store". Two bugs in that contract — independent of any sync work:
//
// 1. The row-level `bump_page_generation_trg` (migration v91) sets
// `NEW.generation = OLD.generation + 1` on UPDATE. Updating a
// NON-MAX page didn't advance MAX(generation). Cache silently
// served stale results for any UPDATE-to-non-max page.
// 2. The trigger is BEFORE INSERT OR UPDATE — DELETE doesn't fire it
// at all. Even an AFTER DELETE wouldn't move MAX (surviving rows
// are untouched).
//
// The fix: single-row counter, bumped per-statement (FOR EACH STATEMENT
// — row-level would turn a 73K-row batch DELETE into 73K UPDATEs on the
// same counter, recreating the bottleneck the sync-delete wave is
// fixing in this same PR). Layer 1 reads page_generation_clock.value
// directly. Per-row pages.generation stays for Layer 2 (per-page
// snapshot via jsonb_each + LEFT JOIN pages) which doesn't care about
// MAX, only per-page advancement.
//
// Seeded with COALESCE(MAX(pages.generation), 0) so existing
// query_cache rows stored under the old MAX semantics aren't all
// instantly invalidated on upgrade. Their max_generation_at_store
// stamp compares cleanly against the seeded clock; future writes bump
// the clock and the bookmark fires correctly.
//
// Mirror lives in src/core/pglite-schema.ts (fresh-install path).
// Forward-reference bootstrap probe in applyForwardReferenceBootstrap
// on both engines so pre-v0.41.25.0 brains pick it up cleanly.
idempotent: true,
sql: `
CREATE TABLE IF NOT EXISTS page_generation_clock (
id INTEGER PRIMARY KEY CHECK (id = 1),
value BIGINT NOT NULL DEFAULT 0
);
INSERT INTO page_generation_clock (id, value)
VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0))
ON CONFLICT (id) DO NOTHING;
CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$
BEGIN
UPDATE page_generation_clock SET value = value + 1 WHERE id = 1;
RETURN NULL;
END;
$func$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages;
CREATE TRIGGER bump_page_generation_clock_trg
AFTER INSERT OR UPDATE OR DELETE ON pages
FOR EACH STATEMENT
EXECUTE FUNCTION bump_page_generation_clock_fn();
`,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
src/core/pglite-engine.ts
+44
View File
@@ -22,6 +22,7 @@ import { logBatchRetry as auditLogBatchRetry, logBatchExhausted as auditLogBatch
import { runMigrations } from './migrate.ts';
import { PGLITE_SCHEMA_SQL, getPGLiteSchema } from './pglite-schema.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
import { acquireLock, releaseLock, type LockHandle } from './pglite-lock.ts';
import type {
Page, PageInput, PageFilters, PageType,
@@ -900,6 +901,49 @@ export class PGLiteEngine implements BrainEngine {
);
}
/**
* v0.41.19.0 — batch delete primitive. See BrainEngine.deletePages JSDoc.
* Parity implementation with PostgresEngine.deletePages. PGLite supports
* `slug = ANY($1)` array-param binding natively (addLinksBatch already
* proves this).
*/
async deletePages(slugs: string[], opts: { sourceId: string }): Promise<string[]> {
if (slugs.length === 0) return [];
if (slugs.length > DELETE_BATCH_SIZE) {
throw new Error(
`deletePages: input size ${slugs.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`,
);
}
const { rows } = await this.db.query<{ slug: string }>(
'DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug',
[slugs, opts.sourceId],
);
return rows.map(r => r.slug);
}
/**
* v0.41.19.0 — batch path → slug resolution. See BrainEngine.resolveSlugsByPaths
* JSDoc.
*/
async resolveSlugsByPaths(
paths: string[],
opts: { sourceId: string },
): Promise<Map<string, string>> {
if (paths.length === 0) return new Map();
if (paths.length > DELETE_BATCH_SIZE) {
throw new Error(
`resolveSlugsByPaths: input size ${paths.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`,
);
}
const { rows } = await this.db.query<{ slug: string; source_path: string }>(
'SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2',
[paths, opts.sourceId],
);
const m = new Map<string, string>();
for (const r of rows) m.set(r.source_path, r.slug);
return m;
}
async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> {
// Idempotent-as-null: only flip rows currently active. Source filter is
// optional; without it the first matching row across sources gets soft-deleted.
src/core/pglite-schema.ts
+26
View File
@@ -144,6 +144,32 @@ CREATE TRIGGER bump_page_generation_trg
FOR EACH ROW
EXECUTE FUNCTION bump_page_generation_fn();
-- v0.41.19.0 (D18/D19, mirror of src/schema.sql): global page-generation
-- clock + statement-level trigger. See src/schema.sql for the full
-- rationale comment. Layer 1 bookmark reads page_generation_clock.value;
-- per-row pages.generation above stays as the Layer 2 (per-page snapshot)
-- substrate.
CREATE TABLE IF NOT EXISTS page_generation_clock (
id INTEGER PRIMARY KEY CHECK (id = 1),
value BIGINT NOT NULL DEFAULT 0
);
INSERT INTO page_generation_clock (id, value)
VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0))
ON CONFLICT (id) DO NOTHING;
CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$
BEGIN
UPDATE page_generation_clock SET value = value + 1 WHERE id = 1;
RETURN NULL;
END;
$func$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages;
CREATE TRIGGER bump_page_generation_clock_trg
AFTER INSERT OR UPDATE OR DELETE ON pages
FOR EACH STATEMENT
EXECUTE FUNCTION bump_page_generation_clock_fn();
CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
src/core/postgres-engine.ts
+48
View File
@@ -57,6 +57,7 @@ import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, pa
import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts';
import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause, buildRecencyComponentSql } from './search/sql-ranking.ts';
import { DEFAULT_EMBEDDING_MODEL, DEFAULT_EMBEDDING_DIMENSIONS } from './ai/defaults.ts';
import { DELETE_BATCH_SIZE } from './engine-constants.ts';
function escapeSqlStringLiteral(value: string): string {
return value.replace(/'/g, "''");
@@ -911,6 +912,53 @@ export class PostgresEngine implements BrainEngine {
await sql`DELETE FROM pages WHERE slug = ${slug} AND source_id = ${sourceId}`;
}
/**
* v0.41.19.0 — batch delete primitive. See BrainEngine.deletePages JSDoc.
* Single SQL round-trip per call; caller is responsible for chunking input
* to <= DELETE_BATCH_SIZE. RETURNING slug projects the actually-deleted set
* so the caller can filter pagesAffected.
*/
async deletePages(slugs: string[], opts: { sourceId: string }): Promise<string[]> {
if (slugs.length === 0) return [];
if (slugs.length > DELETE_BATCH_SIZE) {
throw new Error(
`deletePages: input size ${slugs.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`,
);
}
const sql = this.sql;
const rows = await sql<{ slug: string }[]>`
DELETE FROM pages
WHERE slug = ANY(${slugs}::text[]) AND source_id = ${opts.sourceId}
RETURNING slug
`;
return rows.map(r => r.slug);
}
/**
* v0.41.19.0 — batch path → slug resolution. See BrainEngine.resolveSlugsByPaths
* JSDoc. Single SQL round-trip; folds rows into a Map.
*/
async resolveSlugsByPaths(
paths: string[],
opts: { sourceId: string },
): Promise<Map<string, string>> {
if (paths.length === 0) return new Map();
if (paths.length > DELETE_BATCH_SIZE) {
throw new Error(
`resolveSlugsByPaths: input size ${paths.length} exceeds DELETE_BATCH_SIZE=${DELETE_BATCH_SIZE}. Caller must chunk.`,
);
}
const sql = this.sql;
const rows = await sql<{ slug: string; source_path: string }[]>`
SELECT slug, source_path
FROM pages
WHERE source_path = ANY(${paths}::text[]) AND source_id = ${opts.sourceId}
`;
const m = new Map<string, string>();
for (const r of rows) m.set(r.source_path, r.slug);
return m;
}
async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> {
const sql = this.sql;
const sourceId = opts?.sourceId;
src/core/schema-embedded.ts
+108 -3
View File
@@ -185,6 +185,48 @@ CREATE TRIGGER bump_page_generation_trg
-- CREATE INDEX since the table is empty.
CREATE INDEX IF NOT EXISTS pages_generation_idx ON pages (generation);
-- v0.41.19.0 (D18/D19, codex outside-voice): global page-generation clock.
-- The pre-v0.41.19.0 Layer 1 bookmark read \`MAX(generation) FROM pages\` to
-- detect "writes happened since cache-store". Two bugs in that contract:
-- 1. The row-level trigger above sets \`NEW.generation = OLD.generation + 1\`
-- on UPDATE. Updating a NON-MAX page didn't advance MAX(generation),
-- silently serving stale cached results.
-- 2. The trigger is \`BEFORE INSERT OR UPDATE\` so DELETE doesn't fire it
-- at all — and even if it did, DELETE doesn't touch surviving rows,
-- so MAX(generation) wouldn't budge.
--
-- The fix: a single-row counter, bumped per-statement (FOR EACH STATEMENT
-- — codex CDX-4: per-row would turn 73K-row batch DELETE into 73K UPDATEs
-- on the same counter, recreating the bottleneck this PR is fixing). Layer
-- 1 reads \`page_generation_clock.value\` directly. The per-row
-- \`pages.generation\` column above stays as the Layer 2 (per-page snapshot)
-- substrate.
--
-- Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache
-- rows stored under the old MAX semantics aren't all instantly invalidated
-- on upgrade (their max_generation_at_store stamp compares cleanly against
-- the seeded clock; future writes bump the clock, bookmark fires correctly).
CREATE TABLE IF NOT EXISTS page_generation_clock (
id INTEGER PRIMARY KEY CHECK (id = 1),
value BIGINT NOT NULL DEFAULT 0
);
INSERT INTO page_generation_clock (id, value)
VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0))
ON CONFLICT (id) DO NOTHING;
CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS \$func\$
BEGIN
UPDATE page_generation_clock SET value = value + 1 WHERE id = 1;
RETURN NULL;
END;
\$func\$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages;
CREATE TRIGGER bump_page_generation_clock_trg
AFTER INSERT OR UPDATE OR DELETE ON pages
FOR EACH STATEMENT
EXECUTE FUNCTION bump_page_generation_clock_fn();
CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
@@ -266,6 +308,13 @@ CREATE INDEX IF NOT EXISTS idx_chunks_embedding_image
CREATE INDEX IF NOT EXISTS idx_chunks_search_vector ON content_chunks USING GIN(search_vector);
CREATE INDEX IF NOT EXISTS idx_chunks_symbol_qualified
ON content_chunks(symbol_name_qualified) WHERE symbol_name_qualified IS NOT NULL;
-- v0.41.18.0 (codex finding #9): partial index for \`gbrain embed --stale\`
-- + \`--priority recent\`. content_chunks has no updated_at column (chunks
-- are re-INSERTed on page change, not UPDATEd), so the "recent-first"
-- ORDER BY happens at the JOIN site: outer ORDER BY p.updated_at DESC
-- uses idx_pages_updated_at_desc; inner partial uses this index.
CREATE INDEX IF NOT EXISTS content_chunks_stale_idx
ON content_chunks(page_id, chunk_index) WHERE embedding IS NULL;
-- v0.20.0 Cathedral II: chunk-grain FTS trigger.
-- Weight 'A' on doc_comment + symbol_name_qualified; weight 'B' on chunk_text.
@@ -357,10 +406,16 @@ CREATE TABLE IF NOT EXISTS links (
to_page_id INTEGER NOT NULL REFERENCES pages(id) ON DELETE CASCADE,
link_type TEXT NOT NULL DEFAULT '',
context TEXT NOT NULL DEFAULT '',
-- v0.42.0.0: 'mentions' added for auto-linked body-text mentions
-- v0.41.18.0: 'mentions' added for auto-linked body-text mentions
-- (gbrain extract links --by-mention). Filtered OUT of backlink-count
-- for search ranking; only counts toward orphan-ratio + graph traversal.
link_source TEXT CHECK (link_source IS NULL OR link_source IN ('markdown', 'frontmatter', 'manual', 'mentions')),
-- v0.41.18.0: nullable link_kind distinguishes "plain body mention" from
-- "verb-pattern-derived typed link" within link_source='mentions'.
-- Codex finding #12 design: keep link_source stable; add link_kind
-- so callers can distinguish without breaking existing mentions queries.
-- NULL = legacy / unknown / pre-v98 row (semantically 'plain').
link_kind TEXT CHECK (link_kind IS NULL OR link_kind IN ('plain', 'typed_ner')),
origin_page_id INTEGER REFERENCES pages(id) ON DELETE SET NULL,
origin_field TEXT,
-- v0.18.0 Step 4: 'qualified' when the link was written as
@@ -423,8 +478,10 @@ CREATE TABLE IF NOT EXISTS timeline_entries (
CREATE INDEX IF NOT EXISTS idx_timeline_page ON timeline_entries(page_id);
CREATE INDEX IF NOT EXISTS idx_timeline_date ON timeline_entries(date);
-- Dedup constraint: same (page, date, summary) treated as same event
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary);
-- v0.41.18.0 (codex finding #11): widened from (page_id, date, summary) to
-- include \`source\` so distinct meeting provenance survives. Legacy rows
-- have source='' (schema default) so legacy dedup behavior is preserved.
CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup ON timeline_entries(page_id, date, summary, source);
-- ============================================================
-- page_versions: snapshot history for compiled_truth
@@ -589,6 +646,10 @@ CREATE TABLE IF NOT EXISTS op_checkpoints (
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
ON op_checkpoints (updated_at);
-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676)
-- because its \`job_id BIGINT REFERENCES minion_jobs(id)\` FK requires
-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix.
-- ============================================================
-- files: binary attachments stored in Supabase Storage
-- ============================================================
@@ -777,6 +838,50 @@ CREATE TABLE IF NOT EXISTS minion_attachments (
CREATE INDEX IF NOT EXISTS idx_minion_attachments_job ON minion_attachments (job_id);
ALTER TABLE minion_attachments ALTER COLUMN content SET STORAGE EXTERNAL;
-- ============================================================
-- migration_impact_log: before/after metric stats per onboard remediation
-- ============================================================
-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the
-- onboard remediation pipeline records before/after metric stats so
-- \`gbrain onboard --history --json\` can show "you reduced orphans 47%".
-- delta computed at read time (NOT a stored GENERATED column —
-- zero PGLite parity risk per eng-review D2).
--
-- Attribution columns (job_id, source_id, brain_id, started_at,
-- idempotency_key) per codex finding #10 so concurrent onboard /
-- autopilot / manual runs can't misattribute deltas to the wrong
-- migration when overlapping runs change the same metric.
--
-- v0.41.25.0 SCHEMA_SQL ordering fix: this block lives AFTER the
-- minion_jobs CREATE TABLE so the \`job_id REFERENCES minion_jobs(id)\`
-- FK can resolve on fresh-install schema replay. Originally placed above
-- minion_jobs in v0.41.18.0; that fired ERROR: relation "minion_jobs"
-- does not exist on every fresh-install initSchema() (silent on master
-- because postgres-js's unsafe() continued past the error, but the
-- table never got created so any later query on migration_impact_log
-- threw 42P01 — which cascaded as "relation minion_jobs does not exist"
-- whenever subsequent statements that referenced minion_jobs ran AFTER
-- the failed CREATE TABLE statement, aborting the entire SCHEMA_SQL batch).
CREATE TABLE IF NOT EXISTS migration_impact_log (
id BIGSERIAL PRIMARY KEY,
remediation_id TEXT NOT NULL,
metric_name TEXT NOT NULL,
metric_before NUMERIC,
metric_after NUMERIC,
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
source_id TEXT,
brain_id TEXT,
started_at TIMESTAMPTZ,
idempotency_key TEXT,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
applied_by TEXT,
details JSONB DEFAULT '{}'::jsonb
);
CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
ON migration_impact_log(remediation_id, applied_at DESC);
CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;
-- ============================================================
-- Subagent runtime (v0.16.0) — durable LLM loops
-- ============================================================
src/core/search/query-cache-gate.ts
+72 -31
View File
@@ -1,27 +1,48 @@
/**
* Cache invalidation gate (v0.40.3.0 — D2 + D6 + D11)
* Cache invalidation gate (v0.40.3.0 base — D2 + D6 + D11; v0.41.19.0
* rewrite — D18 + D20, codex CDX-1/CDX-2/CDX-5/CDX-6).
*
* Two pure helpers wired by query-cache.ts at store + lookup time. Pure
* surface lets us unit-test the two-layer gate logic without a real cache.
*
* Layer 1 (cheap bookmark): `MAX(generation) FROM pages` <=
* `query_cache.max_generation_at_store`. If true, brain has not been
* written since this row stored, so the row is fresh corpus-wide.
* Layer 1 (cheap bookmark): `page_generation_clock.value` <=
* `query_cache.max_generation_at_store`. If true, no page write has
* happened since this row stored, so the row is fresh corpus-wide.
*
* Layer 2 (per-page snapshot): if bookmark fires, fall through to the
* `page_generations JSONB` snapshot. For each `(page_id, stored_gen)`
* pair, compare to current `pages.generation`. Any mismatch (page
* deleted, page bumped) invalidates.
* deleted, page bumped) invalidates. **Empty `page_generations = {}`
* does NOT pass Layer 2 in v0.41.19.0+** — empty snapshots have no
* per-page signal to invalidate against, so they MUST rely on Layer 1
* exclusively (CDX-6 fix: pre-v0.41.19.0 the vacuous-valid path let
* empty-result cache rows survive across writes that should have
* invalidated them).
*
* Why the rewrite: pre-v0.41.19.0 Layer 1 read `MAX(generation) FROM
* pages`, but the per-row trigger sets `NEW.generation = OLD.generation
* + 1` on UPDATE. Updating a NON-MAX page didn't advance MAX(generation),
* so the bookmark silently passed stale cache rows (codex CDX-2). DELETE
* doesn't fire the trigger AT ALL, so deletion didn't advance MAX either
* (codex CDX-1). Migration v105 introduces a global single-row counter
* (`page_generation_clock`) bumped per-statement by a separate trigger;
* Layer 1 now reads the counter directly so every INSERT/UPDATE/DELETE
* statement advances the bookmark exactly once regardless of which rows
* changed.
*
* Backward compat: rows stored before v0.40.3.0 have
* `max_generation_at_store = 0` AND `page_generations = '{}'::jsonb`.
* Bookmark check: `MAX <= 0` is false on any populated brain, so we fall
* through to Layer 2; Layer 2 sees `'{}'::jsonb` and is vacuously valid.
* Legacy rows continue to serve naturally (IRON-RULE regression pinned
* in test/e2e/cache-gate-pglite.test.ts).
* Layer 1 check on a populated brain: `clock > 0` so Layer 1 fails.
* Layer 2 v0.41.19.0+ stricter: empty `{}` no longer passes — legacy
* rows invalidate on first post-upgrade lookup. Cache fills back up
* naturally; correct semantics restored. The pre-v0.41.19.0 IRON-RULE
* "legacy rows serve via vacuously-valid Layer 2" is intentionally
* reversed: that path was the CDX-6 bug.
*
* See plan ~/.claude/plans/system-instruction-you-are-working-enchanted-mountain.md
* Phase 2A for full design.
* Per-page `pages.generation` column + its row-level trigger
* (`bump_page_generation_trg`) stay in place. Layer 2 reads from them
* — Layer 2 only needs per-page advancement (which the row-level trigger
* delivers correctly), NOT a MAX-style aggregate.
*/
import type { BrainEngine } from '../engine.ts';
@@ -73,14 +94,20 @@ export async function buildPageGenerationsSnapshot(
try {
if (pageIds.length === 0) {
// Empty-result query: only need the Layer 1 bookmark (clock value).
// Per D20, empty-result cache rows trust Layer 1 exclusively;
// bumping the clock on subsequent writes correctly invalidates them.
const rows = await engine.executeRaw<{ v: number }>(
`SELECT COALESCE(MAX(generation), 0)::bigint AS v FROM pages`,
`SELECT COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v`,
);
snapshot.max_generation_at_store = Number(rows[0]?.v ?? 0);
return snapshot;
}
// Combined query: per-page generation + corpus-state MAX.
// Combined query: per-page generation (Layer 2 substrate) + global
// clock value (Layer 1 bookmark). UNION ALL folds both into one
// round trip. The 'CLOCK' tag row is identified by `is_max = true`
// (field name preserved for back-compat at the call site).
const rows = await engine.executeRaw<{
k: string;
v: number;
@@ -89,8 +116,9 @@ export async function buildPageGenerationsSnapshot(
`SELECT id::text AS k, generation::bigint AS v, FALSE AS is_max
FROM pages WHERE id = ANY($1::int[])
UNION ALL
SELECT 'MAX' AS k, COALESCE(MAX(generation), 0)::bigint AS v, TRUE AS is_max
FROM pages`,
SELECT 'CLOCK' AS k,
COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v,
TRUE AS is_max`,
[pageIds],
);
@@ -104,10 +132,12 @@ export async function buildPageGenerationsSnapshot(
}
return snapshot;
} catch {
// Pre-v91 brain (no `generation` column yet). Return the
// backward-compat empty snapshot with zero bookmark — every cache
// row will fall through to Layer 2 and serve via the `'{}'::jsonb`
// vacuously-valid path. Closes the upgrade-path gap.
// Pre-v105 brain (no `page_generation_clock` table yet). Return the
// empty snapshot with zero bookmark — every cache row will fall
// through to Layer 2 (which is stricter post-v0.41.19.0 and will
// invalidate empty snapshots). Acceptable upgrade-path one-time
// cache miss; migration v105 fills the table within the same
// initSchema() call so this branch is short-lived.
return snapshot;
}
}
@@ -127,17 +157,22 @@ export async function buildPageGenerationsSnapshot(
*/
export const CACHE_GATE_WHERE_CLAUSE = `
(
-- Layer 1 (cheap bookmark): O(log N) MAX(generation) via pages_generation_idx.
-- If no page has been bumped since this row stored, the row is fresh.
(SELECT COALESCE(MAX(generation), 0) FROM pages) <= qc.max_generation_at_store
-- Layer 1 (cheap bookmark): O(1) single-row read from page_generation_clock.
-- Bumped per-statement by bump_page_generation_clock_trg on every INSERT,
-- UPDATE, or DELETE on pages. If no statement has fired since this row
-- stored, the row is fresh corpus-wide.
COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)
<= qc.max_generation_at_store
OR
-- Layer 2 (per-page snapshot): bookmark fired, but maybe this row's
-- specific result set isn't affected. Pre-v0.40.3.0 rows have
-- page_generations = '{}'::jsonb and serve vacuously (legacy compat —
-- IRON-RULE regression in test/e2e/cache-gate-pglite.test.ts).
-- Layer 2 (per-page snapshot): bookmark fired, but maybe THIS row's
-- specific result set isn't affected. v0.41.19.0+ requires the snapshot
-- to be non-empty — empty {} snapshots cannot disprove staleness, so
-- they invalidate when Layer 1 fails (D20 / codex CDX-6 fix). Per-page
-- mismatch (page deleted via LEFT JOIN p.id IS NULL, or page generation
-- bumped) invalidates.
(
qc.page_generations = '{}'::jsonb
OR NOT EXISTS (
qc.page_generations <> '{}'::jsonb
AND NOT EXISTS (
SELECT 1
FROM jsonb_each(qc.page_generations) AS g(page_id, stored_gen)
LEFT JOIN pages p ON p.id = (g.page_id)::int
@@ -167,12 +202,18 @@ export function validateCacheRowAgainstPages(
page_generations: Record<string, number | undefined>;
},
): boolean {
// Layer 1: bookmark.
// Layer 1: bookmark. `current.max_generation` is the global clock value
// (kept named max_generation for back-compat at call sites; the underlying
// read source switched from MAX(pages.generation) to
// page_generation_clock.value in v0.41.19.0).
if (current.max_generation <= snapshot.max_generation_at_store) return true;
// Layer 2: per-page snapshot.
// Layer 2 (v0.41.19.0+ stricter, D20 / codex CDX-6): empty per-page
// snapshots cannot disprove staleness, so they invalidate when Layer 1
// fails. Pre-v0.41.19.0 callers got a "vacuously valid" pass that
// silently served stale empty-result rows across writes.
const ids = Object.keys(snapshot.page_generations);
if (ids.length === 0) return true; // Vacuously valid (legacy + zero-page).
if (ids.length === 0) return false;
for (const id of ids) {
const storedGen = snapshot.page_generations[id];
src/schema.sql
+89 -32
View File
@@ -181,6 +181,48 @@ CREATE TRIGGER bump_page_generation_trg
-- CREATE INDEX since the table is empty.
CREATE INDEX IF NOT EXISTS pages_generation_idx ON pages (generation);
-- v0.41.19.0 (D18/D19, codex outside-voice): global page-generation clock.
-- The pre-v0.41.19.0 Layer 1 bookmark read `MAX(generation) FROM pages` to
-- detect "writes happened since cache-store". Two bugs in that contract:
-- 1. The row-level trigger above sets `NEW.generation = OLD.generation + 1`
-- on UPDATE. Updating a NON-MAX page didn't advance MAX(generation),
-- silently serving stale cached results.
-- 2. The trigger is `BEFORE INSERT OR UPDATE` so DELETE doesn't fire it
-- at all — and even if it did, DELETE doesn't touch surviving rows,
-- so MAX(generation) wouldn't budge.
--
-- The fix: a single-row counter, bumped per-statement (FOR EACH STATEMENT
-- — codex CDX-4: per-row would turn 73K-row batch DELETE into 73K UPDATEs
-- on the same counter, recreating the bottleneck this PR is fixing). Layer
-- 1 reads `page_generation_clock.value` directly. The per-row
-- `pages.generation` column above stays as the Layer 2 (per-page snapshot)
-- substrate.
--
-- Seeded with COALESCE(MAX(pages.generation), 0) so existing query_cache
-- rows stored under the old MAX semantics aren't all instantly invalidated
-- on upgrade (their max_generation_at_store stamp compares cleanly against
-- the seeded clock; future writes bump the clock, bookmark fires correctly).
CREATE TABLE IF NOT EXISTS page_generation_clock (
id INTEGER PRIMARY KEY CHECK (id = 1),
value BIGINT NOT NULL DEFAULT 0
);
INSERT INTO page_generation_clock (id, value)
VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0))
ON CONFLICT (id) DO NOTHING;
CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$
BEGIN
UPDATE page_generation_clock SET value = value + 1 WHERE id = 1;
RETURN NULL;
END;
$func$ LANGUAGE plpgsql;
DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages;
CREATE TRIGGER bump_page_generation_clock_trg
AFTER INSERT OR UPDATE OR DELETE ON pages
FOR EACH STATEMENT
EXECUTE FUNCTION bump_page_generation_clock_fn();
CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type);
CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter);
CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops);
@@ -600,38 +642,9 @@ CREATE TABLE IF NOT EXISTS op_checkpoints (
CREATE INDEX IF NOT EXISTS op_checkpoints_updated_at_idx
ON op_checkpoints (updated_at);
-- ============================================================
-- migration_impact_log: before/after metric stats per onboard remediation
-- ============================================================
-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the
-- onboard remediation pipeline records before/after metric stats so
-- `gbrain onboard --history --json` can show "you reduced orphans 47%".
-- delta computed at read time (NOT a stored GENERATED column —
-- zero PGLite parity risk per eng-review D2).
--
-- Attribution columns (job_id, source_id, brain_id, started_at,
-- idempotency_key) per codex finding #10 so concurrent onboard /
-- autopilot / manual runs can't misattribute deltas to the wrong
-- migration when overlapping runs change the same metric.
CREATE TABLE IF NOT EXISTS migration_impact_log (
id BIGSERIAL PRIMARY KEY,
remediation_id TEXT NOT NULL,
metric_name TEXT NOT NULL,
metric_before NUMERIC,
metric_after NUMERIC,
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
source_id TEXT,
brain_id TEXT,
started_at TIMESTAMPTZ,
idempotency_key TEXT,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
applied_by TEXT,
details JSONB DEFAULT '{}'::jsonb
);
CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
ON migration_impact_log(remediation_id, applied_at DESC);
CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;
-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676)
-- because its `job_id BIGINT REFERENCES minion_jobs(id)` FK requires
-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix.
-- ============================================================
-- files: binary attachments stored in Supabase Storage
@@ -821,6 +834,50 @@ CREATE TABLE IF NOT EXISTS minion_attachments (
CREATE INDEX IF NOT EXISTS idx_minion_attachments_job ON minion_attachments (job_id);
ALTER TABLE minion_attachments ALTER COLUMN content SET STORAGE EXTERNAL;
-- ============================================================
-- migration_impact_log: before/after metric stats per onboard remediation
-- ============================================================
-- v0.41.18.0 (gbrain onboard wave). Every completion captured by the
-- onboard remediation pipeline records before/after metric stats so
-- `gbrain onboard --history --json` can show "you reduced orphans 47%".
-- delta computed at read time (NOT a stored GENERATED column —
-- zero PGLite parity risk per eng-review D2).
--
-- Attribution columns (job_id, source_id, brain_id, started_at,
-- idempotency_key) per codex finding #10 so concurrent onboard /
-- autopilot / manual runs can't misattribute deltas to the wrong
-- migration when overlapping runs change the same metric.
--
-- v0.41.25.0 SCHEMA_SQL ordering fix: this block lives AFTER the
-- minion_jobs CREATE TABLE so the `job_id REFERENCES minion_jobs(id)`
-- FK can resolve on fresh-install schema replay. Originally placed above
-- minion_jobs in v0.41.18.0; that fired ERROR: relation "minion_jobs"
-- does not exist on every fresh-install initSchema() (silent on master
-- because postgres-js's unsafe() continued past the error, but the
-- table never got created so any later query on migration_impact_log
-- threw 42P01 — which cascaded as "relation minion_jobs does not exist"
-- whenever subsequent statements that referenced minion_jobs ran AFTER
-- the failed CREATE TABLE statement, aborting the entire SCHEMA_SQL batch).
CREATE TABLE IF NOT EXISTS migration_impact_log (
id BIGSERIAL PRIMARY KEY,
remediation_id TEXT NOT NULL,
metric_name TEXT NOT NULL,
metric_before NUMERIC,
metric_after NUMERIC,
job_id BIGINT REFERENCES minion_jobs(id) ON DELETE SET NULL,
source_id TEXT,
brain_id TEXT,
started_at TIMESTAMPTZ,
idempotency_key TEXT,
applied_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
applied_by TEXT,
details JSONB DEFAULT '{}'::jsonb
);
CREATE INDEX IF NOT EXISTS migration_impact_log_remediation_idx
ON migration_impact_log(remediation_id, applied_at DESC);
CREATE INDEX IF NOT EXISTS migration_impact_log_attribution_idx
ON migration_impact_log(job_id, source_id) WHERE job_id IS NOT NULL;
-- ============================================================
-- Subagent runtime (v0.16.0) — durable LLM loops
-- ============================================================