fix(extract+migrate): kill N+1 hang + v0.12.0 migration timeout (v0.12.1) (#198)

* feat(engine): add addLinksBatch + addTimelineEntriesBatch via unnest()

Multi-row INSERT...SELECT FROM unnest() JOIN pages ON CONFLICT DO NOTHING
RETURNING 1. 4 array-typed bound parameters (links) or 5 (timeline)
regardless of batch size, sidesteps Postgres's 65535-parameter cap.

Returns count of rows actually inserted (excluding ON CONFLICT no-ops
and JOIN-dropped rows whose slugs don't exist).

Per-row addLink / addTimelineEntry signatures and SQL behavior unchanged.
All 10 existing call sites compile and behave identically.

Tests: 11 PGLite cases (empty batch, missing optionals, within-batch dedup,
JOIN drops missing slug, half-existing batch, batch of 100) + 9 E2E
postgres-engine cases against real Postgres+pgvector.

* fix(migrate): pre-create btree helper in v8 + v9 dedup; bump phaseASchema timeout

Production bug: v0.12.0 schema migration timed out at Supabase Management API's
60s ceiling on brains with 80K+ duplicate timeline rows. The DELETE...USING
self-join was O(n²) without an index on the dedup columns.

Fix: pre-create idx_links_dedup_helper / idx_timeline_dedup_helper on the
dedup columns BEFORE the DELETE, drop after. Turns O(n²) into O(n log n).
On 80K+ rows the migration completes in <1s instead of timing out.

Also bumps the v0.12.0 orchestrator's phaseASchema timeout 60s -> 600s as
belt-and-suspenders for unforeseen slowness.

Exports MIGRATIONS for structural test assertions.

Tests: 2 structural assertions (helper-index DDL must appear in v8/v9 SQL
in the right order — catches regression even at 0-row scale) + 2 behavioral
regression tests (1000-row dedup completes <5s).

* perf(extract): kill N+1 dedup pre-load; switch to batched writes

Production bug: gbrain extract hung 10+ minutes producing zero output on
47K-page brains. The pre-load loop called engine.getLinks(slug) (or
getTimeline) once per page across engine.listPages({limit: 100000}) — 47K
serial round-trips over the Supabase pooler before the first file was read.

Both engines already enforced uniqueness at the SQL layer
(UNIQUE(from, to, link_type) on links, idx_timeline_dedup on timeline_entries).
The in-memory dedup Set was redundant insurance that became the bottleneck.

Fix: delete the pre-load entirely. Buffer 100 candidates per file walk,
flush via engine.addLinksBatch / engine.addTimelineEntriesBatch. ~99% fewer
DB round-trips per re-extract.

Also fixes counter accuracy: 'created' now counts rows actually inserted
(via batch RETURNING 1 row count). Re-run on a fully-extracted brain
prints 'Done: 0 links' instead of lying.

Dry-run mode keeps a per-run dedup Set so duplicate candidates from N
markdown files print exactly once, not N times.

Batch errors are visible in BOTH json and human modes — silent loss of
100 rows is worse than per-row error visibility.

Tests: extract-fs.test.ts (idempotency + truthful counter + dry-run dedup
+ perf regression guard <2s).

* chore: bump version + changelog (v0.12.1)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: update CLAUDE.md for v0.12.1 (batch engine API, test counts)

Reflect what shipped in v0.12.1:
- New engine methods addLinksBatch + addTimelineEntriesBatch (PGLite via
  unnest() + manual $N, postgres-engine via INSERT...SELECT FROM
  unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING).
- extract.ts no longer pre-loads dedup set; candidates are buffered 100
  at a time and flushed via the new batch methods.
- v0.12.0 orchestrator phaseASchema timeout bumped 60s to 600s.
- Test counts 1297 unit / 105 E2E to 1412 unit / 119 E2E.
- New test/extract-fs.test.ts covers the N+1 regression guard.
- BrainEngine method count 37/38 to 40.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

src/core/postgres-engine.ts

@@ -1,5 +1,5 @@
 import postgres from 'postgres';
-import type { BrainEngine } from './engine.ts';
+import type { BrainEngine, LinkBatchInput, TimelineBatchInput } from './engine.ts';
 import { MAX_SEARCH_LIMIT, clampSearchLimit } from './engine.ts';
 import { runMigrations } from './migrate.ts';
 import { SCHEMA_SQL } from './schema-embedded.ts';
@@ -372,6 +372,30 @@ export class PostgresEngine implements BrainEngine {
     `;
   }
 
+  async addLinksBatch(links: LinkBatchInput[]): Promise<number> {
+    if (links.length === 0) return 0;
+    const sql = this.sql;
+    // unnest() pattern: 4 array-typed bound parameters regardless of batch size.
+    // Avoids the 65535-parameter cap and the postgres-js sql(rows, ...) helper's
+    // identifier-escape gotcha when used inside a (VALUES) subquery.
+    const fromSlugs = links.map(l => l.from_slug);
+    const toSlugs = links.map(l => l.to_slug);
+    // Normalize optional fields to '' to match per-row addLink + NOT NULL DDL.
+    const linkTypes = links.map(l => l.link_type || '');
+    const contexts = links.map(l => l.context || '');
+    const result = await sql`
+      INSERT INTO links (from_page_id, to_page_id, link_type, context)
+      SELECT f.id, t.id, v.link_type, v.context
+      FROM unnest(${fromSlugs}::text[], ${toSlugs}::text[], ${linkTypes}::text[], ${contexts}::text[])
+        AS v(from_slug, to_slug, link_type, context)
+      JOIN pages f ON f.slug = v.from_slug
+      JOIN pages t ON t.slug = v.to_slug
+      ON CONFLICT (from_page_id, to_page_id, link_type) DO NOTHING
+      RETURNING 1
+    `;
+    return result.length;
+  }
+
   async removeLink(from: string, to: string, linkType?: string): Promise<void> {
     const sql = this.sql;
     if (linkType !== undefined) {
@@ -629,6 +653,28 @@ export class PostgresEngine implements BrainEngine {
     `;
   }
 
+  async addTimelineEntriesBatch(entries: TimelineBatchInput[]): Promise<number> {
+    if (entries.length === 0) return 0;
+    const sql = this.sql;
+    // unnest() pattern: 5 array-typed bound parameters regardless of batch size.
+    const slugs = entries.map(e => e.slug);
+    const dates = entries.map(e => e.date);
+    // Normalize optional fields to '' to match per-row addTimelineEntry + NOT NULL DDL.
+    const sources = entries.map(e => e.source || '');
+    const summaries = entries.map(e => e.summary);
+    const details = entries.map(e => e.detail || '');
+    const result = await sql`
+      INSERT INTO timeline_entries (page_id, date, source, summary, detail)
+      SELECT p.id, v.date::date, v.source, v.summary, v.detail
+      FROM unnest(${slugs}::text[], ${dates}::text[], ${sources}::text[], ${summaries}::text[], ${details}::text[])
+        AS v(slug, date, source, summary, detail)
+      JOIN pages p ON p.slug = v.slug
+      ON CONFLICT (page_id, date, summary) DO NOTHING
+      RETURNING 1
+    `;
+    return result.length;
+  }
+
   async getTimeline(slug: string, opts?: TimelineOpts): Promise<TimelineEntry[]> {
     const sql = this.sql;
     const limit = opts?.limit || 100;