mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
fix(extract+migrate): kill N+1 hang + v0.12.0 migration timeout (v0.12.1) (#198)
* feat(engine): add addLinksBatch + addTimelineEntriesBatch via unnest()
Multi-row INSERT...SELECT FROM unnest() JOIN pages ON CONFLICT DO NOTHING
RETURNING 1. 4 array-typed bound parameters (links) or 5 (timeline)
regardless of batch size, sidesteps Postgres's 65535-parameter cap.
Returns count of rows actually inserted (excluding ON CONFLICT no-ops
and JOIN-dropped rows whose slugs don't exist).
Per-row addLink / addTimelineEntry signatures and SQL behavior unchanged.
All 10 existing call sites compile and behave identically.
Tests: 11 PGLite cases (empty batch, missing optionals, within-batch dedup,
JOIN drops missing slug, half-existing batch, batch of 100) + 9 E2E
postgres-engine cases against real Postgres+pgvector.
* fix(migrate): pre-create btree helper in v8 + v9 dedup; bump phaseASchema timeout
Production bug: v0.12.0 schema migration timed out at Supabase Management API's
60s ceiling on brains with 80K+ duplicate timeline rows. The DELETE...USING
self-join was O(n²) without an index on the dedup columns.
Fix: pre-create idx_links_dedup_helper / idx_timeline_dedup_helper on the
dedup columns BEFORE the DELETE, drop after. Turns O(n²) into O(n log n).
On 80K+ rows the migration completes in <1s instead of timing out.
Also bumps the v0.12.0 orchestrator's phaseASchema timeout 60s -> 600s as
belt-and-suspenders for unforeseen slowness.
Exports MIGRATIONS for structural test assertions.
Tests: 2 structural assertions (helper-index DDL must appear in v8/v9 SQL
in the right order — catches regression even at 0-row scale) + 2 behavioral
regression tests (1000-row dedup completes <5s).
* perf(extract): kill N+1 dedup pre-load; switch to batched writes
Production bug: gbrain extract hung 10+ minutes producing zero output on
47K-page brains. The pre-load loop called engine.getLinks(slug) (or
getTimeline) once per page across engine.listPages({limit: 100000}) — 47K
serial round-trips over the Supabase pooler before the first file was read.
Both engines already enforced uniqueness at the SQL layer
(UNIQUE(from, to, link_type) on links, idx_timeline_dedup on timeline_entries).
The in-memory dedup Set was redundant insurance that became the bottleneck.
Fix: delete the pre-load entirely. Buffer 100 candidates per file walk,
flush via engine.addLinksBatch / engine.addTimelineEntriesBatch. ~99% fewer
DB round-trips per re-extract.
Also fixes counter accuracy: 'created' now counts rows actually inserted
(via batch RETURNING 1 row count). Re-run on a fully-extracted brain
prints 'Done: 0 links' instead of lying.
Dry-run mode keeps a per-run dedup Set so duplicate candidates from N
markdown files print exactly once, not N times.
Batch errors are visible in BOTH json and human modes — silent loss of
100 rows is worse than per-row error visibility.
Tests: extract-fs.test.ts (idempotency + truthful counter + dry-run dedup
+ perf regression guard <2s).
* chore: bump version + changelog (v0.12.1)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: update CLAUDE.md for v0.12.1 (batch engine API, test counts)
Reflect what shipped in v0.12.1:
- New engine methods addLinksBatch + addTimelineEntriesBatch (PGLite via
unnest() + manual $N, postgres-engine via INSERT...SELECT FROM
unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING).
- extract.ts no longer pre-loads dedup set; candidates are buffered 100
at a time and flushed via the new batch methods.
- v0.12.0 orchestrator phaseASchema timeout bumped 60s to 600s.
- Test counts 1297 unit / 105 E2E to 1412 unit / 119 E2E.
- New test/extract-fs.test.ts covers the N+1 regression guard.
- BrainEngine method count 37/38 to 40.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
81b3f7afac
commit
699db50a3d
@@ -285,6 +285,136 @@ describeE2E('E2E: Timeline', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Batch methods (addLinksBatch / addTimelineEntriesBatch)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Postgres-engine batch methods use postgres-js's sql(rows, 'col1', ...) helper,
|
||||
// which is structurally different from PGLite's manual $N placeholder construction
|
||||
// (covered in test/pglite-engine.test.ts). These tests verify the postgres-js code
|
||||
// path against a real Postgres against the same invariants.
|
||||
|
||||
describeE2E('E2E: addLinksBatch (postgres-engine)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
await importFixtures();
|
||||
});
|
||||
afterAll(teardownDB);
|
||||
|
||||
test('empty batch returns 0 with no DB call', async () => {
|
||||
const engine = getEngine();
|
||||
expect(await engine.addLinksBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('within-batch duplicates dedup via ON CONFLICT (no 21000 cardinality error)', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
// Deterministic cleanup so re-runs aren't perturbed by prior fixture state.
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-dup'`;
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-dup' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-dup' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-dup'`;
|
||||
});
|
||||
|
||||
test('rows with missing slug silently dropped by JOIN', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-missing'`;
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/does-not-exist', to_slug: 'companies/novamind', link_type: 'e2e-batch-missing' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-missing' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-missing'`;
|
||||
});
|
||||
|
||||
test('half-existing batch returns count of new only', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-half'`;
|
||||
await engine.addLink('people/sarah-chen', 'companies/novamind', 'pre-existing', 'e2e-batch-half');
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind', link_type: 'e2e-batch-half' },
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'people/marcus-reid', link_type: 'e2e-batch-half' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM links WHERE link_type = 'e2e-batch-half'`;
|
||||
});
|
||||
|
||||
test('missing optional fields normalize to empty strings (NOT NULL safety)', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM links WHERE link_type = ''`;
|
||||
// No link_type, no context — must default to '' to satisfy NOT NULL.
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'people/sarah-chen', to_slug: 'companies/novamind' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
const rows = await conn`
|
||||
SELECT link_type, context FROM links
|
||||
WHERE from_page_id = (SELECT id FROM pages WHERE slug = 'people/sarah-chen')
|
||||
AND to_page_id = (SELECT id FROM pages WHERE slug = 'companies/novamind')
|
||||
AND link_type = ''
|
||||
`;
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].context).toBe('');
|
||||
await conn`DELETE FROM links WHERE link_type = ''`;
|
||||
});
|
||||
});
|
||||
|
||||
describeE2E('E2E: addTimelineEntriesBatch (postgres-engine)', () => {
|
||||
beforeAll(async () => {
|
||||
await setupDB();
|
||||
await importFixtures();
|
||||
});
|
||||
afterAll(teardownDB);
|
||||
|
||||
test('empty batch returns 0', async () => {
|
||||
const engine = getEngine();
|
||||
expect(await engine.addTimelineEntriesBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('within-batch duplicates dedup via ON CONFLICT', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-dup'`;
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-01', summary: 'e2e-batch-tl-dup' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-01', summary: 'e2e-batch-tl-dup' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-dup'`;
|
||||
});
|
||||
|
||||
test('rows with missing slug silently dropped by JOIN', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-missing'`;
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/no-such-page', date: '2025-05-02', summary: 'e2e-batch-tl-missing' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-02', summary: 'e2e-batch-tl-missing' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary = 'e2e-batch-tl-missing'`;
|
||||
});
|
||||
|
||||
test('mix of new + existing returns count of new only', async () => {
|
||||
const engine = getEngine();
|
||||
const conn = getConn();
|
||||
await conn`DELETE FROM timeline_entries WHERE summary IN ('e2e-batch-tl-half-1', 'e2e-batch-tl-half-2')`;
|
||||
await engine.addTimelineEntry('people/sarah-chen', { date: '2025-05-03', summary: 'e2e-batch-tl-half-1' });
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-03', summary: 'e2e-batch-tl-half-1' },
|
||||
{ slug: 'people/sarah-chen', date: '2025-05-04', summary: 'e2e-batch-tl-half-2' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
await conn`DELETE FROM timeline_entries WHERE summary IN ('e2e-batch-tl-half-1', 'e2e-batch-tl-half-2')`;
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Versions
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
/**
|
||||
* Tests for `gbrain extract --source fs` (the default, FS-walking path).
|
||||
*
|
||||
* Companion to test/extract-db.test.ts. Specifically guards against the
|
||||
* v0.12.0 N+1 hang: extractLinksFromDir / extractTimelineFromDir used to
|
||||
* pre-load the entire dedup set with one engine.getLinks() per page across
|
||||
* engine.listPages(), which on a 47K-page brain meant 47K sequential
|
||||
* round-trips before any work happened.
|
||||
*
|
||||
* Verifies:
|
||||
* 1. Single run extracts the expected links + timeline entries.
|
||||
* 2. Second run reports `created: 0` (proves DO NOTHING in batch + accurate
|
||||
* counter via RETURNING).
|
||||
* 3. --dry-run prints the same link found across multiple files exactly
|
||||
* once (proves the dry-run-only dedup Set works).
|
||||
* 4. Second run wall-clock < 2s (regression guard against any future change
|
||||
* that re-introduces the N+1 read pre-load).
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { runExtract } from '../src/commands/extract.ts';
|
||||
import type { PageInput } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let brainDir: string;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
async function truncateAll() {
|
||||
for (const t of ['content_chunks', 'links', 'tags', 'raw_data', 'timeline_entries', 'page_versions', 'ingest_log', 'pages']) {
|
||||
await (engine as any).db.exec(`DELETE FROM ${t}`);
|
||||
}
|
||||
}
|
||||
|
||||
const personPage = (title: string, body = ''): PageInput => ({
|
||||
type: 'person', title, compiled_truth: body, timeline: '',
|
||||
});
|
||||
|
||||
const companyPage = (title: string, body = ''): PageInput => ({
|
||||
type: 'company', title, compiled_truth: body, timeline: '',
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
brainDir = mkdtempSync(join(tmpdir(), 'gbrain-extract-fs-'));
|
||||
});
|
||||
|
||||
function writeFile(rel: string, content: string) {
|
||||
const full = join(brainDir, rel);
|
||||
mkdirSync(join(full, '..'), { recursive: true });
|
||||
writeFileSync(full, content);
|
||||
}
|
||||
|
||||
describe('gbrain extract links --source fs', () => {
|
||||
test('first run inserts links, second run reports 0 (idempotent + truthful counter)', async () => {
|
||||
// Set up brain in DB matching the file structure
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
await engine.putPage('people/bob', personPage('Bob'));
|
||||
await engine.putPage('companies/acme', companyPage('Acme'));
|
||||
|
||||
// Set up matching markdown files on disk
|
||||
writeFile('people/alice.md', '---\ntitle: Alice\n---\n\n[Bob](../people/bob.md) is a friend.\n');
|
||||
writeFile('people/bob.md', '---\ntitle: Bob\n---\n\nWorks at [Acme](../companies/acme.md).\n');
|
||||
writeFile('companies/acme.md', '---\ntitle: Acme\n---\n\nFounded by [Alice](../people/alice.md).\n');
|
||||
|
||||
// First run — write batch path
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const linksAfter1 = (await engine.getLinks('people/alice'))
|
||||
.concat(await engine.getLinks('people/bob'))
|
||||
.concat(await engine.getLinks('companies/acme'));
|
||||
expect(linksAfter1.length).toBeGreaterThanOrEqual(3);
|
||||
|
||||
// Second run — must dedup via ON CONFLICT and report 0 new (truthful counter)
|
||||
const start = Date.now();
|
||||
await runExtract(engine, ['links', '--dir', brainDir]);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
const linksAfter2 = (await engine.getLinks('people/alice'))
|
||||
.concat(await engine.getLinks('people/bob'))
|
||||
.concat(await engine.getLinks('companies/acme'));
|
||||
expect(linksAfter2.length).toBe(linksAfter1.length);
|
||||
|
||||
// Perf regression guard: re-run on tiny fixture must not loop through
|
||||
// listPages + per-page getLinks. ~10 files should complete in well under
|
||||
// 2s even on a slow CI box.
|
||||
expect(elapsedMs).toBeLessThan(2000);
|
||||
});
|
||||
|
||||
test('--dry-run dedups duplicate candidates across files (printed once, not N times)', async () => {
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
await engine.putPage('companies/acme', companyPage('Acme'));
|
||||
|
||||
// Same link target appears in 3 different files. The target file must
|
||||
// exist on disk so the FS extractor's allSlugs Set includes it.
|
||||
writeFile('companies/acme.md', '---\ntitle: Acme\n---\n');
|
||||
writeFile('a.md', '[Acme](companies/acme.md)\n');
|
||||
writeFile('b.md', '[Acme](companies/acme.md)\n');
|
||||
writeFile('c.md', '[Acme](companies/acme.md)\n');
|
||||
|
||||
// Capture stdout to check print frequency
|
||||
const lines: string[] = [];
|
||||
const origLog = console.log;
|
||||
console.log = (...args: unknown[]) => { lines.push(args.join(' ')); };
|
||||
try {
|
||||
await runExtract(engine, ['links', '--dry-run', '--dir', brainDir]);
|
||||
} finally {
|
||||
console.log = origLog;
|
||||
}
|
||||
|
||||
// Each (from, to, link_type) tuple should print at most once.
|
||||
// Three distinct from_slugs (a, b, c) all link to companies/acme, so
|
||||
// we expect 3 link lines (one per source file), not 9.
|
||||
const linkLines = lines.filter(l => l.includes('→') && l.includes('companies/acme'));
|
||||
expect(linkLines.length).toBe(3);
|
||||
|
||||
// No actual writes happened
|
||||
const links = await engine.getLinks('companies/acme');
|
||||
expect(links.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gbrain extract timeline --source fs', () => {
|
||||
test('first run inserts entries, second run reports 0 (idempotent + truthful counter)', async () => {
|
||||
await engine.putPage('people/alice', personPage('Alice'));
|
||||
|
||||
writeFile('people/alice.md', `---
|
||||
title: Alice
|
||||
---
|
||||
|
||||
## Timeline
|
||||
|
||||
- **2024-01-15** | source — Founded NovaMind
|
||||
- **2024-06-01** | source — Raised seed round
|
||||
`);
|
||||
|
||||
await runExtract(engine, ['timeline', '--dir', brainDir]);
|
||||
const after1 = await engine.getTimeline('people/alice');
|
||||
expect(after1.length).toBe(2);
|
||||
|
||||
const start = Date.now();
|
||||
await runExtract(engine, ['timeline', '--dir', brainDir]);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
const after2 = await engine.getTimeline('people/alice');
|
||||
expect(after2.length).toBe(2);
|
||||
|
||||
expect(elapsedMs).toBeLessThan(2000);
|
||||
});
|
||||
});
|
||||
+186
-3
@@ -1,5 +1,6 @@
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { LATEST_VERSION } from '../src/core/migrate.ts';
|
||||
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { LATEST_VERSION, runMigrations, MIGRATIONS } from '../src/core/migrate.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
|
||||
describe('migrate', () => {
|
||||
test('LATEST_VERSION is a number >= 1', () => {
|
||||
@@ -8,10 +9,192 @@ describe('migrate', () => {
|
||||
});
|
||||
|
||||
test('runMigrations is exported and callable', async () => {
|
||||
const { runMigrations } = await import('../src/core/migrate.ts');
|
||||
expect(typeof runMigrations).toBe('function');
|
||||
});
|
||||
|
||||
// Integration tests for actual migration execution require DATABASE_URL
|
||||
// and are covered in the E2E suite (test/e2e/mechanical.test.ts)
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// REGRESSION TESTS — migrations v8 + v9 perf on duplicate-heavy tables
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
//
|
||||
// Garry's production brain hit Supabase Management API's 60s ceiling because
|
||||
// the DELETE...USING self-join in migrations v8 + v9 was O(n²) without an
|
||||
// index on the dedup columns. The fix pre-creates a btree helper index
|
||||
// before the DELETE, then drops it. These tests guard against any future
|
||||
// change that re-introduces the missing helper index.
|
||||
//
|
||||
// Two-layer guard:
|
||||
// 1. Structural — assert the migration SQL literally contains the helper
|
||||
// CREATE INDEX + DROP INDEX (deterministic, fast, catches the regression
|
||||
// even at 0-row scale where wall-clock can't distinguish O(n²) from O(1)).
|
||||
// 2. Behavioral — populate 1000 duplicates and assert the migration completes
|
||||
// under the wall-clock cap. Sanity check at small scale; the structural
|
||||
// assertion is the real guard.
|
||||
|
||||
describe('migrations v8 + v9 — structural guard for helper-index fix', () => {
|
||||
test('migration v8 SQL contains idx_links_dedup_helper CREATE+DROP around the DELETE', () => {
|
||||
const v8 = MIGRATIONS.find(m => m.version === 8);
|
||||
expect(v8).toBeDefined();
|
||||
const sql = v8!.sql;
|
||||
|
||||
// The fix must: (a) create the helper btree, (b) DELETE...USING, (c) drop the helper, (d) add the unique constraint.
|
||||
// If anyone reorders or removes the helper-index lines, this fails.
|
||||
expect(sql).toContain('CREATE INDEX IF NOT EXISTS idx_links_dedup_helper');
|
||||
expect(sql).toContain('ON links(from_page_id, to_page_id, link_type)');
|
||||
expect(sql).toContain('DROP INDEX IF EXISTS idx_links_dedup_helper');
|
||||
expect(sql).toContain('DELETE FROM links a USING links b');
|
||||
expect(sql).toContain('ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique');
|
||||
|
||||
// Order matters: CREATE INDEX before DELETE, DROP INDEX after DELETE, before ADD CONSTRAINT.
|
||||
const createIdx = sql.indexOf('CREATE INDEX IF NOT EXISTS idx_links_dedup_helper');
|
||||
const deleteUsing = sql.indexOf('DELETE FROM links a USING links b');
|
||||
const dropIdx = sql.indexOf('DROP INDEX IF EXISTS idx_links_dedup_helper');
|
||||
const addConstraint = sql.indexOf('ALTER TABLE links ADD CONSTRAINT links_from_to_type_unique');
|
||||
expect(createIdx).toBeLessThan(deleteUsing);
|
||||
expect(deleteUsing).toBeLessThan(dropIdx);
|
||||
expect(dropIdx).toBeLessThan(addConstraint);
|
||||
});
|
||||
|
||||
test('migration v9 SQL contains idx_timeline_dedup_helper CREATE+DROP around the DELETE', () => {
|
||||
const v9 = MIGRATIONS.find(m => m.version === 9);
|
||||
expect(v9).toBeDefined();
|
||||
const sql = v9!.sql;
|
||||
|
||||
expect(sql).toContain('CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper');
|
||||
expect(sql).toContain('ON timeline_entries(page_id, date, summary)');
|
||||
expect(sql).toContain('DROP INDEX IF EXISTS idx_timeline_dedup_helper');
|
||||
expect(sql).toContain('DELETE FROM timeline_entries a USING timeline_entries b');
|
||||
expect(sql).toContain('CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup');
|
||||
|
||||
const createHelper = sql.indexOf('CREATE INDEX IF NOT EXISTS idx_timeline_dedup_helper');
|
||||
const deleteUsing = sql.indexOf('DELETE FROM timeline_entries a USING timeline_entries b');
|
||||
const dropHelper = sql.indexOf('DROP INDEX IF EXISTS idx_timeline_dedup_helper');
|
||||
const createUnique = sql.indexOf('CREATE UNIQUE INDEX IF NOT EXISTS idx_timeline_dedup');
|
||||
expect(createHelper).toBeLessThan(deleteUsing);
|
||||
expect(deleteUsing).toBeLessThan(dropHelper);
|
||||
expect(dropHelper).toBeLessThan(createUnique);
|
||||
});
|
||||
});
|
||||
|
||||
describe('migrate: v8 (links_dedup) regression — must be fast on 1K duplicate rows', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('1000 duplicate links dedup completes in <5s and leaves table deduped', async () => {
|
||||
// Set up: drop the unique constraint so duplicates can be inserted, then reset
|
||||
// version so v8 re-runs. Schema-embedded.ts already has the constraint, so
|
||||
// initSchema() above set it up; explicit DROP makes the test premise valid.
|
||||
const db = (engine as any).db;
|
||||
await db.exec(`ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_unique`);
|
||||
|
||||
// Two pages so the FK is satisfied
|
||||
await engine.putPage('p/from', { type: 'concept', title: 'F', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('p/to', { type: 'concept', title: 'T', compiled_truth: '', timeline: '' });
|
||||
const fromId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/from'`)).rows[0].id;
|
||||
const toId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/to'`)).rows[0].id;
|
||||
|
||||
// Insert 1000 duplicates of the same (from, to, type) row
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
await db.query(
|
||||
`INSERT INTO links (from_page_id, to_page_id, link_type, context) VALUES ($1, $2, $3, $4)`,
|
||||
[fromId, toId, 'mention', `dup-${i}`]
|
||||
);
|
||||
}
|
||||
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c;
|
||||
expect(beforeCount).toBe(1000);
|
||||
|
||||
// Reset version to 7 so v8 + v9 + v10 re-run
|
||||
await engine.setConfig('version', '7');
|
||||
|
||||
// Run migrations and assert wall-clock + correctness
|
||||
const start = Date.now();
|
||||
await runMigrations(engine);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
expect(elapsedMs).toBeLessThan(5000);
|
||||
|
||||
const afterCount = (await db.query(`SELECT COUNT(*)::int AS c FROM links`)).rows[0].c;
|
||||
expect(afterCount).toBe(1); // deduped to one row
|
||||
|
||||
// Unique constraint reinstated
|
||||
const constraints = (await db.query(`
|
||||
SELECT conname FROM pg_constraint
|
||||
WHERE conrelid = 'links'::regclass AND contype = 'u'
|
||||
`)).rows;
|
||||
expect(constraints.some((c: { conname: string }) => c.conname === 'links_from_to_type_unique')).toBe(true);
|
||||
|
||||
// Helper index was dropped after dedup
|
||||
const helperIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'links' AND indexname = 'idx_links_dedup_helper'
|
||||
`)).rows;
|
||||
expect(helperIdx.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('migrate: v9 (timeline_dedup_index) regression — must be fast on 1K duplicate rows', () => {
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
test('1000 duplicate timeline entries dedup completes in <5s and leaves table deduped', async () => {
|
||||
const db = (engine as any).db;
|
||||
await db.exec(`DROP INDEX IF EXISTS idx_timeline_dedup`);
|
||||
|
||||
await engine.putPage('p/timeline', { type: 'concept', title: 'TL', compiled_truth: '', timeline: '' });
|
||||
const pageId = (await db.query(`SELECT id FROM pages WHERE slug = 'p/timeline'`)).rows[0].id;
|
||||
|
||||
// Insert 1000 duplicates of the same (page_id, date, summary) row
|
||||
for (let i = 0; i < 1000; i++) {
|
||||
await db.query(
|
||||
`INSERT INTO timeline_entries (page_id, date, source, summary, detail) VALUES ($1, $2::date, $3, $4, $5)`,
|
||||
[pageId, '2024-01-15', `src-${i}`, 'Founded NovaMind', `detail-${i}`]
|
||||
);
|
||||
}
|
||||
const beforeCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c;
|
||||
expect(beforeCount).toBe(1000);
|
||||
|
||||
await engine.setConfig('version', '7');
|
||||
|
||||
const start = Date.now();
|
||||
await runMigrations(engine);
|
||||
const elapsedMs = Date.now() - start;
|
||||
|
||||
expect(elapsedMs).toBeLessThan(5000);
|
||||
|
||||
const afterCount = (await db.query(`SELECT COUNT(*)::int AS c FROM timeline_entries`)).rows[0].c;
|
||||
expect(afterCount).toBe(1);
|
||||
|
||||
const uniqueIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'timeline_entries' AND indexname = 'idx_timeline_dedup'
|
||||
`)).rows;
|
||||
expect(uniqueIdx.length).toBe(1);
|
||||
|
||||
const helperIdx = (await db.query(`
|
||||
SELECT indexname FROM pg_indexes
|
||||
WHERE tablename = 'timeline_entries' AND indexname = 'idx_timeline_dedup_helper'
|
||||
`)).rows;
|
||||
expect(helperIdx.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -350,6 +350,118 @@ describe('PGLiteEngine: Timeline', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Batch methods (addLinksBatch / addTimelineEntriesBatch)
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
describe('PGLiteEngine: addLinksBatch', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
await engine.putPage('a', { type: 'concept', title: 'A', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('b', { type: 'concept', title: 'B', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('c', { type: 'concept', title: 'C', compiled_truth: '', timeline: '' });
|
||||
});
|
||||
|
||||
test('empty batch returns 0 with no DB call', async () => {
|
||||
expect(await engine.addLinksBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('batch of 1 with missing optional fields inserts row with empty defaults', async () => {
|
||||
const inserted = await engine.addLinksBatch([{ from_slug: 'a', to_slug: 'b' }]);
|
||||
expect(inserted).toBe(1);
|
||||
const links = await engine.getLinks('a');
|
||||
expect(links.length).toBe(1);
|
||||
expect(links[0].context).toBe('');
|
||||
expect(links[0].link_type).toBe('');
|
||||
});
|
||||
|
||||
test('within-batch duplicates are deduped via ON CONFLICT (no 21000 error)', async () => {
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'c', link_type: 'mention' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
|
||||
test('rows with missing slug are silently dropped by JOIN', async () => {
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'doesnt-exist', to_slug: 'b' },
|
||||
{ from_slug: 'a', to_slug: 'b' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('half-existing batch returns count of new only', async () => {
|
||||
await engine.addLink('a', 'b', '', 'mention');
|
||||
const inserted = await engine.addLinksBatch([
|
||||
{ from_slug: 'a', to_slug: 'b', link_type: 'mention' },
|
||||
{ from_slug: 'a', to_slug: 'c', link_type: 'mention' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('batch of 100 fresh rows returns 100', async () => {
|
||||
// Create 100 target pages
|
||||
for (let i = 0; i < 100; i++) {
|
||||
await engine.putPage(`target/${i}`, { type: 'concept', title: `T${i}`, compiled_truth: '', timeline: '' });
|
||||
}
|
||||
const batch = Array.from({ length: 100 }, (_, i) => ({
|
||||
from_slug: 'a', to_slug: `target/${i}`, link_type: 'mention',
|
||||
}));
|
||||
expect(await engine.addLinksBatch(batch)).toBe(100);
|
||||
});
|
||||
});
|
||||
|
||||
describe('PGLiteEngine: addTimelineEntriesBatch', () => {
|
||||
beforeEach(async () => {
|
||||
await truncateAll();
|
||||
await engine.putPage('p1', { type: 'concept', title: 'P1', compiled_truth: '', timeline: '' });
|
||||
await engine.putPage('p2', { type: 'concept', title: 'P2', compiled_truth: '', timeline: '' });
|
||||
});
|
||||
|
||||
test('empty batch returns 0', async () => {
|
||||
expect(await engine.addTimelineEntriesBatch([])).toBe(0);
|
||||
});
|
||||
|
||||
test('batch of 1 with missing optionals inserts with empty defaults', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
const entries = await engine.getTimeline('p1');
|
||||
expect(entries.length).toBe(1);
|
||||
expect(entries[0].source).toBe('');
|
||||
expect(entries[0].detail).toBe('');
|
||||
});
|
||||
|
||||
test('within-batch duplicates are deduped via ON CONFLICT', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-02-01', summary: 'Launched' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
|
||||
test('rows with missing slug are silently dropped by JOIN', async () => {
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'no-such-page', date: '2024-01-15', summary: 'Phantom' },
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Real' },
|
||||
]);
|
||||
expect(inserted).toBe(1);
|
||||
});
|
||||
|
||||
test('mix of new + existing returns count of new only', async () => {
|
||||
await engine.addTimelineEntry('p1', { date: '2024-01-15', summary: 'Founded' });
|
||||
const inserted = await engine.addTimelineEntriesBatch([
|
||||
{ slug: 'p1', date: '2024-01-15', summary: 'Founded' },
|
||||
{ slug: 'p1', date: '2024-02-01', summary: 'Launched' },
|
||||
{ slug: 'p2', date: '2024-03-01', summary: 'Spun off' },
|
||||
]);
|
||||
expect(inserted).toBe(2);
|
||||
});
|
||||
});
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Raw Data, Versions, Config, IngestLog
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user