mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-28 06:23:01 +00:00
* feat(v0.26.5): destructive operation guard — impact preview, confirmation gate, soft-delete
Three-layer protection against accidental data loss:
1. **Impact preview**: Every destructive operation (sources remove, purge)
now shows a formatted preview of exactly what will be destroyed —
page count, chunk count, embedding count, file count — BEFORE acting.
2. **--confirm-destructive flag**: `--yes` alone is no longer sufficient
when a source has data. Must pass `--confirm-destructive` to proceed
with permanent deletion. Prevents scripted/reflexive destroys.
3. **Soft-delete with 72h TTL**: New `gbrain sources archive <id>`
hides a source from search and federation without destroying any data.
Data preserved for 72 hours. Restorable via `gbrain sources restore <id>`.
Expired archives purged via `gbrain sources purge`.
New subcommands:
- `gbrain sources archive <id>` — soft-delete (hide, preserve 72h)
- `gbrain sources restore <id>` — un-archive, re-federate
- `gbrain sources archived` — list soft-deleted sources + TTL
- `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete
Behavioral changes:
- `sources remove` with data now requires `--confirm-destructive` (not just `--yes`)
- `sources remove --dry-run` shows full impact preview without side effects
- Impact box format shows source name, id, and all cascade counts
New files:
- src/core/destructive-guard.ts — impact assessment, confirmation gate,
soft-delete/restore/purge logic, display formatters
* chore(release): v0.26.5 — destructive operation guard
Bump VERSION + package.json to 0.26.5 and add the v0.26.5 CHANGELOG entry
on top of the destructive-guard feature commit cherry-picked from PR #595.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.26.5): page-level soft-delete + autopilot purge + search visibility
Closes the destructive-guard posture across every gbrain destructive surface.
PR #595 cherry-pick covered the CLI source-remove path; this commit closes
the higher-velocity MCP `delete_page` agent footgun and the three internal
correctness gaps the CEO+Eng review surfaced:
- Gap 1: archived sources were not actually filtered from search. Now they
are, via `buildVisibilityClause` in `searchKeyword`/`searchKeywordChunks`/
`searchVector` for both engines.
- Gap 2: 72h TTL was honor-system. Now wired into a new autopilot `purge`
phase (9th in ALL_PHASES) that calls `purgeExpiredSources` + `engine.
purgeDeletedPages(72)`. Manual escape hatch: `gbrain pages purge-deleted`.
- Gap 3: zero tests for safety-critical code. ~30 cases now in
`test/destructive-guard.test.ts`, `test/pages-soft-delete.test.ts`, and
`test/sql-ranking.test.ts` covering the boundary truth table, JSONB→column
migration, soft-delete/restore/purge round-trip, multi-source isolation,
cascade verification, and the Q3 IRON-rule contract test.
Schema migration v33 (`destructive_guard_columns`): adds `pages.deleted_at`
+ partial purge index, promotes `archived` from `sources.config` JSONB to
real columns (`sources.archived BOOLEAN`, `archived_at`, `archive_expires_at`),
backfills any pre-v0.26.5 JSONB shape. Engine-aware: Postgres uses CREATE
INDEX CONCURRENTLY, PGLite uses plain CREATE INDEX. Forward-reference
bootstrap extended in both engines so pre-v0.26.5 brains don't crash on the
embedded-schema replay.
BrainEngine surface: new `softDeletePage` / `restorePage` /
`purgeDeletedPages` methods + `includeDeleted` flag on `getPage`/`listPages`.
MCP ops: `delete_page` rewired to soft-delete (description string updated);
new `restore_page` (scope: write) + `purge_deleted_pages` (scope: admin,
localOnly: true).
Q3 contract (eng-review lynchpin): `get_page(slug)` returns null for
soft-deleted by default; `get_page(slug, {include_deleted: true})` surfaces
the row with `deleted_at` populated. Same flag for `list_pages`. Mirrors
the search-filter contract end-to-end.
Issue 5 (eng-review): `archived` is now a real column on `sources`, not a
JSONB key. No reserved-key footgun. Faster filter. Visibility clause
compiles to a column lookup, not JSONB containment.
Verification:
- bun run typecheck: PASS
- bun run build:schema + bun run build:llms: regenerated
- targeted test runs: 90 pass / 0 fail across destructive-guard,
pages-soft-delete, sql-ranking, schema-bootstrap-coverage, build-llms
- full bun test: 16 pre-existing failures inherited from v0.26.2 (sync,
sync-parallel, queue-child-done, etc — already filed in TODOS.md as
"Fix 22 pre-existing test failures unrelated to OAuth")
CHANGELOG, CLAUDE.md (Key Files + Commands), TODOS.md updated. The plan
file at ~/.claude/plans/take-a-look-and-gentle-pine.md captures the full
review trail (CEO=C, Eng-Q3=A, Eng-Issue5=a, 8 defaults applied).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.26.5): CI fallout — getStats excludes soft-deleted; tests use --confirm-destructive
Two CI failures from the v0.26.5 ship:
1. **Tier 1 (Postgres E2E):** `E2E: Page CRUD > delete_page removes page and
others survive` failed because `delete_page` now soft-deletes (sets
deleted_at) but `getStats.page_count` was still counting all rows. The
test seeds 16 pages, deletes one, and asserts page_count is 15. Fix:
`getStats` now filters `WHERE deleted_at IS NULL` for page_count in both
engines. This matches the visibility-filter contract — soft-deleted pages
are hidden everywhere the user looks (search, get_page, list_pages, stats).
Chunks and links stay raw because they still occupy storage until the
autopilot purge phase runs.
2. **Test 2 (PGLite unit):** `multi-source-integration.test.ts:184` and
`e2e/multi-source.test.ts:274` called `runSources(engine, ['remove', X,
'--yes'])` against populated sources. v0.26.5's destructive guard rejects
`--yes` alone on populated sources and calls `process.exit(5)`, which
killed the bun test runner mid-suite (CI exit 5). Both test sites now
pass `--confirm-destructive` per the v0.26.5 contract.
Verification: 115/0 pass across destructive-guard, pages-soft-delete,
sql-ranking, schema-bootstrap-coverage, sources, repos-alias, and
multi-source-integration test files. typecheck PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test): cycle phase count is 9 (v0.26.5 added `purge` phase)
CI failure: `runCycle — yieldBetweenPhases hook` tests asserted exactly 8
phases. v0.26.5 added the autopilot `purge` phase as the 9th, so:
- `test/core/cycle.test.ts:381` — `hookCalls` is now 9 (one yield per phase)
- `test/core/cycle.test.ts:392` — `report.phases.length` is now 9
- `test/e2e/cycle.test.ts:101` — same update for the dry-run E2E
The `purge` phase invocation was already visible in the failing log output:
the cycle ran 9 phases end-to-end; the test assertions hadn't been updated.
Verification: bun run typecheck PASS. cycle.test.ts: 28/0 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
610 lines
26 KiB
TypeScript
610 lines
26 KiB
TypeScript
/**
|
|
* E2E: v0.18.0 multi-source migrations against REAL Postgres.
|
|
*
|
|
* PGLite doesn't have a files table (see pglite-schema.ts header), so the
|
|
* v23 migration's files.source_id + files.page_id rewrite + ledger seed
|
|
* is NEVER executed by the PGLite integration test. This file closes
|
|
* that gap by exercising the full v20-v23 chain against a real Postgres
|
|
* DB with pre-existing data.
|
|
*
|
|
* Also covers the gaps in the PR's pre-shipping test matrix that the
|
|
* author self-audited:
|
|
* - files.page_slug → page_id backfill against real rows
|
|
* - file_migration_ledger seeding
|
|
* - cascade delete via sources.remove (pages + chunks + timeline +
|
|
* files + links all gone)
|
|
* - sync --source <id> routing reads + writes per-source sync anchors
|
|
* instead of the global config keys
|
|
*
|
|
* Gated by DATABASE_URL — skips gracefully when unset, per the CLAUDE.md
|
|
* E2E lifecycle pattern.
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
|
import { runSources } from '../../src/commands/sources.ts';
|
|
import { performSync } from '../../src/commands/sync.ts';
|
|
import { runStorageBackfill } from '../../src/commands/migrations/v0_18_0-storage-backfill.ts';
|
|
import type { StorageBackend } from '../../src/core/storage.ts';
|
|
import { hasDatabase, setupDB, teardownDB, getConn, getEngine } from './helpers.ts';
|
|
|
|
const SKIP = !hasDatabase();
|
|
const describeE2E = SKIP ? describe.skip : describe;
|
|
|
|
describeE2E('v0.18.0 multi-source — Postgres schema shape (fresh install)', () => {
|
|
beforeAll(async () => {
|
|
await setupDB();
|
|
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
|
|
// residual rows from prior test runs can shadow new INSERTs. Wipe
|
|
// non-default sources at the top of every describe to keep each
|
|
// block hermetic. file_migration_ledger cascades from files which
|
|
// setupDB already truncates, but wipe explicitly in case files did
|
|
// not cascade it.
|
|
const conn = getConn();
|
|
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
|
|
await conn.unsafe(`DELETE FROM file_migration_ledger`);
|
|
});
|
|
afterAll(async () => {
|
|
await teardownDB();
|
|
});
|
|
|
|
test("sources('default') exists after initSchema + migration chain", async () => {
|
|
const conn = getConn();
|
|
const rows = await conn.unsafe(
|
|
`SELECT id, name, config FROM sources WHERE id = 'default'`,
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
expect(rows[0].name).toBe('default');
|
|
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
|
|
expect(config.federated).toBe(true);
|
|
});
|
|
|
|
test('pages.source_id NOT NULL with DEFAULT default (v21)', async () => {
|
|
const conn = getConn();
|
|
const rows = await conn.unsafe(
|
|
`SELECT column_name, column_default, is_nullable
|
|
FROM information_schema.columns
|
|
WHERE table_name = 'pages' AND column_name = 'source_id'`,
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
expect(rows[0].is_nullable).toBe('NO');
|
|
expect(String(rows[0].column_default)).toContain('default');
|
|
});
|
|
|
|
test('composite UNIQUE pages(source_id, slug) replaces global UNIQUE(slug)', async () => {
|
|
const conn = getConn();
|
|
const composite = await conn.unsafe(
|
|
`SELECT conname FROM pg_constraint WHERE conname = 'pages_source_slug_key'`,
|
|
);
|
|
expect(composite.length).toBe(1);
|
|
const oldGlobal = await conn.unsafe(
|
|
`SELECT conname FROM pg_constraint WHERE conname = 'pages_slug_key'`,
|
|
);
|
|
expect(oldGlobal.length).toBe(0);
|
|
});
|
|
|
|
test('links.resolution_type column exists with CHECK (v22)', async () => {
|
|
const conn = getConn();
|
|
const rows = await conn.unsafe(
|
|
`SELECT column_name FROM information_schema.columns
|
|
WHERE table_name = 'links' AND column_name = 'resolution_type'`,
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
const check = await conn.unsafe(
|
|
`SELECT conname FROM pg_constraint WHERE conname = 'links_resolution_type_check'`,
|
|
);
|
|
expect(check.length).toBe(1);
|
|
});
|
|
|
|
test('files.source_id + files.page_id columns exist (v23, Postgres-only)', async () => {
|
|
const conn = getConn();
|
|
const cols = await conn.unsafe(
|
|
`SELECT column_name FROM information_schema.columns
|
|
WHERE table_name = 'files' AND column_name IN ('source_id', 'page_id')`,
|
|
);
|
|
// postgres.js returns RowList with an iterable-row shape; cast via
|
|
// unknown before narrowing to plain objects (TS2352 otherwise).
|
|
const names = new Set(
|
|
(cols as unknown as Array<{ column_name: string }>).map(r => r.column_name),
|
|
);
|
|
expect(names.has('source_id')).toBe(true);
|
|
expect(names.has('page_id')).toBe(true);
|
|
});
|
|
|
|
test('file_migration_ledger table exists with status CHECK (v23)', async () => {
|
|
const conn = getConn();
|
|
const tables = await conn.unsafe(
|
|
`SELECT table_name FROM information_schema.tables
|
|
WHERE table_name = 'file_migration_ledger'`,
|
|
);
|
|
expect(tables.length).toBe(1);
|
|
const check = await conn.unsafe(
|
|
`SELECT conname FROM pg_constraint WHERE conname = 'chk_ledger_status'`,
|
|
);
|
|
expect(check.length).toBe(1);
|
|
});
|
|
});
|
|
|
|
describeE2E('v0.18.0 multi-source — composite UNIQUE semantics on real Postgres', () => {
|
|
beforeAll(async () => {
|
|
await setupDB();
|
|
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
|
|
// residual rows from prior test runs can shadow new INSERTs. Wipe
|
|
// non-default sources at the top of every describe to keep each
|
|
// block hermetic. file_migration_ledger cascades from files which
|
|
// setupDB already truncates, but wipe explicitly in case files did
|
|
// not cascade it.
|
|
const conn = getConn();
|
|
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
|
|
await conn.unsafe(`DELETE FROM file_migration_ledger`);
|
|
});
|
|
afterAll(async () => {
|
|
await teardownDB();
|
|
});
|
|
|
|
test('same slug in two sources coexists (REGRESSION GUARD — Codex critical)', async () => {
|
|
const conn = getConn();
|
|
// Create a second source.
|
|
const engine = getEngine();
|
|
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['add', 'wiki', '--federated']);
|
|
|
|
// Insert the same slug under 'default' (via putPage) and 'wiki' (raw INSERT).
|
|
await engine.putPage('topics/ai', {
|
|
type: 'concept', title: 'AI from default', compiled_truth: 'default source take',
|
|
});
|
|
await conn.unsafe(
|
|
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
|
|
VALUES ('wiki', 'topics/ai', 'concept', 'AI from wiki', 'wiki source take', '', '{}'::jsonb, 'wikihash')`,
|
|
);
|
|
|
|
const rows = await conn.unsafe(
|
|
`SELECT source_id, slug, title FROM pages WHERE slug = 'topics/ai' ORDER BY source_id`,
|
|
);
|
|
expect(rows.length).toBe(2);
|
|
expect(rows.map((r: any) => r.source_id).sort()).toEqual(['default', 'wiki']);
|
|
});
|
|
|
|
test('duplicate (source_id, slug) hits composite UNIQUE', async () => {
|
|
const conn = getConn();
|
|
let err: Error | null = null;
|
|
try {
|
|
await conn.unsafe(
|
|
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
|
|
VALUES ('wiki', 'topics/ai', 'concept', 'dup', '', '', '{}'::jsonb, 'dup')`,
|
|
);
|
|
} catch (e) {
|
|
err = e as Error;
|
|
}
|
|
expect(err).not.toBeNull();
|
|
expect(err!.message.toLowerCase()).toMatch(/unique|duplicate/);
|
|
});
|
|
|
|
test('putPage (engine API) targets default source by schema DEFAULT', async () => {
|
|
const engine = getEngine();
|
|
await engine.putPage('topics/from-putpage', {
|
|
type: 'note', title: 'Via putPage', compiled_truth: 'body',
|
|
});
|
|
const conn = getConn();
|
|
const rows = await conn.unsafe(
|
|
`SELECT source_id FROM pages WHERE slug = 'topics/from-putpage'`,
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
expect(rows[0].source_id).toBe('default');
|
|
});
|
|
});
|
|
|
|
describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row', () => {
|
|
beforeAll(async () => {
|
|
await setupDB();
|
|
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
|
|
// residual rows from prior test runs can shadow new INSERTs. Wipe
|
|
// non-default sources at the top of every describe to keep each
|
|
// block hermetic. file_migration_ledger cascades from files which
|
|
// setupDB already truncates, but wipe explicitly in case files did
|
|
// not cascade it.
|
|
const conn = getConn();
|
|
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
|
|
await conn.unsafe(`DELETE FROM file_migration_ledger`);
|
|
});
|
|
afterAll(async () => {
|
|
await teardownDB();
|
|
});
|
|
|
|
test('sources remove cascades to pages + chunks + timeline + links + files', async () => {
|
|
const conn = getConn();
|
|
const engine = getEngine();
|
|
|
|
// Build a fully populated source: page, chunks, timeline entries,
|
|
// links, a file row. Then remove the source and verify nothing
|
|
// for that source survives.
|
|
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['add', 'cascadetest', '--federated']);
|
|
|
|
// Page under cascadetest
|
|
await conn.unsafe(
|
|
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
|
|
VALUES ('cascadetest', 'people/alice', 'person', 'Alice', 'Alice body', '', '{}'::jsonb, 'h1')`,
|
|
);
|
|
const alicePage = await conn.unsafe(
|
|
`SELECT id FROM pages WHERE source_id = 'cascadetest' AND slug = 'people/alice'`,
|
|
);
|
|
const aliceId = alicePage[0].id as number;
|
|
|
|
// A second page for link target
|
|
await conn.unsafe(
|
|
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
|
|
VALUES ('cascadetest', 'companies/acme', 'company', 'Acme', 'Acme body', '', '{}'::jsonb, 'h2')`,
|
|
);
|
|
const acmePage = await conn.unsafe(
|
|
`SELECT id FROM pages WHERE source_id = 'cascadetest' AND slug = 'companies/acme'`,
|
|
);
|
|
const acmeId = acmePage[0].id as number;
|
|
|
|
// Chunk
|
|
await conn.unsafe(
|
|
`INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source)
|
|
VALUES (${aliceId}, 0, 'Alice body chunk', 'compiled_truth')`,
|
|
);
|
|
|
|
// Timeline
|
|
await conn.unsafe(
|
|
`INSERT INTO timeline_entries (page_id, date, source, summary, detail)
|
|
VALUES (${aliceId}, '2026-01-15', 'test', 'Joined Acme', 'detail')`,
|
|
);
|
|
|
|
// Link Alice → Acme
|
|
await conn.unsafe(
|
|
`INSERT INTO links (from_page_id, to_page_id, link_type, link_source)
|
|
VALUES (${aliceId}, ${acmeId}, 'works_at', 'markdown')`,
|
|
);
|
|
|
|
// File row pointing at Alice
|
|
await conn.unsafe(
|
|
`INSERT INTO files (source_id, page_id, filename, storage_path, content_hash)
|
|
VALUES ('cascadetest', ${aliceId}, 'alice.pdf', 'cascadetest/people/alice/alice.pdf', 'fh1')`,
|
|
);
|
|
|
|
// Sanity: everything exists
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(2);
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM content_chunks WHERE page_id = ${aliceId}`))[0].n).toBe(1);
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM timeline_entries WHERE page_id = ${aliceId}`))[0].n).toBe(1);
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM links WHERE from_page_id = ${aliceId}`))[0].n).toBe(1);
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(1);
|
|
|
|
// Remove the source.
|
|
// v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected.
|
|
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['remove', 'cascadetest', '--confirm-destructive']);
|
|
|
|
// Everything for that source is gone.
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(0);
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM content_chunks WHERE page_id = ${aliceId}`))[0].n).toBe(0);
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM timeline_entries WHERE page_id = ${aliceId}`))[0].n).toBe(0);
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM links WHERE from_page_id = ${aliceId}`))[0].n).toBe(0);
|
|
expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(0);
|
|
|
|
// The sources row itself is gone.
|
|
const src = await conn.unsafe(`SELECT id FROM sources WHERE id = 'cascadetest'`);
|
|
expect(src.length).toBe(0);
|
|
});
|
|
});
|
|
|
|
describeE2E('v0.18.0 multi-source — sync --source routes through sources table', () => {
|
|
beforeAll(async () => {
|
|
await setupDB();
|
|
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
|
|
// residual rows from prior test runs can shadow new INSERTs. Wipe
|
|
// non-default sources at the top of every describe to keep each
|
|
// block hermetic. file_migration_ledger cascades from files which
|
|
// setupDB already truncates, but wipe explicitly in case files did
|
|
// not cascade it.
|
|
const conn = getConn();
|
|
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
|
|
await conn.unsafe(`DELETE FROM file_migration_ledger`);
|
|
});
|
|
afterAll(async () => {
|
|
await teardownDB();
|
|
});
|
|
|
|
test('performSync with sourceId reads local_path from sources row', async () => {
|
|
const engine = getEngine();
|
|
const conn = getConn();
|
|
|
|
// Register a source with a bogus path (we're not actually walking a
|
|
// repo — this test asserts that performSync correctly RESOLVES the
|
|
// source row vs hitting the global config).
|
|
await runSources(engine as unknown as Parameters<typeof runSources>[0], [
|
|
'add', 'syncsrc', '--path', '/nonexistent/syncsrc/path', '--no-federated',
|
|
]);
|
|
|
|
// Also set a DIFFERENT path in the global config so we can verify
|
|
// sourceId actually disambiguates.
|
|
await engine.setConfig('sync.repo_path', '/some/other/default/path');
|
|
|
|
// performSync({sourceId: 'syncsrc'}) should attempt to use
|
|
// /nonexistent/syncsrc/path, NOT /some/other/default/path.
|
|
let err: Error | null = null;
|
|
try {
|
|
await performSync(engine, { sourceId: 'syncsrc' });
|
|
} catch (e) {
|
|
err = e as Error;
|
|
}
|
|
expect(err).not.toBeNull();
|
|
// The error message references the source-scoped path, not the
|
|
// global config path. (Could be "Not a git repository"
|
|
// or "No commits in repo" — either way the path it cites should
|
|
// be the source's.)
|
|
expect(err!.message).toContain('/nonexistent/syncsrc/path');
|
|
expect(err!.message).not.toContain('/some/other/default/path');
|
|
});
|
|
|
|
test('performSync with no sourceId falls back to global sync.repo_path', async () => {
|
|
const engine = getEngine();
|
|
// Global config is still '/some/other/default/path' from the
|
|
// previous test. Without --source, performSync uses it.
|
|
let err: Error | null = null;
|
|
try {
|
|
await performSync(engine, {});
|
|
} catch (e) {
|
|
err = e as Error;
|
|
}
|
|
expect(err).not.toBeNull();
|
|
expect(err!.message).toContain('/some/other/default/path');
|
|
});
|
|
});
|
|
|
|
describeE2E('v0.18.0 multi-source — sources table surface', () => {
|
|
beforeAll(async () => {
|
|
await setupDB();
|
|
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
|
|
// residual rows from prior test runs can shadow new INSERTs. Wipe
|
|
// non-default sources at the top of every describe to keep each
|
|
// block hermetic. file_migration_ledger cascades from files which
|
|
// setupDB already truncates, but wipe explicitly in case files did
|
|
// not cascade it.
|
|
const conn = getConn();
|
|
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
|
|
await conn.unsafe(`DELETE FROM file_migration_ledger`);
|
|
});
|
|
afterAll(async () => {
|
|
await teardownDB();
|
|
});
|
|
|
|
test('default source is seeded federated=true; new sources default to isolated', async () => {
|
|
const conn = getConn();
|
|
const engine = getEngine();
|
|
|
|
const def = await conn.unsafe(`SELECT config FROM sources WHERE id = 'default'`);
|
|
const defConfig = typeof def[0].config === 'string' ? JSON.parse(def[0].config) : def[0].config;
|
|
expect(defConfig.federated).toBe(true);
|
|
|
|
// Defensive cleanup: sources isn't in helpers.ALL_TABLES, so residual
|
|
// rows from prior test runs can shadow this INSERT via ON CONFLICT
|
|
// DO NOTHING. Delete first, then create.
|
|
await conn.unsafe(`DELETE FROM sources WHERE id = 'isolatedsrc'`);
|
|
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['add', 'isolatedsrc']);
|
|
const iso = await conn.unsafe(`SELECT config FROM sources WHERE id = 'isolatedsrc'`);
|
|
const isoConfig = typeof iso[0].config === 'string' ? JSON.parse(iso[0].config) : iso[0].config;
|
|
expect(isoConfig.federated).toBeUndefined(); // omitted → isolated-by-default
|
|
});
|
|
|
|
test('federate / unfederate flips config.federated on real DB', async () => {
|
|
const conn = getConn();
|
|
const engine = getEngine();
|
|
|
|
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['federate', 'isolatedsrc']);
|
|
let row = await conn.unsafe(`SELECT config FROM sources WHERE id = 'isolatedsrc'`);
|
|
let config = typeof row[0].config === 'string' ? JSON.parse(row[0].config) : row[0].config;
|
|
expect(config.federated).toBe(true);
|
|
|
|
await runSources(engine as unknown as Parameters<typeof runSources>[0], ['unfederate', 'isolatedsrc']);
|
|
row = await conn.unsafe(`SELECT config FROM sources WHERE id = 'isolatedsrc'`);
|
|
config = typeof row[0].config === 'string' ? JSON.parse(row[0].config) : row[0].config;
|
|
expect(config.federated).toBe(false);
|
|
});
|
|
|
|
test('rename changes name, id stays stable', async () => {
|
|
const conn = getConn();
|
|
const engine = getEngine();
|
|
|
|
await runSources(engine as unknown as Parameters<typeof runSources>[0], [
|
|
'rename', 'isolatedsrc', 'My Isolated Source',
|
|
]);
|
|
const row = await conn.unsafe(`SELECT id, name FROM sources WHERE id = 'isolatedsrc'`);
|
|
expect(row[0].id).toBe('isolatedsrc');
|
|
expect(row[0].name).toBe('My Isolated Source');
|
|
});
|
|
});
|
|
|
|
describeE2E('v0.18.0 multi-source — storage backfill against file_migration_ledger', () => {
|
|
beforeAll(async () => {
|
|
await setupDB();
|
|
// sources + file_migration_ledger are not in helpers.ALL_TABLES, so
|
|
// residual rows from prior test runs can shadow new INSERTs. Wipe
|
|
// non-default sources at the top of every describe to keep each
|
|
// block hermetic. file_migration_ledger cascades from files which
|
|
// setupDB already truncates, but wipe explicitly in case files did
|
|
// not cascade it.
|
|
const conn = getConn();
|
|
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
|
|
await conn.unsafe(`DELETE FROM file_migration_ledger`);
|
|
});
|
|
afterAll(async () => {
|
|
await teardownDB();
|
|
});
|
|
|
|
test('seeded ledger + stub storage: pending → complete end-to-end', async () => {
|
|
const conn = getConn();
|
|
const engine = getEngine();
|
|
|
|
// Seed a page + file (via raw INSERT so the test doesn't depend on
|
|
// sync running).
|
|
await conn.unsafe(
|
|
`INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash)
|
|
VALUES ('default', 'topics/storage', 'note', 'Storage test', 'body', '', '{}'::jsonb, 'sh1')`,
|
|
);
|
|
const pageRow = await conn.unsafe(
|
|
`SELECT id FROM pages WHERE source_id = 'default' AND slug = 'topics/storage'`,
|
|
);
|
|
const pageId = pageRow[0].id as number;
|
|
|
|
await conn.unsafe(
|
|
`INSERT INTO files (source_id, page_id, filename, storage_path, content_hash)
|
|
VALUES ('default', ${pageId}, 'doc.pdf', 'topics/storage/doc.pdf', 'fh1')`,
|
|
);
|
|
const fileRow = await conn.unsafe(
|
|
`SELECT id FROM files WHERE storage_path = 'topics/storage/doc.pdf'`,
|
|
);
|
|
const fileId = fileRow[0].id as number;
|
|
|
|
// Seed the ledger manually so we don't depend on the v23 seed SQL
|
|
// (the TRUNCATE CASCADE in setupDB wipes ledger rows).
|
|
await conn.unsafe(
|
|
`INSERT INTO file_migration_ledger (file_id, storage_path_old, storage_path_new, status)
|
|
VALUES (${fileId}, 'topics/storage/doc.pdf', 'default/topics/storage/doc.pdf', 'pending')
|
|
ON CONFLICT (file_id) DO NOTHING`,
|
|
);
|
|
|
|
// Stub storage: downloads return bytes, uploads track what was written.
|
|
const uploaded = new Set<string>();
|
|
const stub: StorageBackend = {
|
|
upload: async (p: string) => { uploaded.add(p); },
|
|
download: async (p: string) => Buffer.from('bytes-for:' + p),
|
|
delete: async (p: string) => { uploaded.delete(p); },
|
|
exists: async (p: string) => uploaded.has(p),
|
|
list: async () => [],
|
|
getUrl: async (p) => `https://stub/${p}`,
|
|
};
|
|
|
|
const report = await runStorageBackfill(engine, stub);
|
|
expect(report.total).toBe(1);
|
|
expect(report.nowComplete).toBe(1);
|
|
expect(report.failed).toBe(0);
|
|
|
|
// Ledger row transitioned to complete.
|
|
const ledger = await conn.unsafe(
|
|
`SELECT status FROM file_migration_ledger WHERE file_id = ${fileId}`,
|
|
);
|
|
expect(ledger[0].status).toBe('complete');
|
|
|
|
// Files row now points at the new path.
|
|
const filesAfter = await conn.unsafe(
|
|
`SELECT storage_path FROM files WHERE id = ${fileId}`,
|
|
);
|
|
expect(filesAfter[0].storage_path).toBe('default/topics/storage/doc.pdf');
|
|
|
|
// Stub storage saw the upload happen at the new path.
|
|
expect(uploaded.has('default/topics/storage/doc.pdf')).toBe(true);
|
|
});
|
|
});
|
|
|
|
// v0.18.0: real-Postgres regression guard for the addLinksBatch /
|
|
// addTimelineEntriesBatch JOIN fan-out bug. Before the fix, the JOIN was
|
|
// `pages.slug = v.from_slug` unqualified — so two pages sharing the same
|
|
// slug across sources would silently duplicate edges and timeline rows.
|
|
// postgres-js binds arrays through `unnest()` rather than inline VALUES,
|
|
// so the query shape is structurally different from PGLite's and gets its
|
|
// own coverage.
|
|
describeE2E('v0.18.0 multi-source — addLinksBatch / addTimelineEntriesBatch source-awareness', () => {
|
|
beforeAll(async () => {
|
|
await setupDB();
|
|
const conn = getConn();
|
|
await conn.unsafe(`DELETE FROM sources WHERE id != 'default'`);
|
|
await conn.unsafe(`DELETE FROM file_migration_ledger`);
|
|
});
|
|
afterAll(async () => { await teardownDB(); });
|
|
|
|
async function seedSameSlugTwoSources() {
|
|
const conn = getConn();
|
|
const engine = getEngine() as PostgresEngine;
|
|
// Second source alongside 'default'.
|
|
await conn.unsafe(
|
|
`INSERT INTO sources (id, name) VALUES ('alt', 'alt') ON CONFLICT (id) DO NOTHING`
|
|
);
|
|
// Create same-slug pages in both sources. putPage defaults to 'default'.
|
|
await engine.putPage('topics/ai', { type: 'concept', title: 'AI (default)', compiled_truth: '', timeline: '' });
|
|
await engine.putPage('topics/ml', { type: 'concept', title: 'ML (default)', compiled_truth: '', timeline: '' });
|
|
await conn.unsafe(
|
|
`INSERT INTO pages (slug, type, title, compiled_truth, timeline, frontmatter, content_hash, source_id, updated_at)
|
|
VALUES ('topics/ai', 'concept', 'AI (alt)', '', '', '{}'::jsonb, 'alt-ai-hash', 'alt', now()),
|
|
('topics/ml', 'concept', 'ML (alt)', '', '', '{}'::jsonb, 'alt-ml-hash', 'alt', now())`
|
|
);
|
|
}
|
|
|
|
test('addLinksBatch without explicit source_id does NOT fan out across sources', async () => {
|
|
await seedSameSlugTwoSources();
|
|
const conn = getConn();
|
|
const engine = getEngine() as PostgresEngine;
|
|
// Reset links from any prior describe block.
|
|
await conn.unsafe(`DELETE FROM links`);
|
|
const inserted = await engine.addLinksBatch([
|
|
{ from_slug: 'topics/ai', to_slug: 'topics/ml', link_type: 'mention' },
|
|
]);
|
|
// Exactly one edge (default → default). Before the fix this was 2.
|
|
expect(inserted).toBe(1);
|
|
const rows = await conn.unsafe(
|
|
`SELECT f.source_id AS from_src, t.source_id AS to_src
|
|
FROM links l
|
|
JOIN pages f ON f.id = l.from_page_id
|
|
JOIN pages t ON t.id = l.to_page_id`
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
expect(rows[0].from_src).toBe('default');
|
|
expect(rows[0].to_src).toBe('default');
|
|
});
|
|
|
|
test('addLinksBatch supports cross-source edges when explicit source_ids differ', async () => {
|
|
const conn = getConn();
|
|
const engine = getEngine() as PostgresEngine;
|
|
await conn.unsafe(`DELETE FROM links`);
|
|
const inserted = await engine.addLinksBatch([
|
|
{
|
|
from_slug: 'topics/ai', to_slug: 'topics/ml', link_type: 'mention',
|
|
from_source_id: 'default', to_source_id: 'alt',
|
|
},
|
|
]);
|
|
expect(inserted).toBe(1);
|
|
const rows = await conn.unsafe(
|
|
`SELECT f.source_id AS from_src, t.source_id AS to_src
|
|
FROM links l
|
|
JOIN pages f ON f.id = l.from_page_id
|
|
JOIN pages t ON t.id = l.to_page_id`
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
expect(rows[0].from_src).toBe('default');
|
|
expect(rows[0].to_src).toBe('alt');
|
|
});
|
|
|
|
test('addTimelineEntriesBatch without explicit source_id does NOT fan out across sources', async () => {
|
|
const conn = getConn();
|
|
const engine = getEngine() as PostgresEngine;
|
|
await conn.unsafe(`DELETE FROM timeline_entries`);
|
|
const inserted = await engine.addTimelineEntriesBatch([
|
|
{ slug: 'topics/ai', date: '2024-01-15', summary: 'Founded' },
|
|
]);
|
|
expect(inserted).toBe(1);
|
|
const rows = await conn.unsafe(
|
|
`SELECT p.source_id
|
|
FROM timeline_entries te
|
|
JOIN pages p ON p.id = te.page_id`
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
expect(rows[0].source_id).toBe('default');
|
|
});
|
|
|
|
test('addTimelineEntriesBatch with explicit alt source_id lands only in alt', async () => {
|
|
const conn = getConn();
|
|
const engine = getEngine() as PostgresEngine;
|
|
await conn.unsafe(`DELETE FROM timeline_entries`);
|
|
const inserted = await engine.addTimelineEntriesBatch([
|
|
{ slug: 'topics/ai', date: '2024-02-01', summary: 'Alt-only event', source_id: 'alt' },
|
|
]);
|
|
expect(inserted).toBe(1);
|
|
const rows = await conn.unsafe(
|
|
`SELECT p.source_id
|
|
FROM timeline_entries te
|
|
JOIN pages p ON p.id = te.page_id`
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
expect(rows[0].source_id).toBe('alt');
|
|
});
|
|
});
|