mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(v0.26.5): destructive operation guard — impact preview, confirmation gate, soft-delete
Three-layer protection against accidental data loss:
1. **Impact preview**: Every destructive operation (sources remove, purge)
now shows a formatted preview of exactly what will be destroyed —
page count, chunk count, embedding count, file count — BEFORE acting.
2. **--confirm-destructive flag**: `--yes` alone is no longer sufficient
when a source has data. Must pass `--confirm-destructive` to proceed
with permanent deletion. Prevents scripted/reflexive destroys.
3. **Soft-delete with 72h TTL**: New `gbrain sources archive <id>`
hides a source from search and federation without destroying any data.
Data preserved for 72 hours. Restorable via `gbrain sources restore <id>`.
Expired archives purged via `gbrain sources purge`.
New subcommands:
- `gbrain sources archive <id>` — soft-delete (hide, preserve 72h)
- `gbrain sources restore <id>` — un-archive, re-federate
- `gbrain sources archived` — list soft-deleted sources + TTL
- `gbrain sources purge [<id>] [--confirm-destructive]` — permanent delete
Behavioral changes:
- `sources remove` with data now requires `--confirm-destructive` (not just `--yes`)
- `sources remove --dry-run` shows full impact preview without side effects
- Impact box format shows source name, id, and all cascade counts
New files:
- src/core/destructive-guard.ts — impact assessment, confirmation gate,
soft-delete/restore/purge logic, display formatters
* chore(release): v0.26.5 — destructive operation guard
Bump VERSION + package.json to 0.26.5 and add the v0.26.5 CHANGELOG entry
on top of the destructive-guard feature commit cherry-picked from PR #595.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(v0.26.5): page-level soft-delete + autopilot purge + search visibility
Closes the destructive-guard posture across every gbrain destructive surface.
PR #595 cherry-pick covered the CLI source-remove path; this commit closes
the higher-velocity MCP `delete_page` agent footgun and the three internal
correctness gaps the CEO+Eng review surfaced:
- Gap 1: archived sources were not actually filtered from search. Now they
are, via `buildVisibilityClause` in `searchKeyword`/`searchKeywordChunks`/
`searchVector` for both engines.
- Gap 2: 72h TTL was honor-system. Now wired into a new autopilot `purge`
phase (9th in ALL_PHASES) that calls `purgeExpiredSources` + `engine.
purgeDeletedPages(72)`. Manual escape hatch: `gbrain pages purge-deleted`.
- Gap 3: zero tests for safety-critical code. ~30 cases now in
`test/destructive-guard.test.ts`, `test/pages-soft-delete.test.ts`, and
`test/sql-ranking.test.ts` covering the boundary truth table, JSONB→column
migration, soft-delete/restore/purge round-trip, multi-source isolation,
cascade verification, and the Q3 IRON-rule contract test.
Schema migration v33 (`destructive_guard_columns`): adds `pages.deleted_at`
+ partial purge index, promotes `archived` from `sources.config` JSONB to
real columns (`sources.archived BOOLEAN`, `archived_at`, `archive_expires_at`),
backfills any pre-v0.26.5 JSONB shape. Engine-aware: Postgres uses CREATE
INDEX CONCURRENTLY, PGLite uses plain CREATE INDEX. Forward-reference
bootstrap extended in both engines so pre-v0.26.5 brains don't crash on the
embedded-schema replay.
BrainEngine surface: new `softDeletePage` / `restorePage` /
`purgeDeletedPages` methods + `includeDeleted` flag on `getPage`/`listPages`.
MCP ops: `delete_page` rewired to soft-delete (description string updated);
new `restore_page` (scope: write) + `purge_deleted_pages` (scope: admin,
localOnly: true).
Q3 contract (eng-review lynchpin): `get_page(slug)` returns null for
soft-deleted by default; `get_page(slug, {include_deleted: true})` surfaces
the row with `deleted_at` populated. Same flag for `list_pages`. Mirrors
the search-filter contract end-to-end.
Issue 5 (eng-review): `archived` is now a real column on `sources`, not a
JSONB key. No reserved-key footgun. Faster filter. Visibility clause
compiles to a column lookup, not JSONB containment.
Verification:
- bun run typecheck: PASS
- bun run build:schema + bun run build:llms: regenerated
- targeted test runs: 90 pass / 0 fail across destructive-guard,
pages-soft-delete, sql-ranking, schema-bootstrap-coverage, build-llms
- full bun test: 16 pre-existing failures inherited from v0.26.2 (sync,
sync-parallel, queue-child-done, etc — already filed in TODOS.md as
"Fix 22 pre-existing test failures unrelated to OAuth")
CHANGELOG, CLAUDE.md (Key Files + Commands), TODOS.md updated. The plan
file at ~/.claude/plans/take-a-look-and-gentle-pine.md captures the full
review trail (CEO=C, Eng-Q3=A, Eng-Issue5=a, 8 defaults applied).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(v0.26.5): CI fallout — getStats excludes soft-deleted; tests use --confirm-destructive
Two CI failures from the v0.26.5 ship:
1. **Tier 1 (Postgres E2E):** `E2E: Page CRUD > delete_page removes page and
others survive` failed because `delete_page` now soft-deletes (sets
deleted_at) but `getStats.page_count` was still counting all rows. The
test seeds 16 pages, deletes one, and asserts page_count is 15. Fix:
`getStats` now filters `WHERE deleted_at IS NULL` for page_count in both
engines. This matches the visibility-filter contract — soft-deleted pages
are hidden everywhere the user looks (search, get_page, list_pages, stats).
Chunks and links stay raw because they still occupy storage until the
autopilot purge phase runs.
2. **Test 2 (PGLite unit):** `multi-source-integration.test.ts:184` and
`e2e/multi-source.test.ts:274` called `runSources(engine, ['remove', X,
'--yes'])` against populated sources. v0.26.5's destructive guard rejects
`--yes` alone on populated sources and calls `process.exit(5)`, which
killed the bun test runner mid-suite (CI exit 5). Both test sites now
pass `--confirm-destructive` per the v0.26.5 contract.
Verification: 115/0 pass across destructive-guard, pages-soft-delete,
sql-ranking, schema-bootstrap-coverage, sources, repos-alias, and
multi-source-integration test files. typecheck PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test): cycle phase count is 9 (v0.26.5 added `purge` phase)
CI failure: `runCycle — yieldBetweenPhases hook` tests asserted exactly 8
phases. v0.26.5 added the autopilot `purge` phase as the 9th, so:
- `test/core/cycle.test.ts:381` — `hookCalls` is now 9 (one yield per phase)
- `test/core/cycle.test.ts:392` — `report.phases.length` is now 9
- `test/e2e/cycle.test.ts:101` — same update for the dry-run E2E
The `purge` phase invocation was already visible in the failing log output:
the cycle ran 9 phases end-to-end; the test assertions hadn't been updated.
Verification: bun run typecheck PASS. cycle.test.ts: 28/0 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
326 lines
12 KiB
TypeScript
326 lines
12 KiB
TypeScript
/**
|
|
* v0.26.5 — destructive-guard unit tests.
|
|
*
|
|
* Source-level guard against accidental data loss. Three layers:
|
|
* 1. Impact assessment (counts pages/chunks/embeddings/files for a source)
|
|
* 2. Confirmation gate (`--confirm-destructive` required when data exists;
|
|
* `--yes` alone rejected)
|
|
* 3. Soft-delete with 72h TTL (column-based as of v0.26.5; JSONB shape was
|
|
* migrated in v33)
|
|
*
|
|
* Run against PGLite — the contract logic is identical on Postgres but
|
|
* PGLite is fast + DATABASE_URL-free. Postgres-specific paths (CONCURRENTLY,
|
|
* RLS) are covered separately by E2E tests.
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
|
import {
|
|
assessDestructiveImpact,
|
|
checkDestructiveConfirmation,
|
|
softDeleteSource,
|
|
restoreSource,
|
|
listArchivedSources,
|
|
purgeExpiredSources,
|
|
formatImpact,
|
|
formatSoftDelete,
|
|
SOFT_DELETE_TTL_HOURS,
|
|
type DestructiveImpact,
|
|
} from '../src/core/destructive-guard.ts';
|
|
|
|
// Tier 3 opt-out — these tests need the cold-init schema path so the v33
|
|
// migration columns exist on the brain under test.
|
|
delete process.env.GBRAIN_PGLITE_SNAPSHOT;
|
|
|
|
async function setupBrain(): Promise<PGLiteEngine> {
|
|
const engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
return engine;
|
|
}
|
|
|
|
async function seedSource(engine: PGLiteEngine, id: string, opts?: { withPages?: number }): Promise<void> {
|
|
await engine.executeRaw(
|
|
`INSERT INTO sources (id, name) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`,
|
|
[id, id],
|
|
);
|
|
const count = opts?.withPages ?? 0;
|
|
for (let i = 0; i < count; i++) {
|
|
await engine.executeRaw(
|
|
`INSERT INTO pages (source_id, slug, type, title) VALUES ($1, $2, 'note', $3)`,
|
|
[id, `${id}/page-${i}`, `Page ${i}`],
|
|
);
|
|
}
|
|
}
|
|
|
|
describe('assessDestructiveImpact', () => {
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = await setupBrain();
|
|
}, 30000);
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
test('returns null for a non-existent source', async () => {
|
|
const impact = await assessDestructiveImpact(engine, 'aim-does-not-exist');
|
|
expect(impact).toBeNull();
|
|
});
|
|
|
|
test('counts zero across the board for an empty source', async () => {
|
|
await seedSource(engine, 'aim-empty', { withPages: 0 });
|
|
const impact = await assessDestructiveImpact(engine, 'aim-empty');
|
|
expect(impact).not.toBeNull();
|
|
expect(impact!.pageCount).toBe(0);
|
|
expect(impact!.chunkCount).toBe(0);
|
|
expect(impact!.embeddingCount).toBe(0);
|
|
expect(impact!.fileCount).toBe(0);
|
|
// Empty-source summary is the safe message, not the "permanently delete" warning.
|
|
expect(impact!.summary).toContain('safe to remove');
|
|
});
|
|
|
|
test('counts pages correctly for a populated source', async () => {
|
|
await seedSource(engine, 'aim-populated', { withPages: 3 });
|
|
const impact = await assessDestructiveImpact(engine, 'aim-populated');
|
|
expect(impact!.pageCount).toBe(3);
|
|
expect(impact!.summary).toContain('3 pages');
|
|
expect(impact!.summary).toContain('permanently delete');
|
|
});
|
|
|
|
test('source-scopes pages — multi-source isolation', async () => {
|
|
await seedSource(engine, 'aim-src-a', { withPages: 2 });
|
|
await seedSource(engine, 'aim-src-b', { withPages: 5 });
|
|
const a = await assessDestructiveImpact(engine, 'aim-src-a');
|
|
const b = await assessDestructiveImpact(engine, 'aim-src-b');
|
|
expect(a!.pageCount).toBe(2);
|
|
expect(b!.pageCount).toBe(5);
|
|
});
|
|
});
|
|
|
|
describe('checkDestructiveConfirmation (gate truth table)', () => {
|
|
const populated: DestructiveImpact = {
|
|
sourceId: 'has-data',
|
|
sourceName: 'has-data',
|
|
pageCount: 100,
|
|
chunkCount: 500,
|
|
embeddingCount: 500,
|
|
fileCount: 0,
|
|
summary: '⚠️ This will permanently delete: 100 pages, 500 chunks, 500 embeddings',
|
|
};
|
|
|
|
const empty: DestructiveImpact = {
|
|
sourceId: 'no-data',
|
|
sourceName: 'no-data',
|
|
pageCount: 0,
|
|
chunkCount: 0,
|
|
embeddingCount: 0,
|
|
fileCount: 0,
|
|
summary: 'Source "no-data" has no data (safe to remove).',
|
|
};
|
|
|
|
test('dry-run always passes regardless of flags', () => {
|
|
expect(checkDestructiveConfirmation(populated, { dryRun: true })).toBeNull();
|
|
expect(checkDestructiveConfirmation(populated, { yes: true, dryRun: true })).toBeNull();
|
|
});
|
|
|
|
test('empty source passes without --confirm-destructive', () => {
|
|
expect(checkDestructiveConfirmation(empty, {})).toBeNull();
|
|
expect(checkDestructiveConfirmation(empty, { yes: true })).toBeNull();
|
|
});
|
|
|
|
test('--confirm-destructive passes regardless of --yes', () => {
|
|
expect(checkDestructiveConfirmation(populated, { confirmDestructive: true })).toBeNull();
|
|
expect(checkDestructiveConfirmation(populated, { yes: true, confirmDestructive: true })).toBeNull();
|
|
});
|
|
|
|
test('--yes alone with data is REJECTED with guidance message', () => {
|
|
const msg = checkDestructiveConfirmation(populated, { yes: true });
|
|
expect(msg).not.toBeNull();
|
|
expect(msg).toContain('--confirm-destructive');
|
|
expect(msg).toContain('archive');
|
|
});
|
|
|
|
test('no flags + populated source rejects', () => {
|
|
const msg = checkDestructiveConfirmation(populated, {});
|
|
expect(msg).not.toBeNull();
|
|
expect(msg).toContain('--confirm-destructive');
|
|
});
|
|
});
|
|
|
|
describe('soft-delete + restore lifecycle (column-based v0.26.5)', () => {
|
|
// ONE engine for the whole describe — cold init runs ~29 migrations, ~3s.
|
|
// Each test uses a unique source id so they don't cross-pollute.
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = await setupBrain();
|
|
}, 30000);
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
test('softDeleteSource flips column shape + sets TTL', async () => {
|
|
const id = 'sd-flips';
|
|
await seedSource(engine, id, { withPages: 2 });
|
|
const before = Date.now();
|
|
const result = await softDeleteSource(engine, id);
|
|
const after = Date.now();
|
|
expect(result).not.toBeNull();
|
|
expect(result!.id).toBe(id);
|
|
expect(result!.pageCount).toBe(2);
|
|
const ttlMs = SOFT_DELETE_TTL_HOURS * 60 * 60 * 1000;
|
|
expect(result!.expiresAt.getTime()).toBeGreaterThanOrEqual(before + ttlMs - 1000);
|
|
expect(result!.expiresAt.getTime()).toBeLessThanOrEqual(after + ttlMs + 1000);
|
|
const rows = await engine.executeRaw<{ archived: boolean; archived_at: string }>(
|
|
`SELECT archived, archived_at FROM sources WHERE id = $1`,
|
|
[id],
|
|
);
|
|
expect(rows[0].archived).toBe(true);
|
|
expect(rows[0].archived_at).not.toBeNull();
|
|
});
|
|
|
|
test('softDeleteSource is idempotent-as-null on already-archived', async () => {
|
|
const id = 'sd-idem';
|
|
await seedSource(engine, id, { withPages: 1 });
|
|
await softDeleteSource(engine, id);
|
|
expect(await softDeleteSource(engine, id)).toBeNull();
|
|
});
|
|
|
|
test('softDeleteSource returns null for unknown source', async () => {
|
|
expect(await softDeleteSource(engine, 'sd-unknown-xyz')).toBeNull();
|
|
});
|
|
|
|
test('softDeleteSource flips federated:false in JSONB but archived state is column-based', async () => {
|
|
const id = 'sd-jsonb';
|
|
await seedSource(engine, id, { withPages: 1 });
|
|
await softDeleteSource(engine, id);
|
|
const rows = await engine.executeRaw<{ config: any; archived: boolean }>(
|
|
`SELECT config, archived FROM sources WHERE id = $1`,
|
|
[id],
|
|
);
|
|
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
|
|
expect(config.federated).toBe(false);
|
|
expect(rows[0].archived).toBe(true);
|
|
// Issue 5 contract: archived must NOT live in config any more.
|
|
expect(config.archived).toBeUndefined();
|
|
expect(config.archived_at).toBeUndefined();
|
|
});
|
|
|
|
test('restoreSource clears the column state and re-federates by default', async () => {
|
|
const id = 'sd-restore-fed';
|
|
await seedSource(engine, id, { withPages: 1 });
|
|
await softDeleteSource(engine, id);
|
|
expect(await restoreSource(engine, id)).toBe(true);
|
|
const rows = await engine.executeRaw<{ archived: boolean; archived_at: string | null; config: any }>(
|
|
`SELECT archived, archived_at, config FROM sources WHERE id = $1`,
|
|
[id],
|
|
);
|
|
expect(rows[0].archived).toBe(false);
|
|
expect(rows[0].archived_at).toBeNull();
|
|
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
|
|
expect(config.federated).toBe(true);
|
|
});
|
|
|
|
test('restoreSource respects --no-federate (refederate=false)', async () => {
|
|
const id = 'sd-no-fed';
|
|
await seedSource(engine, id, { withPages: 1 });
|
|
await softDeleteSource(engine, id);
|
|
await restoreSource(engine, id, false);
|
|
const rows = await engine.executeRaw<{ config: any }>(
|
|
`SELECT config FROM sources WHERE id = $1`,
|
|
[id],
|
|
);
|
|
const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config;
|
|
expect(config.federated).toBe(false);
|
|
});
|
|
|
|
test('restoreSource is idempotent-as-false on already-active', async () => {
|
|
const id = 'sd-active';
|
|
await seedSource(engine, id);
|
|
expect(await restoreSource(engine, id)).toBe(false);
|
|
});
|
|
|
|
test('listArchivedSources filters via the archived column, not JSONB', async () => {
|
|
const archivedId = 'la-archived';
|
|
const liveId = 'la-live';
|
|
await seedSource(engine, archivedId, { withPages: 3 });
|
|
await seedSource(engine, liveId, { withPages: 1 });
|
|
await softDeleteSource(engine, archivedId);
|
|
const archived = await listArchivedSources(engine);
|
|
const ids = archived.map((a) => a.id);
|
|
expect(ids).toContain(archivedId);
|
|
expect(ids).not.toContain(liveId);
|
|
const archivedRow = archived.find((a) => a.id === archivedId)!;
|
|
expect(archivedRow.pageCount).toBe(3);
|
|
});
|
|
|
|
test('purgeExpiredSources only deletes rows with archive_expires_at <= now()', async () => {
|
|
const expiredId = 'pe-expired';
|
|
const recoverableId = 'pe-recoverable';
|
|
await seedSource(engine, expiredId, { withPages: 2 });
|
|
await seedSource(engine, recoverableId, { withPages: 1 });
|
|
await softDeleteSource(engine, expiredId);
|
|
await softDeleteSource(engine, recoverableId);
|
|
await engine.executeRaw(
|
|
`UPDATE sources SET archive_expires_at = now() - INTERVAL '1 hour' WHERE id = $1`,
|
|
[expiredId],
|
|
);
|
|
const purged = await purgeExpiredSources(engine);
|
|
expect(purged).toContain(expiredId);
|
|
expect(purged).not.toContain(recoverableId);
|
|
const remainingPages = await engine.executeRaw<{ n: number }>(
|
|
`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`,
|
|
[expiredId],
|
|
);
|
|
expect(remainingPages[0].n).toBe(0);
|
|
});
|
|
|
|
test('purgeExpiredSources is no-op when nothing is past TTL', async () => {
|
|
// After all earlier tests, there may still be archived rows whose
|
|
// archive_expires_at is in the future. Force-update any leftover-past
|
|
// rows OUT of expiration before this assertion (we only want to test
|
|
// the no-op return here, not interfere with prior test state).
|
|
await engine.executeRaw(
|
|
`UPDATE sources SET archive_expires_at = now() + INTERVAL '72 hours' WHERE archived = true`,
|
|
);
|
|
const purged = await purgeExpiredSources(engine);
|
|
expect(purged).toEqual([]);
|
|
});
|
|
});
|
|
|
|
describe('formatters (display helpers)', () => {
|
|
test('formatImpact renders the boxed preview with the source id and counts', () => {
|
|
const impact: DestructiveImpact = {
|
|
sourceId: 'media-corpus',
|
|
sourceName: 'Media Corpus',
|
|
pageCount: 5033,
|
|
chunkCount: 22000,
|
|
embeddingCount: 22000,
|
|
fileCount: 0,
|
|
summary: '⚠️ This will permanently delete: 5,033 pages, 22,000 chunks, 22,000 embeddings',
|
|
};
|
|
const out = formatImpact(impact);
|
|
expect(out).toContain('Media Corpus');
|
|
expect(out).toContain('media-corpus');
|
|
expect(out).toContain('5,033');
|
|
expect(out).toContain('22,000');
|
|
expect(out).toContain('DESTRUCTIVE OPERATION');
|
|
});
|
|
|
|
test('formatSoftDelete renders the post-archive guidance with restore command', () => {
|
|
const out = formatSoftDelete({
|
|
id: 'src-a',
|
|
name: 'src-a',
|
|
deletedAt: new Date(),
|
|
expiresAt: new Date(Date.now() + 72 * 60 * 60 * 1000),
|
|
pageCount: 100,
|
|
});
|
|
expect(out).toContain('archived');
|
|
expect(out).toContain('restore src-a');
|
|
expect(out).toContain('72');
|
|
});
|
|
});
|