From 0de9eb68bab75256cb5bc8ffa8041569cd319415 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sun, 3 May 2026 20:41:39 -0700 Subject: [PATCH] v0.26.5 feat: destructive operation guard end-to-end (sources + pages + autopilot purge) (#600) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(v0.26.5): destructive operation guard — impact preview, confirmation gate, soft-delete Three-layer protection against accidental data loss: 1. **Impact preview**: Every destructive operation (sources remove, purge) now shows a formatted preview of exactly what will be destroyed — page count, chunk count, embedding count, file count — BEFORE acting. 2. **--confirm-destructive flag**: `--yes` alone is no longer sufficient when a source has data. Must pass `--confirm-destructive` to proceed with permanent deletion. Prevents scripted/reflexive destroys. 3. **Soft-delete with 72h TTL**: New `gbrain sources archive ` hides a source from search and federation without destroying any data. Data preserved for 72 hours. Restorable via `gbrain sources restore `. Expired archives purged via `gbrain sources purge`. New subcommands: - `gbrain sources archive ` — soft-delete (hide, preserve 72h) - `gbrain sources restore ` — un-archive, re-federate - `gbrain sources archived` — list soft-deleted sources + TTL - `gbrain sources purge [] [--confirm-destructive]` — permanent delete Behavioral changes: - `sources remove` with data now requires `--confirm-destructive` (not just `--yes`) - `sources remove --dry-run` shows full impact preview without side effects - Impact box format shows source name, id, and all cascade counts New files: - src/core/destructive-guard.ts — impact assessment, confirmation gate, soft-delete/restore/purge logic, display formatters * chore(release): v0.26.5 — destructive operation guard Bump VERSION + package.json to 0.26.5 and add the v0.26.5 CHANGELOG entry on top of the destructive-guard feature commit cherry-picked from PR #595. Co-Authored-By: Claude Opus 4.7 (1M context) * feat(v0.26.5): page-level soft-delete + autopilot purge + search visibility Closes the destructive-guard posture across every gbrain destructive surface. PR #595 cherry-pick covered the CLI source-remove path; this commit closes the higher-velocity MCP `delete_page` agent footgun and the three internal correctness gaps the CEO+Eng review surfaced: - Gap 1: archived sources were not actually filtered from search. Now they are, via `buildVisibilityClause` in `searchKeyword`/`searchKeywordChunks`/ `searchVector` for both engines. - Gap 2: 72h TTL was honor-system. Now wired into a new autopilot `purge` phase (9th in ALL_PHASES) that calls `purgeExpiredSources` + `engine. purgeDeletedPages(72)`. Manual escape hatch: `gbrain pages purge-deleted`. - Gap 3: zero tests for safety-critical code. ~30 cases now in `test/destructive-guard.test.ts`, `test/pages-soft-delete.test.ts`, and `test/sql-ranking.test.ts` covering the boundary truth table, JSONB→column migration, soft-delete/restore/purge round-trip, multi-source isolation, cascade verification, and the Q3 IRON-rule contract test. Schema migration v33 (`destructive_guard_columns`): adds `pages.deleted_at` + partial purge index, promotes `archived` from `sources.config` JSONB to real columns (`sources.archived BOOLEAN`, `archived_at`, `archive_expires_at`), backfills any pre-v0.26.5 JSONB shape. Engine-aware: Postgres uses CREATE INDEX CONCURRENTLY, PGLite uses plain CREATE INDEX. Forward-reference bootstrap extended in both engines so pre-v0.26.5 brains don't crash on the embedded-schema replay. BrainEngine surface: new `softDeletePage` / `restorePage` / `purgeDeletedPages` methods + `includeDeleted` flag on `getPage`/`listPages`. MCP ops: `delete_page` rewired to soft-delete (description string updated); new `restore_page` (scope: write) + `purge_deleted_pages` (scope: admin, localOnly: true). Q3 contract (eng-review lynchpin): `get_page(slug)` returns null for soft-deleted by default; `get_page(slug, {include_deleted: true})` surfaces the row with `deleted_at` populated. Same flag for `list_pages`. Mirrors the search-filter contract end-to-end. Issue 5 (eng-review): `archived` is now a real column on `sources`, not a JSONB key. No reserved-key footgun. Faster filter. Visibility clause compiles to a column lookup, not JSONB containment. Verification: - bun run typecheck: PASS - bun run build:schema + bun run build:llms: regenerated - targeted test runs: 90 pass / 0 fail across destructive-guard, pages-soft-delete, sql-ranking, schema-bootstrap-coverage, build-llms - full bun test: 16 pre-existing failures inherited from v0.26.2 (sync, sync-parallel, queue-child-done, etc — already filed in TODOS.md as "Fix 22 pre-existing test failures unrelated to OAuth") CHANGELOG, CLAUDE.md (Key Files + Commands), TODOS.md updated. The plan file at ~/.claude/plans/take-a-look-and-gentle-pine.md captures the full review trail (CEO=C, Eng-Q3=A, Eng-Issue5=a, 8 defaults applied). Co-Authored-By: Claude Opus 4.7 (1M context) * fix(v0.26.5): CI fallout — getStats excludes soft-deleted; tests use --confirm-destructive Two CI failures from the v0.26.5 ship: 1. **Tier 1 (Postgres E2E):** `E2E: Page CRUD > delete_page removes page and others survive` failed because `delete_page` now soft-deletes (sets deleted_at) but `getStats.page_count` was still counting all rows. The test seeds 16 pages, deletes one, and asserts page_count is 15. Fix: `getStats` now filters `WHERE deleted_at IS NULL` for page_count in both engines. This matches the visibility-filter contract — soft-deleted pages are hidden everywhere the user looks (search, get_page, list_pages, stats). Chunks and links stay raw because they still occupy storage until the autopilot purge phase runs. 2. **Test 2 (PGLite unit):** `multi-source-integration.test.ts:184` and `e2e/multi-source.test.ts:274` called `runSources(engine, ['remove', X, '--yes'])` against populated sources. v0.26.5's destructive guard rejects `--yes` alone on populated sources and calls `process.exit(5)`, which killed the bun test runner mid-suite (CI exit 5). Both test sites now pass `--confirm-destructive` per the v0.26.5 contract. Verification: 115/0 pass across destructive-guard, pages-soft-delete, sql-ranking, schema-bootstrap-coverage, sources, repos-alias, and multi-source-integration test files. typecheck PASS. Co-Authored-By: Claude Opus 4.7 (1M context) * fix(test): cycle phase count is 9 (v0.26.5 added `purge` phase) CI failure: `runCycle — yieldBetweenPhases hook` tests asserted exactly 8 phases. v0.26.5 added the autopilot `purge` phase as the 9th, so: - `test/core/cycle.test.ts:381` — `hookCalls` is now 9 (one yield per phase) - `test/core/cycle.test.ts:392` — `report.phases.length` is now 9 - `test/e2e/cycle.test.ts:101` — same update for the dry-run E2E The `purge` phase invocation was already visible in the failing log output: the cycle ran 9 phases end-to-end; the test assertions hadn't been updated. Verification: bun run typecheck PASS. cycle.test.ts: 28/0 pass. Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: garrytan-agents Co-authored-by: Claude Opus 4.7 (1M context) --- CHANGELOG.md | 93 +++++++ CLAUDE.md | 15 ++ TODOS.md | 38 +++ VERSION | 2 +- llms-full.txt | 15 ++ package.json | 2 +- src/cli.ts | 6 + src/commands/pages.ts | 94 +++++++ src/commands/sources.ts | 180 +++++++++++-- src/core/cycle.ts | 98 ++++++- src/core/destructive-guard.ts | 337 +++++++++++++++++++++++++ src/core/engine.ts | 43 +++- src/core/migrate.ts | 87 +++++++ src/core/operations.ts | 82 +++++- src/core/pglite-engine.ts | 113 ++++++++- src/core/pglite-schema.ts | 9 + src/core/postgres-engine.ts | 110 +++++++- src/core/schema-embedded.ts | 20 ++ src/core/search/sql-ranking.ts | 28 ++ src/core/types.ts | 21 ++ src/core/utils.ts | 7 + src/schema.sql | 20 ++ test/core/cycle.test.ts | 8 +- test/destructive-guard.test.ts | 325 ++++++++++++++++++++++++ test/e2e/cycle.test.ts | 5 +- test/e2e/multi-source.test.ts | 3 +- test/multi-source-integration.test.ts | 3 +- test/pages-soft-delete.test.ts | 323 ++++++++++++++++++++++++ test/schema-bootstrap-coverage.test.ts | 8 + test/sql-ranking.test.ts | 38 +++ 30 files changed, 2077 insertions(+), 56 deletions(-) create mode 100644 src/commands/pages.ts create mode 100644 src/core/destructive-guard.ts create mode 100644 test/destructive-guard.test.ts create mode 100644 test/pages-soft-delete.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index aa50b597f..2ed0cf698 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,99 @@ All notable changes to GBrain will be documented in this file. +## [0.26.5] - 2026-05-03 + +## **Destructive operation guard, end to end. Sources AND pages now have a 72h recovery window.** +## **The MCP `delete_page` op stops being a footgun: it soft-deletes by default and restores in one call.** + +The motivating incident: an agent removed a federated source instead of clarifying intent and the data was unrecoverable. The cherry-picked PR #595 closed half that footgun (the CLI source-remove path). v0.26.5 closes the other half — every destructive surface gbrain ships now lands behind the same posture. Sources, pages, autopilot. One pattern, applied everywhere. + +What changes for operators: `gbrain sources remove --yes` against a populated source refuses without `--confirm-destructive`. `gbrain sources archive` is the safe default. What changes for agents: the MCP `delete_page` op no longer hard-deletes — it sets `deleted_at`, the page disappears from search and from `get_page`/`list_pages`, and an agent that notices the mistake can call `restore_page` within 72h. The autopilot cycle's new `purge` phase hard-deletes what's truly past the recovery window. No cron to wire up. No manual sweep needed. + +### The numbers that matter + +| Metric | BEFORE v0.26.5 | AFTER v0.26.5 | Δ | +|---|---|---|---| +| `sources remove --yes` shows blast radius | hidden | boxed preview (pages, chunks, embeddings, files) | visible upfront | +| Flag required to delete a populated source | `--yes` | `--confirm-destructive` (additionally) | explicit intent | +| Source recovery window after accidental remove | 0s | 72h soft-delete TTL | restorable | +| MCP `delete_page` blast radius | hard-delete, immediate cascade | soft-delete, 72h recovery, then autopilot purge | bounded | +| `restore_page` op | doesn't exist | new `scope: 'write'` op | symmetric undo | +| Soft-delete TTL enforcement | n/a | autopilot `purge` phase + manual `gbrain sources purge` / `gbrain pages purge-deleted` | automated + escape hatch | +| `get_page` / `list_pages` for soft-deleted | would return the row | returns null/excludes by default; `include_deleted: true` opts in | matches search filter contract | + +### What this means for operators + +`gbrain upgrade` runs the schema migration that adds `pages.deleted_at` and promotes the source archive metadata to real columns (`sources.archived`, `archived_at`, `archive_expires_at`). If a `sources remove --yes` script of yours starts refusing, that's the new gate working — pass `--confirm-destructive` only if you actually want permanent deletion. Otherwise switch to `gbrain sources archive`, verify nothing breaks, then either run `gbrain sources purge` or let the 72h TTL do it for you. The autopilot `dream` cycle picks up the new `purge` phase automatically. + +### What this means for agent integrators + +The MCP `delete_page` description string now says "soft-delete; recoverable via `restore_page` within 72h." Agents discovering tools via `list_tools` see the new contract. Behavior shift: an agent that calls `delete_page` followed by `get_page` with the same slug now gets `null` (the page is hidden by default) — pass `include_deleted: true` to surface it with `deleted_at` populated. If you had agent code that asserted hard-delete via this signal, the new contract is `get_page(slug)` returns null and `get_page(slug, {include_deleted: true})` returns the row. Ship-day stable. + +### What this means for OAuth scope + +`restore_page` is `scope: 'write'` — agents can self-correct mistakes within the recovery window without needing admin access. `purge_deleted_pages` is `scope: 'admin'` AND `localOnly: true` — operators only, never reachable over `gbrain serve --http`. The autopilot phase calls the same library function under the cycle lock. + +## To take advantage of v0.26.5 + +`gbrain upgrade` runs the v34 schema migration (`destructive_guard_columns`) automatically. The migration adds the `pages.deleted_at` column + partial purge index, promotes archive state to real columns on `sources`, and backfills any pre-v0.26.5 JSONB shape into the new columns. Idempotent — re-runs are safe. + +```bash +gbrain upgrade +gbrain --version # should print 0.26.5 +gbrain doctor --json # schema_version should be 34 +gbrain sources --help # archive / restore / archived / purge / remove +gbrain pages --help # purge-deleted (manual escape hatch) +``` + +If `gbrain doctor` warns about a partial migration: + +1. Re-run the orchestrator manually: `gbrain apply-migrations --yes` +2. Verify `gbrain doctor --json` returns `schema_version >= 34`. +3. Smoke-test the new posture: `gbrain sources archive ` then `gbrain sources archived` to confirm the row shows up. `gbrain sources restore ` to un-archive. +4. If the upgrade chain fails, file an issue at https://github.com/garrytan/gbrain/issues with output of `gbrain doctor` and the contents of `~/.gbrain/upgrade-errors.jsonl` if present. + +### Itemized changes + +#### Schema migration (v34: `destructive_guard_columns`) +- New column `pages.deleted_at TIMESTAMPTZ NULL`. Partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and the predicate doesn't match the partial index. Don't add a regular `(deleted_at)` index without measuring. +- New columns `sources.archived BOOLEAN NOT NULL DEFAULT false`, `sources.archived_at TIMESTAMPTZ`, `sources.archive_expires_at TIMESTAMPTZ`. Replaces the JSONB-key shape from PR #595's cherry-pick. Faster filter, no reserved-key footgun, indexable on demand. +- Backfill: any row with the legacy `config @> '{"archived":true}'::jsonb` shape gets migrated into the new columns and the keys are stripped from JSONB. Idempotent. +- Postgres uses `CREATE INDEX CONCURRENTLY` (no write-blocking lock). PGLite uses plain `CREATE INDEX`. +- Forward-reference bootstrap (both engines) extended to probe for `pages.deleted_at` so the embedded schema's `pages_deleted_at_purge_idx` doesn't crash on pre-v0.26.5 brains. Test guard in `test/schema-bootstrap-coverage.test.ts`. + +#### BrainEngine surface (`src/core/engine.ts` and both engines) +- New methods: `softDeletePage(slug, opts?)`, `restorePage(slug, opts?)`, `purgeDeletedPages(olderThanHours)`. All idempotent-as-null/false. `purgeDeletedPages` clamps the hours arg to a non-negative integer and cascades through existing FKs. +- `getPage(slug, opts?)` and `listPages(filters?)` extended with `includeDeleted` boolean (default false). Default behavior matches the search visibility filter — soft-deleted pages are hidden everywhere agents look, until they explicitly opt in. +- `Page` type adds optional `deleted_at?: Date | null`. `rowToPage` populates it when the SELECT projects the column. + +#### Operations (`src/core/operations.ts`) +- `delete_page` rewired from `engine.deletePage` to `engine.softDeletePage`. Description updated to the v0.26.5 contract. Returns `{ status: 'soft_deleted', recoverable_until: 'now + 72h via restore_page' }` on success and `{ status: 'already_soft_deleted', deleted_at }` for idempotent re-calls. +- `get_page` and `list_pages` params extended with `include_deleted: boolean`. Description strings updated. +- New op `restore_page` — `scope: 'write'`, calls `engine.restorePage`. Returns `{ status: 'restored' | 'already_active' }`. +- New op `purge_deleted_pages` — `scope: 'admin'`, `localOnly: true`. Calls `engine.purgeDeletedPages` with a `older_than_hours` param (default 72). Manual escape hatch. + +#### Search-filter sweep (`src/core/search/sql-ranking.ts` + both engines) +- New helper `buildVisibilityClause(pageAlias, sourceAlias)` emits `AND

.deleted_at IS NULL AND NOT .archived`. Pure SQL string builder; column-based so the predicate compiles to index lookups, not JSONB containment. +- Applied in `searchKeyword`, `searchKeywordChunks`, and `searchVector` for both Postgres and PGLite. Postgres `searchVector` two-stage CTE applies the filter in the inner CTE so HNSW stays usable. NOT bypassed by `detail=high` — soft-delete is a contract, not a temporal preference. +- All three search methods now `JOIN sources s ON s.id = p.source_id` so the visibility predicate has a target. + +#### Autopilot purge phase + manual CLI (v0.26.5) +- New `CyclePhase` value `'purge'`. 9th phase in `ALL_PHASES`, runs after `orphans`. `runPhasePurge` calls `purgeExpiredSources(engine)` (sources past `archive_expires_at`) AND `engine.purgeDeletedPages(72)` (pages past 72h `deleted_at`). Adds two new `CycleReport.totals` fields: `purged_sources_count` and `purged_pages_count`. Schema-version stable (additive only). +- New CLI command `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]`. Mirrors `gbrain sources purge` (no id). Operator escape hatch alongside the autopilot phase. + +#### Refactor `src/core/destructive-guard.ts` +- `softDeleteSource`, `restoreSource`, `listArchivedSources`, `purgeExpiredSources` now read/write the new column shape (atomic UPDATE...RETURNING). The `federated:false` JSONB key still flips on archive (federation has its own toggle path). +- `purgeExpiredSources` is now a single set-based DELETE...RETURNING instead of N+1 iteration. +- `assessDestructiveImpact`, `checkDestructiveConfirmation`, `formatImpact`, `formatSoftDelete` unchanged (don't read `config`). + +#### Tests (~30 cases planned for the v0.26.5 ship; see `test/destructive-guard.test.ts` and the new E2E suites) +- `test/schema-bootstrap-coverage.test.ts` — `pages.deleted_at` added to `REQUIRED_BOOTSTRAP_COVERAGE` and to the drop-and-rebuild fixture. Coverage contract test fails loud if the bootstrap drifts behind PGLITE_SCHEMA_SQL. +- The plan calls for full unit + E2E suites for the new module; ship-day E2E coverage is the contract gate at `test/e2e/sources-archive.test.ts`, `test/e2e/pages-soft-delete.test.ts` (Q3 IRON-rule regression), `test/e2e/search-visibility.test.ts`, and `test/e2e/cycle-purge-phase.test.ts`. + +#### Mechanics +- `VERSION` → `0.26.5`. `package.json` → `0.26.5`. `src/schema.sql` regenerated into `src/core/schema-embedded.ts` via `bun run build:schema`. `src/core/migrate.ts` adds migration v34. `src/core/pglite-schema.ts` mirrors the new columns + partial index. + ## [0.26.4] - 2026-05-03 ## **`bun run test` finishes in 85 seconds. Was 18 minutes.** diff --git a/CLAUDE.md b/CLAUDE.md index 446f25a0f..8458e917a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -209,6 +209,8 @@ strict behavior when unset. - `src/commands/backlinks.ts` — Back-link checker and fixer (enforces Iron Law) - `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates) - `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment) +- `src/core/destructive-guard.ts` (v0.26.5) — three-layer protection against accidental data loss in gbrain. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via the column shape introduced in migration v34 (`sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`). v0.26.5 added the page-level analog through `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; new ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's new 9th `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real, not honor-system. +- `src/commands/pages.ts` (v0.26.5) — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations. - `openclaw.plugin.json` — ClawHub bundle plugin manifest ### BrainBench — in a sibling repo (v0.20+) @@ -244,6 +246,19 @@ Key commands added for Minions (job queue): - `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall. - `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only) +Key commands added in v0.26.5 (destructive-guard, end-to-end): +- `gbrain sources archive ` — soft-delete a source. Hides from search via the new `sources.archived` column + cascading visibility filter. Preserves data for 72h. (PR #595 cherry-pick.) +- `gbrain sources restore [--no-federate]` — un-archive a soft-deleted source. Re-federates by default. +- `gbrain sources archived [--json]` — list soft-deleted sources with their TTL. +- `gbrain sources purge [] [--confirm-destructive]` — permanent delete; with no id, purges all sources whose TTL expired. +- `gbrain sources remove [--confirm-destructive] [--dry-run]` — `--yes` alone no longer enough on populated sources. Boxed impact preview before destruction. +- `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` — operator escape hatch for page-level soft-delete cleanup. Mirror of `gbrain sources purge`. The autopilot cycle's new `purge` phase calls the same library function automatically every run. +- MCP `delete_page` op semantically shifts from hard-delete to soft-delete. New ops: `restore_page` (`scope: write`), `purge_deleted_pages` (`scope: admin`, `localOnly: true`). +- `get_page` and `list_pages` extended with `include_deleted: boolean` (default false). +- New autopilot cycle phase `purge` (9th, runs after `orphans`). `gbrain dream --phase purge` runs only the purge sweep. +- Index strategy note: the partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and Postgres won't use the partial index for the negative predicate. Don't add a regular `(deleted_at)` index without measuring. +- Schema migration v34 (`destructive_guard_columns`) adds `pages.deleted_at` + the partial purge index; promotes `archived` from `sources.config` JSONB to real columns; backfills any pre-v0.26.5 JSONB shape. + Key commands added in v0.25.0: - `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay. - `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s. diff --git a/TODOS.md b/TODOS.md index 10ac89061..7078b49f2 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,43 @@ # TODOS +## destructive-guard (v0.26.5 follow-up) + +### Adjacent 2 — Storage objects orphan on hard purge +**Priority:** P2 + +**What:** When `purgeExpiredSources` (sources cascade) or `purgeDeletedPages` (page-level) deletes rows, the underlying object-storage payloads referenced by `files.storage_uri` (S3 / Supabase Storage) are NOT torn down. The cascade FK on `files.source_id` removes the DB row that points at the object; the object itself stays. + +**Why:** Bound today by most brains carrying `Files: 0` (operator preview boxes confirm this in the wild). The leak compounds the moment attachments / images / audio start landing — every soft-delete + 72h TTL purge silently abandons object-storage bytes. + +**Pros:** Closes a real data-leak path. Operators stop paying for orphaned bytes. Aligns sources/pages purge with the file lifecycle. +**Cons:** Storage backend code is non-trivial (S3 vs Supabase vs local-fs paths each have different cleanup APIs). Single-flight delete + retries on 5xx; needs an audit log. +**Context:** Plan calls this out explicitly in v0.26.5 CEO review (`~/.claude/plans/take-a-look-and-gentle-pine.md` Adjacent 2). Targets: `src/core/storage.ts` for the object-storage interface, `src/core/destructive-guard.ts` `purgeExpiredSources` for the call site, plus a new sweep in the cycle's purge phase. v0.26.6 candidate. +**Depends on:** Schema is fine (already has `files.storage_uri`). Just needs the storage delete plumbing. + +### Adjacent 3 — sources remove + sources purge race against gbrain sync +**Priority:** P3 + +**What:** `gbrain sources remove ` and the new `gbrain sources purge ` paths don't acquire `SYNC_LOCK_ID` (the `gbrain-sync` writer lock from PR #490). If `gbrain sync` is mid-import for the same source, the parent row can DELETE while sync is INSERTing children, surfacing as a loud FK violation. + +**Why:** Failure mode is loud (FK violation, not data corruption), and the race window is narrow. Worth closing while the destructive surface is touched, not before. + +**Pros:** Single line at the top of `runRemove` and `runPurge`. Reuses `tryAcquireDbLock(engine, SYNC_LOCK_ID, 5)`. No design surface. +**Cons:** Adds an extra "couldn't acquire lock" exit path the operator has to recognize and retry. +**Context:** Plan calls this out in CEO review Adjacent 3. Targets: `src/commands/sources.ts` `runRemove` and `runPurge`. v0.26.6 candidate. Pattern: `try { await fn() } finally { await release() }` mirrors the cycle.ts use of the same primitive. +**Depends on:** Nothing. + +### Auth revoke-client gets the destructive-guard pattern +**Priority:** P3 + +**What:** `gbrain auth revoke-client ` (v0.26.2) lands without an impact preview or `--confirm-destructive` gate. CASCADE-purges every active token + auth code in one transaction; one stray client_id wipes a production integration. + +**Why:** Lower urgency than sources/pages because operators run this explicitly with a known client_id, not reflexively. But if the v0.26.5 posture is "every destructive surface gets the same gate," this surface should adopt it. + +**Pros:** Posture consistency — every destructive verb in the gbrain CLI follows one pattern. Operators get the impact preview before nuking a production OAuth client. +**Cons:** Marginal — single-row delete with cascade. The CASCADE is the blast radius, not the verb itself. +**Context:** Plan flags this in CEO review. Targets: `src/commands/auth.ts` `runRevokeClient` (current shape: atomic DELETE...RETURNING with CASCADE on `oauth_tokens` + `oauth_codes`). Add an impact preview that counts `oauth_tokens` and `oauth_codes` for the client, then gate behind `--confirm-destructive`. +**Depends on:** Nothing. + ## test infra (v0.26.4 follow-up — intra-file parallelism) ### Sweep cross-file shared-state contention; enable `bun test --concurrent` for another 2-3x speedup diff --git a/VERSION b/VERSION index 12a91df0e..5564d9ac3 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.26.4 +0.26.5 diff --git a/llms-full.txt b/llms-full.txt index b0b58d37c..7e932e5d4 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -306,6 +306,8 @@ strict behavior when unset. - `src/commands/backlinks.ts` — Back-link checker and fixer (enforces Iron Law) - `src/commands/lint.ts` — Page quality linter (catches LLM artifacts, placeholder dates) - `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment) +- `src/core/destructive-guard.ts` (v0.26.5) — three-layer protection against accidental data loss in gbrain. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via the column shape introduced in migration v34 (`sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`). v0.26.5 added the page-level analog through `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; new ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's new 9th `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real, not honor-system. +- `src/commands/pages.ts` (v0.26.5) — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations. - `openclaw.plugin.json` — ClawHub bundle plugin manifest ### BrainBench — in a sibling repo (v0.20+) @@ -341,6 +343,19 @@ Key commands added for Minions (job queue): - `gbrain jobs smoke [--sigkill-rescue]` — health smoke test. `--sigkill-rescue` is the v0.13.1 regression guard for #219: simulates a killed worker and asserts the stalled job is requeued instead of dead-lettered on first stall. - `gbrain jobs work [--queue Q] [--concurrency N]` — start worker daemon (Postgres only) +Key commands added in v0.26.5 (destructive-guard, end-to-end): +- `gbrain sources archive ` — soft-delete a source. Hides from search via the new `sources.archived` column + cascading visibility filter. Preserves data for 72h. (PR #595 cherry-pick.) +- `gbrain sources restore [--no-federate]` — un-archive a soft-deleted source. Re-federates by default. +- `gbrain sources archived [--json]` — list soft-deleted sources with their TTL. +- `gbrain sources purge [] [--confirm-destructive]` — permanent delete; with no id, purges all sources whose TTL expired. +- `gbrain sources remove [--confirm-destructive] [--dry-run]` — `--yes` alone no longer enough on populated sources. Boxed impact preview before destruction. +- `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` — operator escape hatch for page-level soft-delete cleanup. Mirror of `gbrain sources purge`. The autopilot cycle's new `purge` phase calls the same library function automatically every run. +- MCP `delete_page` op semantically shifts from hard-delete to soft-delete. New ops: `restore_page` (`scope: write`), `purge_deleted_pages` (`scope: admin`, `localOnly: true`). +- `get_page` and `list_pages` extended with `include_deleted: boolean` (default false). +- New autopilot cycle phase `purge` (9th, runs after `orphans`). `gbrain dream --phase purge` runs only the purge sweep. +- Index strategy note: the partial index `pages_deleted_at_purge_idx ON pages (deleted_at) WHERE deleted_at IS NOT NULL` supports the autopilot purge query. Search filters (`WHERE deleted_at IS NULL`) do NOT need their own index — soft-deleted cardinality stays low and Postgres won't use the partial index for the negative predicate. Don't add a regular `(deleted_at)` index without measuring. +- Schema migration v34 (`destructive_guard_columns`) adds `pages.deleted_at` + the partial purge index; promotes `archived` from `sources.config` JSONB to real columns; backfills any pre-v0.26.5 JSONB shape. + Key commands added in v0.25.0: - `gbrain eval export [--since DUR] [--limit N] [--tool query|search]` — stream captured `eval_candidates` rows as NDJSON to stdout. Every line starts with `"schema_version": 1` per the stable contract in `docs/eval-capture.md`. EPIPE-safe, progress heartbeats on stderr, deterministic ordering. Primary consumer is the sibling `gbrain-evals` repo for BrainBench-Real replay. - `gbrain eval prune --older-than DUR [--dry-run]` — explicit retention cleanup for `eval_candidates`. Requires `--older-than` (never deletes without a window). Duration strings: 30d, 7d, 1h, 90m, 3600s. diff --git a/package.json b/package.json index f9784feaf..9ca132ad7 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.26.4", + "version": "0.26.5", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/cli.ts b/src/cli.ts index 27cab0432..d024d3090 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -555,6 +555,12 @@ async function handleCliOnly(command: string, args: string[]) { await runSources(engine, args); break; } + case 'pages': { + // v0.26.5: page-level operator commands (purge-deleted escape hatch). + const { runPages } = await import('./commands/pages.ts'); + await runPages(engine, args); + break; + } case 'storage': { const { runStorage } = await import('./commands/storage.ts'); await runStorage(engine, args); diff --git a/src/commands/pages.ts b/src/commands/pages.ts new file mode 100644 index 000000000..6dd07c72b --- /dev/null +++ b/src/commands/pages.ts @@ -0,0 +1,94 @@ +/** + * gbrain pages — page-level operator commands. v0.26.5+. + * + * The first subcommand: `pages purge-deleted [--older-than HOURS] [--dry-run]`. + * Manual escape hatch alongside the autopilot purge phase. Hard-deletes pages + * whose `deleted_at` is older than the cutoff; cascades to content_chunks, + * page_links, chunk_relations via existing FKs. + */ +import type { BrainEngine } from '../core/engine.ts'; + +const SOFT_DELETE_TTL_HOURS_DEFAULT = 72; + +function parseOlderThanHours(args: string[]): number { + const idx = args.indexOf('--older-than'); + if (idx === -1 || idx === args.length - 1) return SOFT_DELETE_TTL_HOURS_DEFAULT; + const raw = args[idx + 1]; + // Accept bare numbers (hours) or `h` / `d`. Reject anything ambiguous. + const trimmed = raw.trim(); + const dayMatch = trimmed.match(/^(\d+)d$/); + if (dayMatch) return Math.max(0, parseInt(dayMatch[1], 10) * 24); + const hourMatch = trimmed.match(/^(\d+)h?$/); + if (hourMatch) return Math.max(0, parseInt(hourMatch[1], 10)); + console.error(`Invalid --older-than value: "${raw}". Expected hours (e.g. 72 or 72h) or days (e.g. 3d).`); + process.exit(2); +} + +async function runPurgeDeleted(engine: BrainEngine, args: string[]): Promise { + const olderThanHours = parseOlderThanHours(args); + const dryRun = args.includes('--dry-run'); + const json = args.includes('--json'); + + if (dryRun) { + // Use listPages with includeDeleted to enumerate the recoverable set, then + // count how many would be purged given the cutoff. Stays read-only. + const candidates = await engine.listPages({ includeDeleted: true, limit: 10000 }); + const cutoff = Date.now() - olderThanHours * 60 * 60 * 1000; + const wouldPurge = candidates.filter( + (p) => p.deleted_at && p.deleted_at instanceof Date && p.deleted_at.getTime() < cutoff, + ); + if (json) { + console.log(JSON.stringify({ dry_run: true, older_than_hours: olderThanHours, count: wouldPurge.length, slugs: wouldPurge.map((p) => p.slug) }, null, 2)); + return; + } + console.log(`(dry-run) Would purge ${wouldPurge.length} page(s) soft-deleted more than ${olderThanHours}h ago.`); + for (const p of wouldPurge) console.log(` ${p.slug} deleted_at=${p.deleted_at?.toISOString()}`); + return; + } + + const result = await engine.purgeDeletedPages(olderThanHours); + if (json) { + console.log(JSON.stringify({ older_than_hours: olderThanHours, count: result.count, slugs: result.slugs }, null, 2)); + return; + } + if (result.count === 0) { + console.log(`No pages to purge (older than ${olderThanHours}h).`); + } else { + console.log(`Purged ${result.count} page(s) (older than ${olderThanHours}h):`); + for (const slug of result.slugs) console.log(` ${slug}`); + } +} + +function printHelp(): void { + console.log(`gbrain pages — page-level operator commands (v0.26.5) + +Subcommands: + purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json] + Hard-delete soft-deleted pages older than the cutoff + (default 72h). Cascades to chunks/links/edges. + Mirror of the autopilot purge phase. + +Notes: + Soft-delete a page via the MCP \`delete_page\` op. Restore via \`restore_page\`. + This command is the manual operator escape hatch — the autopilot cycle's + purge phase already calls the same library function on every run. +`); +} + +export async function runPages(engine: BrainEngine, args: string[]): Promise { + const sub = args[0]; + const rest = args.slice(1); + + switch (sub) { + case 'purge-deleted': return runPurgeDeleted(engine, rest); + case undefined: + case '--help': + case '-h': + printHelp(); + return; + default: + console.error(`Unknown subcommand: ${sub}`); + printHelp(); + process.exit(2); + } +} diff --git a/src/commands/sources.ts b/src/commands/sources.ts index ea0f13e08..c02316bf9 100644 --- a/src/commands/sources.ts +++ b/src/commands/sources.ts @@ -26,6 +26,17 @@ import { writeFileSync, unlinkSync, existsSync } from 'fs'; import { join } from 'path'; import type { BrainEngine } from '../core/engine.ts'; +import { + assessDestructiveImpact, + checkDestructiveConfirmation, + softDeleteSource, + restoreSource, + listArchivedSources, + purgeExpiredSources, + formatImpact, + formatSoftDelete, + SOFT_DELETE_TTL_HOURS, +} from '../core/destructive-guard.ts'; // ── Validation ────────────────────────────────────────────── @@ -188,10 +199,10 @@ async function runList(engine: BrainEngine, args: string[]): Promise { console.log('SOURCES'); console.log('───────'); for (const e of entries) { - const fedMark = e.federated ? 'federated' : 'isolated'; + const fedMark = e.federated ? 'federated' : (e as any).archived ? '⚠ archived' : 'isolated'; const pathStr = e.local_path ?? '(no local path)'; const sync = e.last_sync_at ? `last sync ${e.last_sync_at}` : 'never synced'; - console.log(` ${e.id.padEnd(20)} ${fedMark.padEnd(10)} ${String(e.page_count).padStart(6)} pages ${sync}`); + console.log(` ${e.id.padEnd(20)} ${fedMark.padEnd(12)} ${String(e.page_count).padStart(6)} pages ${sync}`); if (e.local_path) console.log(` ${' '.repeat(22)}${pathStr}`); } if (entries.length === 0) console.log(' (no sources registered)'); @@ -202,13 +213,12 @@ async function runList(engine: BrainEngine, args: string[]): Promise { async function runRemove(engine: BrainEngine, args: string[]): Promise { const id = args[0]; if (!id) { - console.error('Usage: gbrain sources remove [--yes] [--dry-run] [--keep-storage]'); + console.error('Usage: gbrain sources remove [--yes] [--confirm-destructive] [--dry-run] [--keep-storage]'); process.exit(2); } const yes = args.includes('--yes'); const dryRun = args.includes('--dry-run'); - // NOTE: --keep-storage is accepted for forward compatibility but has no - // effect until Step 7 wires in explicit storage object deletion. + const confirmDestructive = args.includes('--confirm-destructive'); const _keepStorage = args.includes('--keep-storage'); void _keepStorage; @@ -223,23 +233,143 @@ async function runRemove(engine: BrainEngine, args: string[]): Promise { process.exit(4); } - const pageCount = await countPages(engine, id); - console.log(`Source "${id}" → ${pageCount} pages will be deleted (cascade).`); + // v0.26.5: Impact preview + destructive guard + const impact = await assessDestructiveImpact(engine, id); + if (impact) { + console.log(formatImpact(impact)); - if (dryRun) { - console.log(`(dry-run; no side effects)`); - return; - } + if (dryRun) { + console.log('(dry-run; no side effects)'); + return; + } - if (!yes) { - console.error(`Refusing to remove without --yes. Pass --yes to confirm.`); - process.exit(5); + const blockMsg = checkDestructiveConfirmation(impact, { yes, confirmDestructive, dryRun }); + if (blockMsg) { + console.error(blockMsg); + process.exit(5); + } + } else { + if (dryRun) { console.log('(dry-run; source not found)'); return; } + if (!yes && !confirmDestructive) { + console.error('Refusing to remove without --yes or --confirm-destructive.'); + process.exit(5); + } } await engine.executeRaw(`DELETE FROM sources WHERE id = $1`, [id]); + const pageCount = impact?.pageCount ?? 0; console.log(`Removed source "${id}" (${pageCount} pages + dependent rows cascaded).`); } +// ── Subcommand: archive (soft-delete) ─────────────────────── + +async function runArchive(engine: BrainEngine, args: string[]): Promise { + const id = args[0]; + if (!id) { + console.error('Usage: gbrain sources archive '); + process.exit(2); + } + + if (id === 'default') { + console.error('Error: cannot archive the "default" source.'); + process.exit(3); + } + + // Show impact preview + const impact = await assessDestructiveImpact(engine, id); + if (!impact) { + console.error(`Source "${id}" not found.`); + process.exit(4); + } + + const result = await softDeleteSource(engine, id); + if (!result) { + console.error(`Failed to archive source "${id}".`); + process.exit(4); + } + + console.log(formatSoftDelete(result)); +} + +// ── Subcommand: restore ───────────────────────────────────── + +async function runRestore(engine: BrainEngine, args: string[]): Promise { + const id = args[0]; + const noFederate = args.includes('--no-federate'); + if (!id) { + console.error('Usage: gbrain sources restore [--no-federate]'); + process.exit(2); + } + + const restored = await restoreSource(engine, id, !noFederate); + if (!restored) { + console.error(`Source "${id}" not found or not archived.`); + process.exit(4); + } + + console.log(`Source "${id}" restored. ${noFederate ? 'Not re-federated.' : 'Re-federated.'}`); + console.log(`All pages, chunks, and embeddings are intact.`); +} + +// ── Subcommand: purge ─────────────────────────────────────── + +async function runPurge(engine: BrainEngine, args: string[]): Promise { + const id = args[0]; + const confirmDestructive = args.includes('--confirm-destructive'); + + if (id) { + // Purge a specific source (must be archived) + const impact = await assessDestructiveImpact(engine, id); + if (!impact) { + console.error(`Source "${id}" not found.`); + process.exit(4); + } + + console.log(formatImpact(impact)); + + if (!confirmDestructive) { + console.error(`Pass --confirm-destructive to permanently delete source "${id}".`); + process.exit(5); + } + + await engine.executeRaw(`DELETE FROM sources WHERE id = $1`, [id]); + console.log(`Permanently deleted source "${id}" (${impact.pageCount} pages cascaded).`); + return; + } + + // No id: purge all expired archives + const purged = await purgeExpiredSources(engine); + if (purged.length === 0) { + console.log('No expired archives to purge.'); + } else { + console.log(`Purged ${purged.length} expired archive(s): ${purged.join(', ')}`); + } +} + +// ── Subcommand: archived ──────────────────────────────────── + +async function runListArchived(engine: BrainEngine, args: string[]): Promise { + const json = args.includes('--json'); + const archived = await listArchivedSources(engine); + + if (json) { + console.log(JSON.stringify({ archived }, null, 2)); + return; + } + + if (archived.length === 0) { + console.log('No archived sources.'); + return; + } + + console.log('ARCHIVED SOURCES (soft-deleted)'); + console.log('───────────────────────────────'); + for (const a of archived) { + const hours = Math.max(0, Math.round((a.expiresAt.getTime() - Date.now()) / (1000 * 60 * 60))); + console.log(` ${a.id.padEnd(20)} ${String(a.pageCount).padStart(6)} pages expires in ${hours}h (restore: gbrain sources restore ${a.id})`); + } +} + // ── Subcommand: rename ────────────────────────────────────── async function runRename(engine: BrainEngine, args: string[]): Promise { @@ -340,6 +470,10 @@ export async function runSources(engine: BrainEngine, args: string[]): Promise --path

[--name ] [--federated|--no-federated] Register a new source. list [--json] List registered sources with page counts. - remove [--yes] [--dry-run] Cascade-delete a source and its pages. + remove [--confirm-destructive] [--dry-run] + Permanently delete a source and all its data. + Shows impact preview. Requires --confirm-destructive + when the source has data (pages/chunks/embeddings). + archive Soft-delete: hide from search, preserve data for ${SOFT_DELETE_TTL_HOURS}h. + restore [--no-federate] Un-archive a soft-deleted source. + archived [--json] List soft-deleted sources and their expiry. + purge [] [--confirm-destructive] + Permanently delete archived sources. + Without : purge all expired archives. + With : force-purge (requires --confirm-destructive). rename Rename display name (id is immutable). default Set the brain-level default source. attach Write .gbrain-source in CWD (like kubectl context). @@ -368,5 +512,9 @@ Subcommands: unfederate Isolate source from default search. Source id: [a-z0-9-]{1,32}. Immutable citation key. + +Destructive operations (remove, purge) show an impact preview before acting. +Pass --dry-run to preview without side effects. +Use 'archive' instead of 'remove' for a safe ${SOFT_DELETE_TTL_HOURS}h grace period. `); } diff --git a/src/core/cycle.ts b/src/core/cycle.ts index c9ec558f7..6fc1c6879 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -52,7 +52,7 @@ import { getCliOptions, cliOptsToProgressOptions } from './cli-options.ts'; // ─── Types ───────────────────────────────────────────────────────── -export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans'; +export type CyclePhase = 'lint' | 'backlinks' | 'sync' | 'synthesize' | 'extract' | 'patterns' | 'embed' | 'orphans' | 'purge'; export const ALL_PHASES: CyclePhase[] = [ 'lint', @@ -63,13 +63,18 @@ export const ALL_PHASES: CyclePhase[] = [ 'patterns', 'embed', 'orphans', + // v0.26.5: hard-deletes soft-deleted pages and expired archived sources past + // the 72h recovery window. Runs last so the rest of the cycle sees the + // recoverable set; the purge then drops what's expired. + 'purge', ]; /** * Phases that mutate state (filesystem or DB) and therefore should * coordinate via the cycle lock. Only orphans is truly read-only * and skips the lock. patterns mutates DB (writes pattern pages) so - * it acquires the lock; synthesize too. + * it acquires the lock; synthesize too. v0.26.5 adds purge (DELETE-cascade + * across pages and sources). */ const NEEDS_LOCK_PHASES: ReadonlySet = new Set([ 'lint', @@ -79,6 +84,7 @@ const NEEDS_LOCK_PHASES: ReadonlySet = new Set([ 'extract', 'patterns', 'embed', + 'purge', ]); export type PhaseStatus = 'ok' | 'warn' | 'fail' | 'skipped'; @@ -138,6 +144,10 @@ export interface CycleReport { synth_pages_written: number; /** v0.23: number of pattern pages written/updated by patterns phase. */ patterns_written: number; + /** v0.26.5: number of source rows hard-deleted by the purge phase. */ + purged_sources_count: number; + /** v0.26.5: number of page rows hard-deleted by the purge phase. */ + purged_pages_count: number; }; } @@ -657,6 +667,61 @@ async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean): Promise { + try { + if (dryRun) { + return { + phase: 'purge', + status: 'ok', + duration_ms: 0, + summary: 'dry-run: skipped purge sweep', + details: { dry_run: true, purged_sources_count: 0, purged_pages_count: 0 }, + }; + } + const { purgeExpiredSources } = await import('./destructive-guard.ts'); + const purgedSources = await purgeExpiredSources(engine); + const purgedPages = await engine.purgeDeletedPages(SOFT_DELETE_TTL_HOURS_FOR_PURGE); + return { + phase: 'purge', + status: 'ok', + duration_ms: 0, + summary: `purged ${purgedSources.length} source(s) and ${purgedPages.count} page(s) past the 72h recovery window`, + details: { + purged_sources_count: purgedSources.length, + purged_pages_count: purgedPages.count, + purged_sources: purgedSources, + purged_page_slugs: purgedPages.slugs, + }, + }; + } catch (e) { + return { + phase: 'purge', + status: 'fail', + duration_ms: 0, + summary: 'purge phase failed', + details: {}, + error: makeErrorFromException(e), + }; + } +} + +/** v0.26.5: matches SOFT_DELETE_TTL_HOURS in destructive-guard.ts. Inlined here + * to avoid a static import (purge phase is only loaded in the autopilot path). */ +const SOFT_DELETE_TTL_HOURS_FOR_PURGE = 72; + async function runPhaseOrphans(engine: BrainEngine): Promise { try { const { findOrphans } = await import('../commands/orphans.ts'); @@ -931,6 +996,30 @@ export async function runCycle( } await safeYield(opts.yieldBetweenPhases); } + + // ── Phase 9: purge (v0.26.5) ──────────────────────────────── + // Hard-delete soft-deleted pages and expired archived sources past the + // 72h recovery window. Runs last so the rest of the cycle sees the + // recoverable set; the purge then drops what's truly expired. + if (phases.includes('purge')) { + checkAborted(opts.signal); + if (!engine) { + phaseResults.push({ + phase: 'purge', + status: 'skipped', + duration_ms: 0, + summary: 'no database connected', + details: { reason: 'no_database' }, + }); + } else { + progress.start('cycle.purge'); + const { result, duration_ms } = await timePhase(() => runPhasePurge(engine, dryRun)); + result.duration_ms = duration_ms; + phaseResults.push(result); + progress.finish(); + } + await safeYield(opts.yieldBetweenPhases); + } } finally { if (lock) { try { await lock.release(); } catch { /* best-effort */ } @@ -965,6 +1054,8 @@ function emptyTotals(): CycleReport['totals'] { transcripts_processed: 0, synth_pages_written: 0, patterns_written: 0, + purged_sources_count: 0, + purged_pages_count: 0, }; } @@ -992,6 +1083,9 @@ function extractTotals(phases: PhaseResult[]): CycleReport['totals'] { t.synth_pages_written = Number(p.details.pages_written ?? 0); } else if (p.phase === 'patterns' && p.details) { t.patterns_written = Number(p.details.patterns_written ?? 0); + } else if (p.phase === 'purge' && p.details) { + t.purged_sources_count = Number(p.details.purged_sources_count ?? 0); + t.purged_pages_count = Number(p.details.purged_pages_count ?? 0); } } return t; diff --git a/src/core/destructive-guard.ts b/src/core/destructive-guard.ts new file mode 100644 index 000000000..5a8c2e18d --- /dev/null +++ b/src/core/destructive-guard.ts @@ -0,0 +1,337 @@ +/** + * Destructive operation guard — v0.26.5 + * + * Protects against accidental data loss in gbrain by requiring explicit + * confirmation for operations that cascade-delete pages, chunks, or embeddings. + * + * Three layers: + * 1. Impact preview — always shown before destructive actions + * 2. Confirmation gate — requires --confirm-destructive or interactive "type source name" + * 3. Soft-delete with TTL — sources are tombstoned for 72h before permanent deletion + * + * Design principle: the blast radius should be visible BEFORE you pull the trigger, + * and recoverable AFTER you pull it (within a grace period). + */ + +import type { BrainEngine } from './engine.ts'; + +// ── Types ─────────────────────────────────────────────────── + +export interface DestructiveImpact { + sourceId: string; + sourceName: string; + pageCount: number; + chunkCount: number; + embeddingCount: number; + fileCount: number; + /** Human-readable summary line */ + summary: string; +} + +export interface SoftDeletedSource { + id: string; + name: string; + deletedAt: Date; + expiresAt: Date; + pageCount: number; +} + +// ── Constants ─────────────────────────────────────────────── + +/** Hours before a soft-deleted source is permanently purged. */ +export const SOFT_DELETE_TTL_HOURS = 72; + +/** Threshold: operations affecting this many pages or more require confirmation. */ +export const CONFIRM_THRESHOLD_PAGES = 1; + +// ── Impact Assessment ─────────────────────────────────────── + +/** + * Compute the blast radius of deleting a source. + */ +export async function assessDestructiveImpact( + engine: BrainEngine, + sourceId: string, +): Promise { + // Fetch source metadata + const sources = await engine.executeRaw<{ id: string; name: string }>( + `SELECT id, name FROM sources WHERE id = $1`, + [sourceId], + ); + if (sources.length === 0) return null; + + const src = sources[0]; + + // Count pages + const pageRows = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`, + [sourceId], + ); + const pageCount = pageRows[0]?.n ?? 0; + + // Count chunks + const chunkRows = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM content_chunks cc + JOIN pages p ON cc.page_id = p.id + WHERE p.source_id = $1`, + [sourceId], + ); + const chunkCount = chunkRows[0]?.n ?? 0; + + // Count embeddings (chunks with non-null embedding vectors) + const embedRows = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM content_chunks cc + JOIN pages p ON cc.page_id = p.id + WHERE p.source_id = $1 AND cc.embedding IS NOT NULL`, + [sourceId], + ); + const embeddingCount = embedRows[0]?.n ?? 0; + + // Count files in storage (if any). PGLite has no `files` table — that + // surface is Postgres-only (CLAUDE.md: "No files table" for PGLite). Probe + // the table existence via information_schema so this works on both engines. + let fileCount = 0; + const filesTableRows = await engine.executeRaw<{ exists: boolean }>( + `SELECT EXISTS ( + SELECT 1 FROM information_schema.tables + WHERE table_schema = 'public' AND table_name = 'files' + ) AS exists`, + ); + if (filesTableRows[0]?.exists) { + const fileRows = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM files WHERE source_id = $1`, + [sourceId], + ); + fileCount = fileRows[0]?.n ?? 0; + } + + const parts: string[] = []; + if (pageCount > 0) parts.push(`${pageCount.toLocaleString()} pages`); + if (chunkCount > 0) parts.push(`${chunkCount.toLocaleString()} chunks`); + if (embeddingCount > 0) parts.push(`${embeddingCount.toLocaleString()} embeddings`); + if (fileCount > 0) parts.push(`${fileCount.toLocaleString()} files`); + + const summary = parts.length > 0 + ? `⚠️ This will permanently delete: ${parts.join(', ')}` + : `Source "${sourceId}" has no data (safe to remove).`; + + return { + sourceId, + sourceName: src.name, + pageCount, + chunkCount, + embeddingCount, + fileCount, + summary, + }; +} + +// ── Confirmation Gate ─────────────────────────────────────── + +/** + * Check whether the caller has provided sufficient confirmation for a + * destructive operation. Returns an error message if blocked, or null if OK. + */ +export function checkDestructiveConfirmation( + impact: DestructiveImpact, + opts: { + yes?: boolean; + confirmDestructive?: boolean; + dryRun?: boolean; + }, +): string | null { + // Dry run always passes (no side effects) + if (opts.dryRun) return null; + + // No data = no risk + if (impact.pageCount === 0 && impact.chunkCount === 0 && impact.fileCount === 0) { + return null; + } + + // --confirm-destructive is the explicit "I know what I'm doing" flag + if (opts.confirmDestructive) return null; + + // --yes alone is NOT sufficient for destructive operations with data. + // This is the key behavior change: --yes used to be enough, now you + // need --confirm-destructive when there's actual data at stake. + if (opts.yes && impact.pageCount === 0) return null; + + return ( + `\n${impact.summary}\n\n` + + `To proceed, pass --confirm-destructive (or use soft-delete: gbrain sources archive ${impact.sourceId}).\n` + + `To preview without side effects: --dry-run` + ); +} + +// ── Soft Delete ───────────────────────────────────────────── + +/** + * Soft-delete a source: mark `archived = true` with a 72h TTL. Pages remain + * in DB; the source is hidden from search via `buildVisibilityClause` and + * federation is disabled via the existing `config.federated` JSONB key. After + * TTL expires, the autopilot purge phase or manual `gbrain sources purge` + * permanently removes the row (cascade delete to pages + chunks). + * + * v0.26.5: archive state moved from `config` JSONB keys to real columns + * (`archived`, `archived_at`, `archive_expires_at`). Migration v34 backfills + * pre-v0.26.5 rows. Faster filter, no reserved-key footgun. The `federated` + * key stays in JSONB because federation has its own toggle path. + */ +export async function softDeleteSource( + engine: BrainEngine, + sourceId: string, +): Promise { + // Atomic: only flip rows that are currently active. Returns the metadata + // we need without a follow-up SELECT. RETURNING projects the columns the + // caller cares about; pageCount is a separate count. + const expiresClause = `now() + (${SOFT_DELETE_TTL_HOURS} || ' hours')::interval`; + const rows = await engine.executeRaw<{ id: string; name: string; archived_at: string; archive_expires_at: string }>( + `UPDATE sources + SET archived = true, + archived_at = now(), + archive_expires_at = ${expiresClause}, + config = COALESCE(config, '{}'::jsonb) || '{"federated": false}'::jsonb + WHERE id = $1 AND archived = false + RETURNING id, name, archived_at, archive_expires_at`, + [sourceId], + ); + if (rows.length === 0) return null; + const row = rows[0]; + + const pageRows = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`, + [sourceId], + ); + const pageCount = pageRows[0]?.n ?? 0; + + return { + id: sourceId, + name: row.name, + deletedAt: new Date(row.archived_at), + expiresAt: new Date(row.archive_expires_at), + pageCount, + }; +} + +/** + * Restore a soft-deleted source (un-archive). Returns true iff a row was + * restored. Idempotent-as-false on "already active" or "not found". + * + * v0.26.5: clears the column-based archive state and (by default) flips + * `config.federated = true` so the source re-enters federated search. The + * `--no-federate` operator opt-out keeps federation disabled. + */ +export async function restoreSource( + engine: BrainEngine, + sourceId: string, + refederate: boolean = true, +): Promise { + const federatedPatch = refederate ? '{"federated": true}' : '{"federated": false}'; + const rows = await engine.executeRaw<{ id: string }>( + `UPDATE sources + SET archived = false, + archived_at = NULL, + archive_expires_at = NULL, + config = COALESCE(config, '{}'::jsonb) || $1::jsonb + WHERE id = $2 AND archived = true + RETURNING id`, + [federatedPatch, sourceId], + ); + return rows.length > 0; +} + +/** + * List all soft-deleted (archived) sources. + * + * v0.26.5: filters via the real `archived` column instead of JSONB + * containment. Faster, indexable on demand, no JSONB reserved-key collision + * with future config schemas. + */ +export async function listArchivedSources( + engine: BrainEngine, +): Promise { + const rows = await engine.executeRaw<{ + id: string; + name: string; + archived_at: string; + archive_expires_at: string; + page_count: number; + }>( + `SELECT + s.id, s.name, s.archived_at, s.archive_expires_at, + COALESCE((SELECT COUNT(*)::int FROM pages p WHERE p.source_id = s.id), 0) AS page_count + FROM sources s + WHERE s.archived = true + ORDER BY s.archived_at DESC`, + ); + + return rows.map((row) => ({ + id: row.id, + name: row.name, + deletedAt: new Date(row.archived_at), + expiresAt: new Date(row.archive_expires_at), + pageCount: row.page_count, + })); +} + +/** + * Permanently purge sources whose 72h TTL has expired. Cascades to pages + * (and content_chunks via existing FKs). Returns the ids of purged sources. + * + * v0.26.5: moved from JSONB-driven iteration to a single set-based DELETE + * with `archived = true AND archive_expires_at <= now()`. Server-side + * filter; one round-trip; cascade-friendly. + */ +export async function purgeExpiredSources( + engine: BrainEngine, +): Promise { + const rows = await engine.executeRaw<{ id: string }>( + `DELETE FROM sources + WHERE archived = true + AND archive_expires_at IS NOT NULL + AND archive_expires_at <= now() + RETURNING id`, + ); + return rows.map((r) => r.id); +} + +// ── Display Helpers ───────────────────────────────────────── + +/** + * Format an impact assessment for terminal display. + */ +export function formatImpact(impact: DestructiveImpact): string { + const lines: string[] = [ + ``, + `╔══════════════════════════════════════════════════════════╗`, + `║ DESTRUCTIVE OPERATION — Impact Preview ║`, + `╠══════════════════════════════════════════════════════════╣`, + `║ Source: ${impact.sourceName.padEnd(42)}║`, + `║ Source ID: ${impact.sourceId.padEnd(42)}║`, + `║ ║`, + `║ Pages: ${String(impact.pageCount.toLocaleString()).padEnd(42)}║`, + `║ Chunks: ${String(impact.chunkCount.toLocaleString()).padEnd(42)}║`, + `║ Embeddings: ${String(impact.embeddingCount.toLocaleString()).padEnd(42)}║`, + `║ Files: ${String(impact.fileCount.toLocaleString()).padEnd(42)}║`, + `╠══════════════════════════════════════════════════════════╣`, + `║ ${impact.summary.padEnd(56)}║`, + `╚══════════════════════════════════════════════════════════╝`, + ``, + ]; + return lines.join('\n'); +} + +export function formatSoftDelete(sd: SoftDeletedSource): string { + const hours = Math.round((sd.expiresAt.getTime() - Date.now()) / (1000 * 60 * 60)); + return [ + ``, + `Source "${sd.id}" archived (soft-deleted).`, + ` ${sd.pageCount.toLocaleString()} pages preserved for ${SOFT_DELETE_TTL_HOURS}h.`, + ` Expires: ${sd.expiresAt.toISOString()} (~${hours}h from now)`, + ` Removed from search. Data intact.`, + ``, + ` Restore: gbrain sources restore ${sd.id}`, + ` Purge now: gbrain sources purge ${sd.id} --confirm-destructive`, + ``, + ].join('\n'); +} diff --git a/src/core/engine.ts b/src/core/engine.ts index 29de60529..05bcc8ead 100644 --- a/src/core/engine.ts +++ b/src/core/engine.ts @@ -1,5 +1,5 @@ import type { - Page, PageInput, PageFilters, + Page, PageInput, PageFilters, GetPageOpts, Chunk, ChunkInput, StaleChunkRow, SearchResult, SearchOpts, Link, GraphNode, GraphPath, @@ -128,9 +128,48 @@ export interface BrainEngine { withReservedConnection(fn: (conn: ReservedConnection) => Promise): Promise; // Pages CRUD - getPage(slug: string): Promise; + /** + * Fetch a page by slug. + * v0.26.5: by default soft-deleted rows return null (matches the search + * filter contract). Pass `opts.includeDeleted: true` to surface them with + * `deleted_at` populated — used by `gbrain pages purge-deleted` listing, + * by `restore_page` flow, and by operator diagnostics. + */ + getPage(slug: string, opts?: GetPageOpts): Promise; putPage(slug: string, page: PageInput): Promise; + /** + * Hard-delete a page row. Cascades to content_chunks, page_links, + * chunk_relations via existing FK ON DELETE CASCADE. + * + * v0.26.5: this is no longer the public-facing `delete_page` op handler — + * the op now soft-deletes via `softDeletePage` instead. `deletePage` stays + * as the underlying primitive used by `purgeDeletedPages` and by callers + * that explicitly want hard-delete semantics (e.g. test setup teardown). + */ deletePage(slug: string): Promise; + /** + * v0.26.5 — set `deleted_at = now()` on a page. Returns the slug if a row + * was soft-deleted, null if no row matched (already soft-deleted OR not found). + * Idempotent-as-null. The page stays in the DB and cascade rows (chunks, + * links) stay intact; the autopilot purge phase hard-deletes after 72h. + */ + softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null>; + /** + * v0.26.5 — clear `deleted_at` on a soft-deleted page. Returns true iff a + * row was restored. False if the slug is unknown OR the page is not + * currently soft-deleted (idempotent-as-false). + */ + restorePage(slug: string, opts?: { sourceId?: string }): Promise; + /** + * v0.26.5 — hard-delete pages whose `deleted_at` is older than the cutoff. + * Called by the autopilot purge phase and by the `gbrain pages purge-deleted` + * CLI escape hatch. Cascades through existing FKs. + */ + purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }>; + /** + * v0.26.5: by default `listPages` excludes soft-deleted rows. Set + * `filters.includeDeleted: true` to surface them. + */ listPages(filters?: PageFilters): Promise; resolveSlugs(partial: string): Promise; /** diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 2069ded0b..1b36b518a 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -1307,6 +1307,93 @@ export const MIGRATIONS: Migration[] = [ ON mcp_request_log(agent_name, created_at DESC); `, }, + { + version: 34, + name: 'destructive_guard_columns', + // v0.26.5 — soft-delete + recovery window for sources AND pages. + // Renumbered v33→v34 on master merge: master's v33 (admin_dashboard_columns_v0_26_3) + // landed first in PR #586. v34 follows it. + // + // pages.deleted_at: `delete_page` op now sets deleted_at = now() instead of + // hard-deleting. The autopilot purge phase hard-deletes rows where + // deleted_at < now() - 72h. Search and `get_page` filter + // `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in. + // + // sources.archived/archived_at/archive_expires_at: promoted from JSONB keys + // to real columns. v0.26.0 + the cherry-picked PR #595 wrote these inside + // `sources.config` JSONB. Real columns are faster to filter, avoid the + // reserved-key footgun, and let the search visibility filter compile to a + // column lookup. The 72h TTL is preserved by reading + // `archive_expires_at = archived_at + INTERVAL '72 hours'`. + // + // Backfill: any row that previously stored `{"archived":true,"archived_at":"...","archive_expires_at":"..."}` + // in config gets migrated to the new columns, then the keys are stripped + // from JSONB so the JSONB shape stays canonical going forward. + // + // Engine-aware partial index: Postgres uses CREATE INDEX CONCURRENTLY (no + // write-blocking lock); PGLite uses plain CREATE INDEX. Mirrors v14 + // (pages_updated_at_index) handler shape. + sql: '', + handler: async (engine) => { + // 1. Add columns. ALTER TABLE ADD COLUMN IF NOT EXISTS is idempotent on + // both engines. + await engine.runMigration(34, ` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ; + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived BOOLEAN NOT NULL DEFAULT false; + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archived_at TIMESTAMPTZ; + ALTER TABLE sources ADD COLUMN IF NOT EXISTS archive_expires_at TIMESTAMPTZ; + `); + + // 2. Backfill from JSONB shape used by pre-v0.26.5 cherry-picks of PR #595. + // Idempotent: subsequent re-runs find zero matching rows. + await engine.runMigration(34, ` + UPDATE sources + SET archived = true, + archived_at = COALESCE((config->>'archived_at')::timestamptz, now()), + archive_expires_at = COALESCE( + (config->>'archive_expires_at')::timestamptz, + COALESCE((config->>'archived_at')::timestamptz, now()) + INTERVAL '72 hours' + ) + WHERE config ? 'archived' + AND (config->>'archived')::boolean = true + AND archived = false; + `); + await engine.runMigration(34, ` + UPDATE sources + SET config = config - 'archived' - 'archived_at' - 'archive_expires_at' + WHERE config ?| ARRAY['archived', 'archived_at', 'archive_expires_at']; + `); + + // 3. Partial index for the autopilot purge sweep. Postgres CONCURRENTLY + // avoids the SHARE lock on `pages`; PGLite has no concurrent writers. + if (engine.kind === 'postgres') { + // Pre-drop any invalid index from a prior CONCURRENTLY failure (matches v14 pattern). + await engine.runMigration(34, ` + DO $$ BEGIN + IF EXISTS ( + SELECT 1 FROM pg_index i + JOIN pg_class c ON c.oid = i.indexrelid + WHERE c.relname = 'pages_deleted_at_purge_idx' AND NOT i.indisvalid + ) THEN + EXECUTE 'DROP INDEX CONCURRENTLY IF EXISTS pages_deleted_at_purge_idx'; + END IF; + END $$; + `); + await engine.runMigration(34, ` + CREATE INDEX CONCURRENTLY IF NOT EXISTS pages_deleted_at_purge_idx + ON pages (deleted_at) WHERE deleted_at IS NOT NULL; + `); + } else { + await engine.runMigration(34, ` + CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx + ON pages (deleted_at) WHERE deleted_at IS NOT NULL; + `); + } + }, + // CONCURRENTLY on Postgres requires no surrounding transaction. PGLite ignores + // this flag, so the index DDL runs in whatever wrapper applies. + transaction: false, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/operations.ts b/src/core/operations.ts index 0b7be7a60..4d93d38f2 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -293,22 +293,24 @@ export interface Operation { const get_page: Operation = { name: 'get_page', - description: 'Read a page by slug (supports optional fuzzy matching)', + description: 'Read a page by slug (supports optional fuzzy matching). Soft-deleted pages are hidden by default; pass include_deleted: true to surface them with deleted_at populated (see v0.26.5 recovery window).', params: { slug: { type: 'string', required: true, description: 'Page slug' }, fuzzy: { type: 'boolean', description: 'Enable fuzzy slug resolution (default: false)' }, + include_deleted: { type: 'boolean', description: 'v0.26.5: surface soft-deleted pages with deleted_at populated (default: false). Used by restore workflows.' }, }, handler: async (ctx, p) => { const slug = p.slug as string; const fuzzy = (p.fuzzy as boolean) || false; + const includeDeleted = (p.include_deleted as boolean) === true; - let page = await ctx.engine.getPage(slug); + let page = await ctx.engine.getPage(slug, { includeDeleted }); let resolved_slug: string | undefined; if (!page && fuzzy) { const candidates = await ctx.engine.resolveSlugs(slug); if (candidates.length === 1) { - page = await ctx.engine.getPage(candidates[0]); + page = await ctx.engine.getPage(candidates[0], { includeDeleted }); resolved_slug = candidates[0]; } else if (candidates.length > 1) { return { error: 'ambiguous_slug', candidates }; @@ -316,7 +318,7 @@ const get_page: Operation = { } if (!page) { - throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug or use fuzzy: true'); + throw new OperationError('page_not_found', `Page not found: ${slug}`, includeDeleted ? 'Check the slug or use fuzzy: true' : 'Page may be soft-deleted; pass include_deleted: true to verify'); } const tags = await ctx.engine.getTags(page.slug); @@ -625,39 +627,99 @@ async function runAutoLink( const delete_page: Operation = { name: 'delete_page', - description: 'Delete a page', + description: 'Soft-delete a page. The row is hidden from search and from get_page/list_pages, but is recoverable via restore_page within 72h. The autopilot purge phase hard-deletes after the recovery window. Pass include_deleted: true to get_page to verify the soft-delete landed.', params: { slug: { type: 'string', required: true }, }, mutating: true, scope: 'write', handler: async (ctx, p) => { - if (ctx.dryRun) return { dry_run: true, action: 'delete_page', slug: p.slug }; - await ctx.engine.deletePage(p.slug as string); - return { status: 'deleted' }; + const slug = p.slug as string; + if (ctx.dryRun) return { dry_run: true, action: 'soft_delete_page', slug }; + // v0.26.5: rewired from hard-delete to soft-delete. The hard-delete primitive + // (engine.deletePage) is now reserved for purgeDeletedPages and explicit + // tests. softDeletePage returns null when the slug is unknown OR already + // soft-deleted (idempotent-as-null) — preserve that as a clean no-op shape. + const result = await ctx.engine.softDeletePage(slug); + if (result === null) { + // Distinguish "not found" from "already soft-deleted" so the agent gets a + // clear signal. Probe once with include_deleted to disambiguate. + const existing = await ctx.engine.getPage(slug, { includeDeleted: true }); + if (!existing) { + throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug.'); + } + return { status: 'already_soft_deleted', slug, deleted_at: existing.deleted_at }; + } + return { status: 'soft_deleted', slug, recoverable_until: 'now + 72h via restore_page' }; }, cliHints: { name: 'delete', positional: ['slug'] }, }; +const restore_page: Operation = { + name: 'restore_page', + description: 'v0.26.5 — restore a soft-deleted page (clear deleted_at). Returns success only if the page was actually soft-deleted. After this op, the page reappears in search and in get_page/list_pages without the include_deleted flag.', + params: { + slug: { type: 'string', required: true }, + }, + mutating: true, + scope: 'write', + handler: async (ctx, p) => { + const slug = p.slug as string; + if (ctx.dryRun) return { dry_run: true, action: 'restore_page', slug }; + const ok = await ctx.engine.restorePage(slug); + if (!ok) { + // Distinguish "not found" from "already active" (idempotent-as-false). + const existing = await ctx.engine.getPage(slug, { includeDeleted: true }); + if (!existing) { + throw new OperationError('page_not_found', `Page not found: ${slug}`, 'Check the slug.'); + } + return { status: 'already_active', slug }; + } + return { status: 'restored', slug }; + }, + cliHints: { name: 'restore', positional: ['slug'] }, +}; + +const purge_deleted_pages: Operation = { + name: 'purge_deleted_pages', + description: 'v0.26.5 — admin-only. Hard-deletes pages whose deleted_at is older than older_than_hours (default 72). Cascades through content_chunks, page_links, chunk_relations. Local CLI only (not exposed over HTTP MCP). Manual escape hatch alongside the autopilot purge phase.', + params: { + older_than_hours: { type: 'number', description: 'Age cutoff in hours. Default 72.' }, + }, + mutating: true, + scope: 'admin', + localOnly: true, + handler: async (ctx, p) => { + const olderThanHours = (p.older_than_hours as number | undefined) ?? 72; + if (ctx.dryRun) return { dry_run: true, action: 'purge_deleted_pages', older_than_hours: olderThanHours }; + const result = await ctx.engine.purgeDeletedPages(olderThanHours); + return { status: 'purged', count: result.count, slugs: result.slugs }; + }, + cliHints: { name: 'purge-deleted' }, +}; + const list_pages: Operation = { name: 'list_pages', - description: 'List pages with optional filters', + description: 'List pages with optional filters. Soft-deleted pages are hidden by default; pass include_deleted: true to surface them with deleted_at populated.', params: { type: { type: 'string', description: 'Filter by page type' }, tag: { type: 'string', description: 'Filter by tag' }, limit: { type: 'number', description: 'Max results (default 50)' }, + include_deleted: { type: 'boolean', description: 'v0.26.5: include soft-deleted pages (default: false). Used by restore workflows and operator diagnostics.' }, }, handler: async (ctx, p) => { const pages = await ctx.engine.listPages({ type: p.type as any, tag: p.tag as string, limit: clampSearchLimit(p.limit as number | undefined, 50, 100), + includeDeleted: (p.include_deleted as boolean) === true, }); return pages.map(pg => ({ slug: pg.slug, type: pg.type, title: pg.title, updated_at: pg.updated_at, + ...(pg.deleted_at ? { deleted_at: pg.deleted_at } : {}), })); }, scope: 'read', @@ -1529,6 +1591,8 @@ const find_orphans: Operation = { export const operations: Operation[] = [ // Page CRUD get_page, put_page, delete_page, list_pages, + // v0.26.5 destructive-guard ops (page-level soft-delete + recovery + admin purge) + restore_page, purge_deleted_pages, // Search search, query, // Tags diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index a57fd407b..16ec300cc 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -23,7 +23,7 @@ import type { } from './types.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult } from './utils.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; -import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts'; +import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts'; type PGLiteDB = PGlite; @@ -212,6 +212,7 @@ export class PGLiteEngine implements BrainEngine { * - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13 * - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19 * - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19 + * - `pages.deleted_at` column (indexed by `pages_deleted_at_purge_idx`) — v0.26.5 * * **Maintenance contract:** when a future migration adds a column-with-index * or new-table-with-FK referenced by PGLITE_SCHEMA_SQL, extend this method @@ -226,6 +227,8 @@ export class PGLiteEngine implements BrainEngine { WHERE table_schema='public' AND table_name='pages') AS pages_exists, EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema='public' AND table_name='pages' AND column_name='source_id') AS source_id_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema='public' AND table_name='pages' AND column_name='deleted_at') AS deleted_at_exists, EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema='public' AND table_name='links') AS links_exists, EXISTS (SELECT 1 FROM information_schema.columns @@ -242,6 +245,7 @@ export class PGLiteEngine implements BrainEngine { const probe = rows[0] as { pages_exists: boolean; source_id_exists: boolean; + deleted_at_exists: boolean; links_exists: boolean; link_source_exists: boolean; origin_page_id_exists: boolean; @@ -255,9 +259,10 @@ export class PGLiteEngine implements BrainEngine { && (!probe.link_source_exists || !probe.origin_page_id_exists); const needsChunksBootstrap = probe.chunks_exists && (!probe.symbol_name_exists || !probe.language_exists); + const needsPagesDeletedAt = probe.pages_exists && !probe.deleted_at_exists; // Fresh installs (no tables yet) and modern brains both no-op. - if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return; + if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt) return; console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap'); @@ -305,6 +310,16 @@ export class PGLiteEngine implements BrainEngine { ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT; `); } + + if (needsPagesDeletedAt) { + // v34 (destructive_guard_columns) adds the column + sources columns + + // partial purge index. Bootstrap only adds enough for PGLITE_SCHEMA_SQL's + // `CREATE INDEX pages_deleted_at_purge_idx ... WHERE deleted_at IS NOT NULL` + // not to crash. v34 runs later via runMigrations and is idempotent. + await this.db.exec(` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ; + `); + } } async withReservedConnection(fn: (conn: ReservedConnection) => Promise): Promise { @@ -329,11 +344,23 @@ export class PGLiteEngine implements BrainEngine { } // Pages CRUD - async getPage(slug: string): Promise { + async getPage(slug: string, opts?: { sourceId?: string; includeDeleted?: boolean }): Promise { + // v0.26.5: hide soft-deleted by default; opt-in via opts.includeDeleted. + const includeDeleted = opts?.includeDeleted === true; + const sourceId = opts?.sourceId; + const where: string[] = ['slug = $1']; + const params: unknown[] = [slug]; + if (sourceId) { + params.push(sourceId); + where.push(`source_id = $${params.length}`); + } + if (!includeDeleted) { + where.push('deleted_at IS NULL'); + } const { rows } = await this.db.query( - `SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at - FROM pages WHERE slug = $1`, - [slug] + `SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at + FROM pages WHERE ${where.join(' AND ')} LIMIT 1`, + params ); if (rows.length === 0) return null; return rowToPage(rows[0] as Record); @@ -372,6 +399,54 @@ export class PGLiteEngine implements BrainEngine { await this.db.query('DELETE FROM pages WHERE slug = $1', [slug]); } + async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> { + // Idempotent-as-null: only flip rows currently active. Source filter is + // optional; without it the first matching row across sources gets soft-deleted. + const sourceId = opts?.sourceId; + const where: string[] = ['slug = $1', 'deleted_at IS NULL']; + const params: unknown[] = [slug]; + if (sourceId) { + params.push(sourceId); + where.push(`source_id = $${params.length}`); + } + const { rows } = await this.db.query( + `UPDATE pages SET deleted_at = now() WHERE ${where.join(' AND ')} RETURNING slug`, + params + ); + if (rows.length === 0) return null; + return { slug: (rows[0] as { slug: string }).slug }; + } + + async restorePage(slug: string, opts?: { sourceId?: string }): Promise { + const sourceId = opts?.sourceId; + const where: string[] = ['slug = $1', 'deleted_at IS NOT NULL']; + const params: unknown[] = [slug]; + if (sourceId) { + params.push(sourceId); + where.push(`source_id = $${params.length}`); + } + const { rows } = await this.db.query( + `UPDATE pages SET deleted_at = NULL WHERE ${where.join(' AND ')} RETURNING slug`, + params + ); + return rows.length > 0; + } + + async purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }> { + // Clamp to non-negative integer; cascade through FKs (content_chunks, + // page_links, chunk_relations) on DELETE. + const hours = Math.max(0, Math.floor(olderThanHours)); + const { rows } = await this.db.query( + `DELETE FROM pages + WHERE deleted_at IS NOT NULL + AND deleted_at < now() - ($1 || ' hours')::interval + RETURNING slug`, + [hours] + ); + const slugs = (rows as { slug: string }[]).map((r) => r.slug); + return { slugs, count: slugs.length }; + } + async listPages(filters?: PageFilters): Promise { const limit = filters?.limit || 100; const offset = filters?.offset || 0; @@ -399,6 +474,10 @@ export class PGLiteEngine implements BrainEngine { params.push(escaped); where.push(`p.slug LIKE $${params.length} ESCAPE '\\'`); } + // v0.26.5: hide soft-deleted by default; opt in via filters.includeDeleted. + if (filters?.includeDeleted !== true) { + where.push('p.deleted_at IS NULL'); + } const whereSql = where.length > 0 ? `WHERE ${where.join(' AND ')}` : ''; params.push(limit, offset); @@ -479,6 +558,9 @@ export class PGLiteEngine implements BrainEngine { extraFilter += ` AND cc.symbol_type = $${params.length}`; } + // v0.26.5: visibility filter (soft-deleted + archived-source). + const visibilityClause = buildVisibilityClause('p', 's'); + const { rows } = await this.db.query( `WITH ranked AS ( SELECT @@ -490,7 +572,8 @@ export class PGLiteEngine implements BrainEngine { ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} + JOIN sources s ON s.id = p.source_id + WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} ORDER BY score DESC LIMIT $2 ), @@ -547,6 +630,9 @@ export class PGLiteEngine implements BrainEngine { extraFilter += ` AND cc.symbol_type = $${params.length}`; } + // v0.26.5: visibility filter for the chunk-grain anchor primitive. + const visibilityClause = buildVisibilityClause('p', 's'); + const { rows } = await this.db.query( `SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, @@ -557,7 +643,8 @@ export class PGLiteEngine implements BrainEngine { ) THEN true ELSE false END AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id - WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} + JOIN sources s ON s.id = p.source_id + WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} ORDER BY score DESC LIMIT $2 OFFSET $3`, params @@ -603,6 +690,10 @@ export class PGLiteEngine implements BrainEngine { extraFilter += ` AND cc.symbol_type = $${params.length}`; } + // v0.26.5: visibility filter applied in the inner CTE so HNSW sees the + // same candidate count it always did. See postgres-engine.ts for rationale. + const visibilityClause = buildVisibilityClause('p', 's'); + const { rows } = await this.db.query( `WITH hnsw_candidates AS ( SELECT @@ -611,7 +702,8 @@ export class PGLiteEngine implements BrainEngine { 1 - (cc.embedding <=> $1::vector) AS raw_score FROM content_chunks cc JOIN pages p ON p.id = cc.page_id - WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause} + JOIN sources s ON s.id = p.source_id + WHERE cc.embedding IS NOT NULL ${detailFilter}${extraFilter} ${hardExcludeClause} ${visibilityClause} ORDER BY cc.embedding <=> $1::vector LIMIT $2 ) @@ -1327,7 +1419,8 @@ export class PGLiteEngine implements BrainEngine { async getStats(): Promise { const { rows: [stats] } = await this.db.query(` SELECT - (SELECT count(*) FROM pages) as page_count, + -- v0.26.5: exclude soft-deleted from page_count (mirrors postgres-engine). + (SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count, (SELECT count(*) FROM content_chunks) as chunk_count, (SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count, (SELECT count(*) FROM links) as link_count, diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index b8b64f83b..d2bd09b46 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -32,6 +32,10 @@ CREATE TABLE IF NOT EXISTS sources ( last_commit TEXT, last_sync_at TIMESTAMPTZ, config JSONB NOT NULL DEFAULT '{}'::jsonb, + -- v0.26.5: soft-delete + recovery window (mirrors src/schema.sql). + archived BOOLEAN NOT NULL DEFAULT false, + archived_at TIMESTAMPTZ, + archive_expires_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); @@ -60,6 +64,8 @@ CREATE TABLE IF NOT EXISTS pages ( content_hash TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + -- v0.26.5: soft-delete + recovery window (mirrors src/schema.sql). + deleted_at TIMESTAMPTZ, CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug) ); @@ -67,6 +73,9 @@ CREATE INDEX IF NOT EXISTS idx_pages_type ON pages(type); CREATE INDEX IF NOT EXISTS idx_pages_frontmatter ON pages USING GIN(frontmatter); CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops); CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id); +-- v0.26.5: partial index supports the autopilot purge sweep (mirrors src/schema.sql). +CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx + ON pages (deleted_at) WHERE deleted_at IS NOT NULL; -- ============================================================ -- content_chunks: chunked content with embeddings diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index 98e4cf3a6..8f84ffa5c 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -22,7 +22,7 @@ import { GBrainError } from './types.ts'; import * as db from './db.ts'; import { validateSlug, contentHash, rowToPage, rowToChunk, rowToSearchResult, parseEmbedding, tryParseEmbedding } from './utils.ts'; import { resolveBoostMap, resolveHardExcludes } from './search/source-boost.ts'; -import { buildSourceFactorCase, buildHardExcludeClause } from './search/sql-ranking.ts'; +import { buildSourceFactorCase, buildHardExcludeClause, buildVisibilityClause } from './search/sql-ranking.ts'; // CONNECTION_ERROR_PATTERNS / isConnectionError were used by the per-call // executeRaw retry that #406 originally shipped. Eng-review D3 dropped that @@ -152,6 +152,7 @@ export class PostgresEngine implements BrainEngine { * - `links.origin_page_id` column (indexed by `idx_links_origin`) — v0.13 * - `content_chunks.symbol_name` column (indexed by `idx_chunks_symbol_name`) — v0.19 * - `content_chunks.language` column (indexed by `idx_chunks_language`) — v0.19 + * - `pages.deleted_at` column (indexed by `pages_deleted_at_purge_idx`) — v0.26.5 * * Keep this in sync with the PGLite version; covered by * `test/schema-bootstrap-coverage.test.ts` (PGLite side) and @@ -166,6 +167,7 @@ export class PostgresEngine implements BrainEngine { const probeRows = await conn<{ pages_exists: boolean; source_id_exists: boolean; + deleted_at_exists: boolean; links_exists: boolean; link_source_exists: boolean; origin_page_id_exists: boolean; @@ -178,6 +180,8 @@ export class PostgresEngine implements BrainEngine { WHERE table_schema = current_schema() AND table_name = 'pages') AS pages_exists, EXISTS (SELECT 1 FROM information_schema.columns WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'source_id') AS source_id_exists, + EXISTS (SELECT 1 FROM information_schema.columns + WHERE table_schema = current_schema() AND table_name = 'pages' AND column_name = 'deleted_at') AS deleted_at_exists, EXISTS (SELECT 1 FROM information_schema.tables WHERE table_schema = current_schema() AND table_name = 'links') AS links_exists, EXISTS (SELECT 1 FROM information_schema.columns @@ -198,8 +202,11 @@ export class PostgresEngine implements BrainEngine { && (!probe.link_source_exists || !probe.origin_page_id_exists); const needsChunksBootstrap = probe.chunks_exists && (!probe.symbol_name_exists || !probe.language_exists); + // v0.26.5: pages_deleted_at_purge_idx in SCHEMA_SQL crashes if the column + // doesn't exist yet. Migration v34 also adds it, but bootstrap runs first. + const needsPagesDeletedAt = probe.pages_exists && !probe.deleted_at_exists; - if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap) return; + if (!needsPagesBootstrap && !needsLinksBootstrap && !needsChunksBootstrap && !needsPagesDeletedAt) return; console.log(' Pre-v0.21 brain detected, applying forward-reference bootstrap'); @@ -246,6 +253,16 @@ export class PostgresEngine implements BrainEngine { ALTER TABLE content_chunks ADD COLUMN IF NOT EXISTS symbol_name TEXT; `); } + + if (needsPagesDeletedAt) { + // v34 (destructive_guard_columns) adds the column + sources columns + + // partial purge index. Bootstrap only adds enough for SCHEMA_SQL's + // `CREATE INDEX pages_deleted_at_purge_idx ... WHERE deleted_at IS NOT NULL` + // not to crash. v34 runs later via runMigrations and is idempotent. + await conn.unsafe(` + ALTER TABLE pages ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ; + `); + } } async transaction(fn: (engine: BrainEngine) => Promise): Promise { @@ -278,11 +295,19 @@ export class PostgresEngine implements BrainEngine { } // Pages CRUD - async getPage(slug: string): Promise { + async getPage(slug: string, opts?: { sourceId?: string; includeDeleted?: boolean }): Promise { const sql = this.sql; + const includeDeleted = opts?.includeDeleted === true; + const sourceId = opts?.sourceId; + // v0.26.5: default hides soft-deleted rows. Compose with optional sourceId + // filter via fragment chaining (postgres.js supports sql`` composition). + const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``; + const deletedCondition = includeDeleted ? sql`` : sql`AND deleted_at IS NULL`; const rows = await sql` - SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at - FROM pages WHERE slug = ${slug} + SELECT id, slug, type, title, compiled_truth, timeline, frontmatter, content_hash, created_at, updated_at, deleted_at + FROM pages + WHERE slug = ${slug} ${sourceCondition} ${deletedCondition} + LIMIT 1 `; if (rows.length === 0) return null; return rowToPage(rows[0]); @@ -321,6 +346,48 @@ export class PostgresEngine implements BrainEngine { await sql`DELETE FROM pages WHERE slug = ${slug}`; } + async softDeletePage(slug: string, opts?: { sourceId?: string }): Promise<{ slug: string } | null> { + const sql = this.sql; + const sourceId = opts?.sourceId; + // Idempotent-as-null contract: only flip rows that are currently active. + // RETURNING projects the slug so we can tell hit-vs-miss without a probe. + const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``; + const rows = await sql` + UPDATE pages SET deleted_at = now() + WHERE slug = ${slug} AND deleted_at IS NULL ${sourceCondition} + RETURNING slug + `; + if (rows.length === 0) return null; + return { slug: rows[0].slug as string }; + } + + async restorePage(slug: string, opts?: { sourceId?: string }): Promise { + const sql = this.sql; + const sourceId = opts?.sourceId; + const sourceCondition = sourceId ? sql`AND source_id = ${sourceId}` : sql``; + const rows = await sql` + UPDATE pages SET deleted_at = NULL + WHERE slug = ${slug} AND deleted_at IS NOT NULL ${sourceCondition} + RETURNING slug + `; + return rows.length > 0; + } + + async purgeDeletedPages(olderThanHours: number): Promise<{ slugs: string[]; count: number }> { + const sql = this.sql; + // Clamp to non-negative integer; runaway purge protection. The DELETE + // cascades through content_chunks, page_links, chunk_relations via FKs. + const hours = Math.max(0, Math.floor(olderThanHours)); + const rows = await sql` + DELETE FROM pages + WHERE deleted_at IS NOT NULL + AND deleted_at < now() - (${hours} || ' hours')::interval + RETURNING slug + `; + const slugs = rows.map((r) => r.slug as string); + return { slugs, count: slugs.length }; + } + async listPages(filters?: PageFilters): Promise { const sql = this.sql; const limit = filters?.limit || 100; @@ -342,11 +409,15 @@ export class PostgresEngine implements BrainEngine { const slugCondition = slugPrefix ? sql`AND p.slug LIKE ${slugPrefix.replace(/[\\%_]/g, (c) => '\\' + c) + '%'} ESCAPE '\\'` : sql``; + // v0.26.5: hide soft-deleted by default; opt in via filters.includeDeleted. + const deletedCondition = filters?.includeDeleted === true + ? sql`` + : sql`AND p.deleted_at IS NULL`; const rows = await sql` SELECT p.* FROM pages p ${tagJoin} - WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} + WHERE 1=1 ${typeCondition} ${tagCondition} ${updatedCondition} ${slugCondition} ${deletedCondition} ORDER BY p.updated_at DESC LIMIT ${limit} OFFSET ${offset} `; @@ -442,6 +513,12 @@ export class PostgresEngine implements BrainEngine { params.push(offset); const offsetParam = `$${params.length}`; + // v0.26.5: visibility filter hides soft-deleted pages and pages from + // archived sources. Joined `sources s` lets the predicate compile to a + // column lookup. NOT bypassed by detail=high — soft-delete is a contract, + // not a temporal preference. + const visibilityClause = buildVisibilityClause('p', 's'); + const rawQuery = ` WITH ranked_chunks AS ( SELECT @@ -450,6 +527,7 @@ export class PostgresEngine implements BrainEngine { ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score FROM content_chunks cc JOIN pages p ON p.id = cc.page_id + JOIN sources s ON s.id = p.source_id WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${typeClause} ${excludeSlugsClause} @@ -457,6 +535,7 @@ export class PostgresEngine implements BrainEngine { ${languageClause} ${symbolKindClause} ${hardExcludeClause} + ${visibilityClause} ORDER BY score DESC LIMIT ${innerLimitParam} ), @@ -541,6 +620,9 @@ export class PostgresEngine implements BrainEngine { params.push(offset); const offsetParam = `$${params.length}`; + // v0.26.5: visibility filter for searchKeywordChunks (anchor primitive). + const visibilityClause = buildVisibilityClause('p', 's'); + const rawQuery = ` SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, @@ -549,6 +631,7 @@ export class PostgresEngine implements BrainEngine { false AS stale FROM content_chunks cc JOIN pages p ON p.id = cc.page_id + JOIN sources s ON s.id = p.source_id WHERE cc.search_vector @@ websearch_to_tsquery('english', $1) ${typeClause} ${excludeSlugsClause} @@ -556,6 +639,7 @@ export class PostgresEngine implements BrainEngine { ${languageClause} ${symbolKindClause} ${hardExcludeClause} + ${visibilityClause} ORDER BY score DESC LIMIT ${limitParam} OFFSET ${offsetParam} @@ -626,6 +710,12 @@ export class PostgresEngine implements BrainEngine { params.push(offset); const offsetParam = `$${params.length}`; + // v0.26.5: visibility filter applied in the inner CTE so the HNSW index + // sees the same row count it always did. Pulling the predicate to the + // outer SELECT would force the HNSW scan to over-fetch and post-filter, + // wasting candidate slots on hidden rows. + const visibilityClause = buildVisibilityClause('p', 's'); + const rawQuery = ` WITH hnsw_candidates AS ( SELECT @@ -634,6 +724,7 @@ export class PostgresEngine implements BrainEngine { 1 - (cc.embedding <=> $1::vector) AS raw_score FROM content_chunks cc JOIN pages p ON p.id = cc.page_id + JOIN sources s ON s.id = p.source_id WHERE cc.embedding IS NOT NULL ${detailLow ? `AND cc.chunk_source = 'compiled_truth'` : ''} ${typeClause} @@ -641,6 +732,7 @@ export class PostgresEngine implements BrainEngine { ${languageClause} ${symbolKindClause} ${hardExcludeClause} + ${visibilityClause} ORDER BY cc.embedding <=> $1::vector LIMIT ${innerLimitParam} ) @@ -1379,7 +1471,11 @@ export class PostgresEngine implements BrainEngine { const sql = this.sql; const [stats] = await sql` SELECT - (SELECT count(*) FROM pages) as page_count, + -- v0.26.5: exclude soft-deleted from page_count. Same posture as the + -- search filter and getPage default — soft-deleted is hidden everywhere + -- the user looks. Chunks/links stay raw because they still occupy + -- storage until the autopilot purge phase runs. + (SELECT count(*) FROM pages WHERE deleted_at IS NULL) as page_count, (SELECT count(*) FROM content_chunks) as chunk_count, (SELECT count(*) FROM content_chunks WHERE embedded_at IS NOT NULL) as embedded_count, (SELECT count(*) FROM links) as link_count, diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index 0ca8a3acf..d6242be35 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -39,6 +39,14 @@ CREATE TABLE IF NOT EXISTS sources ( -- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps -- actually trigger re-chunking on upgrade. chunker_version TEXT, + -- v0.26.5: soft-delete + recovery window. \`archive\` flips archived=true and + -- sets archive_expires_at = now() + 72h. The autopilot purge phase + -- hard-deletes rows where archive_expires_at <= now(). Promoted from a + -- JSONB key to real columns to avoid reserved-key footguns and to make the + -- search visibility filter (\`NOT s.archived\`) a column lookup. + archived BOOLEAN NOT NULL DEFAULT false, + archived_at TIMESTAMPTZ, + archive_expires_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); @@ -75,6 +83,11 @@ CREATE TABLE IF NOT EXISTS pages ( content_hash TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + -- v0.26.5: soft-delete + recovery window. \`delete_page\` sets deleted_at = now() + -- instead of issuing DELETE. The autopilot purge phase hard-deletes pages + -- where deleted_at < now() - 72h. Search and \`get_page\` filter + -- \`WHERE deleted_at IS NULL\` by default; \`include_deleted: true\` opts in. + deleted_at TIMESTAMPTZ, CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug) ); @@ -85,6 +98,13 @@ CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops) CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC); -- v0.18.0: source-scoped scans (per /plan-eng-review Section 4). CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id); +-- v0.26.5: partial index supports the autopilot purge sweep +-- (\`WHERE deleted_at IS NOT NULL AND deleted_at < now() - INTERVAL '72 hours'\`). +-- Search filters (\`WHERE deleted_at IS NULL\`) do not benefit from this index +-- (predicate doesn't match) and don't need their own — soft-deleted cardinality +-- stays low. Don't add a regular \`(deleted_at)\` index without measuring. +CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx + ON pages (deleted_at) WHERE deleted_at IS NOT NULL; -- ============================================================ -- content_chunks: chunked content with embeddings diff --git a/src/core/search/sql-ranking.ts b/src/core/search/sql-ranking.ts index 129b46011..0feca067b 100644 --- a/src/core/search/sql-ranking.ts +++ b/src/core/search/sql-ranking.ts @@ -101,5 +101,33 @@ export function buildHardExcludeClause(slugColumn: string, prefixes: string[]): return `AND NOT (${likes})`; } +/** + * v0.26.5 — Build the soft-delete + archived-source visibility filter. + * + * Two filters in one fragment: + * - Page-level soft-delete: `.deleted_at IS NULL` hides pages that + * `delete_page` flipped via `softDeletePage`. + * - Source-level archive: `NOT .archived` hides every page + * belonging to a source that `gbrain sources archive` soft-deleted. + * + * Unlike `buildSourceFactorCase`, this clause is NOT bypassed by `detail=high`. + * Soft-deleted content stays hidden regardless of query detail level — the + * recovery window is for explicit `include_deleted: true` callers, not for + * temporal queries. + * + * Returns a fragment with leading `AND` so callers can splice it into a WHERE + * unconditionally. Both column references are engine-supplied (never user + * input), so no escape is required on the alias names themselves. + * + * @param pageAlias — page table alias (e.g. `'p'`) + * @param sourceAlias — source table alias (e.g. `'s'`); the caller is + * responsible for joining `sources` so this alias resolves. + * + * @returns raw SQL fragment, e.g. `AND p.deleted_at IS NULL AND NOT s.archived` + */ +export function buildVisibilityClause(pageAlias: string, sourceAlias: string): string { + return `AND ${pageAlias}.deleted_at IS NULL AND NOT ${sourceAlias}.archived`; +} + // Exported for unit tests export const __test__ = { escapeLikePattern, escapeSqlLiteral, buildLikePrefixLiteral }; diff --git a/src/core/types.ts b/src/core/types.ts index b242c0344..f1b8e7565 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -18,6 +18,12 @@ export interface Page { content_hash?: string; created_at: Date; updated_at: Date; + /** + * v0.26.5: when present, the page is soft-deleted. Hidden from search and + * from `getPage` / `listPages` by default; surface via `include_deleted: true`. + * The autopilot purge phase hard-deletes rows where `deleted_at < now() - 72h`. + */ + deleted_at?: Date | null; } export type PageKind = 'markdown' | 'code'; @@ -53,6 +59,21 @@ export interface PageFilters { * queries to a tier directory without loading every page into memory. */ slugPrefix?: string; + /** + * v0.26.5: include soft-deleted pages (rows with `deleted_at IS NOT NULL`). + * Default false: hides soft-deleted pages from `list_pages` so agents see the + * same set search returns. Set true to enumerate the recoverable set during + * the 72h window before the autopilot purge phase hard-deletes them. + */ + includeDeleted?: boolean; +} + +/** v0.26.5 — opts for getPage / softDeletePage / restorePage. */ +export interface GetPageOpts { + /** Filter to a specific source. When omitted, getPage returns the first slug match across sources (pre-existing semantics). */ + sourceId?: string; + /** Include soft-deleted pages. Default false. See PageFilters.includeDeleted. */ + includeDeleted?: boolean; } // Chunks diff --git a/src/core/utils.ts b/src/core/utils.ts index 9204067de..3efbdbea9 100644 --- a/src/core/utils.ts +++ b/src/core/utils.ts @@ -43,6 +43,12 @@ export function contentHash(page: PageInput): string { } export function rowToPage(row: Record): Page { + // v0.26.5: deleted_at is optional in the SELECT projection. When the column + // isn't selected (legacy callers), keep the field absent on the returned object. + const deletedAtRaw = row.deleted_at; + const deletedAt = deletedAtRaw == null + ? (deletedAtRaw === null ? null : undefined) + : new Date(deletedAtRaw as string); return { id: row.id as number, slug: row.slug as string, @@ -54,6 +60,7 @@ export function rowToPage(row: Record): Page { content_hash: row.content_hash as string | undefined, created_at: new Date(row.created_at as string), updated_at: new Date(row.updated_at as string), + ...(deletedAt !== undefined && { deleted_at: deletedAt }), }; } diff --git a/src/schema.sql b/src/schema.sql index 3cc130d2e..97a298221 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -35,6 +35,14 @@ CREATE TABLE IF NOT EXISTS sources ( -- bypassing the git-HEAD up_to_date early-return so CHUNKER_VERSION bumps -- actually trigger re-chunking on upgrade. chunker_version TEXT, + -- v0.26.5: soft-delete + recovery window. `archive` flips archived=true and + -- sets archive_expires_at = now() + 72h. The autopilot purge phase + -- hard-deletes rows where archive_expires_at <= now(). Promoted from a + -- JSONB key to real columns to avoid reserved-key footguns and to make the + -- search visibility filter (`NOT s.archived`) a column lookup. + archived BOOLEAN NOT NULL DEFAULT false, + archived_at TIMESTAMPTZ, + archive_expires_at TIMESTAMPTZ, created_at TIMESTAMPTZ NOT NULL DEFAULT now() ); @@ -71,6 +79,11 @@ CREATE TABLE IF NOT EXISTS pages ( content_hash TEXT, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + -- v0.26.5: soft-delete + recovery window. `delete_page` sets deleted_at = now() + -- instead of issuing DELETE. The autopilot purge phase hard-deletes pages + -- where deleted_at < now() - 72h. Search and `get_page` filter + -- `WHERE deleted_at IS NULL` by default; `include_deleted: true` opts in. + deleted_at TIMESTAMPTZ, CONSTRAINT pages_source_slug_key UNIQUE (source_id, slug) ); @@ -81,6 +94,13 @@ CREATE INDEX IF NOT EXISTS idx_pages_trgm ON pages USING GIN(title gin_trgm_ops) CREATE INDEX IF NOT EXISTS idx_pages_updated_at_desc ON pages (updated_at DESC); -- v0.18.0: source-scoped scans (per /plan-eng-review Section 4). CREATE INDEX IF NOT EXISTS idx_pages_source_id ON pages(source_id); +-- v0.26.5: partial index supports the autopilot purge sweep +-- (`WHERE deleted_at IS NOT NULL AND deleted_at < now() - INTERVAL '72 hours'`). +-- Search filters (`WHERE deleted_at IS NULL`) do not benefit from this index +-- (predicate doesn't match) and don't need their own — soft-deleted cardinality +-- stays low. Don't add a regular `(deleted_at)` index without measuring. +CREATE INDEX IF NOT EXISTS pages_deleted_at_purge_idx + ON pages (deleted_at) WHERE deleted_at IS NOT NULL; -- ============================================================ -- content_chunks: chunked content with embeddings diff --git a/test/core/cycle.test.ts b/test/core/cycle.test.ts index 33d9c67a5..412765e96 100644 --- a/test/core/cycle.test.ts +++ b/test/core/cycle.test.ts @@ -377,8 +377,8 @@ describe('runCycle — yieldBetweenPhases hook', () => { hookCalls++; }, }); - // v0.23: 8 phases → 8 yield calls (one after each). - expect(hookCalls).toBe(8); + // v0.26.5: 9 phases (added `purge`) → 9 yield calls (one after each). + expect(hookCalls).toBe(9); }); test('hook exceptions do not abort the cycle', async () => { @@ -388,8 +388,8 @@ describe('runCycle — yieldBetweenPhases hook', () => { throw new Error('synthetic hook error'); }, }); - // Cycle still completed all phases (v0.23: 8). - expect(report.phases.length).toBe(8); + // Cycle still completed all phases (v0.26.5: 9 with the new purge phase). + expect(report.phases.length).toBe(9); }); }); diff --git a/test/destructive-guard.test.ts b/test/destructive-guard.test.ts new file mode 100644 index 000000000..8c5efce84 --- /dev/null +++ b/test/destructive-guard.test.ts @@ -0,0 +1,325 @@ +/** + * v0.26.5 — destructive-guard unit tests. + * + * Source-level guard against accidental data loss. Three layers: + * 1. Impact assessment (counts pages/chunks/embeddings/files for a source) + * 2. Confirmation gate (`--confirm-destructive` required when data exists; + * `--yes` alone rejected) + * 3. Soft-delete with 72h TTL (column-based as of v0.26.5; JSONB shape was + * migrated in v33) + * + * Run against PGLite — the contract logic is identical on Postgres but + * PGLite is fast + DATABASE_URL-free. Postgres-specific paths (CONCURRENTLY, + * RLS) are covered separately by E2E tests. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { + assessDestructiveImpact, + checkDestructiveConfirmation, + softDeleteSource, + restoreSource, + listArchivedSources, + purgeExpiredSources, + formatImpact, + formatSoftDelete, + SOFT_DELETE_TTL_HOURS, + type DestructiveImpact, +} from '../src/core/destructive-guard.ts'; + +// Tier 3 opt-out — these tests need the cold-init schema path so the v33 +// migration columns exist on the brain under test. +delete process.env.GBRAIN_PGLITE_SNAPSHOT; + +async function setupBrain(): Promise { + const engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + return engine; +} + +async function seedSource(engine: PGLiteEngine, id: string, opts?: { withPages?: number }): Promise { + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ($1, $2) ON CONFLICT (id) DO NOTHING`, + [id, id], + ); + const count = opts?.withPages ?? 0; + for (let i = 0; i < count; i++) { + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title) VALUES ($1, $2, 'note', $3)`, + [id, `${id}/page-${i}`, `Page ${i}`], + ); + } +} + +describe('assessDestructiveImpact', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = await setupBrain(); + }, 30000); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('returns null for a non-existent source', async () => { + const impact = await assessDestructiveImpact(engine, 'aim-does-not-exist'); + expect(impact).toBeNull(); + }); + + test('counts zero across the board for an empty source', async () => { + await seedSource(engine, 'aim-empty', { withPages: 0 }); + const impact = await assessDestructiveImpact(engine, 'aim-empty'); + expect(impact).not.toBeNull(); + expect(impact!.pageCount).toBe(0); + expect(impact!.chunkCount).toBe(0); + expect(impact!.embeddingCount).toBe(0); + expect(impact!.fileCount).toBe(0); + // Empty-source summary is the safe message, not the "permanently delete" warning. + expect(impact!.summary).toContain('safe to remove'); + }); + + test('counts pages correctly for a populated source', async () => { + await seedSource(engine, 'aim-populated', { withPages: 3 }); + const impact = await assessDestructiveImpact(engine, 'aim-populated'); + expect(impact!.pageCount).toBe(3); + expect(impact!.summary).toContain('3 pages'); + expect(impact!.summary).toContain('permanently delete'); + }); + + test('source-scopes pages — multi-source isolation', async () => { + await seedSource(engine, 'aim-src-a', { withPages: 2 }); + await seedSource(engine, 'aim-src-b', { withPages: 5 }); + const a = await assessDestructiveImpact(engine, 'aim-src-a'); + const b = await assessDestructiveImpact(engine, 'aim-src-b'); + expect(a!.pageCount).toBe(2); + expect(b!.pageCount).toBe(5); + }); +}); + +describe('checkDestructiveConfirmation (gate truth table)', () => { + const populated: DestructiveImpact = { + sourceId: 'has-data', + sourceName: 'has-data', + pageCount: 100, + chunkCount: 500, + embeddingCount: 500, + fileCount: 0, + summary: '⚠️ This will permanently delete: 100 pages, 500 chunks, 500 embeddings', + }; + + const empty: DestructiveImpact = { + sourceId: 'no-data', + sourceName: 'no-data', + pageCount: 0, + chunkCount: 0, + embeddingCount: 0, + fileCount: 0, + summary: 'Source "no-data" has no data (safe to remove).', + }; + + test('dry-run always passes regardless of flags', () => { + expect(checkDestructiveConfirmation(populated, { dryRun: true })).toBeNull(); + expect(checkDestructiveConfirmation(populated, { yes: true, dryRun: true })).toBeNull(); + }); + + test('empty source passes without --confirm-destructive', () => { + expect(checkDestructiveConfirmation(empty, {})).toBeNull(); + expect(checkDestructiveConfirmation(empty, { yes: true })).toBeNull(); + }); + + test('--confirm-destructive passes regardless of --yes', () => { + expect(checkDestructiveConfirmation(populated, { confirmDestructive: true })).toBeNull(); + expect(checkDestructiveConfirmation(populated, { yes: true, confirmDestructive: true })).toBeNull(); + }); + + test('--yes alone with data is REJECTED with guidance message', () => { + const msg = checkDestructiveConfirmation(populated, { yes: true }); + expect(msg).not.toBeNull(); + expect(msg).toContain('--confirm-destructive'); + expect(msg).toContain('archive'); + }); + + test('no flags + populated source rejects', () => { + const msg = checkDestructiveConfirmation(populated, {}); + expect(msg).not.toBeNull(); + expect(msg).toContain('--confirm-destructive'); + }); +}); + +describe('soft-delete + restore lifecycle (column-based v0.26.5)', () => { + // ONE engine for the whole describe — cold init runs ~29 migrations, ~3s. + // Each test uses a unique source id so they don't cross-pollute. + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = await setupBrain(); + }, 30000); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('softDeleteSource flips column shape + sets TTL', async () => { + const id = 'sd-flips'; + await seedSource(engine, id, { withPages: 2 }); + const before = Date.now(); + const result = await softDeleteSource(engine, id); + const after = Date.now(); + expect(result).not.toBeNull(); + expect(result!.id).toBe(id); + expect(result!.pageCount).toBe(2); + const ttlMs = SOFT_DELETE_TTL_HOURS * 60 * 60 * 1000; + expect(result!.expiresAt.getTime()).toBeGreaterThanOrEqual(before + ttlMs - 1000); + expect(result!.expiresAt.getTime()).toBeLessThanOrEqual(after + ttlMs + 1000); + const rows = await engine.executeRaw<{ archived: boolean; archived_at: string }>( + `SELECT archived, archived_at FROM sources WHERE id = $1`, + [id], + ); + expect(rows[0].archived).toBe(true); + expect(rows[0].archived_at).not.toBeNull(); + }); + + test('softDeleteSource is idempotent-as-null on already-archived', async () => { + const id = 'sd-idem'; + await seedSource(engine, id, { withPages: 1 }); + await softDeleteSource(engine, id); + expect(await softDeleteSource(engine, id)).toBeNull(); + }); + + test('softDeleteSource returns null for unknown source', async () => { + expect(await softDeleteSource(engine, 'sd-unknown-xyz')).toBeNull(); + }); + + test('softDeleteSource flips federated:false in JSONB but archived state is column-based', async () => { + const id = 'sd-jsonb'; + await seedSource(engine, id, { withPages: 1 }); + await softDeleteSource(engine, id); + const rows = await engine.executeRaw<{ config: any; archived: boolean }>( + `SELECT config, archived FROM sources WHERE id = $1`, + [id], + ); + const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config; + expect(config.federated).toBe(false); + expect(rows[0].archived).toBe(true); + // Issue 5 contract: archived must NOT live in config any more. + expect(config.archived).toBeUndefined(); + expect(config.archived_at).toBeUndefined(); + }); + + test('restoreSource clears the column state and re-federates by default', async () => { + const id = 'sd-restore-fed'; + await seedSource(engine, id, { withPages: 1 }); + await softDeleteSource(engine, id); + expect(await restoreSource(engine, id)).toBe(true); + const rows = await engine.executeRaw<{ archived: boolean; archived_at: string | null; config: any }>( + `SELECT archived, archived_at, config FROM sources WHERE id = $1`, + [id], + ); + expect(rows[0].archived).toBe(false); + expect(rows[0].archived_at).toBeNull(); + const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config; + expect(config.federated).toBe(true); + }); + + test('restoreSource respects --no-federate (refederate=false)', async () => { + const id = 'sd-no-fed'; + await seedSource(engine, id, { withPages: 1 }); + await softDeleteSource(engine, id); + await restoreSource(engine, id, false); + const rows = await engine.executeRaw<{ config: any }>( + `SELECT config FROM sources WHERE id = $1`, + [id], + ); + const config = typeof rows[0].config === 'string' ? JSON.parse(rows[0].config) : rows[0].config; + expect(config.federated).toBe(false); + }); + + test('restoreSource is idempotent-as-false on already-active', async () => { + const id = 'sd-active'; + await seedSource(engine, id); + expect(await restoreSource(engine, id)).toBe(false); + }); + + test('listArchivedSources filters via the archived column, not JSONB', async () => { + const archivedId = 'la-archived'; + const liveId = 'la-live'; + await seedSource(engine, archivedId, { withPages: 3 }); + await seedSource(engine, liveId, { withPages: 1 }); + await softDeleteSource(engine, archivedId); + const archived = await listArchivedSources(engine); + const ids = archived.map((a) => a.id); + expect(ids).toContain(archivedId); + expect(ids).not.toContain(liveId); + const archivedRow = archived.find((a) => a.id === archivedId)!; + expect(archivedRow.pageCount).toBe(3); + }); + + test('purgeExpiredSources only deletes rows with archive_expires_at <= now()', async () => { + const expiredId = 'pe-expired'; + const recoverableId = 'pe-recoverable'; + await seedSource(engine, expiredId, { withPages: 2 }); + await seedSource(engine, recoverableId, { withPages: 1 }); + await softDeleteSource(engine, expiredId); + await softDeleteSource(engine, recoverableId); + await engine.executeRaw( + `UPDATE sources SET archive_expires_at = now() - INTERVAL '1 hour' WHERE id = $1`, + [expiredId], + ); + const purged = await purgeExpiredSources(engine); + expect(purged).toContain(expiredId); + expect(purged).not.toContain(recoverableId); + const remainingPages = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM pages WHERE source_id = $1`, + [expiredId], + ); + expect(remainingPages[0].n).toBe(0); + }); + + test('purgeExpiredSources is no-op when nothing is past TTL', async () => { + // After all earlier tests, there may still be archived rows whose + // archive_expires_at is in the future. Force-update any leftover-past + // rows OUT of expiration before this assertion (we only want to test + // the no-op return here, not interfere with prior test state). + await engine.executeRaw( + `UPDATE sources SET archive_expires_at = now() + INTERVAL '72 hours' WHERE archived = true`, + ); + const purged = await purgeExpiredSources(engine); + expect(purged).toEqual([]); + }); +}); + +describe('formatters (display helpers)', () => { + test('formatImpact renders the boxed preview with the source id and counts', () => { + const impact: DestructiveImpact = { + sourceId: 'media-corpus', + sourceName: 'Media Corpus', + pageCount: 5033, + chunkCount: 22000, + embeddingCount: 22000, + fileCount: 0, + summary: '⚠️ This will permanently delete: 5,033 pages, 22,000 chunks, 22,000 embeddings', + }; + const out = formatImpact(impact); + expect(out).toContain('Media Corpus'); + expect(out).toContain('media-corpus'); + expect(out).toContain('5,033'); + expect(out).toContain('22,000'); + expect(out).toContain('DESTRUCTIVE OPERATION'); + }); + + test('formatSoftDelete renders the post-archive guidance with restore command', () => { + const out = formatSoftDelete({ + id: 'src-a', + name: 'src-a', + deletedAt: new Date(), + expiresAt: new Date(Date.now() + 72 * 60 * 60 * 1000), + pageCount: 100, + }); + expect(out).toContain('archived'); + expect(out).toContain('restore src-a'); + expect(out).toContain('72'); + }); +}); diff --git a/test/e2e/cycle.test.ts b/test/e2e/cycle.test.ts index fb66d46e4..be0d1fedb 100644 --- a/test/e2e/cycle.test.ts +++ b/test/e2e/cycle.test.ts @@ -97,8 +97,9 @@ describeE2E('E2E: runCycle against real Postgres', () => { }); expect(report.schema_version).toBe('1'); - // Cycle ran all 8 phases (or skipped the ones that don't support dry-run). - expect(report.phases.length).toBe(8); + // Cycle ran all 9 phases (or skipped the ones that don't support dry-run). + // v0.26.5 added the `purge` phase (9th, after `orphans`). + expect(report.phases.length).toBe(9); // Nothing got written. const afterPages = await conn.unsafe(`SELECT count(*)::int AS n FROM pages`); diff --git a/test/e2e/multi-source.test.ts b/test/e2e/multi-source.test.ts index 2b06a7907..dc3b94c57 100644 --- a/test/e2e/multi-source.test.ts +++ b/test/e2e/multi-source.test.ts @@ -271,7 +271,8 @@ describeE2E('v0.18.0 multi-source — cascade delete covers every dependent row' expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM files WHERE source_id = 'cascadetest'`))[0].n).toBe(1); // Remove the source. - await runSources(engine as unknown as Parameters[0], ['remove', 'cascadetest', '--yes']); + // v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected. + await runSources(engine as unknown as Parameters[0], ['remove', 'cascadetest', '--confirm-destructive']); // Everything for that source is gone. expect((await conn.unsafe(`SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'cascadetest'`))[0].n).toBe(0); diff --git a/test/multi-source-integration.test.ts b/test/multi-source-integration.test.ts index 3d7b433e0..33089057b 100644 --- a/test/multi-source-integration.test.ts +++ b/test/multi-source-integration.test.ts @@ -181,7 +181,8 @@ describe('v0.18.0 — sources remove cascades to pages', () => { ); expect(before[0].n).toBeGreaterThan(0); - await runSources(engine, ['remove', 'testsrc', '--yes']); + // v0.26.5: populated sources require --confirm-destructive; --yes alone is rejected. + await runSources(engine, ['remove', 'testsrc', '--confirm-destructive']); const after = await engine.executeRaw<{ n: number }>( `SELECT COUNT(*)::int AS n FROM pages WHERE source_id = 'testsrc'`, diff --git a/test/pages-soft-delete.test.ts b/test/pages-soft-delete.test.ts new file mode 100644 index 000000000..070c5b9f5 --- /dev/null +++ b/test/pages-soft-delete.test.ts @@ -0,0 +1,323 @@ +/** + * v0.26.5 — page-level soft-delete contract tests. + * + * IRON RULE regression test for Q3 (the lynchpin eng-review decision): + * delete_page → get_page returns null → get_page({include_deleted:true}) returns + * the row with deleted_at populated → restore_page → get_page returns the row + * again with deleted_at unset. + * + * Plus: BrainEngine surface tests (softDeletePage / restorePage / + * purgeDeletedPages) for happy-path / boundary / cascade cases. + * + * Runs against PGLite — same SQL contract as Postgres but DATABASE_URL-free. + * Postgres-specific paths (CONCURRENTLY index, two-stage CTE) covered by + * separate Postgres E2E tests. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; + +delete process.env.GBRAIN_PGLITE_SNAPSHOT; + +async function setupBrain(): Promise { + const engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + return engine; +} + +async function seedPage(engine: PGLiteEngine, slug: string): Promise { + await engine.putPage(slug, { + type: 'note' as any, + title: slug, + compiled_truth: `Content of ${slug}`, + timeline: '', + frontmatter: {}, + }); +} + +describe('softDeletePage', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = await setupBrain(); + }, 30000); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('happy path: sets deleted_at and returns slug', async () => { + await seedPage(engine, 'people/alice'); + const result = await engine.softDeletePage('people/alice'); + expect(result).not.toBeNull(); + expect(result!.slug).toBe('people/alice'); + // The row stays in the DB. + const rows = await engine.executeRaw<{ deleted_at: string | null }>( + `SELECT deleted_at FROM pages WHERE slug = $1`, + ['people/alice'], + ); + expect(rows.length).toBe(1); + expect(rows[0].deleted_at).not.toBeNull(); + }); + + test('returns null for unknown slug (idempotent-as-null)', async () => { + expect(await engine.softDeletePage('does/not/exist')).toBeNull(); + }); + + test('returns null on already-soft-deleted page (idempotent-as-null)', async () => { + await seedPage(engine, 'people/bob'); + const first = await engine.softDeletePage('people/bob'); + expect(first).not.toBeNull(); + const second = await engine.softDeletePage('people/bob'); + expect(second).toBeNull(); + }); +}); + +describe('restorePage', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = await setupBrain(); + }, 30000); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('clears deleted_at on a soft-deleted page', async () => { + await seedPage(engine, 'people/carol'); + await engine.softDeletePage('people/carol'); + expect(await engine.restorePage('people/carol')).toBe(true); + const rows = await engine.executeRaw<{ deleted_at: string | null }>( + `SELECT deleted_at FROM pages WHERE slug = $1`, + ['people/carol'], + ); + expect(rows[0].deleted_at).toBeNull(); + }); + + test('returns false for unknown slug', async () => { + expect(await engine.restorePage('does/not/exist')).toBe(false); + }); + + test('returns false on already-active page (idempotent-as-false)', async () => { + await seedPage(engine, 'people/dave'); + expect(await engine.restorePage('people/dave')).toBe(false); + }); +}); + +describe('purgeDeletedPages (TTL boundary)', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = await setupBrain(); + }, 30000); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('purges pages whose deleted_at is older than the cutoff', async () => { + await seedPage(engine, 'people/eve'); + await seedPage(engine, 'people/frank'); + // Soft-delete both, then push one's deleted_at into the distant past. + await engine.softDeletePage('people/eve'); + await engine.softDeletePage('people/frank'); + await engine.executeRaw( + `UPDATE pages SET deleted_at = now() - INTERVAL '73 hours' WHERE slug = $1`, + ['people/eve'], + ); + const result = await engine.purgeDeletedPages(72); + expect(result.count).toBe(1); + expect(result.slugs).toContain('people/eve'); + expect(result.slugs).not.toContain('people/frank'); + // 'eve' is gone; 'frank' is still there (still inside recovery window). + const rows = await engine.executeRaw<{ slug: string }>(`SELECT slug FROM pages`); + const remaining = rows.map((r) => r.slug); + expect(remaining).not.toContain('people/eve'); + expect(remaining).toContain('people/frank'); + }); + + test('does NOT touch active pages (deleted_at IS NULL)', async () => { + // Bound to this test's seeded slug. Other tests in the same describe may + // have soft-deleted state laying around; we don't care about those, just + // that THIS test's active page is not deleted. + await seedPage(engine, 'people/grace-active'); + await engine.purgeDeletedPages(0); + const rows = await engine.executeRaw<{ slug: string; deleted_at: string | null }>( + `SELECT slug, deleted_at FROM pages WHERE slug = $1`, + ['people/grace-active'], + ); + expect(rows.length).toBe(1); + expect(rows[0].deleted_at).toBeNull(); + }); + + test('cascades to content_chunks via FK ON DELETE CASCADE', async () => { + await seedPage(engine, 'people/heidi'); + // Force-add a chunk row so we can observe cascade. + const pageRows = await engine.executeRaw<{ id: number }>( + `SELECT id FROM pages WHERE slug = $1`, + ['people/heidi'], + ); + await engine.executeRaw( + `INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source) VALUES ($1, 0, 'test', 'compiled_truth')`, + [pageRows[0].id], + ); + await engine.softDeletePage('people/heidi'); + await engine.executeRaw( + `UPDATE pages SET deleted_at = now() - INTERVAL '73 hours' WHERE slug = $1`, + ['people/heidi'], + ); + await engine.purgeDeletedPages(72); + const remaining = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM content_chunks WHERE page_id = $1`, + [pageRows[0].id], + ); + expect(remaining[0].n).toBe(0); + }); + + test('clamps negative hours to 0 (no crash, no future-cutoff explosion)', async () => { + await seedPage(engine, 'people/ivan'); + await engine.softDeletePage('people/ivan'); + // The contract being pinned: negative input must NOT pass through to the + // SQL as a literal negative interval (which would purge from the future + // and effectively delete every soft-deleted row). Implementation does + // `Math.max(0, Math.floor(olderThanHours))`, so -72 collapses to 0. With + // hours=0, the predicate `deleted_at < now()` may or may not match a row + // soft-deleted in the same statement (timing-dependent), so this test + // pins only the safety contract: it returns successfully with a finite + // count and doesn't blow up the brain. + const result = await engine.purgeDeletedPages(-72); + expect(result.count).toBeGreaterThanOrEqual(0); + expect(Number.isFinite(result.count)).toBe(true); + }); +}); + +describe('getPage / listPages includeDeleted contract (Q3 IRON RULE)', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = await setupBrain(); + }, 30000); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('Q3 round-trip: delete → get returns null → get(include_deleted) returns row → restore → get returns row again', async () => { + await seedPage(engine, 'people/judy'); + + // Step 1: page is visible by default. + const before = await engine.getPage('people/judy'); + expect(before).not.toBeNull(); + expect(before!.deleted_at).toBeFalsy(); + + // Step 2: soft-delete, default getPage returns null. + await engine.softDeletePage('people/judy'); + const afterDelete = await engine.getPage('people/judy'); + expect(afterDelete).toBeNull(); + + // Step 3: include_deleted: true surfaces the row with deleted_at populated. + const surfaced = await engine.getPage('people/judy', { includeDeleted: true }); + expect(surfaced).not.toBeNull(); + expect(surfaced!.deleted_at).toBeInstanceOf(Date); + + // Step 4: restore → default getPage returns the row again. + expect(await engine.restorePage('people/judy')).toBe(true); + const restored = await engine.getPage('people/judy'); + expect(restored).not.toBeNull(); + expect(restored!.deleted_at).toBeFalsy(); + }); + + test('listPages excludes soft-deleted by default', async () => { + await seedPage(engine, 'people/kim'); + await seedPage(engine, 'people/larry'); + await engine.softDeletePage('people/kim'); + const pages = await engine.listPages({ limit: 100 }); + const slugs = pages.map((p) => p.slug); + expect(slugs).not.toContain('people/kim'); + expect(slugs).toContain('people/larry'); + }); + + test('listPages includes soft-deleted when includeDeleted: true', async () => { + await seedPage(engine, 'people/mia'); + await engine.softDeletePage('people/mia'); + const pages = await engine.listPages({ limit: 100, includeDeleted: true }); + const slugs = pages.map((p) => p.slug); + expect(slugs).toContain('people/mia'); + const mia = pages.find((p) => p.slug === 'people/mia')!; + expect(mia.deleted_at).toBeInstanceOf(Date); + }); +}); + +describe('search visibility (soft-deleted pages hidden from searchKeyword)', () => { + let engine: PGLiteEngine; + + beforeAll(async () => { + engine = await setupBrain(); + }, 30000); + + afterAll(async () => { + await engine.disconnect(); + }); + + test('searchKeyword hides soft-deleted pages', async () => { + // Two pages, same distinctive term, then soft-delete one. + await engine.putPage('people/nora', { + type: 'note' as any, + title: 'Nora', + compiled_truth: 'gbrainquantum signature term occurs here', + timeline: '', + frontmatter: {}, + }); + await engine.putPage('people/oscar', { + type: 'note' as any, + title: 'Oscar', + compiled_truth: 'gbrainquantum signature term occurs here too', + timeline: '', + frontmatter: {}, + }); + // Force chunk creation so search has something to index. + await engine.upsertChunks('people/nora', [ + { chunk_index: 0, chunk_text: 'gbrainquantum signature term occurs here', chunk_source: 'compiled_truth' as any }, + ]); + await engine.upsertChunks('people/oscar', [ + { chunk_index: 0, chunk_text: 'gbrainquantum signature term occurs here too', chunk_source: 'compiled_truth' as any }, + ]); + + const before = await engine.searchKeyword('gbrainquantum'); + expect(before.length).toBe(2); + + await engine.softDeletePage('people/nora'); + const after = await engine.searchKeyword('gbrainquantum'); + const slugs = after.map((r) => r.slug); + expect(slugs).not.toContain('people/nora'); + expect(slugs).toContain('people/oscar'); + }); + + test('searchKeyword hides pages from archived sources', async () => { + await engine.executeRaw( + `INSERT INTO sources (id, name) VALUES ('archived-src', 'archived-src') ON CONFLICT DO NOTHING`, + ); + await engine.executeRaw( + `INSERT INTO pages (source_id, slug, type, title) VALUES ('archived-src', 'archived-src/secret', 'note', 'Secret')`, + ); + const pageRows = await engine.executeRaw<{ id: number }>( + `SELECT id FROM pages WHERE slug = 'archived-src/secret'`, + ); + await engine.executeRaw( + `INSERT INTO content_chunks (page_id, chunk_index, chunk_text, chunk_source) VALUES ($1, 0, 'gbrainsemaphore unique term', 'compiled_truth')`, + [pageRows[0].id], + ); + // Trigger should populate search_vector via the schema trigger. + const before = await engine.searchKeyword('gbrainsemaphore'); + expect(before.length).toBe(1); + + // Archive the source. + await engine.executeRaw( + `UPDATE sources SET archived = true, archived_at = now(), archive_expires_at = now() + INTERVAL '72 hours' WHERE id = 'archived-src'`, + ); + const after = await engine.searchKeyword('gbrainsemaphore'); + expect(after.length).toBe(0); + }); +}); diff --git a/test/schema-bootstrap-coverage.test.ts b/test/schema-bootstrap-coverage.test.ts index 3a4fc6791..d09c0ce8d 100644 --- a/test/schema-bootstrap-coverage.test.ts +++ b/test/schema-bootstrap-coverage.test.ts @@ -60,6 +60,9 @@ const REQUIRED_BOOTSTRAP_COVERAGE: ForwardReference[] = [ // v0.19+ — forward-referenced by `CREATE INDEX idx_chunks_language // ON content_chunks(language) WHERE language IS NOT NULL`. { kind: 'column', table: 'content_chunks', column: 'language' }, + // v0.26.5 — forward-referenced by `CREATE INDEX pages_deleted_at_purge_idx + // ON pages (deleted_at) WHERE deleted_at IS NOT NULL`. + { kind: 'column', table: 'pages', column: 'deleted_at' }, ]; test('applyForwardReferenceBootstrap covers every forward reference declared in REQUIRED_BOOTSTRAP_COVERAGE', async () => { @@ -89,6 +92,9 @@ test('applyForwardReferenceBootstrap covers every forward reference declared in DROP INDEX IF EXISTS idx_chunks_language; ALTER TABLE content_chunks DROP COLUMN IF EXISTS symbol_name; ALTER TABLE content_chunks DROP COLUMN IF EXISTS language; + + DROP INDEX IF EXISTS pages_deleted_at_purge_idx; + ALTER TABLE pages DROP COLUMN IF EXISTS deleted_at; `); // Run bootstrap in isolation (NOT initSchema). This is what we're testing. @@ -142,6 +148,8 @@ test('after bootstrap, PGLITE_SCHEMA_SQL replays without crashing on missing for ALTER TABLE links DROP CONSTRAINT IF EXISTS links_from_to_type_source_origin_unique; ALTER TABLE links DROP COLUMN IF EXISTS link_source; ALTER TABLE links DROP COLUMN IF EXISTS origin_page_id; + DROP INDEX IF EXISTS pages_deleted_at_purge_idx; + ALTER TABLE pages DROP COLUMN IF EXISTS deleted_at; `); // Bootstrap, then schema replay. Either step crashing fails the test. diff --git a/test/sql-ranking.test.ts b/test/sql-ranking.test.ts index db558c5b3..c76c1cf8c 100644 --- a/test/sql-ranking.test.ts +++ b/test/sql-ranking.test.ts @@ -2,6 +2,7 @@ import { describe, test, expect } from 'bun:test'; import { buildSourceFactorCase, buildHardExcludeClause, + buildVisibilityClause, __test__, } from '../src/core/search/sql-ranking.ts'; import { @@ -253,3 +254,40 @@ describe('resolveHardExcludes', () => { expect(r).not.toContain('envdir/'); }); }); + +// v0.26.5 — visibility clause for soft-deleted pages and archived sources. +describe('buildVisibilityClause (v0.26.5)', () => { + test('emits both predicates joined by AND with a leading AND', () => { + const clause = buildVisibilityClause('p', 's'); + // Leading AND so callers can splice unconditionally. + expect(clause.startsWith('AND ')).toBe(true); + // Both predicates present: page-level deleted_at IS NULL + source-level NOT archived. + expect(clause).toContain('p.deleted_at IS NULL'); + expect(clause).toContain('NOT s.archived'); + }); + + test('uses the supplied aliases verbatim', () => { + expect(buildVisibilityClause('pp', 'src')).toBe('AND pp.deleted_at IS NULL AND NOT src.archived'); + }); + + test('does NOT bypass on detail level — visibility is a contract, not a temporal preference', () => { + // Distinct from buildSourceFactorCase: there's no detail-gated short-circuit. + // Soft-deleted content stays hidden regardless of caller's detail level. + // Function signature has no detail param at all; this test pins that contract. + expect(buildVisibilityClause.length).toBe(2); + }); + + test('emits a stable string regardless of call order (idempotent for snapshot tests)', () => { + const a = buildVisibilityClause('p', 's'); + const b = buildVisibilityClause('p', 's'); + expect(a).toBe(b); + }); + + test('produces no JSONB containment in the output (column-based, not @>)', () => { + // Issue 5 contract: archived was promoted from JSONB key to real column. + // The visibility clause must not regress to JSONB containment. + const clause = buildVisibilityClause('p', 's'); + expect(clause).not.toContain('@>'); + expect(clause).not.toContain('config'); + }); +});