diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
index 3857043ba..74717c3ea 100644
--- a/.github/workflows/e2e.yml
+++ b/.github/workflows/e2e.yml
@@ -79,7 +79,22 @@ jobs:
bun-version: 1.3.13
- run: bun install
- name: Install OpenClaw
- run: npm install -g openclaw@2026.4.9
+ # Bound + retry the install: a transient npm/registry stall here used to
+ # hang unbounded and (since the v0.42.50.0 job timeout) burn the entire
+ # 30m Tier 2 budget before failing — even though the install normally
+ # finishes in well under a minute. `timeout` kills a hung attempt fast;
+ # up to 3 attempts ride out a flaky registry. Step cap is a backstop.
+ timeout-minutes: 8
+ run: |
+ for attempt in 1 2 3; do
+ if timeout 120 npm install -g openclaw@2026.4.9; then
+ exit 0
+ fi
+ echo "::warning::openclaw install attempt $attempt failed or timed out; retrying in 10s" >&2
+ sleep 10
+ done
+ echo "::error::openclaw install failed after 3 attempts" >&2
+ exit 1
- name: Configure OpenClaw MCP
run: |
mkdir -p ~/.openclaw
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 380ae0a75..565627272 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,22 @@
All notable changes to GBrain will be documented in this file.
+## [0.42.51.0] - 2026-06-17
+
+**`gbrain sync` stops bottlenecking all its workers on a single database row, a malformed checkpoint can no longer wedge a source, and `gbrain doctor` tells an actively-running sync apart from a stuck one.** A slow source that fell behind HEAD could read as permanently stale even while it imported every cycle: sync was single-core-bound at the database layer, so handing it more workers didn't help, and the freshness check couldn't see that a sync was in fact running.
+
+The root cause was the page-generation clock that backs the search cache. Every page write bumped a single locked counter row, so concurrent sync workers serialized on one another's commits no matter how many you ran. It is now a contention-free sequence: the cache invalidation contract is unchanged (it still over-invalidates rather than ever serving stale), but writers no longer wait in line. The other fixes harden checkpoint state and make the freshness signal honest.
+
+### Changed
+- **Sync writes scale across cores.** The page-generation clock moved from a single locked counter row to a contention-free sequence, so parallel sync workers stop serializing on each other. A large `gbrain sync` now uses the workers you give it instead of collapsing to roughly one.
+- **`gbrain doctor` distinguishes in-progress from stale.** A source holding a live sync lock is reported as actively syncing (naming the running process), not flagged stale. A genuinely stuck, blocked, or never-completed sync still reports stale — the signal is the live lock, so a stopped sync is never masked.
+
+### Fixed
+- **A malformed checkpoint record can no longer wedge a source.** Checkpoint state is structurally constrained, repaired automatically on upgrade, and the loader survives a bad record instead of discarding all banked progress for that source.
+- **`gbrain sync --force-break-lock` is honest when there is no lock.** It now says plainly that nothing was held and points at how to inspect a genuinely wedged sync, instead of a terse no-op that read like a successful unwedge.
+
+### To take advantage of v0.42.51.0
+`gbrain upgrade`, then `gbrain doctor`. Existing brains pick up the contention-free clock and the checkpoint integrity constraint automatically on the next migration; the search cache rebuilds itself on first query. Nothing to configure.
## [0.42.50.0] - 2026-06-17
**CI reliability hardening — a wedged job can no longer run for six hours, a superseded run no longer reports a stale flaky failure, and broken workflow YAML is caught before it ships.** gbrain's CI already had the deep machinery (content-hash run-skip cache, weight-aware shard balancing, test-isolation guards, hermetic E2E). What it lacked was the cheap GitHub-Actions hygiene that was already wired into `heavy-tests.yml` but never into the two hot-path workflows. This pass closes that gap, porting the patterns from the sibling GStack project's CI-reliability work.
diff --git a/VERSION b/VERSION
index 0578db35f..6a6c1a02a 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-0.42.50.0
+0.42.51.0
diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md
index d6b1251c5..b2d7fe7d2 100644
--- a/docs/architecture/KEY_FILES.md
+++ b/docs/architecture/KEY_FILES.md
@@ -287,7 +287,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/core/ai/model-resolver.ts:parseModelId` — gateway-side resolver accepts both colon and slash form (`provider:model` and `provider/model`) so a slash-form id resolves to the same recipe at every gateway entry point (chat / embed / rerank) instead of throwing `AIConfigError: model id must be in format provider:model`. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form.
- `src/commands/transcripts.ts` — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`.
- `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). Batch-load fast path on Postgres uses a single SQL query (fixes the PgBouncer round-trip timeout, ~60s → ~6s), gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1`. Batch projection is `SELECT ... ORDER BY source_id, slug` (NOT `SELECT DISTINCT ON (slug)`, which collapsed same-slug-different-source pages into one scan) so multi-source brains scan each `(source, slug)` row independently. Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`; batch + sequential paths report the same page count on multi-source brains.
-- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check at `doctor.ts:1011-1043` consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings in BOTH `doctor.ts` and `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source ` matches. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. IRON-RULE regression banning stale verb names, the sync_freshness boundary matrix, the D4 regression guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases).
+- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check at `doctor.ts:1011-1043` consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings in BOTH `doctor.ts` and `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source ` matches. A source holding a LIVE, non-expired per-source sync lock (`inspectLock(engine, syncLockId(source.id))` from `src/core/db-lock.ts`) is reported as actively syncing (the message names the holder pid + host) and counted in `synced_recently_count`, NOT flagged stale — the live lock is the only honest in-progress signal (checkpoint banking can't distinguish in-progress from wedged: a blocked sync banks its files but writes no anchor). A blocked/failed sync's process has exited (no lock row) and a wedged holder stops refreshing (TTL lapses), so either falls through to the stale path and is never masked; the dynamic `db-lock` import is swallowed to a no-op on a stub engine or pre-lock-table brain, so this can only ADD an in-progress verdict, never suppress a real stale one. The in-progress note is appended to whatever verdict the buckets produce and is empty when nothing is syncing, so steady-state messages stay byte-for-byte unchanged. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. IRON-RULE regression banning stale verb names, the sync_freshness boundary matrix, the D4 regression guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases).
- `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). `Migration` interface carries `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses in a transaction; ignored on PGLite). Key migrations: v14 (handler branches on `engine.kind` for CONCURRENTLY-on-Postgres with invalid-remnant pre-drop via `pg_index.indisvalid`, plain `CREATE INDEX` on PGLite); v15 (`minion_jobs.max_stalled` default 1→5 + backfill non-terminal rows); v24 `rls_backfill_missing_tables` (`sqlFor: { pglite: '' }` no-op — PGLite has no RLS engine, targets subagent tables absent from pglite-schema.ts); v30 `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))` (RLS-enabled under BYPASSRLS; synthesize reads/writes to avoid re-judging); v35 auto-RLS event trigger `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` running `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on new `public.*` tables (no FORCE) + one-time backfill on every existing `public.*` base table whose comment doesn't match `^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}` (per-table failure aborts the offending CREATE TABLE; no EXCEPTION wrap; PGLite no-op via `sqlFor.pglite: ''`; breaking change: intentionally-RLS-off public tables need the GBRAIN:RLS_EXEMPT comment before upgrade); v40 `pages_emotional_weight` (`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`, column-only metadata-only); v46 `mcp_request_log_params_jsonb_normalize` (`UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`, idempotent); v60-v65 six-migration chain wiring source-scoping into `oauth_clients` — v60 (`oauth_clients_source_id_fk`: `source_id TEXT` NULL→`'default'` backfill + FK to `sources(id) ON DELETE SET NULL`), v61 (`federated_read TEXT[] NOT NULL DEFAULT '{}'`), v62 (explicit-CASE backfill so `source_id IS NULL` → `'{}'`), v63 (fail-loud check every row's source_id is in its federated_read array), v64 (FK flipped to `ON DELETE RESTRICT`), v65 (GIN index for array-containment); v68 `eval_candidates_embedding_column` (`eval_candidates.embedding_column TEXT NULL` per-row provenance for `gbrain eval replay` to reproduce the same retrieval space; NULL-tolerant); v108 `pages_embedding_signature` (`pages.embedding_signature TEXT NULL` = `:` stamped via `setPageEmbeddingSignature`; GRANDFATHER — stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current` so NULL is NEVER stale and upgrade never re-embeds the whole corpus; no index; metadata-only); v109 `sources_newest_content_at` (`sources.newest_content_at TIMESTAMPTZ` durable newest-COMMIT HEAD committer time written by `writeSyncAnchor`, read by the REMOTE staleness path instead of shelling to git; mirror in pglite-schema.ts + schema.sql + bootstrap probe); v110 `page_aliases` (`(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` + lookup indexes on `(source_id, alias_norm)` and `(source_id, slug)`; `alias_norm` is `normalizeAlias()` output so WRITE/READ key on the same form; also in `src/core/pglite-schema.ts`); v111 `search_telemetry_rank1_columns` (`ADD COLUMN IF NOT EXISTS` on both engines: `sum_rank1_score`, `count_rank1`, three buckets `rank1_lt_solid`/`rank1_solid`/`rank1_high` on `search_telemetry` — aggregate not per-query rows so rank-1 median drift is bounded-growth; ALTERs right after v57 which created the table); v114 `links_link_source_check_kebab_regex` (#1941, opens `link_source` from the closed allowlist to a kebab-case format gate `^[a-z][a-z0-9]*(-[a-z0-9]+)*$` + `char_length<=64`; Postgres branch uses `NOT VALID` + `VALIDATE CONSTRAINT` with `transaction:false`, PGLite plain DROP+ADD; existing built-ins all satisfy the regex so VALIDATE never fails on existing data); v116 `code_edges_source_backfill_and_callee_index` (#2073, idempotent: backfills NULL `code_edges_symbol`/`code_edges_chunk` `source_id` from each edge's `from_chunk` page — NULL never matched a scoped `AND source_id = …` filter so scoped `code-callers`/`code-callees` returned 0 rows on multi-source brains — plus plain `CREATE INDEX` on `from_symbol_qualified` for both edge tables, which had no index and seq-scanned per BFS node). The dedup-index self-heal (`timeline_dedup_index`, see `timeline-dedup-repair.ts`) is NOT version-gated: `runMigrations` invokes `repairTimelineDedupIndex` on every pass (including the no-pending early-return path) because a merge-renumbered migration can leave the version counter past the index change while the index stays the old shape.
- `src/core/timeline-dedup-repair.ts` (#2038) — schema-drift self-heal for `idx_timeline_dedup`. The migration that widened the dedup index from `(page_id, date, summary)` to `(page_id, date, summary, source)` was renumbered during a master merge, so a brain that ran the old variant has its version counter stamped past the change while the index keeps the 3-column shape — and every `addTimelineEntry` batch then fails its 4-column `ON CONFLICT`, silently breaking timeline writes brain-wide. The version counter can't detect this, so the repair is keyed off the actual index SHAPE: `checkTimelineDedupIndex(engine)` returns `{tablePresent, indexPresent, columns, needsRepair}` (read-only; powers the `timeline_dedup_index` doctor check) and `repairTimelineDedupIndex(engine)` dedupes-then-rebuilds the index. `runMigrations` invokes the repair on every pass (including the no-pending early-return path); idempotent no-op when the index is already 4-column. `gbrain apply-migrations --force-schema` triggers it on demand. Pinned by `test/timeline-dedup-repair.test.ts`.
- `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY `\r`-rewriting; non-TTY plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. `emitHumanLine` is prefix-aware — inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts` it prepends `[id] ` (and TTY-rewrite mode `\r\x1b[2K` carries the prefix inside the clear-to-EOL escape); `emitJson` is intentionally NOT prefixed so NDJSON consumers don't choke on a `[id] {...}` shape.
@@ -383,7 +383,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment)
- `src/core/destructive-guard.ts` — three-layer protection against accidental data loss. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via `sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`. Page-level analog: `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real.
- `src/commands/pages.ts` — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations.
-- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred.
+- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on `jsonb_typeof(completed_keys) = 'array'` so a non-array (scalar) parent row can't make `jsonb_array_elements_text` throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_array` CHECK (`jsonb_typeof(completed_keys) = 'array'`) — a DB-enforced, always-on guard that makes the scalar-corruption class structurally impossible going forward; the migration repairs any pre-existing scalar to `'[]'` under `LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE` and `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` ship the same CHECK on fresh installs (a loader hit now implies schema drift, a disabled constraint, or an out-of-band writer). `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean, and the scalar-parent guard). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred.
- `src/core/brain-score-recommendations.ts` — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (references stable ids, not check names — so plan order is reproducible). `classifyChecks(report)` triages every doctor check three-state into `remediable | human_only | blocked` (`human_only` covers RLS warnings and other human-judgment gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty/under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` (synthesize/patterns/consolidate) and `embedding-pricing.ts` (embed jobs). Pinned by `test/brain-score-recommendations.test.ts` (~27 cases incl. determinism, content-hash idempotency, DB-backed checkpoint provenance, three-state triage).
- `src/commands/doctor.ts` extension — `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`.
- `src/commands/jobs.ts` extension — registers 11 Minion handlers: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. Phase wrappers delegate to `runCycle({phases:[name]})` so `src/core/cycle.ts` stays the single source of truth for phase semantics. The standalone `sync` handler passes `noExtract: true` to match `runPhaseSync`'s contract (doctor's remediation plan emitting `[sync, extract]` would otherwise double-extract).
diff --git a/package.json b/package.json
index 4dd05db81..82b43c3bc 100644
--- a/package.json
+++ b/package.json
@@ -143,5 +143,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
- "version": "0.42.50.0"
+ "version": "0.42.51.0"
}
diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts
index f27772ac8..f931b5486 100644
--- a/src/commands/doctor.ts
+++ b/src/commands/doctor.ts
@@ -3394,6 +3394,40 @@ export async function checkSyncFreshness(
let hasWarnings = false;
let hasFailures = false;
+ // BUG 4 (v0.42.x): a source with a LIVE, non-expired per-source sync lock is
+ // actively syncing RIGHT NOW — it must not read as stale or never-synced.
+ // The live lock is the only honest "in progress" signal. Checkpoint banking
+ // is NOT usable: a blocked sync banks the good files then writes no anchor
+ // (test/sync-resumable-import.serial.test.ts), so banking can't tell
+ // in-progress from wedged. A blocked/failed sync's process has exited (no
+ // lock row) and a wedged holder stops refreshing (TTL lapses), so either
+ // correctly falls through to the stale path and is NEVER masked. Same
+ // dynamic import as the stale_locks check; any throw (stub engine in unit
+ // tests, pre-lock-table brain) is swallowed to false, so this can only ADD
+ // an in-progress verdict, never suppress a real stale one.
+ // Notes for sources caught actively syncing (surfaced in the result
+ // message so the operator sees "in progress", not just a silent healthy
+ // bucket). Empty when nothing is syncing — keeps the steady-state messages
+ // byte-for-byte unchanged.
+ const inProgress: string[] = [];
+ let liveSyncSnap: (sourceId: string) => Promise<{ holder_pid: number; holder_host: string } | null> =
+ async () => null;
+ try {
+ const { inspectLock, syncLockId } = await import('../core/db-lock.ts');
+ liveSyncSnap = async (sourceId: string) => {
+ try {
+ const snap = await inspectLock(engine, syncLockId(sourceId));
+ return snap && !snap.ttl_expired
+ ? { holder_pid: snap.holder_pid, holder_host: snap.holder_host }
+ : null;
+ } catch {
+ return null;
+ }
+ };
+ } catch {
+ /* db-lock unavailable — skip in-progress detection, staleness stands. */
+ }
+
for (const source of sources) {
// Embed source.id in user-visible messages so `gbrain sync --source `
// matches what the user copy-pastes. Show display name in parens when set.
@@ -3401,6 +3435,15 @@ export async function checkSyncFreshness(
? `'${source.id}' (${source.name})`
: `'${source.id}'`;
+ // BUG 4: actively syncing (live lock) → healthy, count as synced_recently
+ // and skip the staleness checks. Keeps the 3-bucket invariant intact.
+ const liveSnap = await liveSyncSnap(source.id);
+ if (liveSnap) {
+ inProgress.push(`${display} sync in progress (pid ${liveSnap.holder_pid} on ${liveSnap.holder_host})`);
+ synced_recently_count++;
+ continue;
+ }
+
if (!source.last_sync_at) {
issues.push(`Source ${display} has never been synced`);
hasFailures = true;
@@ -3486,12 +3529,15 @@ export async function checkSyncFreshness(
// D6 invariant: every source incremented exactly one bucket.
const details = { unchanged_count, synced_recently_count, stale_count };
+ // BUG 4: append in-progress context when any source is actively syncing.
+ // Empty otherwise, so steady-state messages are byte-for-byte unchanged.
+ const inProgressNote = inProgress.length ? `. ${inProgress.join('; ')}` : '';
if (hasFailures) {
return {
name: 'sync_freshness',
status: 'fail',
- message: `${issues.join('; ')}. Run \`gbrain sync --source \` for each stale source`,
+ message: `${issues.join('; ')}. Run \`gbrain sync --source \` for each stale source${inProgressNote}`,
details,
};
}
@@ -3499,7 +3545,7 @@ export async function checkSyncFreshness(
return {
name: 'sync_freshness',
status: 'warn',
- message: `${issues.join('; ')}. Run \`gbrain sync --source \` to refresh`,
+ message: `${issues.join('; ')}. Run \`gbrain sync --source \` to refresh${inProgressNote}`,
details,
};
}
@@ -3510,7 +3556,7 @@ export async function checkSyncFreshness(
return {
name: 'sync_freshness',
status: 'ok',
- message: `All ${sources.length} federated source(s) up to date (no new commits since last sync)`,
+ message: `All ${sources.length} federated source(s) up to date (no new commits since last sync)${inProgressNote}`,
details,
};
}
@@ -3518,14 +3564,14 @@ export async function checkSyncFreshness(
return {
name: 'sync_freshness',
status: 'ok',
- message: `${sources.length} federated source(s): ${synced_recently_count} synced recently, ${unchanged_count} unchanged since last sync`,
+ message: `${sources.length} federated source(s): ${synced_recently_count} synced recently, ${unchanged_count} unchanged since last sync${inProgressNote}`,
details,
};
}
return {
name: 'sync_freshness',
status: 'ok',
- message: `All ${sources.length} federated source(s) synced recently`,
+ message: `All ${sources.length} federated source(s) synced recently${inProgressNote}`,
details,
};
} catch (e) {
diff --git a/src/commands/sync.ts b/src/commands/sync.ts
index d788cd201..856900bf0 100644
--- a/src/commands/sync.ts
+++ b/src/commands/sync.ts
@@ -1224,7 +1224,7 @@ async function formatLockBusyMessage(engine: BrainEngine, lockKey: string): Prom
* with another break-lock or with TTL-eviction can't produce confusing
* post-conditions.
*/
-async function runBreakLock(
+export async function runBreakLock(
engine: BrainEngine,
lockKey: string,
sourceId: string,
@@ -1243,6 +1243,25 @@ async function runBreakLock(
}
if (!snap) {
+ // BUG 5 (v0.42.x): --force-break-lock used to emit the same terse "not
+ // held" line and exit 0 even when a sync was genuinely wedged — sending the
+ // operator down a dead end (the wedge was not a held lock). Keep rc=0
+ // (breaking a non-existent lock is idempotently successful; flipping the
+ // exit code would break automation that treats it as success), but under
+ // --force say plainly that nothing was broken and point at the real next
+ // step. The non-force path message is unchanged.
+ if (opts.force) {
+ const wedgeHint =
+ `No lock is held on ${lockKey} — nothing to break. If a sync still ` +
+ `appears wedged, the cause is not a held lock; inspect checkpoint/resume ` +
+ `state with \`gbrain sync --source ${sourceId}\` or \`gbrain doctor\`.`;
+ if (opts.json) {
+ console.log(JSON.stringify({ status: 'absent', lock: lockKey, source_id: sourceId, wedge_hint: wedgeHint }));
+ } else {
+ console.log(wedgeHint);
+ }
+ return 0;
+ }
if (opts.json) console.log(JSON.stringify({ status: 'absent', lock: lockKey, source_id: sourceId }));
else console.log(`Lock ${lockKey} is not held (nothing to break).`);
return 0;
diff --git a/src/core/migrate.ts b/src/core/migrate.ts
index 019f02ea4..acc57aec2 100644
--- a/src/core/migrate.ts
+++ b/src/core/migrate.ts
@@ -5270,6 +5270,103 @@ export const MIGRATIONS: Migration[] = [
ON context_volunteer_events (source_id, slug);
`,
},
+ {
+ version: 118,
+ name: 'page_generation_clock_sequence_swap',
+ // v0.42.x — contention-free page-generation clock. The v107 single-row
+ // `UPDATE page_generation_clock SET value = value + 1 WHERE id = 1` took a
+ // transaction-length RowExclusiveLock on one tuple, serializing every
+ // concurrent page writer on the prior writer's COMMIT (sync ran at ~0.8
+ // cores regardless of worker count). Swap to a SEQUENCE: nextval() takes a
+ // microsecond LWLock, never a row lock. The Layer-1 cache bookmark reads
+ // `last_value` instead of the row.
+ //
+ // Correctness: `last_value` is non-transactional — it can reflect
+ // rolled-back or concurrent-uncommitted writers. That is the SAFE direction
+ // (cache OVER-invalidates, never serves stale). The clock's only contract is
+ // monotonic advancement on any page INSERT/UPDATE/DELETE.
+ //
+ // The setval is LOAD-BEARING: a fresh CREATE SEQUENCE has is_called=false,
+ // so the first nextval() returns the start value (1) and last_value would
+ // not visibly advance. The 2-arg setval (is_called=true) makes the first
+ // post-seed write strictly exceed the seed. Floor 1 (sequence MINVALUE);
+ // seed >= old clock and MAX(pages.generation) so monotonicity holds.
+ //
+ // We keep the table + trigger + function NAMES; only the function body and
+ // the three readers in query-cache-gate.ts repoint. DELETE FROM query_cache
+ // so no bookmark stamped under the old table-clock survives the swap.
+ // Mirrors: src/schema.sql, src/core/pglite-schema.ts (and the generated
+ // src/core/schema-embedded.ts) ship the sequence on fresh install.
+ //
+ // pages.generation (Layer 2) is assigned by the SEPARATE row-level trigger
+ // bump_page_generation_fn — untouched here.
+ idempotent: true,
+ sql: `
+ CREATE SEQUENCE IF NOT EXISTS page_generation_clock_seq;
+
+ SELECT setval('page_generation_clock_seq', GREATEST(
+ 1,
+ COALESCE((SELECT last_value FROM page_generation_clock_seq), 0),
+ COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0),
+ COALESCE((SELECT MAX(generation) FROM pages), 0)
+ ));
+
+ CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$
+ BEGIN
+ PERFORM nextval('page_generation_clock_seq');
+ RETURN NULL;
+ END;
+ $func$ LANGUAGE plpgsql;
+
+ DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages;
+ CREATE TRIGGER bump_page_generation_clock_trg
+ AFTER INSERT OR UPDATE OR DELETE ON pages
+ FOR EACH STATEMENT
+ EXECUTE FUNCTION bump_page_generation_clock_fn();
+
+ DELETE FROM query_cache;
+ `,
+ },
+ {
+ version: 119,
+ name: 'op_checkpoints_completed_keys_array_check',
+ // v0.42.x — make the op_checkpoints scalar-corruption class structurally
+ // impossible. completed_keys is JSONB and the loader runs
+ // jsonb_array_elements_text(completed_keys); a non-array (scalar) value
+ // makes that throw "cannot extract elements from a scalar", which takes
+ // down the whole UNION load (including the valid op_checkpoint_paths child
+ // rows) and loses all checkpoint progress for that key. No current writer
+ // can produce a scalar, but an older binary / external script / future bug
+ // could — the CHECK is a DB-enforced, always-on guard (the correct pattern
+ // vs a migration verify-hook, which would not run on already-stamped
+ // brains). LOCK first so an out-of-band scalar write can't land between the
+ // repair and the ADD CONSTRAINT (no-op on single-connection PGLite). The
+ // repair resets any pre-existing scalar to '[]'; op_checkpoint_paths child
+ // rows are the append-only source of truth, so the reset loses nothing.
+ // Mirrored in src/schema.sql, src/core/pglite-schema.ts, and the generated
+ // src/core/schema-embedded.ts so fresh installs carry the same CHECK.
+ idempotent: true,
+ sql: `
+ LOCK TABLE op_checkpoints IN SHARE ROW EXCLUSIVE MODE;
+
+ UPDATE op_checkpoints
+ SET completed_keys = '[]'::jsonb, updated_at = now()
+ WHERE jsonb_typeof(completed_keys) <> 'array';
+
+ DO $$
+ BEGIN
+ IF NOT EXISTS (
+ SELECT 1 FROM pg_constraint
+ WHERE conname = 'op_checkpoints_completed_keys_array'
+ AND conrelid = 'op_checkpoints'::regclass
+ ) THEN
+ ALTER TABLE op_checkpoints
+ ADD CONSTRAINT op_checkpoints_completed_keys_array
+ CHECK (jsonb_typeof(completed_keys) = 'array');
+ END IF;
+ END $$;
+ `,
+ },
];
export const LATEST_VERSION = MIGRATIONS.length > 0
diff --git a/src/core/op-checkpoint.ts b/src/core/op-checkpoint.ts
index a338d20b6..f1c0c1e2e 100644
--- a/src/core/op-checkpoint.ts
+++ b/src/core/op-checkpoint.ts
@@ -122,18 +122,40 @@ export async function loadOpCheckpoint(
// dedupes, so we skip a server-side dedup sort over up to 204K rows on every
// resume. `jsonb_array_elements_text` expands the legacy array server-side,
// which also removes the old postgres.js-vs-PGLite string/array handling.
- const rows = await engine.executeRaw<{ ckey: unknown }>(
- `SELECT path AS ckey FROM op_checkpoint_paths
+ //
+ // v0.42.x (BUG 3 guard): the legacy arm is gated on
+ // `jsonb_typeof(completed_keys) = 'array'`. Without it a non-array (scalar)
+ // parent row makes jsonb_array_elements_text throw "cannot extract elements
+ // from a scalar", which kills the WHOLE union — including the valid child
+ // rows — and loses all checkpoint progress for the key. Skipping the scalar
+ // keeps the child rows; the third arm flags the corruption so we log it once
+ // (migration v119's CHECK makes this impossible going forward; a hit implies
+ // schema drift / disabled constraint / an out-of-band writer).
+ const rows = await engine.executeRaw<{ ckey: unknown; corrupt: number }>(
+ `SELECT path AS ckey, 0 AS corrupt FROM op_checkpoint_paths
WHERE op = $1 AND fingerprint = $2
UNION ALL
- SELECT jsonb_array_elements_text(completed_keys) AS ckey FROM op_checkpoints
- WHERE op = $1 AND fingerprint = $2`,
+ SELECT jsonb_array_elements_text(completed_keys) AS ckey, 0 AS corrupt FROM op_checkpoints
+ WHERE op = $1 AND fingerprint = $2 AND jsonb_typeof(completed_keys) = 'array'
+ UNION ALL
+ SELECT NULL AS ckey, 1 AS corrupt FROM op_checkpoints
+ WHERE op = $1 AND fingerprint = $2 AND jsonb_typeof(completed_keys) <> 'array'`,
[key.op, key.fingerprint],
);
const set = new Set();
+ let corruptParent = false;
for (const r of rows) {
+ if (Number(r.corrupt) === 1) {
+ corruptParent = true;
+ continue;
+ }
if (typeof r.ckey === 'string') set.add(r.ckey);
}
+ if (corruptParent) {
+ console.error(
+ `[op-checkpoint] WARNING: op_checkpoints.completed_keys for (${key.op}, ${key.fingerprint}) is a non-array (scalar) and was skipped to protect the load — child op_checkpoint_paths rows still applied. This implies schema drift, a disabled CHECK constraint, or an out-of-band writer.`,
+ );
+ }
return [...set];
} catch (e) {
console.error(`[op-checkpoint] load failed (${key.op}, ${key.fingerprint}):`, (e as Error).message);
diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts
index b0d7da947..1a0bc6c19 100644
--- a/src/core/pglite-schema.ts
+++ b/src/core/pglite-schema.ts
@@ -164,9 +164,24 @@ INSERT INTO page_generation_clock (id, value)
VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0))
ON CONFLICT (id) DO NOTHING;
+-- v0.42.x: contention-free clock. nextval() takes a microsecond LWLock, not a
+-- transaction-length row lock. Load-bearing setval (2-arg -> is_called=true) so
+-- the first write strictly exceeds the seed; floor 1 (sequence MINVALUE).
+-- Layer-1 reads last_value. Table + trigger names retained.
+CREATE SEQUENCE IF NOT EXISTS page_generation_clock_seq;
+-- Monotonic seed: GREATEST over the sequence's OWN last_value too, so replaying
+-- this blob on an already-upgraded brain (initSchema is re-runnable) can never
+-- move last_value BACKWARD below a stored query_cache bookmark.
+SELECT setval('page_generation_clock_seq', GREATEST(
+ 1,
+ COALESCE((SELECT last_value FROM page_generation_clock_seq), 0),
+ COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0),
+ COALESCE((SELECT MAX(generation) FROM pages), 0)
+));
+
CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$
BEGIN
- UPDATE page_generation_clock SET value = value + 1 WHERE id = 1;
+ PERFORM nextval('page_generation_clock_seq');
RETURN NULL;
END;
$func$ LANGUAGE plpgsql;
@@ -924,7 +939,11 @@ CREATE TABLE IF NOT EXISTS oauth_codes (
CREATE TABLE IF NOT EXISTS op_checkpoints (
op TEXT NOT NULL,
fingerprint TEXT NOT NULL,
- completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb,
+ -- v0.42.x: must be a JSONB array. The loader runs jsonb_array_elements_text
+ -- over it; a scalar would throw and wipe the whole checkpoint load. CHECK is
+ -- the DB-enforced always-on guard (mirrors migration v119).
+ completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb
+ CONSTRAINT op_checkpoints_completed_keys_array CHECK (jsonb_typeof(completed_keys) = 'array'),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (op, fingerprint)
);
diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts
index e0f313dc8..32beb37d0 100644
--- a/src/core/schema-embedded.ts
+++ b/src/core/schema-embedded.ts
@@ -227,9 +227,26 @@ INSERT INTO page_generation_clock (id, value)
VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0))
ON CONFLICT (id) DO NOTHING;
+-- v0.42.x: contention-free clock. nextval() takes a microsecond LWLock, not a
+-- transaction-length row lock, so concurrent page writers no longer serialize
+-- on one tuple's COMMIT. Load-bearing setval (2-arg -> is_called=true) so the
+-- first write strictly exceeds the seed; floor 1 (sequence MINVALUE). Layer-1
+-- reads last_value. The table + trigger names are retained.
+CREATE SEQUENCE IF NOT EXISTS page_generation_clock_seq;
+-- Monotonic seed: GREATEST over the sequence's OWN last_value too, so replaying
+-- this blob on an already-upgraded brain (initSchema is re-runnable) can never
+-- move last_value BACKWARD below a stored query_cache bookmark (which would let
+-- Layer 1 serve stale rows). Mirrors the old table's ON CONFLICT DO NOTHING.
+SELECT setval('page_generation_clock_seq', GREATEST(
+ 1,
+ COALESCE((SELECT last_value FROM page_generation_clock_seq), 0),
+ COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0),
+ COALESCE((SELECT MAX(generation) FROM pages), 0)
+));
+
CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS \$func\$
BEGIN
- UPDATE page_generation_clock SET value = value + 1 WHERE id = 1;
+ PERFORM nextval('page_generation_clock_seq');
RETURN NULL;
END;
\$func\$ LANGUAGE plpgsql;
@@ -689,7 +706,11 @@ CREATE INDEX IF NOT EXISTS idx_mcp_log_agent_time ON mcp_request_log(agent_name,
CREATE TABLE IF NOT EXISTS op_checkpoints (
op TEXT NOT NULL,
fingerprint TEXT NOT NULL,
- completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb,
+ -- v0.42.x: must be a JSONB array. The loader runs jsonb_array_elements_text
+ -- over it; a scalar would throw and wipe the whole checkpoint load. CHECK is
+ -- the DB-enforced always-on guard (mirrors migration v119).
+ completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb
+ CONSTRAINT op_checkpoints_completed_keys_array CHECK (jsonb_typeof(completed_keys) = 'array'),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (op, fingerprint)
);
diff --git a/src/core/search/query-cache-gate.ts b/src/core/search/query-cache-gate.ts
index be2ec3607..6b72bd310 100644
--- a/src/core/search/query-cache-gate.ts
+++ b/src/core/search/query-cache-gate.ts
@@ -5,9 +5,14 @@
* Two pure helpers wired by query-cache.ts at store + lookup time. Pure
* surface lets us unit-test the two-layer gate logic without a real cache.
*
- * Layer 1 (cheap bookmark): `page_generation_clock.value` <=
+ * Layer 1 (cheap bookmark): `page_generation_clock_seq` last_value <=
* `query_cache.max_generation_at_store`. If true, no page write has
* happened since this row stored, so the row is fresh corpus-wide.
+ * v0.42.x: the bookmark source switched from a locked single-row counter
+ * (`page_generation_clock.value`) to a contention-free SEQUENCE bumped by
+ * `nextval()` in the statement trigger. `last_value` is non-transactional —
+ * it can reflect a rolled-back or concurrent-uncommitted writer, which only
+ * ever OVER-invalidates (loses a cache hit), never serves stale.
*
* Layer 2 (per-page snapshot): if bookmark fires, fall through to the
* `page_generations JSONB` snapshot. For each `(page_id, stored_gen)`
@@ -98,7 +103,7 @@ export async function buildPageGenerationsSnapshot(
// Per D20, empty-result cache rows trust Layer 1 exclusively;
// bumping the clock on subsequent writes correctly invalidates them.
const rows = await engine.executeRaw<{ v: number }>(
- `SELECT COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v`,
+ `SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`,
);
snapshot.max_generation_at_store = Number(rows[0]?.v ?? 0);
return snapshot;
@@ -117,7 +122,7 @@ export async function buildPageGenerationsSnapshot(
FROM pages WHERE id = ANY($1::int[])
UNION ALL
SELECT 'CLOCK' AS k,
- COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v,
+ COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v,
TRUE AS is_max`,
[pageIds],
);
@@ -132,12 +137,12 @@ export async function buildPageGenerationsSnapshot(
}
return snapshot;
} catch {
- // Pre-v105 brain (no `page_generation_clock` table yet). Return the
- // empty snapshot with zero bookmark — every cache row will fall
- // through to Layer 2 (which is stricter post-v0.41.19.0 and will
- // invalidate empty snapshots). Acceptable upgrade-path one-time
- // cache miss; migration v105 fills the table within the same
- // initSchema() call so this branch is short-lived.
+ // Pre-sequence brain mid-upgrade (no `page_generation_clock_seq` yet).
+ // Return the empty snapshot with zero bookmark — every cache row will
+ // fall through to Layer 2 (which is stricter post-v0.41.19.0 and will
+ // invalidate empty snapshots). Acceptable upgrade-path one-time cache
+ // miss; migration v118 creates the sequence within the same initSchema()
+ // call so this branch is short-lived.
return snapshot;
}
}
@@ -157,11 +162,12 @@ export async function buildPageGenerationsSnapshot(
*/
export const CACHE_GATE_WHERE_CLAUSE = `
(
- -- Layer 1 (cheap bookmark): O(1) single-row read from page_generation_clock.
- -- Bumped per-statement by bump_page_generation_clock_trg on every INSERT,
- -- UPDATE, or DELETE on pages. If no statement has fired since this row
- -- stored, the row is fresh corpus-wide.
- COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)
+ -- Layer 1 (cheap bookmark): O(1) read of page_generation_clock_seq.last_value.
+ -- The sequence is advanced (nextval) per-statement by bump_page_generation_clock_trg
+ -- on every INSERT, UPDATE, or DELETE on pages. If no statement has fired
+ -- since this row stored, last_value is unchanged and the row is fresh
+ -- corpus-wide. Non-transactional read: only ever over-invalidates.
+ COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)
<= qc.max_generation_at_store
OR
-- Layer 2 (per-page snapshot): bookmark fired, but maybe THIS row's
diff --git a/src/schema.sql b/src/schema.sql
index d05b45117..dcf9455e1 100644
--- a/src/schema.sql
+++ b/src/schema.sql
@@ -223,9 +223,26 @@ INSERT INTO page_generation_clock (id, value)
VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0))
ON CONFLICT (id) DO NOTHING;
+-- v0.42.x: contention-free clock. nextval() takes a microsecond LWLock, not a
+-- transaction-length row lock, so concurrent page writers no longer serialize
+-- on one tuple's COMMIT. Load-bearing setval (2-arg -> is_called=true) so the
+-- first write strictly exceeds the seed; floor 1 (sequence MINVALUE). Layer-1
+-- reads last_value. The table + trigger names are retained.
+CREATE SEQUENCE IF NOT EXISTS page_generation_clock_seq;
+-- Monotonic seed: GREATEST over the sequence's OWN last_value too, so replaying
+-- this blob on an already-upgraded brain (initSchema is re-runnable) can never
+-- move last_value BACKWARD below a stored query_cache bookmark (which would let
+-- Layer 1 serve stale rows). Mirrors the old table's ON CONFLICT DO NOTHING.
+SELECT setval('page_generation_clock_seq', GREATEST(
+ 1,
+ COALESCE((SELECT last_value FROM page_generation_clock_seq), 0),
+ COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0),
+ COALESCE((SELECT MAX(generation) FROM pages), 0)
+));
+
CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$
BEGIN
- UPDATE page_generation_clock SET value = value + 1 WHERE id = 1;
+ PERFORM nextval('page_generation_clock_seq');
RETURN NULL;
END;
$func$ LANGUAGE plpgsql;
@@ -685,7 +702,11 @@ CREATE INDEX IF NOT EXISTS idx_mcp_log_agent_time ON mcp_request_log(agent_name,
CREATE TABLE IF NOT EXISTS op_checkpoints (
op TEXT NOT NULL,
fingerprint TEXT NOT NULL,
- completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb,
+ -- v0.42.x: must be a JSONB array. The loader runs jsonb_array_elements_text
+ -- over it; a scalar would throw and wipe the whole checkpoint load. CHECK is
+ -- the DB-enforced always-on guard (mirrors migration v119).
+ completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb
+ CONSTRAINT op_checkpoints_completed_keys_array CHECK (jsonb_typeof(completed_keys) = 'array'),
updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (op, fingerprint)
);
diff --git a/test/conversation-parser/llm-base.test.ts b/test/conversation-parser/llm-base.test.ts
index c09ececff..ad51cbb8c 100644
--- a/test/conversation-parser/llm-base.test.ts
+++ b/test/conversation-parser/llm-base.test.ts
@@ -12,7 +12,7 @@
*/
import { describe, expect, test, beforeEach } from 'bun:test';
-import { withEnv } from '../helpers/with-env.ts';
+import { withEnv, emptyHome } from '../helpers/with-env.ts';
import {
runLlmCall,
parseLlmJson,
@@ -29,7 +29,7 @@ beforeEach(() => {
describe('probeLlmAvailability', () => {
test('returns null when ANTHROPIC_API_KEY is unset', async () => {
await withEnv(
- { ANTHROPIC_API_KEY: undefined as unknown as string },
+ { ANTHROPIC_API_KEY: undefined as unknown as string, GBRAIN_HOME: emptyHome() },
async () => {
expect(probeLlmAvailability('claude-haiku-4-5')).toBeNull();
expect(probeLlmAvailability('anthropic:claude-haiku-4-5')).toBeNull();
@@ -94,7 +94,7 @@ describe('runLlmCall — happy path', () => {
describe('runLlmCall — fail-open paths', () => {
test('provider unavailable returns null without calling transport', async () => {
await withEnv(
- { ANTHROPIC_API_KEY: undefined as unknown as string },
+ { ANTHROPIC_API_KEY: undefined as unknown as string, GBRAIN_HOME: emptyHome() },
async () => {
let calls = 0;
const result = await runLlmCall({
diff --git a/test/conversation-parser/llm-fallback.test.ts b/test/conversation-parser/llm-fallback.test.ts
index cebbed249..b7121643a 100644
--- a/test/conversation-parser/llm-fallback.test.ts
+++ b/test/conversation-parser/llm-fallback.test.ts
@@ -12,7 +12,7 @@
*/
import { describe, expect, test, beforeEach } from 'bun:test';
-import { withEnv } from '../helpers/with-env.ts';
+import { withEnv, emptyHome } from '../helpers/with-env.ts';
import { runLlmFallback } from '../../src/core/conversation-parser/llm-fallback.ts';
import { _resetLlmCacheForTests } from '../../src/core/conversation-parser/llm-base.ts';
import { makeChatResult } from './helpers.ts';
@@ -71,7 +71,7 @@ describe('runLlmFallback', () => {
test('provider unavailable: returns null without calling transport', async () => {
await withEnv(
- { ANTHROPIC_API_KEY: undefined as unknown as string },
+ { ANTHROPIC_API_KEY: undefined as unknown as string, GBRAIN_HOME: emptyHome() },
async () => {
let calls = 0;
const result = await runLlmFallback({
diff --git a/test/doctor-ze-checks.test.ts b/test/doctor-ze-checks.test.ts
index d98059e3b..aebf5d33c 100644
--- a/test/doctor-ze-checks.test.ts
+++ b/test/doctor-ze-checks.test.ts
@@ -11,7 +11,7 @@
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
-import { withEnv } from './helpers/with-env.ts';
+import { withEnv, emptyHome } from './helpers/with-env.ts';
import {
checkZeEmbeddingHealth,
checkEmbeddingWidthConsistency,
@@ -58,8 +58,10 @@ describe('checkZeEmbeddingHealth', () => {
embedding_dimensions: 1280,
env: { ...process.env, ZEROENTROPY_API_KEY: undefined as any },
});
- // Clear the env var for the no-key path (user's real env may have it set).
- await withEnv({ ZEROENTROPY_API_KEY: undefined }, async () => {
+ // Clear the env var AND isolate GBRAIN_HOME for the no-key path: the check
+ // reads ZEROENTROPY_API_KEY from env OR the gbrain config file, so a dev
+ // machine whose real ~/.gbrain/config.json holds the key needs both cleared.
+ await withEnv({ ZEROENTROPY_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => {
const check = await checkZeEmbeddingHealth(engine);
expect(check.status).toBe('warn');
expect(check.message).toContain('ZEROENTROPY_API_KEY');
diff --git a/test/doctor.test.ts b/test/doctor.test.ts
index fab1201d0..5b68a6931 100644
--- a/test/doctor.test.ts
+++ b/test/doctor.test.ts
@@ -1330,3 +1330,114 @@ describe('v0.42 (#1699) — quarantined_pages + flagged_pages checks', () => {
expect(source).toMatch(/name: 'flagged_pages'/);
});
});
+
+// ============================================================================
+// BUG 4 (v0.42.x) — doctor reports an actively-running sync via the live lock,
+// not stale freshness. Uses a REAL PGLiteEngine so inspectLock/syncLockId run
+// against actual gbrain_cycle_locks rows (the stub engine can't model a lock).
+// ============================================================================
+describe('BUG 4 — in-progress sync via live lock, not stale freshness', () => {
+ let engine: any;
+ let syncLockId: (s: string) => string;
+
+ beforeAll(async () => {
+ const { PGLiteEngine } = await import('../src/core/pglite-engine.ts');
+ ({ syncLockId } = await import('../src/core/db-lock.ts'));
+ engine = new PGLiteEngine();
+ await engine.connect({});
+ await engine.initSchema();
+ });
+
+ afterAll(async () => {
+ await engine.disconnect();
+ });
+
+ beforeEach(async () => {
+ const { resetPgliteState } = await import('./helpers/reset-pglite.ts');
+ await resetPgliteState(engine);
+ await engine.executeRaw(`DELETE FROM gbrain_cycle_locks`);
+ });
+
+ const staleDate = () => new Date(Date.now() - 5 * 24 * 60 * 60 * 1000); // 5d ago
+
+ async function addSource(id: string, lastSyncAt: Date | null) {
+ await engine.executeRaw(
+ `INSERT INTO sources (id, name, local_path, last_sync_at, config)
+ VALUES ($1, $1, $2, $3, '{"federated":true}'::jsonb)
+ ON CONFLICT (id) DO UPDATE SET last_sync_at = EXCLUDED.last_sync_at`,
+ [id, `/tmp/${id}`, lastSyncAt],
+ );
+ }
+
+ // ttlMinutes > 0 → live lock; <= 0 → already-expired (wedged) holder.
+ async function holdLock(sourceId: string, ttlMinutes: number) {
+ await engine.executeRaw(
+ `INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at)
+ VALUES ($1, 4242, 'testhost', now(), now() + ($2 || ' minutes')::interval, now())`,
+ [syncLockId(sourceId), String(ttlMinutes)],
+ );
+ }
+
+ test('stale source with NO live lock → fail (blocked/wedged is not masked)', async () => {
+ const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
+ await addSource('wiki', staleDate());
+ const result = await checkSyncFreshness(engine);
+ expect(result.status).toBe('fail');
+ expect(result.message).toContain(`'wiki'`);
+ });
+
+ test('stale source WITH a live (non-expired) lock → ok (sync in progress)', async () => {
+ const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
+ await addSource('wiki', staleDate());
+ await holdLock('wiki', 30);
+ const result = await checkSyncFreshness(engine);
+ expect(result.status).toBe('ok');
+ expect(result.details?.synced_recently_count).toBe(1);
+ expect(result.details?.stale_count).toBe(0);
+ // BUG 4: operator sees the in-progress holder, not silence.
+ expect(result.message).toContain('sync in progress');
+ expect(result.message).toContain('pid 4242');
+ });
+
+ test('never-synced source WITH a live lock → ok (initial sync in progress)', async () => {
+ const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
+ await addSource('wiki', null);
+ await holdLock('wiki', 30);
+ const result = await checkSyncFreshness(engine);
+ expect(result.status).toBe('ok');
+ expect(result.details?.synced_recently_count).toBe(1);
+ expect(result.message).toContain('sync in progress');
+ });
+
+ test('never-synced source with NO lock → fail (unchanged behavior)', async () => {
+ const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
+ await addSource('wiki', null);
+ const result = await checkSyncFreshness(engine);
+ expect(result.status).toBe('fail');
+ expect(result.message).toContain('never been synced');
+ });
+
+ test('expired-TTL lock does NOT mask staleness (wedged-but-not-refreshing holder)', async () => {
+ const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
+ await addSource('wiki', staleDate());
+ await holdLock('wiki', -5); // ttl_expires_at 5 min in the past
+ const result = await checkSyncFreshness(engine);
+ expect(result.status).toBe('fail');
+ });
+
+ test('blocked source with banked checkpoint rows but NO live lock → still fail', async () => {
+ const { checkSyncFreshness } = await import('../src/commands/doctor.ts');
+ await addSource('wiki', staleDate());
+ // A blocked sync banks the good files then exits without an anchor and
+ // without a held lock. Banking must NOT be read as "in progress".
+ await engine.executeRaw(
+ `INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at)
+ VALUES ('sync', 'fp-blocked', '[]'::jsonb, now())`,
+ );
+ await engine.executeRaw(
+ `INSERT INTO op_checkpoint_paths (op, fingerprint, path) VALUES ('sync', 'fp-blocked', 'banked.md')`,
+ );
+ const result = await checkSyncFreshness(engine);
+ expect(result.status).toBe('fail');
+ });
+});
diff --git a/test/helpers/with-env.ts b/test/helpers/with-env.ts
index f4c877f51..5b2253c81 100644
--- a/test/helpers/with-env.ts
+++ b/test/helpers/with-env.ts
@@ -1,3 +1,7 @@
+import { mkdtempSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+
/**
* Run a callback with `process.env` mutations applied, then restore the prior
* values via try/finally. The canonical pattern for env-touching tests in this
@@ -69,3 +73,17 @@ export async function withEnv(
}
}
}
+
+/**
+ * A fresh empty temp dir for `GBRAIN_HOME`, so `loadConfig()` / `configDir()`
+ * resolve to a directory with no config.json. Pair with a `withEnv` override
+ * (`GBRAIN_HOME: emptyHome()`) on any "no key" assertion: `hasAnthropicKey()`
+ * and the ZE/embedding key probes read BOTH the env var AND the gbrain config
+ * file, so clearing only the env var is NOT hermetic on a dev machine whose
+ * real `~/.gbrain/config.json` holds a key — the assertion flips and the test
+ * fails locally while passing in key-less CI. The dir is tiny and intentionally
+ * leaked (test process is short-lived); the OS reaps tmp.
+ */
+export function emptyHome(): string {
+ return mkdtempSync(join(tmpdir(), 'gbrain-nokey-home-'));
+}
diff --git a/test/op-checkpoint.test.ts b/test/op-checkpoint.test.ts
index 6c821e38c..b5fe46127 100644
--- a/test/op-checkpoint.test.ts
+++ b/test/op-checkpoint.test.ts
@@ -243,6 +243,82 @@ describe('resumeFilter (pure)', () => {
});
});
+describe('BUG 3: completed_keys array-shape guard (v119 CHECK + defensive loader)', () => {
+ const CONSTRAINT = 'op_checkpoints_completed_keys_array';
+
+ test('CHECK rejects a scalar completed_keys write — exactly one constraint (no blob+migration dupe)', async () => {
+ // Fresh PGLite install: the schema blob ships the NAMED inline CHECK and
+ // migration v119's IF NOT EXISTS skips re-adding it. Exactly one constraint.
+ const c = await engine.executeRaw<{ n: number }>(
+ `SELECT count(*)::int AS n FROM pg_constraint WHERE conname = $1`,
+ [CONSTRAINT],
+ );
+ expect(Number(c[0].n)).toBe(1);
+
+ let threw = false;
+ try {
+ await engine.executeRaw(
+ `INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at)
+ VALUES ('embed', 'fp-reject', '"not-an-array"'::jsonb, now())`,
+ );
+ } catch {
+ threw = true;
+ }
+ expect(threw).toBe(true);
+ });
+
+ test('loader survives a scalar parent: returns child rows (not []), does not throw', async () => {
+ const key = { op: 'sync', fingerprint: 'fp-scalar-survive' };
+ // Bypass the CHECK to simulate pre-migration / out-of-band corruption.
+ await engine.executeRaw(`ALTER TABLE op_checkpoints DROP CONSTRAINT ${CONSTRAINT}`);
+ try {
+ await engine.executeRaw(
+ `INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at)
+ VALUES ('sync', 'fp-scalar-survive', '"corrupt-scalar"'::jsonb, now())`,
+ );
+ await engine.executeRaw(
+ `INSERT INTO op_checkpoint_paths (op, fingerprint, path)
+ VALUES ('sync', 'fp-scalar-survive', 'child-a.md')`,
+ );
+ // Pre-guard, jsonb_array_elements_text on the scalar threw and the catch
+ // returned [] — losing child-a.md. The typeof guard skips the scalar so
+ // the valid child survives.
+ const loaded = await loadOpCheckpoint(engine, key);
+ expect(loaded).toEqual(['child-a.md']);
+ } finally {
+ await engine.executeRaw(
+ `UPDATE op_checkpoints SET completed_keys = '[]'::jsonb WHERE jsonb_typeof(completed_keys) <> 'array'`,
+ );
+ await engine.executeRaw(
+ `ALTER TABLE op_checkpoints ADD CONSTRAINT ${CONSTRAINT} CHECK (jsonb_typeof(completed_keys) = 'array')`,
+ );
+ }
+ });
+
+ test('v119 repair converts a scalar parent to an empty array', async () => {
+ await engine.executeRaw(`ALTER TABLE op_checkpoints DROP CONSTRAINT ${CONSTRAINT}`);
+ try {
+ await engine.executeRaw(
+ `INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at)
+ VALUES ('embed', 'fp-repair', '"scalar"'::jsonb, now())`,
+ );
+ // Migration v119's repair statement.
+ await engine.executeRaw(
+ `UPDATE op_checkpoints SET completed_keys = '[]'::jsonb, updated_at = now()
+ WHERE jsonb_typeof(completed_keys) <> 'array'`,
+ );
+ const typ = await engine.executeRaw<{ t: string }>(
+ `SELECT jsonb_typeof(completed_keys) AS t FROM op_checkpoints WHERE op = 'embed' AND fingerprint = 'fp-repair'`,
+ );
+ expect(typ[0].t).toBe('array');
+ } finally {
+ await engine.executeRaw(
+ `ALTER TABLE op_checkpoints ADD CONSTRAINT ${CONSTRAINT} CHECK (jsonb_typeof(completed_keys) = 'array')`,
+ );
+ }
+ });
+});
+
describe('purgeStaleCheckpoints', () => {
test('no stale rows: returns 0', async () => {
await recordCompleted(engine, { op: 'embed', fingerprint: 'fresh' }, ['x']);
diff --git a/test/page-generation-counter.test.ts b/test/page-generation-counter.test.ts
index 45748f72f..25f57f94b 100644
--- a/test/page-generation-counter.test.ts
+++ b/test/page-generation-counter.test.ts
@@ -44,8 +44,11 @@ beforeEach(async () => {
});
async function clockValue(): Promise {
+ // v0.42.x: the Layer-1 bookmark moved from the locked single-row
+ // page_generation_clock table to a contention-free SEQUENCE bumped by
+ // nextval() in the statement trigger. Read last_value.
const rows = await engine.executeRaw<{ value: number }>(
- `SELECT value FROM page_generation_clock WHERE id = 1`,
+ `SELECT last_value AS value FROM page_generation_clock_seq`,
);
return Number(rows[0]?.value ?? -1);
}
@@ -69,13 +72,20 @@ describe('page_generation_clock table + statement-level trigger', () => {
expect(threw).toBe(true);
});
- test('seed: clock starts at COALESCE(MAX(pages.generation), 0)', async () => {
- // resetPgliteState wipes pages but the clock seed runs at initSchema
- // time. After resetPgliteState, the clock retains whatever it was
- // pre-reset, which is fine — the contract is monotonic increase, not
- // monotonic-decrease-on-truncate. (Production resets don't happen.)
+ test('seed: clock sequence starts at >= 1 with is_called=true', async () => {
+ // v0.42.x: the sequence is seeded via setval(GREATEST(1, MAX(generation)))
+ // at initSchema with is_called=true (2-arg setval), so the FIRST write's
+ // nextval strictly exceeds the seed. Without is_called=true a fresh
+ // sequence's first nextval returns the start value and last_value would not
+ // visibly advance — that would let a fresh install serve a stale cache row.
+ // resetPgliteState does NOT reset the sequence (sequences aren't pg_tables),
+ // so last_value only ever increases — monotonic, never decrease-on-truncate.
const v = await clockValue();
- expect(v).toBeGreaterThanOrEqual(0);
+ expect(v).toBeGreaterThanOrEqual(1);
+ const meta = await engine.executeRaw<{ is_called: boolean }>(
+ `SELECT is_called FROM page_generation_clock_seq`,
+ );
+ expect(meta[0].is_called).toBe(true);
});
test('INSERT bumps clock by exactly 1 (single-row insert via raw SQL)', async () => {
@@ -225,3 +235,81 @@ describe('query-cache integration (D14 + CDX-6 + CDX-7 end-to-end)', () => {
expect(afterClock).toBe(beforeClock + 1);
});
});
+
+describe('v0.42.x sequence-backed clock (BUG 1: contention removal)', () => {
+ test('mechanism: trigger function uses nextval, NOT a locked row UPDATE', async () => {
+ // The contention source was `UPDATE page_generation_clock SET value=value+1
+ // WHERE id=1` (a transaction-length RowExclusiveLock on one tuple). Prove at
+ // the schema level that it is gone and replaced by nextval (a microsecond
+ // LWLock). This is the deterministic, PGLite-runnable contention proof.
+ const rows = await engine.executeRaw<{ src: string }>(
+ `SELECT prosrc AS src FROM pg_proc WHERE proname = 'bump_page_generation_clock_fn'`,
+ );
+ expect(rows.length).toBe(1);
+ expect(rows[0].src).toContain("nextval('page_generation_clock_seq')");
+ expect(rows[0].src).not.toContain('UPDATE page_generation_clock');
+ });
+
+ test('rollback still advances the sequence (over-invalidation is the SAFE direction)', async () => {
+ const before = await clockValue();
+ // Aborted import: the statement trigger fires nextval (sequences are
+ // non-transactional), so last_value advances even though the page never
+ // commits. A cache row stamped before this now fails Layer 1 and
+ // re-validates — a LOST HIT, never a stale serve.
+ try {
+ await engine.transaction(async (tx) => {
+ await tx.executeRaw(
+ `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter)
+ VALUES ('default', 'test/rollback-page', 'note', 't', 'body', '', '{}'::jsonb)`,
+ );
+ throw new Error('abort');
+ });
+ } catch {
+ /* expected */
+ }
+ const after = await clockValue();
+ expect(after).toBeGreaterThan(before);
+ // The page itself did NOT persist — rollback worked.
+ const pages = await engine.executeRaw<{ n: number }>(
+ `SELECT COUNT(*)::int AS n FROM pages WHERE slug = 'test/rollback-page' AND source_id = 'default'`,
+ );
+ expect(Number(pages[0].n)).toBe(0);
+ });
+
+ test('PGLite supports sequences: CREATE / nextval / setval / last_value round-trip', async () => {
+ // codex flagged: no existing sequence usage in the repo — prove (not assert)
+ // that PGLite's WASM Postgres supports the constructs migration v118 relies
+ // on, including the is_called gotcha the load-bearing setval guards against.
+ await engine.executeRaw(`CREATE SEQUENCE IF NOT EXISTS test_probe_seq`);
+ // Fresh sequence (is_called=false): first nextval returns the START value 1,
+ // and last_value does NOT visibly advance past it — the exact trap.
+ const n1 = await engine.executeRaw<{ v: number }>(`SELECT nextval('test_probe_seq') AS v`);
+ expect(Number(n1[0].v)).toBe(1);
+ const n2 = await engine.executeRaw<{ v: number }>(`SELECT nextval('test_probe_seq') AS v`);
+ expect(Number(n2[0].v)).toBe(2);
+ // 2-arg setval → last_value=N, is_called=true; the next nextval = N+1.
+ await engine.executeRaw(`SELECT setval('test_probe_seq', 100)`);
+ const lv = await engine.executeRaw<{ v: number }>(`SELECT last_value AS v FROM test_probe_seq`);
+ expect(Number(lv[0].v)).toBe(100);
+ const n3 = await engine.executeRaw<{ v: number }>(`SELECT nextval('test_probe_seq') AS v`);
+ expect(Number(n3[0].v)).toBe(101);
+ await engine.executeRaw(`DROP SEQUENCE test_probe_seq`);
+ });
+
+ test('re-seeding is monotonic: the GREATEST guard never moves last_value backward', async () => {
+ // Regression for the codex P1: initSchema replays the schema blob, whose
+ // setval must NOT reset the clock below its current value — a backward move
+ // would let a stored query_cache bookmark serve stale rows. Push the
+ // sequence high, then run the EXACT monotonic seed the blob + v118 use.
+ await engine.executeRaw(`SELECT setval('page_generation_clock_seq', 999999)`);
+ await engine.executeRaw(
+ `SELECT setval('page_generation_clock_seq', GREATEST(
+ 1,
+ COALESCE((SELECT last_value FROM page_generation_clock_seq), 0),
+ COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0),
+ COALESCE((SELECT MAX(generation) FROM pages), 0)
+ ))`,
+ );
+ expect(await clockValue()).toBeGreaterThanOrEqual(999999);
+ });
+});
diff --git a/test/query-cache-gate.test.ts b/test/query-cache-gate.test.ts
index 4dcb0c624..b7db9ae22 100644
--- a/test/query-cache-gate.test.ts
+++ b/test/query-cache-gate.test.ts
@@ -252,13 +252,16 @@ describe('buildPageGenerationsSnapshot (PGLite-backed)', () => {
});
describe('CACHE_GATE_WHERE_CLAUSE (SQL shape regression)', () => {
- test('v0.41.19.0: Layer 1 reads page_generation_clock (not MAX(generation))', () => {
- expect(CACHE_GATE_WHERE_CLAUSE).toContain('page_generation_clock');
+ test('v0.42.x: Layer 1 reads page_generation_clock_seq.last_value (not the locked row, not MAX)', () => {
+ expect(CACHE_GATE_WHERE_CLAUSE).toContain('page_generation_clock_seq');
+ expect(CACHE_GATE_WHERE_CLAUSE).toContain('last_value');
expect(CACHE_GATE_WHERE_CLAUSE).toContain('qc.max_generation_at_store');
// Negative regression guard: the old MAX(generation) read shape MUST
// be gone (codex CDX-1/CDX-2: it silently served stale on
// UPDATE-to-non-max and DELETE).
expect(CACHE_GATE_WHERE_CLAUSE).not.toContain('MAX(generation) FROM pages');
+ // The locked single-row read (the BUG 1 contention source) MUST be gone.
+ expect(CACHE_GATE_WHERE_CLAUSE).not.toContain('value FROM page_generation_clock WHERE id');
});
test('contains Layer 2 per-page snapshot (jsonb_each + LEFT JOIN)', () => {
diff --git a/test/sync-break-lock-all.test.ts b/test/sync-break-lock-all.test.ts
index b910495b9..d8d50e04d 100644
--- a/test/sync-break-lock-all.test.ts
+++ b/test/sync-break-lock-all.test.ts
@@ -246,3 +246,67 @@ describe('R6 regression: schema bootstrap includes last_refreshed_at column', ()
expect(rows[0].is_nullable).toBe('YES');
});
});
+
+// ============================================================================
+// BUG 5 (v0.42.x) — honest --force-break-lock diagnostic when no lock is held.
+// Previously --force-break-lock emitted the same terse "not held (nothing to
+// break)" line and exited 0, sending operators down a dead end when a sync was
+// wedged for a reason other than a held lock.
+// ============================================================================
+describe('BUG 5 — --force-break-lock honest no-lock diagnostic', () => {
+ const LOCK = 'gbrain-sync:wiki';
+
+ test('force + no lock → wedge_hint in JSON, status absent, rc 0', async () => {
+ const { runBreakLock } = await import('../src/commands/sync.ts');
+ const logs: string[] = [];
+ const orig = console.log;
+ console.log = (...a: unknown[]) => { logs.push(a.map(String).join(' ')); };
+ let rc: number;
+ try {
+ rc = await runBreakLock(engine, LOCK, 'wiki', { force: true, json: true });
+ } finally {
+ console.log = orig;
+ }
+ expect(rc).toBe(0);
+ const parsed = JSON.parse(logs[0]);
+ expect(parsed.status).toBe('absent');
+ expect(parsed.lock).toBe(LOCK);
+ expect(typeof parsed.wedge_hint).toBe('string');
+ expect(parsed.wedge_hint).toContain('not a held lock');
+ });
+
+ test('force + no lock → human output carries the wedge hint, not the terse line', async () => {
+ const { runBreakLock } = await import('../src/commands/sync.ts');
+ const logs: string[] = [];
+ const orig = console.log;
+ console.log = (...a: unknown[]) => { logs.push(a.map(String).join(' ')); };
+ let rc: number;
+ try {
+ rc = await runBreakLock(engine, LOCK, 'wiki', { force: true, json: false });
+ } finally {
+ console.log = orig;
+ }
+ expect(rc).toBe(0);
+ const out = logs.join('\n');
+ expect(out).toContain('nothing to break');
+ expect(out).toContain('gbrain doctor');
+ expect(out).not.toBe(`Lock ${LOCK} is not held (nothing to break).`);
+ });
+
+ test('non-force + no lock → unchanged terse line, no wedge_hint', async () => {
+ const { runBreakLock } = await import('../src/commands/sync.ts');
+ const logs: string[] = [];
+ const orig = console.log;
+ console.log = (...a: unknown[]) => { logs.push(a.map(String).join(' ')); };
+ let rc: number;
+ try {
+ rc = await runBreakLock(engine, LOCK, 'wiki', { force: false, json: true });
+ } finally {
+ console.log = orig;
+ }
+ expect(rc).toBe(0);
+ const parsed = JSON.parse(logs[0]);
+ expect(parsed.status).toBe('absent');
+ expect(parsed.wedge_hint).toBeUndefined();
+ });
+});
diff --git a/test/think-gateway-adapter.test.ts b/test/think-gateway-adapter.test.ts
index d9788af04..dbe830ce6 100644
--- a/test/think-gateway-adapter.test.ts
+++ b/test/think-gateway-adapter.test.ts
@@ -18,7 +18,7 @@
import { describe, test, expect } from 'bun:test';
import { __thinkAdapter } from '../src/core/think/index.ts';
import { resetGateway } from '../src/core/ai/gateway.ts';
-import { withEnv } from './helpers/with-env.ts';
+import { withEnv, emptyHome } from './helpers/with-env.ts';
describe('think gateway adapter — response shape conversion', () => {
test('chatResultToMessage maps ChatResult.text to Anthropic.Message content[0].text', () => {
@@ -76,7 +76,7 @@ describe('think gateway adapter — model-id normalization', () => {
});
test('tryBuildGatewayClient returns null when ANTHROPIC_API_KEY is absent (preserves legacy NO_ANTHROPIC_API_KEY signal)', async () => {
- await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => {
+ await withEnv({ ANTHROPIC_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => {
const client = await __thinkAdapter.tryBuildGatewayClient('claude-opus-4-7');
expect(client).toBeNull();
});
@@ -86,7 +86,7 @@ describe('think gateway adapter — model-id normalization', () => {
await withEnv({ ANTHROPIC_API_KEY: 'sk-test-key' }, async () => {
expect(__thinkAdapter.hasAnthropicKey()).toBe(true);
});
- await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => {
+ await withEnv({ ANTHROPIC_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => {
expect(__thinkAdapter.hasAnthropicKey()).toBe(false);
});
});
@@ -115,7 +115,7 @@ describe('think gateway adapter — #1698 slash form + explicit-model fork', ()
});
test('explicit anthropic model with no key THROWS (unavailable)', async () => {
- await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => {
+ await withEnv({ ANTHROPIC_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => {
await expect(
__thinkAdapter.tryBuildGatewayClient('anthropic:claude-sonnet-4-6', { explicitModel: true }),
).rejects.toThrow(/not usable.*unavailable/);
@@ -164,7 +164,7 @@ describe('think gateway adapter — #1698 slash form + explicit-model fork', ()
// 'no LLM available' stub). A future refactor that turns this into a graceful path fails here.
test('D1 backstop: explicit non-anthropic model, no key → BUILDS then create() THROWS (never a stub)', async () => {
await withEnv(
- { ANTHROPIC_API_KEY: undefined, DEEPSEEK_API_KEY: undefined, OPENAI_API_KEY: undefined },
+ { ANTHROPIC_API_KEY: undefined, DEEPSEEK_API_KEY: undefined, OPENAI_API_KEY: undefined, GBRAIN_HOME: emptyHome() },
async () => {
resetGateway(); // unconfigured → gateway.chat() throws AIConfigError at create()
// deepseek:deepseek-chat passes validateModelId (real recipe + chat touchpoint) — the