From 9bf96db807c2f050449142f2f0b05726f58e5054 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Wed, 17 Jun 2026 14:02:47 -0700 Subject: [PATCH] v0.42.51.0 fix(sync): contention-free clock + checkpoint integrity + honest sync freshness (#2255) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(sync): contention-free page-generation clock — sequence swap The page-generation clock backed the query-cache Layer-1 bookmark via a FOR EACH STATEMENT trigger running `UPDATE page_generation_clock SET value=value+1 WHERE id=1`. That took a transaction-length RowExclusiveLock on one tuple, so every concurrent page writer serialized on the prior writer's COMMIT — sync ran at ~0.8 cores regardless of worker count. Swap to a SEQUENCE bumped by nextval() (a microsecond LWLock, never a row lock). The clock's only contract is monotonic advancement on any page INSERT/UPDATE/DELETE; last_value is non-transactional, so rolled-back or concurrent-uncommitted writers only OVER-invalidate the cache (lose a hit), never serve stale. - migration v118: CREATE SEQUENCE + load-bearing 2-arg setval (is_called= true, floor 1, seeded >= old clock and MAX(generation)) + repoint the trigger function body + DELETE query_cache so no old-clock bookmark survives the swap. v107 left immutable. - query-cache-gate.ts: 3 readers -> SELECT last_value FROM page_generation_clock_seq. - schema.sql + pglite-schema.ts (+ regenerated schema-embedded.ts) ship the sequence on fresh install; table + trigger names retained. - tests: clockValue reads last_value; mechanism proof (trigger fn uses nextval not the row UPDATE); rollback-advances-clock safety pin; real PGLite sequence round-trip (is_called gotcha); shape test requires _seq. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(sync): op_checkpoints array-shape guard — CHECK + repair + defensive loader completed_keys is JSONB and the checkpoint loader runs jsonb_array_elements_text over it. A non-array (scalar) value makes that throw "cannot extract elements from a scalar", which takes down the whole UNION load — including the valid op_checkpoint_paths child rows — and loses all checkpoint progress for that key. No current writer produces a scalar, but an older binary / external script / future bug could. Make the corruption class structurally impossible and self-healing: - migration v119: LOCK TABLE (so an out-of-band scalar can't land between repair and constrain; no-op on single-connection PGLite), repair any pre-existing scalar to '[]' (op_checkpoint_paths child rows are the append-only source of truth, so the reset loses nothing), then add the named CHECK (jsonb_typeof(completed_keys) = 'array') via a pg_constraint IF NOT EXISTS guard. A DB-enforced always-on guard — the correct pattern vs a migration verify-hook, which never runs on already-stamped brains. - schema.sql + pglite-schema.ts (+ regenerated schema-embedded.ts) ship the same NAMED inline CHECK so fresh installs match migrated brains and v119 skips the duplicate. - op-checkpoint.ts loader: gate the legacy arm on jsonb_typeof = 'array' so a scalar parent is skipped (children still load) instead of throwing the whole union, and log a specific corruption warning when one is seen. - tests: CHECK rejects a scalar (exactly one constraint, no blob+migration dupe); loader survives a scalar parent and returns the children; v119 repair converts a scalar to '[]'. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(doctor): report actively-running sync via live lock, not stale freshness A slow source that makes partial progress every cycle but never fully completes used to read as permanently "stale" / "never synced" because last_sync_at only advances on a full successful sync. The naive fix (treat recent checkpoint banking as "in progress") is unsafe: a blocked sync banks the good files then writes no anchor, so banking can't tell in-progress from wedged. Use the only honest signal: a LIVE, non-expired per-source sync lock (inspectLock + syncLockId against gbrain_cycle_locks). Every non-skipLock sync holds it and refreshes it; a blocked/failed sync's process has exited (no lock row) and a wedged holder stops refreshing (TTL lapses), so either correctly falls through to the stale path and is NEVER masked. An actively-syncing source (including a never-synced source doing its first sync) counts as synced_recently, preserving the pinned 3-bucket invariant. The lock lookup reuses doctor's existing dynamic db-lock import and swallows any throw (stub engine, pre-lock-table brain) to false, so it can only ADD an in-progress verdict, never suppress a real stale one. Tests (real PGLiteEngine + real lock rows): stale+no-lock -> fail; stale+live-lock -> ok; never-synced+live-lock -> ok; never-synced+no-lock -> fail; expired-TTL lock -> fail (wedged not masked); blocked source with banked checkpoint rows but no lock -> still fail. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(sync): honest --force-break-lock diagnostic when no lock is held --force-break-lock used to emit the same terse "Lock ... is not held (nothing to break)" line and exit 0 even when a sync was genuinely wedged, sending the operator down a dead end — the wedge was not a held lock. Keep rc=0 (breaking a non-existent lock is idempotently successful; flipping the exit code would break automation), but under --force say plainly that nothing was broken and point at the real next step (gbrain sync / gbrain doctor) plus a `wedge_hint` field in --json output. The non-force path is byte-for-byte unchanged. runBreakLock is exported for the test. Tests: force+no-lock -> wedge_hint JSON + human hint, rc 0; non-force+no-lock -> unchanged terse line, no hint. Co-Authored-By: Claude Opus 4.8 (1M context) * fix(doctor): surface the in-progress sync holder in the freshness message Plan-completion follow-up to the BUG 4 live-lock signal: when a source is actively syncing, name the holder (pid + host) in the check message instead of silently folding it into synced_recently. The note is appended only when something is in progress, so steady-state messages stay byte-for-byte unchanged (the pinned exact-message + 3-bucket-invariant tests still pass). Co-Authored-By: Claude Opus 4.8 (1M context) * fix(sync): pre-landing review fixes — monotonic clock seed, scoped CHECK guard Adversarial (codex) review of the implementation diff caught three: - P1 (correctness): the fresh-schema setval was not monotonic. initSchema replays the schema blob, and the unconditional setval(MAX(generation)) could move page_generation_clock_seq.last_value BACKWARD on an already-upgraded brain, letting a stored query_cache bookmark serve stale rows. Seed via GREATEST over the sequence's OWN last_value (+ old table value + MAX(generation)) in all 3 fresh schemas and migration v118, so a replay is idempotent — mirrors the old table's ON CONFLICT DO NOTHING. Pinned by a new monotonic regression test. - P2: v119's CHECK-exists guard keyed on conname only (not globally unique). Scope it to conrelid = 'op_checkpoints'::regclass. - P3: in-progress note ran into the prior sentence in fail/warn doctor messages; separate it with '. '. Co-Authored-By: Claude Opus 4.8 (1M context) * test: make Anthropic/ZE no-key tests hermetic against a dev config key These "no key" tests cleared only ANTHROPIC_API_KEY / ZEROENTROPY_API_KEY from the env, but hasAnthropicKey() and checkZeEmbeddingHealth() also read the key from ~/.gbrain/config.json. On a dev machine whose real config holds a key, the no-key assertions flipped and the tests failed locally (they passed only in key-less CI). Add a shared with-env emptyHome() helper and point GBRAIN_HOME at an empty dir in every no-key path so loadConfig finds nothing — matching the already-hermetic anthropic-key / gateway-probe tests. Co-Authored-By: Claude Opus 4.8 (1M context) * chore: bump version and changelog (v0.44.1.0) Co-Authored-By: Claude Opus 4.8 (1M context) * docs(key-files): sync doctor + op-checkpoint entries to v0.44.1.0 truth checkSyncFreshness now reports an actively-running sync via the live per-source lock (names holder pid+host, counts as synced_recently) instead of flagging it stale; loadOpCheckpoint gates the legacy union arm on jsonb_typeof = 'array' so a scalar parent can't take down the whole load, and migration v119's CHECK constraint makes the corruption class structurally impossible. Reference docs describe current behavior only — both entries updated in place, no release-clause appends. Guard + llms freshness test green. Co-Authored-By: Claude Opus 4.8 (1M context) * chore: re-version to v0.42.51.0 (natural next-off-master) Maintainer override of the queue allocator's leap to 0.44.1.0 (it jumped past in-flight sibling PR claims at 0.42.50/0.43.0/0.44.0). Take the natural next slot in the 0.42.x line above the immediate sibling claim (0.42.50.0); a merge re-bump resolves any collision if a cathedral PR lands first. Co-Authored-By: Claude Opus 4.8 (1M context) * ci(e2e): bound + retry the OpenClaw install so a transient npm hang can't burn the Tier 2 budget The Tier 2 (LLM Skills) job failed at 30m16s — the `npm install -g openclaw@2026.4.9` step hung on a transient npm/registry stall (orphan `npm install openclaw` was still running at cancel time) and consumed the entire 30m job budget that v0.42.50.0 (#2254) introduced. The install normally finishes in under a minute (Tier 2 is ~4m end to end on master), so this is flaky-install infra, not a test failure. Wrap the install in `timeout 120` + a 3-attempt retry loop with an 8-minute step backstop: a hung attempt is killed in 2 min and retried instead of eating the whole job. Same bound-the-hang philosophy as #2254's job timeouts. Co-Authored-By: Claude Opus 4.8 (1M context) --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .github/workflows/e2e.yml | 17 ++- CHANGELOG.md | 16 +++ VERSION | 2 +- docs/architecture/KEY_FILES.md | 4 +- package.json | 2 +- src/commands/doctor.ts | 56 ++++++++- src/commands/sync.ts | 21 +++- src/core/migrate.ts | 97 +++++++++++++++ src/core/op-checkpoint.ts | 30 ++++- src/core/pglite-schema.ts | 23 +++- src/core/schema-embedded.ts | 25 +++- src/core/search/query-cache-gate.ts | 34 +++--- src/schema.sql | 25 +++- test/conversation-parser/llm-base.test.ts | 6 +- test/conversation-parser/llm-fallback.test.ts | 4 +- test/doctor-ze-checks.test.ts | 8 +- test/doctor.test.ts | 111 ++++++++++++++++++ test/helpers/with-env.ts | 18 +++ test/op-checkpoint.test.ts | 76 ++++++++++++ test/page-generation-counter.test.ts | 102 ++++++++++++++-- test/query-cache-gate.test.ts | 7 +- test/sync-break-lock-all.test.ts | 64 ++++++++++ test/think-gateway-adapter.test.ts | 10 +- 23 files changed, 701 insertions(+), 57 deletions(-) diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 3857043ba..74717c3ea 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -79,7 +79,22 @@ jobs: bun-version: 1.3.13 - run: bun install - name: Install OpenClaw - run: npm install -g openclaw@2026.4.9 + # Bound + retry the install: a transient npm/registry stall here used to + # hang unbounded and (since the v0.42.50.0 job timeout) burn the entire + # 30m Tier 2 budget before failing — even though the install normally + # finishes in well under a minute. `timeout` kills a hung attempt fast; + # up to 3 attempts ride out a flaky registry. Step cap is a backstop. + timeout-minutes: 8 + run: | + for attempt in 1 2 3; do + if timeout 120 npm install -g openclaw@2026.4.9; then + exit 0 + fi + echo "::warning::openclaw install attempt $attempt failed or timed out; retrying in 10s" >&2 + sleep 10 + done + echo "::error::openclaw install failed after 3 attempts" >&2 + exit 1 - name: Configure OpenClaw MCP run: | mkdir -p ~/.openclaw diff --git a/CHANGELOG.md b/CHANGELOG.md index 380ae0a75..565627272 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,22 @@ All notable changes to GBrain will be documented in this file. +## [0.42.51.0] - 2026-06-17 + +**`gbrain sync` stops bottlenecking all its workers on a single database row, a malformed checkpoint can no longer wedge a source, and `gbrain doctor` tells an actively-running sync apart from a stuck one.** A slow source that fell behind HEAD could read as permanently stale even while it imported every cycle: sync was single-core-bound at the database layer, so handing it more workers didn't help, and the freshness check couldn't see that a sync was in fact running. + +The root cause was the page-generation clock that backs the search cache. Every page write bumped a single locked counter row, so concurrent sync workers serialized on one another's commits no matter how many you ran. It is now a contention-free sequence: the cache invalidation contract is unchanged (it still over-invalidates rather than ever serving stale), but writers no longer wait in line. The other fixes harden checkpoint state and make the freshness signal honest. + +### Changed +- **Sync writes scale across cores.** The page-generation clock moved from a single locked counter row to a contention-free sequence, so parallel sync workers stop serializing on each other. A large `gbrain sync` now uses the workers you give it instead of collapsing to roughly one. +- **`gbrain doctor` distinguishes in-progress from stale.** A source holding a live sync lock is reported as actively syncing (naming the running process), not flagged stale. A genuinely stuck, blocked, or never-completed sync still reports stale — the signal is the live lock, so a stopped sync is never masked. + +### Fixed +- **A malformed checkpoint record can no longer wedge a source.** Checkpoint state is structurally constrained, repaired automatically on upgrade, and the loader survives a bad record instead of discarding all banked progress for that source. +- **`gbrain sync --force-break-lock` is honest when there is no lock.** It now says plainly that nothing was held and points at how to inspect a genuinely wedged sync, instead of a terse no-op that read like a successful unwedge. + +### To take advantage of v0.42.51.0 +`gbrain upgrade`, then `gbrain doctor`. Existing brains pick up the contention-free clock and the checkpoint integrity constraint automatically on the next migration; the search cache rebuilds itself on first query. Nothing to configure. ## [0.42.50.0] - 2026-06-17 **CI reliability hardening — a wedged job can no longer run for six hours, a superseded run no longer reports a stale flaky failure, and broken workflow YAML is caught before it ships.** gbrain's CI already had the deep machinery (content-hash run-skip cache, weight-aware shard balancing, test-isolation guards, hermetic E2E). What it lacked was the cheap GitHub-Actions hygiene that was already wired into `heavy-tests.yml` but never into the two hot-path workflows. This pass closes that gap, porting the patterns from the sibling GStack project's CI-reliability work. diff --git a/VERSION b/VERSION index 0578db35f..6a6c1a02a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.50.0 +0.42.51.0 diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index d6b1251c5..b2d7fe7d2 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -287,7 +287,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/ai/model-resolver.ts:parseModelId` — gateway-side resolver accepts both colon and slash form (`provider:model` and `provider/model`) so a slash-form id resolves to the same recipe at every gateway entry point (chat / embed / rerank) instead of throwing `AIConfigError: model id must be in format provider:model`. Bare names without ANY separator still throw — gateway routing always needs an explicit provider. Pinned by `test/ai/model-resolver-slash.test.ts` including a `resolveRecipe` round-trip asserting slash form resolves to the same recipe object as colon form. - `src/commands/transcripts.ts` — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). Batch-load fast path on Postgres uses a single SQL query (fixes the PgBouncer round-trip timeout, ~60s → ~6s), gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1`. Batch projection is `SELECT ... ORDER BY source_id, slug` (NOT `SELECT DISTINCT ON (slug)`, which collapsed same-slug-different-source pages into one scan) so multi-source brains scan each `(source, slug)` row independently. Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`; batch + sequential paths report the same page count on multi-source brains. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check at `doctor.ts:1011-1043` consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings in BOTH `doctor.ts` and `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source ` matches. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. IRON-RULE regression banning stale verb names, the sync_freshness boundary matrix, the D4 regression guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases). +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. Checks include `jsonb_integrity` + `markdown_body_completeness` (reliability), `schema_version` (fails loudly when `version=0`, routes to `gbrain apply-migrations --yes`), `queue_health` (Postgres-only: stalled-forever active jobs started_at > 1h, waiting-depth-per-name > threshold default 10 via `GBRAIN_QUEUE_WAITING_THRESHOLD`, and dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier in last 24h), `sync_failures` (`[CODE=N, ...]` breakdown for unacked-warn + acked-ok; severity comes from the shared `decideSyncFailureSeverity` in `src/core/sync-failure-ledger.ts` so the LOCAL and REMOTE/thin-client doctor surfaces can never drift — a stuck bookmark escalates to FAIL once an OPEN failure has blocked past the staleness window or ≥10 files block, while already `auto_skipped` rows stay a visible WARN), `rls_event_trigger` (healthy `evtenabled` set is `('O','A')` only; fix hint `gbrain apply-migrations --force-retry 35`), `graph_coverage` (short-circuits to ok when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0; WARN hint is `gbrain extract all`), `embedding_column_registry` (probes each declared column via Postgres `format_type(atttypid, atttypmod)` to catch dim mismatch with a paste-ready `gbrain config set embedding_columns '{...}'` hint, probes HNSW index presence via `pg_indexes`, computes default-column population via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` warning below 90% except empty brains where chunk_count=0 short-circuits to ok; PGLite parity via `executeRaw`), and `skill_brain_first` (walks SKILL.md via `autoDetectSkillsDirReadOnly`, calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file with structured `Check.issues[]`; warn states `missing_brain_first`/`brain_first_typo`, ok states `compliant_callout`/`compliant_phase`/`compliant_position`/`exempt_frontmatter`/`no_external`; snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl`). `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts via `src/core/dry-fix.ts` (and MISSING_RULE_PATTERNS for the brain-first callout); `--fix --dry-run` previews. `--index-audit` (Postgres-only, informational, no auto-drop) reports zero-scan indexes from `pg_stat_user_indexes`. Every DB check runs under a progress phase; `markdown_body_completeness` runs under a 1s heartbeat. `runDoctor` uses `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`; install-path fallback so `cd ~ && gbrain doctor` finds bundled skills); `--fix` carries a D6 install-path safety gate that refuses auto-repair when `detected.source === 'install_path'` (would rewrite the bundled tree). The Lane D supervisor check at `doctor.ts:1011-1043` consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` (warn at `>=1` real crash; ok message has `clean_exits_24h=N`; warn message has `runtime=A oom=B unknown=C legacy=D` per-cause breakdown) so OOM/runtime/unknown crashes are distinguishable from clean code=0 worker drains; cross-surface parity with `gbrain jobs supervisor status` is pinned by source-grep wiring assertions requiring the breakdown substrings in BOTH `doctor.ts` and `jobs.ts`. `checkSyncFreshness` (exported, in `runDoctor` local + `doctorReportRemote` thin-client) is a staleness probe: warns at 24h, fails at 72h or never-synced; future-`last_sync_at` warns ("clock skew") instead of falling through ok; env overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS`/`GBRAIN_SYNC_FRESHNESS_FAIL_HOURS` (invalid fall back with once-per-process stderr warn via `_resolveSyncFreshnessHours`); failure messages embed `source.id` so the printed `gbrain sync --source ` matches. A source holding a LIVE, non-expired per-source sync lock (`inspectLock(engine, syncLockId(source.id))` from `src/core/db-lock.ts`) is reported as actively syncing (the message names the holder pid + host) and counted in `synced_recently_count`, NOT flagged stale — the live lock is the only honest in-progress signal (checkpoint banking can't distinguish in-progress from wedged: a blocked sync banks its files but writes no anchor). A blocked/failed sync's process has exited (no lock row) and a wedged holder stops refreshing (TTL lapses), so either falls through to the stale path and is never masked; the dynamic `db-lock` import is swallowed to a no-op on a stub engine or pre-lock-table brain, so this can only ADD an in-progress verdict, never suppress a real stale one. The in-progress note is appended to whatever verdict the buckets produce and is empty when nothing is syncing, so steady-state messages stay byte-for-byte unchanged. It has a `localOnly`-gated git short-circuit (`runDoctor` passes `localOnly: true`; `doctorReportRemote` runs in the HTTP MCP server `src/commands/serve-http.ts` and keeps default `false` so that path never walks DB-supplied `local_path` via subprocess — trust boundary). The local predicate mirrors sync's "do work?" gate (HEAD == `last_commit` AND working tree clean via `requireCleanWorkingTree: 'ignore-untracked'` so a quiet repo with only untracked dirs is `unchanged` not SEVERE, AND `chunker_version === CURRENT`); the inline SELECT carries `last_commit + chunker_version + newest_content_at`. The REMOTE path computes lag via `lagFromContentMs(newest_content_at, lastSync, now)` from the stored column, NO git subprocess; LOCAL fall-through and the `< 0` clock-skew check stay on raw wall-clock. Three-bucket count math populates `Check.details = {unchanged_count, synced_recently_count, stale_count}` with the invariant `sum === sources.length`. `checkCycleFreshness` is DELIBERATELY NOT git-short-circuited or content-relativized (`last_commit == HEAD` can't answer "did the full cycle complete?"; a sync can succeed while later cycle phases fail; different axis `last_full_cycle_at`). Pinned by `test/doctor.test.ts` (incl. IRON-RULE regression banning stale verb names, the sync_freshness boundary matrix, the D4 regression guard verifying git probes are NEVER called when `localOnly` is unset/false, the three-bucket invariant, and the untracked-folders / remote-never-shells-out trust-boundary cases). - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). `Migration` interface carries `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses in a transaction; ignored on PGLite). Key migrations: v14 (handler branches on `engine.kind` for CONCURRENTLY-on-Postgres with invalid-remnant pre-drop via `pg_index.indisvalid`, plain `CREATE INDEX` on PGLite); v15 (`minion_jobs.max_stalled` default 1→5 + backfill non-terminal rows); v24 `rls_backfill_missing_tables` (`sqlFor: { pglite: '' }` no-op — PGLite has no RLS engine, targets subagent tables absent from pglite-schema.ts); v30 `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))` (RLS-enabled under BYPASSRLS; synthesize reads/writes to avoid re-judging); v35 auto-RLS event trigger `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` running `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on new `public.*` tables (no FORCE) + one-time backfill on every existing `public.*` base table whose comment doesn't match `^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}` (per-table failure aborts the offending CREATE TABLE; no EXCEPTION wrap; PGLite no-op via `sqlFor.pglite: ''`; breaking change: intentionally-RLS-off public tables need the GBRAIN:RLS_EXEMPT comment before upgrade); v40 `pages_emotional_weight` (`pages.emotional_weight REAL NOT NULL DEFAULT 0.0`, column-only metadata-only); v46 `mcp_request_log_params_jsonb_normalize` (`UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`, idempotent); v60-v65 six-migration chain wiring source-scoping into `oauth_clients` — v60 (`oauth_clients_source_id_fk`: `source_id TEXT` NULL→`'default'` backfill + FK to `sources(id) ON DELETE SET NULL`), v61 (`federated_read TEXT[] NOT NULL DEFAULT '{}'`), v62 (explicit-CASE backfill so `source_id IS NULL` → `'{}'`), v63 (fail-loud check every row's source_id is in its federated_read array), v64 (FK flipped to `ON DELETE RESTRICT`), v65 (GIN index for array-containment); v68 `eval_candidates_embedding_column` (`eval_candidates.embedding_column TEXT NULL` per-row provenance for `gbrain eval replay` to reproduce the same retrieval space; NULL-tolerant); v108 `pages_embedding_signature` (`pages.embedding_signature TEXT NULL` = `:` stamped via `setPageEmbeddingSignature`; GRANDFATHER — stale predicate is `embedding_signature IS NOT NULL AND embedding_signature <> $current` so NULL is NEVER stale and upgrade never re-embeds the whole corpus; no index; metadata-only); v109 `sources_newest_content_at` (`sources.newest_content_at TIMESTAMPTZ` durable newest-COMMIT HEAD committer time written by `writeSyncAnchor`, read by the REMOTE staleness path instead of shelling to git; mirror in pglite-schema.ts + schema.sql + bootstrap probe); v110 `page_aliases` (`(id, source_id, alias_norm, slug, ...)` with `UNIQUE (source_id, alias_norm, slug)` + lookup indexes on `(source_id, alias_norm)` and `(source_id, slug)`; `alias_norm` is `normalizeAlias()` output so WRITE/READ key on the same form; also in `src/core/pglite-schema.ts`); v111 `search_telemetry_rank1_columns` (`ADD COLUMN IF NOT EXISTS` on both engines: `sum_rank1_score`, `count_rank1`, three buckets `rank1_lt_solid`/`rank1_solid`/`rank1_high` on `search_telemetry` — aggregate not per-query rows so rank-1 median drift is bounded-growth; ALTERs right after v57 which created the table); v114 `links_link_source_check_kebab_regex` (#1941, opens `link_source` from the closed allowlist to a kebab-case format gate `^[a-z][a-z0-9]*(-[a-z0-9]+)*$` + `char_length<=64`; Postgres branch uses `NOT VALID` + `VALIDATE CONSTRAINT` with `transaction:false`, PGLite plain DROP+ADD; existing built-ins all satisfy the regex so VALIDATE never fails on existing data); v116 `code_edges_source_backfill_and_callee_index` (#2073, idempotent: backfills NULL `code_edges_symbol`/`code_edges_chunk` `source_id` from each edge's `from_chunk` page — NULL never matched a scoped `AND source_id = …` filter so scoped `code-callers`/`code-callees` returned 0 rows on multi-source brains — plus plain `CREATE INDEX` on `from_symbol_qualified` for both edge tables, which had no index and seq-scanned per BFS node). The dedup-index self-heal (`timeline_dedup_index`, see `timeline-dedup-repair.ts`) is NOT version-gated: `runMigrations` invokes `repairTimelineDedupIndex` on every pass (including the no-pending early-return path) because a merge-renumbered migration can leave the version counter past the index change while the index stays the old shape. - `src/core/timeline-dedup-repair.ts` (#2038) — schema-drift self-heal for `idx_timeline_dedup`. The migration that widened the dedup index from `(page_id, date, summary)` to `(page_id, date, summary, source)` was renumbered during a master merge, so a brain that ran the old variant has its version counter stamped past the change while the index keeps the 3-column shape — and every `addTimelineEntry` batch then fails its 4-column `ON CONFLICT`, silently breaking timeline writes brain-wide. The version counter can't detect this, so the repair is keyed off the actual index SHAPE: `checkTimelineDedupIndex(engine)` returns `{tablePresent, indexPresent, columns, needsRepair}` (read-only; powers the `timeline_dedup_index` doctor check) and `repairTimelineDedupIndex(engine)` dedupes-then-rebuilds the index. `runMigrations` invokes the repair on every pass (including the no-pending early-return path); idempotent no-op when the index is already 4-column. `gbrain apply-migrations --force-schema` triggers it on demand. Pinned by `test/timeline-dedup-repair.test.ts`. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY `\r`-rewriting; non-TTY plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. `emitHumanLine` is prefix-aware — inside a `withSourcePrefix(id, ...)` scope from `src/core/console-prefix.ts` it prepends `[id] ` (and TTY-rewrite mode `\r\x1b[2K` carries the prefix inside the clear-to-EOL escape); `emitJson` is intentionally NOT prefixed so NDJSON consumers don't choke on a `[id] {...}` shape. @@ -383,7 +383,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/commands/report.ts` — Structured report saver (audit trail for maintenance/enrichment) - `src/core/destructive-guard.ts` — three-layer protection against accidental data loss. `assessDestructiveImpact(engine, sourceId)` counts pages/chunks/embeddings/files for a source. `checkDestructiveConfirmation(impact, opts)` is the fail-closed gate (`--confirm-destructive` required when data is present; `--yes` alone is rejected). `softDeleteSource` / `restoreSource` / `listArchivedSources` / `purgeExpiredSources` drive the source-level archive lifecycle via `sources.archived BOOLEAN`, `archived_at TIMESTAMPTZ`, `archive_expires_at TIMESTAMPTZ`. Page-level analog: `BrainEngine.softDeletePage` / `restorePage` / `purgeDeletedPages` plus `pages.deleted_at TIMESTAMPTZ` and a partial purge index. The MCP `delete_page` op rewires to `softDeletePage`; ops `restore_page` (`scope: write`) and `purge_deleted_pages` (`scope: admin`, `localOnly: true`) round out the surface. Search visibility (`buildVisibilityClause` in `src/core/search/sql-ranking.ts`) hides soft-deleted pages and archived sources from `searchKeyword` / `searchKeywordChunks` / `searchVector` in both engines. The autopilot cycle's `purge` phase calls `purgeExpiredSources` + `engine.purgeDeletedPages(72)` so the 72h TTL is real. - `src/commands/pages.ts` — `gbrain pages purge-deleted [--older-than HOURS|Nd] [--dry-run] [--json]` operator escape hatch. Mirror of `gbrain sources purge` for the page-level lifecycle. Hard-deletes pages whose `deleted_at` is older than the cutoff; cascades to content_chunks/page_links/chunk_relations. -- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred. +- `src/core/op-checkpoint.ts` — DB-backed checkpoint primitive for long-running ops. Migration v67 introduces `op_checkpoints (op TEXT, fingerprint TEXT, completed_keys JSONB, updated_at TIMESTAMPTZ, PK(op, fingerprint))`. Per-op fingerprint helpers (`embedFingerprint`, `extractFingerprint`, `reindexFingerprint`, `integrityFingerprint`, `purgeFingerprint`) compute `sha8(canonical-JSON(relevant-params))` so re-running with the same params resumes from `completed_keys` and re-running with different params (e.g. `--limit 100` vs `--limit 200`) starts fresh. Cross-worker safe on Postgres (DB row, no file-lock race); PGLite degrades gracefully. Replaces per-op file-backed JSON checkpoints scattered across `import.ts`, `embed.ts`, `reindex.ts`. The 7-day TTL GC runs in the cycle's `purge` phase. All writes (`recordCompleted`, `clearOpCheckpoint`) route through `engine.executeRawDirect` + `withRetry(BULK_RETRY_OPTS)` so they survive Supavisor pool exhaustion, and `recordCompleted` returns `boolean` (banked vs failed-after-retries) — the 9 non-sync consumers keep its REPLACE-into-`completed_keys` semantics. Resumable sync uses the additive `appendCompleted(key, deltaKeys)` / `appendCompletedOnce` (the latter no-retry for the SIGTERM path) which INSERT a delta into the `op_checkpoint_paths` child table (migration v115: `(op, fingerprint, path)` PK, FK to `op_checkpoints` ON DELETE CASCADE) via a single writable-CTE `unnest($3::text[])` write — O(delta), killing the old O(N²) full-set rewrite. `loadOpCheckpoint` returns the `UNION ALL` of legacy `completed_keys` + child-table paths (deduped in JS), so an in-flight upgrade loses nothing. The legacy arm is gated on `jsonb_typeof(completed_keys) = 'array'` so a non-array (scalar) parent row can't make `jsonb_array_elements_text` throw "cannot extract elements from a scalar" and take down the whole union (which would discard the valid child rows and lose all banked progress for the key); a third union arm flags the corruption so the loader logs it once and keeps the child rows. Migration v119 adds the `op_checkpoints_completed_keys_array` CHECK (`jsonb_typeof(completed_keys) = 'array'`) — a DB-enforced, always-on guard that makes the scalar-corruption class structurally impossible going forward; the migration repairs any pre-existing scalar to `'[]'` under `LOCK TABLE ... IN SHARE ROW EXCLUSIVE MODE` and `src/core/schema-embedded.ts` + `src/core/pglite-schema.ts` ship the same CHECK on fresh installs (a loader hit now implies schema drift, a disabled constraint, or an out-of-band writer). `syncFingerprint({sourceId, lastCommit})` keys the sync rows. Pinned by `test/op-checkpoint.test.ts` (incl. delta-append, union read, cascade clear, durable-write boolean, and the scalar-parent guard). `import-checkpoint.ts` was NOT migrated to this primitive — both checkpoint systems coexist without conflict; migrating requires async-propagating 4 sync call sites in `src/commands/import.ts` and rewriting 18 tests, deferred. - `src/core/brain-score-recommendations.ts` — pure data layer consumed by both `gbrain doctor --remediation-plan` / `--remediate` and `gbrain features`. `computeRecommendations(checks, opts)` returns `Remediation[]` with stable `id`, content-hash `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on` (references stable ids, not check names — so plan order is reproducible). `classifyChecks(report)` triages every doctor check three-state into `remediable | human_only | blocked` (`human_only` covers RLS warnings and other human-judgment gates; `blocked` covers dependency chains where a parent check failed). `maxReachableScore(checks)` computes the ceiling for empty/under-configured brains (no entity pages → graph_coverage caps at 70; no embedding key → embedding_coverage caps at 60). Cost estimates pull from `anthropic-pricing.ts` (synthesize/patterns/consolidate) and `embedding-pricing.ts` (embed jobs). Pinned by `test/brain-score-recommendations.test.ts` (~27 cases incl. determinism, content-hash idempotency, DB-backed checkpoint provenance, three-state triage). - `src/commands/doctor.ts` extension — `--remediation-plan [--json] [--target-score N]` prints what would run (stable `id`, `idempotency_key`, `severity`, `est_seconds`, `est_usd_cost`, `depends_on`); `--remediate [--yes] [--target-score N] [--max-usd N]` submits each plan step as a Minion job in dependency order, re-checking score between steps. `--target-score N` defaults to 90; refuses to start when target exceeds `maxReachableScore()` and lists what's missing. `--max-usd N` is the cron-safety guard — submission refuses when the plan's `est_total_usd_cost` exceeds the cap. JSON envelope adds a `Check.remediation` field (additive, schema_version unchanged). Pinned by tests in `test/doctor.test.ts`. - `src/commands/jobs.ts` extension — registers 11 Minion handlers: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. Phase wrappers delegate to `runCycle({phases:[name]})` so `src/core/cycle.ts` stays the single source of truth for phase semantics. The standalone `sync` handler passes `noExtract: true` to match `runPhaseSync`'s contract (doctor's remediation plan emitting `[sync, extract]` would otherwise double-extract). diff --git a/package.json b/package.json index 4dd05db81..82b43c3bc 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.50.0" + "version": "0.42.51.0" } diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index f27772ac8..f931b5486 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -3394,6 +3394,40 @@ export async function checkSyncFreshness( let hasWarnings = false; let hasFailures = false; + // BUG 4 (v0.42.x): a source with a LIVE, non-expired per-source sync lock is + // actively syncing RIGHT NOW — it must not read as stale or never-synced. + // The live lock is the only honest "in progress" signal. Checkpoint banking + // is NOT usable: a blocked sync banks the good files then writes no anchor + // (test/sync-resumable-import.serial.test.ts), so banking can't tell + // in-progress from wedged. A blocked/failed sync's process has exited (no + // lock row) and a wedged holder stops refreshing (TTL lapses), so either + // correctly falls through to the stale path and is NEVER masked. Same + // dynamic import as the stale_locks check; any throw (stub engine in unit + // tests, pre-lock-table brain) is swallowed to false, so this can only ADD + // an in-progress verdict, never suppress a real stale one. + // Notes for sources caught actively syncing (surfaced in the result + // message so the operator sees "in progress", not just a silent healthy + // bucket). Empty when nothing is syncing — keeps the steady-state messages + // byte-for-byte unchanged. + const inProgress: string[] = []; + let liveSyncSnap: (sourceId: string) => Promise<{ holder_pid: number; holder_host: string } | null> = + async () => null; + try { + const { inspectLock, syncLockId } = await import('../core/db-lock.ts'); + liveSyncSnap = async (sourceId: string) => { + try { + const snap = await inspectLock(engine, syncLockId(sourceId)); + return snap && !snap.ttl_expired + ? { holder_pid: snap.holder_pid, holder_host: snap.holder_host } + : null; + } catch { + return null; + } + }; + } catch { + /* db-lock unavailable — skip in-progress detection, staleness stands. */ + } + for (const source of sources) { // Embed source.id in user-visible messages so `gbrain sync --source ` // matches what the user copy-pastes. Show display name in parens when set. @@ -3401,6 +3435,15 @@ export async function checkSyncFreshness( ? `'${source.id}' (${source.name})` : `'${source.id}'`; + // BUG 4: actively syncing (live lock) → healthy, count as synced_recently + // and skip the staleness checks. Keeps the 3-bucket invariant intact. + const liveSnap = await liveSyncSnap(source.id); + if (liveSnap) { + inProgress.push(`${display} sync in progress (pid ${liveSnap.holder_pid} on ${liveSnap.holder_host})`); + synced_recently_count++; + continue; + } + if (!source.last_sync_at) { issues.push(`Source ${display} has never been synced`); hasFailures = true; @@ -3486,12 +3529,15 @@ export async function checkSyncFreshness( // D6 invariant: every source incremented exactly one bucket. const details = { unchanged_count, synced_recently_count, stale_count }; + // BUG 4: append in-progress context when any source is actively syncing. + // Empty otherwise, so steady-state messages are byte-for-byte unchanged. + const inProgressNote = inProgress.length ? `. ${inProgress.join('; ')}` : ''; if (hasFailures) { return { name: 'sync_freshness', status: 'fail', - message: `${issues.join('; ')}. Run \`gbrain sync --source \` for each stale source`, + message: `${issues.join('; ')}. Run \`gbrain sync --source \` for each stale source${inProgressNote}`, details, }; } @@ -3499,7 +3545,7 @@ export async function checkSyncFreshness( return { name: 'sync_freshness', status: 'warn', - message: `${issues.join('; ')}. Run \`gbrain sync --source \` to refresh`, + message: `${issues.join('; ')}. Run \`gbrain sync --source \` to refresh${inProgressNote}`, details, }; } @@ -3510,7 +3556,7 @@ export async function checkSyncFreshness( return { name: 'sync_freshness', status: 'ok', - message: `All ${sources.length} federated source(s) up to date (no new commits since last sync)`, + message: `All ${sources.length} federated source(s) up to date (no new commits since last sync)${inProgressNote}`, details, }; } @@ -3518,14 +3564,14 @@ export async function checkSyncFreshness( return { name: 'sync_freshness', status: 'ok', - message: `${sources.length} federated source(s): ${synced_recently_count} synced recently, ${unchanged_count} unchanged since last sync`, + message: `${sources.length} federated source(s): ${synced_recently_count} synced recently, ${unchanged_count} unchanged since last sync${inProgressNote}`, details, }; } return { name: 'sync_freshness', status: 'ok', - message: `All ${sources.length} federated source(s) synced recently`, + message: `All ${sources.length} federated source(s) synced recently${inProgressNote}`, details, }; } catch (e) { diff --git a/src/commands/sync.ts b/src/commands/sync.ts index d788cd201..856900bf0 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -1224,7 +1224,7 @@ async function formatLockBusyMessage(engine: BrainEngine, lockKey: string): Prom * with another break-lock or with TTL-eviction can't produce confusing * post-conditions. */ -async function runBreakLock( +export async function runBreakLock( engine: BrainEngine, lockKey: string, sourceId: string, @@ -1243,6 +1243,25 @@ async function runBreakLock( } if (!snap) { + // BUG 5 (v0.42.x): --force-break-lock used to emit the same terse "not + // held" line and exit 0 even when a sync was genuinely wedged — sending the + // operator down a dead end (the wedge was not a held lock). Keep rc=0 + // (breaking a non-existent lock is idempotently successful; flipping the + // exit code would break automation that treats it as success), but under + // --force say plainly that nothing was broken and point at the real next + // step. The non-force path message is unchanged. + if (opts.force) { + const wedgeHint = + `No lock is held on ${lockKey} — nothing to break. If a sync still ` + + `appears wedged, the cause is not a held lock; inspect checkpoint/resume ` + + `state with \`gbrain sync --source ${sourceId}\` or \`gbrain doctor\`.`; + if (opts.json) { + console.log(JSON.stringify({ status: 'absent', lock: lockKey, source_id: sourceId, wedge_hint: wedgeHint })); + } else { + console.log(wedgeHint); + } + return 0; + } if (opts.json) console.log(JSON.stringify({ status: 'absent', lock: lockKey, source_id: sourceId })); else console.log(`Lock ${lockKey} is not held (nothing to break).`); return 0; diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 019f02ea4..acc57aec2 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -5270,6 +5270,103 @@ export const MIGRATIONS: Migration[] = [ ON context_volunteer_events (source_id, slug); `, }, + { + version: 118, + name: 'page_generation_clock_sequence_swap', + // v0.42.x — contention-free page-generation clock. The v107 single-row + // `UPDATE page_generation_clock SET value = value + 1 WHERE id = 1` took a + // transaction-length RowExclusiveLock on one tuple, serializing every + // concurrent page writer on the prior writer's COMMIT (sync ran at ~0.8 + // cores regardless of worker count). Swap to a SEQUENCE: nextval() takes a + // microsecond LWLock, never a row lock. The Layer-1 cache bookmark reads + // `last_value` instead of the row. + // + // Correctness: `last_value` is non-transactional — it can reflect + // rolled-back or concurrent-uncommitted writers. That is the SAFE direction + // (cache OVER-invalidates, never serves stale). The clock's only contract is + // monotonic advancement on any page INSERT/UPDATE/DELETE. + // + // The setval is LOAD-BEARING: a fresh CREATE SEQUENCE has is_called=false, + // so the first nextval() returns the start value (1) and last_value would + // not visibly advance. The 2-arg setval (is_called=true) makes the first + // post-seed write strictly exceed the seed. Floor 1 (sequence MINVALUE); + // seed >= old clock and MAX(pages.generation) so monotonicity holds. + // + // We keep the table + trigger + function NAMES; only the function body and + // the three readers in query-cache-gate.ts repoint. DELETE FROM query_cache + // so no bookmark stamped under the old table-clock survives the swap. + // Mirrors: src/schema.sql, src/core/pglite-schema.ts (and the generated + // src/core/schema-embedded.ts) ship the sequence on fresh install. + // + // pages.generation (Layer 2) is assigned by the SEPARATE row-level trigger + // bump_page_generation_fn — untouched here. + idempotent: true, + sql: ` + CREATE SEQUENCE IF NOT EXISTS page_generation_clock_seq; + + SELECT setval('page_generation_clock_seq', GREATEST( + 1, + COALESCE((SELECT last_value FROM page_generation_clock_seq), 0), + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0), + COALESCE((SELECT MAX(generation) FROM pages), 0) + )); + + CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ + BEGIN + PERFORM nextval('page_generation_clock_seq'); + RETURN NULL; + END; + $func$ LANGUAGE plpgsql; + + DROP TRIGGER IF EXISTS bump_page_generation_clock_trg ON pages; + CREATE TRIGGER bump_page_generation_clock_trg + AFTER INSERT OR UPDATE OR DELETE ON pages + FOR EACH STATEMENT + EXECUTE FUNCTION bump_page_generation_clock_fn(); + + DELETE FROM query_cache; + `, + }, + { + version: 119, + name: 'op_checkpoints_completed_keys_array_check', + // v0.42.x — make the op_checkpoints scalar-corruption class structurally + // impossible. completed_keys is JSONB and the loader runs + // jsonb_array_elements_text(completed_keys); a non-array (scalar) value + // makes that throw "cannot extract elements from a scalar", which takes + // down the whole UNION load (including the valid op_checkpoint_paths child + // rows) and loses all checkpoint progress for that key. No current writer + // can produce a scalar, but an older binary / external script / future bug + // could — the CHECK is a DB-enforced, always-on guard (the correct pattern + // vs a migration verify-hook, which would not run on already-stamped + // brains). LOCK first so an out-of-band scalar write can't land between the + // repair and the ADD CONSTRAINT (no-op on single-connection PGLite). The + // repair resets any pre-existing scalar to '[]'; op_checkpoint_paths child + // rows are the append-only source of truth, so the reset loses nothing. + // Mirrored in src/schema.sql, src/core/pglite-schema.ts, and the generated + // src/core/schema-embedded.ts so fresh installs carry the same CHECK. + idempotent: true, + sql: ` + LOCK TABLE op_checkpoints IN SHARE ROW EXCLUSIVE MODE; + + UPDATE op_checkpoints + SET completed_keys = '[]'::jsonb, updated_at = now() + WHERE jsonb_typeof(completed_keys) <> 'array'; + + DO $$ + BEGIN + IF NOT EXISTS ( + SELECT 1 FROM pg_constraint + WHERE conname = 'op_checkpoints_completed_keys_array' + AND conrelid = 'op_checkpoints'::regclass + ) THEN + ALTER TABLE op_checkpoints + ADD CONSTRAINT op_checkpoints_completed_keys_array + CHECK (jsonb_typeof(completed_keys) = 'array'); + END IF; + END $$; + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/op-checkpoint.ts b/src/core/op-checkpoint.ts index a338d20b6..f1c0c1e2e 100644 --- a/src/core/op-checkpoint.ts +++ b/src/core/op-checkpoint.ts @@ -122,18 +122,40 @@ export async function loadOpCheckpoint( // dedupes, so we skip a server-side dedup sort over up to 204K rows on every // resume. `jsonb_array_elements_text` expands the legacy array server-side, // which also removes the old postgres.js-vs-PGLite string/array handling. - const rows = await engine.executeRaw<{ ckey: unknown }>( - `SELECT path AS ckey FROM op_checkpoint_paths + // + // v0.42.x (BUG 3 guard): the legacy arm is gated on + // `jsonb_typeof(completed_keys) = 'array'`. Without it a non-array (scalar) + // parent row makes jsonb_array_elements_text throw "cannot extract elements + // from a scalar", which kills the WHOLE union — including the valid child + // rows — and loses all checkpoint progress for the key. Skipping the scalar + // keeps the child rows; the third arm flags the corruption so we log it once + // (migration v119's CHECK makes this impossible going forward; a hit implies + // schema drift / disabled constraint / an out-of-band writer). + const rows = await engine.executeRaw<{ ckey: unknown; corrupt: number }>( + `SELECT path AS ckey, 0 AS corrupt FROM op_checkpoint_paths WHERE op = $1 AND fingerprint = $2 UNION ALL - SELECT jsonb_array_elements_text(completed_keys) AS ckey FROM op_checkpoints - WHERE op = $1 AND fingerprint = $2`, + SELECT jsonb_array_elements_text(completed_keys) AS ckey, 0 AS corrupt FROM op_checkpoints + WHERE op = $1 AND fingerprint = $2 AND jsonb_typeof(completed_keys) = 'array' + UNION ALL + SELECT NULL AS ckey, 1 AS corrupt FROM op_checkpoints + WHERE op = $1 AND fingerprint = $2 AND jsonb_typeof(completed_keys) <> 'array'`, [key.op, key.fingerprint], ); const set = new Set(); + let corruptParent = false; for (const r of rows) { + if (Number(r.corrupt) === 1) { + corruptParent = true; + continue; + } if (typeof r.ckey === 'string') set.add(r.ckey); } + if (corruptParent) { + console.error( + `[op-checkpoint] WARNING: op_checkpoints.completed_keys for (${key.op}, ${key.fingerprint}) is a non-array (scalar) and was skipped to protect the load — child op_checkpoint_paths rows still applied. This implies schema drift, a disabled CHECK constraint, or an out-of-band writer.`, + ); + } return [...set]; } catch (e) { console.error(`[op-checkpoint] load failed (${key.op}, ${key.fingerprint}):`, (e as Error).message); diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index b0d7da947..1a0bc6c19 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -164,9 +164,24 @@ INSERT INTO page_generation_clock (id, value) VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) ON CONFLICT (id) DO NOTHING; +-- v0.42.x: contention-free clock. nextval() takes a microsecond LWLock, not a +-- transaction-length row lock. Load-bearing setval (2-arg -> is_called=true) so +-- the first write strictly exceeds the seed; floor 1 (sequence MINVALUE). +-- Layer-1 reads last_value. Table + trigger names retained. +CREATE SEQUENCE IF NOT EXISTS page_generation_clock_seq; +-- Monotonic seed: GREATEST over the sequence's OWN last_value too, so replaying +-- this blob on an already-upgraded brain (initSchema is re-runnable) can never +-- move last_value BACKWARD below a stored query_cache bookmark. +SELECT setval('page_generation_clock_seq', GREATEST( + 1, + COALESCE((SELECT last_value FROM page_generation_clock_seq), 0), + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0), + COALESCE((SELECT MAX(generation) FROM pages), 0) +)); + CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ BEGIN - UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + PERFORM nextval('page_generation_clock_seq'); RETURN NULL; END; $func$ LANGUAGE plpgsql; @@ -924,7 +939,11 @@ CREATE TABLE IF NOT EXISTS oauth_codes ( CREATE TABLE IF NOT EXISTS op_checkpoints ( op TEXT NOT NULL, fingerprint TEXT NOT NULL, - completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb, + -- v0.42.x: must be a JSONB array. The loader runs jsonb_array_elements_text + -- over it; a scalar would throw and wipe the whole checkpoint load. CHECK is + -- the DB-enforced always-on guard (mirrors migration v119). + completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb + CONSTRAINT op_checkpoints_completed_keys_array CHECK (jsonb_typeof(completed_keys) = 'array'), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (op, fingerprint) ); diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index e0f313dc8..32beb37d0 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -227,9 +227,26 @@ INSERT INTO page_generation_clock (id, value) VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) ON CONFLICT (id) DO NOTHING; +-- v0.42.x: contention-free clock. nextval() takes a microsecond LWLock, not a +-- transaction-length row lock, so concurrent page writers no longer serialize +-- on one tuple's COMMIT. Load-bearing setval (2-arg -> is_called=true) so the +-- first write strictly exceeds the seed; floor 1 (sequence MINVALUE). Layer-1 +-- reads last_value. The table + trigger names are retained. +CREATE SEQUENCE IF NOT EXISTS page_generation_clock_seq; +-- Monotonic seed: GREATEST over the sequence's OWN last_value too, so replaying +-- this blob on an already-upgraded brain (initSchema is re-runnable) can never +-- move last_value BACKWARD below a stored query_cache bookmark (which would let +-- Layer 1 serve stale rows). Mirrors the old table's ON CONFLICT DO NOTHING. +SELECT setval('page_generation_clock_seq', GREATEST( + 1, + COALESCE((SELECT last_value FROM page_generation_clock_seq), 0), + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0), + COALESCE((SELECT MAX(generation) FROM pages), 0) +)); + CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS \$func\$ BEGIN - UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + PERFORM nextval('page_generation_clock_seq'); RETURN NULL; END; \$func\$ LANGUAGE plpgsql; @@ -689,7 +706,11 @@ CREATE INDEX IF NOT EXISTS idx_mcp_log_agent_time ON mcp_request_log(agent_name, CREATE TABLE IF NOT EXISTS op_checkpoints ( op TEXT NOT NULL, fingerprint TEXT NOT NULL, - completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb, + -- v0.42.x: must be a JSONB array. The loader runs jsonb_array_elements_text + -- over it; a scalar would throw and wipe the whole checkpoint load. CHECK is + -- the DB-enforced always-on guard (mirrors migration v119). + completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb + CONSTRAINT op_checkpoints_completed_keys_array CHECK (jsonb_typeof(completed_keys) = 'array'), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (op, fingerprint) ); diff --git a/src/core/search/query-cache-gate.ts b/src/core/search/query-cache-gate.ts index be2ec3607..6b72bd310 100644 --- a/src/core/search/query-cache-gate.ts +++ b/src/core/search/query-cache-gate.ts @@ -5,9 +5,14 @@ * Two pure helpers wired by query-cache.ts at store + lookup time. Pure * surface lets us unit-test the two-layer gate logic without a real cache. * - * Layer 1 (cheap bookmark): `page_generation_clock.value` <= + * Layer 1 (cheap bookmark): `page_generation_clock_seq` last_value <= * `query_cache.max_generation_at_store`. If true, no page write has * happened since this row stored, so the row is fresh corpus-wide. + * v0.42.x: the bookmark source switched from a locked single-row counter + * (`page_generation_clock.value`) to a contention-free SEQUENCE bumped by + * `nextval()` in the statement trigger. `last_value` is non-transactional — + * it can reflect a rolled-back or concurrent-uncommitted writer, which only + * ever OVER-invalidates (loses a cache hit), never serves stale. * * Layer 2 (per-page snapshot): if bookmark fires, fall through to the * `page_generations JSONB` snapshot. For each `(page_id, stored_gen)` @@ -98,7 +103,7 @@ export async function buildPageGenerationsSnapshot( // Per D20, empty-result cache rows trust Layer 1 exclusively; // bumping the clock on subsequent writes correctly invalidates them. const rows = await engine.executeRaw<{ v: number }>( - `SELECT COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v`, + `SELECT COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v`, ); snapshot.max_generation_at_store = Number(rows[0]?.v ?? 0); return snapshot; @@ -117,7 +122,7 @@ export async function buildPageGenerationsSnapshot( FROM pages WHERE id = ANY($1::int[]) UNION ALL SELECT 'CLOCK' AS k, - COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0)::bigint AS v, + COALESCE((SELECT last_value FROM page_generation_clock_seq), 0)::bigint AS v, TRUE AS is_max`, [pageIds], ); @@ -132,12 +137,12 @@ export async function buildPageGenerationsSnapshot( } return snapshot; } catch { - // Pre-v105 brain (no `page_generation_clock` table yet). Return the - // empty snapshot with zero bookmark — every cache row will fall - // through to Layer 2 (which is stricter post-v0.41.19.0 and will - // invalidate empty snapshots). Acceptable upgrade-path one-time - // cache miss; migration v105 fills the table within the same - // initSchema() call so this branch is short-lived. + // Pre-sequence brain mid-upgrade (no `page_generation_clock_seq` yet). + // Return the empty snapshot with zero bookmark — every cache row will + // fall through to Layer 2 (which is stricter post-v0.41.19.0 and will + // invalidate empty snapshots). Acceptable upgrade-path one-time cache + // miss; migration v118 creates the sequence within the same initSchema() + // call so this branch is short-lived. return snapshot; } } @@ -157,11 +162,12 @@ export async function buildPageGenerationsSnapshot( */ export const CACHE_GATE_WHERE_CLAUSE = ` ( - -- Layer 1 (cheap bookmark): O(1) single-row read from page_generation_clock. - -- Bumped per-statement by bump_page_generation_clock_trg on every INSERT, - -- UPDATE, or DELETE on pages. If no statement has fired since this row - -- stored, the row is fresh corpus-wide. - COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0) + -- Layer 1 (cheap bookmark): O(1) read of page_generation_clock_seq.last_value. + -- The sequence is advanced (nextval) per-statement by bump_page_generation_clock_trg + -- on every INSERT, UPDATE, or DELETE on pages. If no statement has fired + -- since this row stored, last_value is unchanged and the row is fresh + -- corpus-wide. Non-transactional read: only ever over-invalidates. + COALESCE((SELECT last_value FROM page_generation_clock_seq), 0) <= qc.max_generation_at_store OR -- Layer 2 (per-page snapshot): bookmark fired, but maybe THIS row's diff --git a/src/schema.sql b/src/schema.sql index d05b45117..dcf9455e1 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -223,9 +223,26 @@ INSERT INTO page_generation_clock (id, value) VALUES (1, COALESCE((SELECT MAX(generation) FROM pages), 0)) ON CONFLICT (id) DO NOTHING; +-- v0.42.x: contention-free clock. nextval() takes a microsecond LWLock, not a +-- transaction-length row lock, so concurrent page writers no longer serialize +-- on one tuple's COMMIT. Load-bearing setval (2-arg -> is_called=true) so the +-- first write strictly exceeds the seed; floor 1 (sequence MINVALUE). Layer-1 +-- reads last_value. The table + trigger names are retained. +CREATE SEQUENCE IF NOT EXISTS page_generation_clock_seq; +-- Monotonic seed: GREATEST over the sequence's OWN last_value too, so replaying +-- this blob on an already-upgraded brain (initSchema is re-runnable) can never +-- move last_value BACKWARD below a stored query_cache bookmark (which would let +-- Layer 1 serve stale rows). Mirrors the old table's ON CONFLICT DO NOTHING. +SELECT setval('page_generation_clock_seq', GREATEST( + 1, + COALESCE((SELECT last_value FROM page_generation_clock_seq), 0), + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0), + COALESCE((SELECT MAX(generation) FROM pages), 0) +)); + CREATE OR REPLACE FUNCTION bump_page_generation_clock_fn() RETURNS trigger AS $func$ BEGIN - UPDATE page_generation_clock SET value = value + 1 WHERE id = 1; + PERFORM nextval('page_generation_clock_seq'); RETURN NULL; END; $func$ LANGUAGE plpgsql; @@ -685,7 +702,11 @@ CREATE INDEX IF NOT EXISTS idx_mcp_log_agent_time ON mcp_request_log(agent_name, CREATE TABLE IF NOT EXISTS op_checkpoints ( op TEXT NOT NULL, fingerprint TEXT NOT NULL, - completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb, + -- v0.42.x: must be a JSONB array. The loader runs jsonb_array_elements_text + -- over it; a scalar would throw and wipe the whole checkpoint load. CHECK is + -- the DB-enforced always-on guard (mirrors migration v119). + completed_keys JSONB NOT NULL DEFAULT '[]'::jsonb + CONSTRAINT op_checkpoints_completed_keys_array CHECK (jsonb_typeof(completed_keys) = 'array'), updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), PRIMARY KEY (op, fingerprint) ); diff --git a/test/conversation-parser/llm-base.test.ts b/test/conversation-parser/llm-base.test.ts index c09ececff..ad51cbb8c 100644 --- a/test/conversation-parser/llm-base.test.ts +++ b/test/conversation-parser/llm-base.test.ts @@ -12,7 +12,7 @@ */ import { describe, expect, test, beforeEach } from 'bun:test'; -import { withEnv } from '../helpers/with-env.ts'; +import { withEnv, emptyHome } from '../helpers/with-env.ts'; import { runLlmCall, parseLlmJson, @@ -29,7 +29,7 @@ beforeEach(() => { describe('probeLlmAvailability', () => { test('returns null when ANTHROPIC_API_KEY is unset', async () => { await withEnv( - { ANTHROPIC_API_KEY: undefined as unknown as string }, + { ANTHROPIC_API_KEY: undefined as unknown as string, GBRAIN_HOME: emptyHome() }, async () => { expect(probeLlmAvailability('claude-haiku-4-5')).toBeNull(); expect(probeLlmAvailability('anthropic:claude-haiku-4-5')).toBeNull(); @@ -94,7 +94,7 @@ describe('runLlmCall — happy path', () => { describe('runLlmCall — fail-open paths', () => { test('provider unavailable returns null without calling transport', async () => { await withEnv( - { ANTHROPIC_API_KEY: undefined as unknown as string }, + { ANTHROPIC_API_KEY: undefined as unknown as string, GBRAIN_HOME: emptyHome() }, async () => { let calls = 0; const result = await runLlmCall({ diff --git a/test/conversation-parser/llm-fallback.test.ts b/test/conversation-parser/llm-fallback.test.ts index cebbed249..b7121643a 100644 --- a/test/conversation-parser/llm-fallback.test.ts +++ b/test/conversation-parser/llm-fallback.test.ts @@ -12,7 +12,7 @@ */ import { describe, expect, test, beforeEach } from 'bun:test'; -import { withEnv } from '../helpers/with-env.ts'; +import { withEnv, emptyHome } from '../helpers/with-env.ts'; import { runLlmFallback } from '../../src/core/conversation-parser/llm-fallback.ts'; import { _resetLlmCacheForTests } from '../../src/core/conversation-parser/llm-base.ts'; import { makeChatResult } from './helpers.ts'; @@ -71,7 +71,7 @@ describe('runLlmFallback', () => { test('provider unavailable: returns null without calling transport', async () => { await withEnv( - { ANTHROPIC_API_KEY: undefined as unknown as string }, + { ANTHROPIC_API_KEY: undefined as unknown as string, GBRAIN_HOME: emptyHome() }, async () => { let calls = 0; const result = await runLlmFallback({ diff --git a/test/doctor-ze-checks.test.ts b/test/doctor-ze-checks.test.ts index d98059e3b..aebf5d33c 100644 --- a/test/doctor-ze-checks.test.ts +++ b/test/doctor-ze-checks.test.ts @@ -11,7 +11,7 @@ import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { resetPgliteState } from './helpers/reset-pglite.ts'; -import { withEnv } from './helpers/with-env.ts'; +import { withEnv, emptyHome } from './helpers/with-env.ts'; import { checkZeEmbeddingHealth, checkEmbeddingWidthConsistency, @@ -58,8 +58,10 @@ describe('checkZeEmbeddingHealth', () => { embedding_dimensions: 1280, env: { ...process.env, ZEROENTROPY_API_KEY: undefined as any }, }); - // Clear the env var for the no-key path (user's real env may have it set). - await withEnv({ ZEROENTROPY_API_KEY: undefined }, async () => { + // Clear the env var AND isolate GBRAIN_HOME for the no-key path: the check + // reads ZEROENTROPY_API_KEY from env OR the gbrain config file, so a dev + // machine whose real ~/.gbrain/config.json holds the key needs both cleared. + await withEnv({ ZEROENTROPY_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => { const check = await checkZeEmbeddingHealth(engine); expect(check.status).toBe('warn'); expect(check.message).toContain('ZEROENTROPY_API_KEY'); diff --git a/test/doctor.test.ts b/test/doctor.test.ts index fab1201d0..5b68a6931 100644 --- a/test/doctor.test.ts +++ b/test/doctor.test.ts @@ -1330,3 +1330,114 @@ describe('v0.42 (#1699) — quarantined_pages + flagged_pages checks', () => { expect(source).toMatch(/name: 'flagged_pages'/); }); }); + +// ============================================================================ +// BUG 4 (v0.42.x) — doctor reports an actively-running sync via the live lock, +// not stale freshness. Uses a REAL PGLiteEngine so inspectLock/syncLockId run +// against actual gbrain_cycle_locks rows (the stub engine can't model a lock). +// ============================================================================ +describe('BUG 4 — in-progress sync via live lock, not stale freshness', () => { + let engine: any; + let syncLockId: (s: string) => string; + + beforeAll(async () => { + const { PGLiteEngine } = await import('../src/core/pglite-engine.ts'); + ({ syncLockId } = await import('../src/core/db-lock.ts')); + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }); + + afterAll(async () => { + await engine.disconnect(); + }); + + beforeEach(async () => { + const { resetPgliteState } = await import('./helpers/reset-pglite.ts'); + await resetPgliteState(engine); + await engine.executeRaw(`DELETE FROM gbrain_cycle_locks`); + }); + + const staleDate = () => new Date(Date.now() - 5 * 24 * 60 * 60 * 1000); // 5d ago + + async function addSource(id: string, lastSyncAt: Date | null) { + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path, last_sync_at, config) + VALUES ($1, $1, $2, $3, '{"federated":true}'::jsonb) + ON CONFLICT (id) DO UPDATE SET last_sync_at = EXCLUDED.last_sync_at`, + [id, `/tmp/${id}`, lastSyncAt], + ); + } + + // ttlMinutes > 0 → live lock; <= 0 → already-expired (wedged) holder. + async function holdLock(sourceId: string, ttlMinutes: number) { + await engine.executeRaw( + `INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at, last_refreshed_at) + VALUES ($1, 4242, 'testhost', now(), now() + ($2 || ' minutes')::interval, now())`, + [syncLockId(sourceId), String(ttlMinutes)], + ); + } + + test('stale source with NO live lock → fail (blocked/wedged is not masked)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + await addSource('wiki', staleDate()); + const result = await checkSyncFreshness(engine); + expect(result.status).toBe('fail'); + expect(result.message).toContain(`'wiki'`); + }); + + test('stale source WITH a live (non-expired) lock → ok (sync in progress)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + await addSource('wiki', staleDate()); + await holdLock('wiki', 30); + const result = await checkSyncFreshness(engine); + expect(result.status).toBe('ok'); + expect(result.details?.synced_recently_count).toBe(1); + expect(result.details?.stale_count).toBe(0); + // BUG 4: operator sees the in-progress holder, not silence. + expect(result.message).toContain('sync in progress'); + expect(result.message).toContain('pid 4242'); + }); + + test('never-synced source WITH a live lock → ok (initial sync in progress)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + await addSource('wiki', null); + await holdLock('wiki', 30); + const result = await checkSyncFreshness(engine); + expect(result.status).toBe('ok'); + expect(result.details?.synced_recently_count).toBe(1); + expect(result.message).toContain('sync in progress'); + }); + + test('never-synced source with NO lock → fail (unchanged behavior)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + await addSource('wiki', null); + const result = await checkSyncFreshness(engine); + expect(result.status).toBe('fail'); + expect(result.message).toContain('never been synced'); + }); + + test('expired-TTL lock does NOT mask staleness (wedged-but-not-refreshing holder)', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + await addSource('wiki', staleDate()); + await holdLock('wiki', -5); // ttl_expires_at 5 min in the past + const result = await checkSyncFreshness(engine); + expect(result.status).toBe('fail'); + }); + + test('blocked source with banked checkpoint rows but NO live lock → still fail', async () => { + const { checkSyncFreshness } = await import('../src/commands/doctor.ts'); + await addSource('wiki', staleDate()); + // A blocked sync banks the good files then exits without an anchor and + // without a held lock. Banking must NOT be read as "in progress". + await engine.executeRaw( + `INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at) + VALUES ('sync', 'fp-blocked', '[]'::jsonb, now())`, + ); + await engine.executeRaw( + `INSERT INTO op_checkpoint_paths (op, fingerprint, path) VALUES ('sync', 'fp-blocked', 'banked.md')`, + ); + const result = await checkSyncFreshness(engine); + expect(result.status).toBe('fail'); + }); +}); diff --git a/test/helpers/with-env.ts b/test/helpers/with-env.ts index f4c877f51..5b2253c81 100644 --- a/test/helpers/with-env.ts +++ b/test/helpers/with-env.ts @@ -1,3 +1,7 @@ +import { mkdtempSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; + /** * Run a callback with `process.env` mutations applied, then restore the prior * values via try/finally. The canonical pattern for env-touching tests in this @@ -69,3 +73,17 @@ export async function withEnv( } } } + +/** + * A fresh empty temp dir for `GBRAIN_HOME`, so `loadConfig()` / `configDir()` + * resolve to a directory with no config.json. Pair with a `withEnv` override + * (`GBRAIN_HOME: emptyHome()`) on any "no key" assertion: `hasAnthropicKey()` + * and the ZE/embedding key probes read BOTH the env var AND the gbrain config + * file, so clearing only the env var is NOT hermetic on a dev machine whose + * real `~/.gbrain/config.json` holds a key — the assertion flips and the test + * fails locally while passing in key-less CI. The dir is tiny and intentionally + * leaked (test process is short-lived); the OS reaps tmp. + */ +export function emptyHome(): string { + return mkdtempSync(join(tmpdir(), 'gbrain-nokey-home-')); +} diff --git a/test/op-checkpoint.test.ts b/test/op-checkpoint.test.ts index 6c821e38c..b5fe46127 100644 --- a/test/op-checkpoint.test.ts +++ b/test/op-checkpoint.test.ts @@ -243,6 +243,82 @@ describe('resumeFilter (pure)', () => { }); }); +describe('BUG 3: completed_keys array-shape guard (v119 CHECK + defensive loader)', () => { + const CONSTRAINT = 'op_checkpoints_completed_keys_array'; + + test('CHECK rejects a scalar completed_keys write — exactly one constraint (no blob+migration dupe)', async () => { + // Fresh PGLite install: the schema blob ships the NAMED inline CHECK and + // migration v119's IF NOT EXISTS skips re-adding it. Exactly one constraint. + const c = await engine.executeRaw<{ n: number }>( + `SELECT count(*)::int AS n FROM pg_constraint WHERE conname = $1`, + [CONSTRAINT], + ); + expect(Number(c[0].n)).toBe(1); + + let threw = false; + try { + await engine.executeRaw( + `INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at) + VALUES ('embed', 'fp-reject', '"not-an-array"'::jsonb, now())`, + ); + } catch { + threw = true; + } + expect(threw).toBe(true); + }); + + test('loader survives a scalar parent: returns child rows (not []), does not throw', async () => { + const key = { op: 'sync', fingerprint: 'fp-scalar-survive' }; + // Bypass the CHECK to simulate pre-migration / out-of-band corruption. + await engine.executeRaw(`ALTER TABLE op_checkpoints DROP CONSTRAINT ${CONSTRAINT}`); + try { + await engine.executeRaw( + `INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at) + VALUES ('sync', 'fp-scalar-survive', '"corrupt-scalar"'::jsonb, now())`, + ); + await engine.executeRaw( + `INSERT INTO op_checkpoint_paths (op, fingerprint, path) + VALUES ('sync', 'fp-scalar-survive', 'child-a.md')`, + ); + // Pre-guard, jsonb_array_elements_text on the scalar threw and the catch + // returned [] — losing child-a.md. The typeof guard skips the scalar so + // the valid child survives. + const loaded = await loadOpCheckpoint(engine, key); + expect(loaded).toEqual(['child-a.md']); + } finally { + await engine.executeRaw( + `UPDATE op_checkpoints SET completed_keys = '[]'::jsonb WHERE jsonb_typeof(completed_keys) <> 'array'`, + ); + await engine.executeRaw( + `ALTER TABLE op_checkpoints ADD CONSTRAINT ${CONSTRAINT} CHECK (jsonb_typeof(completed_keys) = 'array')`, + ); + } + }); + + test('v119 repair converts a scalar parent to an empty array', async () => { + await engine.executeRaw(`ALTER TABLE op_checkpoints DROP CONSTRAINT ${CONSTRAINT}`); + try { + await engine.executeRaw( + `INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at) + VALUES ('embed', 'fp-repair', '"scalar"'::jsonb, now())`, + ); + // Migration v119's repair statement. + await engine.executeRaw( + `UPDATE op_checkpoints SET completed_keys = '[]'::jsonb, updated_at = now() + WHERE jsonb_typeof(completed_keys) <> 'array'`, + ); + const typ = await engine.executeRaw<{ t: string }>( + `SELECT jsonb_typeof(completed_keys) AS t FROM op_checkpoints WHERE op = 'embed' AND fingerprint = 'fp-repair'`, + ); + expect(typ[0].t).toBe('array'); + } finally { + await engine.executeRaw( + `ALTER TABLE op_checkpoints ADD CONSTRAINT ${CONSTRAINT} CHECK (jsonb_typeof(completed_keys) = 'array')`, + ); + } + }); +}); + describe('purgeStaleCheckpoints', () => { test('no stale rows: returns 0', async () => { await recordCompleted(engine, { op: 'embed', fingerprint: 'fresh' }, ['x']); diff --git a/test/page-generation-counter.test.ts b/test/page-generation-counter.test.ts index 45748f72f..25f57f94b 100644 --- a/test/page-generation-counter.test.ts +++ b/test/page-generation-counter.test.ts @@ -44,8 +44,11 @@ beforeEach(async () => { }); async function clockValue(): Promise { + // v0.42.x: the Layer-1 bookmark moved from the locked single-row + // page_generation_clock table to a contention-free SEQUENCE bumped by + // nextval() in the statement trigger. Read last_value. const rows = await engine.executeRaw<{ value: number }>( - `SELECT value FROM page_generation_clock WHERE id = 1`, + `SELECT last_value AS value FROM page_generation_clock_seq`, ); return Number(rows[0]?.value ?? -1); } @@ -69,13 +72,20 @@ describe('page_generation_clock table + statement-level trigger', () => { expect(threw).toBe(true); }); - test('seed: clock starts at COALESCE(MAX(pages.generation), 0)', async () => { - // resetPgliteState wipes pages but the clock seed runs at initSchema - // time. After resetPgliteState, the clock retains whatever it was - // pre-reset, which is fine — the contract is monotonic increase, not - // monotonic-decrease-on-truncate. (Production resets don't happen.) + test('seed: clock sequence starts at >= 1 with is_called=true', async () => { + // v0.42.x: the sequence is seeded via setval(GREATEST(1, MAX(generation))) + // at initSchema with is_called=true (2-arg setval), so the FIRST write's + // nextval strictly exceeds the seed. Without is_called=true a fresh + // sequence's first nextval returns the start value and last_value would not + // visibly advance — that would let a fresh install serve a stale cache row. + // resetPgliteState does NOT reset the sequence (sequences aren't pg_tables), + // so last_value only ever increases — monotonic, never decrease-on-truncate. const v = await clockValue(); - expect(v).toBeGreaterThanOrEqual(0); + expect(v).toBeGreaterThanOrEqual(1); + const meta = await engine.executeRaw<{ is_called: boolean }>( + `SELECT is_called FROM page_generation_clock_seq`, + ); + expect(meta[0].is_called).toBe(true); }); test('INSERT bumps clock by exactly 1 (single-row insert via raw SQL)', async () => { @@ -225,3 +235,81 @@ describe('query-cache integration (D14 + CDX-6 + CDX-7 end-to-end)', () => { expect(afterClock).toBe(beforeClock + 1); }); }); + +describe('v0.42.x sequence-backed clock (BUG 1: contention removal)', () => { + test('mechanism: trigger function uses nextval, NOT a locked row UPDATE', async () => { + // The contention source was `UPDATE page_generation_clock SET value=value+1 + // WHERE id=1` (a transaction-length RowExclusiveLock on one tuple). Prove at + // the schema level that it is gone and replaced by nextval (a microsecond + // LWLock). This is the deterministic, PGLite-runnable contention proof. + const rows = await engine.executeRaw<{ src: string }>( + `SELECT prosrc AS src FROM pg_proc WHERE proname = 'bump_page_generation_clock_fn'`, + ); + expect(rows.length).toBe(1); + expect(rows[0].src).toContain("nextval('page_generation_clock_seq')"); + expect(rows[0].src).not.toContain('UPDATE page_generation_clock'); + }); + + test('rollback still advances the sequence (over-invalidation is the SAFE direction)', async () => { + const before = await clockValue(); + // Aborted import: the statement trigger fires nextval (sequences are + // non-transactional), so last_value advances even though the page never + // commits. A cache row stamped before this now fails Layer 1 and + // re-validates — a LOST HIT, never a stale serve. + try { + await engine.transaction(async (tx) => { + await tx.executeRaw( + `INSERT INTO pages (source_id, slug, type, title, compiled_truth, timeline, frontmatter) + VALUES ('default', 'test/rollback-page', 'note', 't', 'body', '', '{}'::jsonb)`, + ); + throw new Error('abort'); + }); + } catch { + /* expected */ + } + const after = await clockValue(); + expect(after).toBeGreaterThan(before); + // The page itself did NOT persist — rollback worked. + const pages = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM pages WHERE slug = 'test/rollback-page' AND source_id = 'default'`, + ); + expect(Number(pages[0].n)).toBe(0); + }); + + test('PGLite supports sequences: CREATE / nextval / setval / last_value round-trip', async () => { + // codex flagged: no existing sequence usage in the repo — prove (not assert) + // that PGLite's WASM Postgres supports the constructs migration v118 relies + // on, including the is_called gotcha the load-bearing setval guards against. + await engine.executeRaw(`CREATE SEQUENCE IF NOT EXISTS test_probe_seq`); + // Fresh sequence (is_called=false): first nextval returns the START value 1, + // and last_value does NOT visibly advance past it — the exact trap. + const n1 = await engine.executeRaw<{ v: number }>(`SELECT nextval('test_probe_seq') AS v`); + expect(Number(n1[0].v)).toBe(1); + const n2 = await engine.executeRaw<{ v: number }>(`SELECT nextval('test_probe_seq') AS v`); + expect(Number(n2[0].v)).toBe(2); + // 2-arg setval → last_value=N, is_called=true; the next nextval = N+1. + await engine.executeRaw(`SELECT setval('test_probe_seq', 100)`); + const lv = await engine.executeRaw<{ v: number }>(`SELECT last_value AS v FROM test_probe_seq`); + expect(Number(lv[0].v)).toBe(100); + const n3 = await engine.executeRaw<{ v: number }>(`SELECT nextval('test_probe_seq') AS v`); + expect(Number(n3[0].v)).toBe(101); + await engine.executeRaw(`DROP SEQUENCE test_probe_seq`); + }); + + test('re-seeding is monotonic: the GREATEST guard never moves last_value backward', async () => { + // Regression for the codex P1: initSchema replays the schema blob, whose + // setval must NOT reset the clock below its current value — a backward move + // would let a stored query_cache bookmark serve stale rows. Push the + // sequence high, then run the EXACT monotonic seed the blob + v118 use. + await engine.executeRaw(`SELECT setval('page_generation_clock_seq', 999999)`); + await engine.executeRaw( + `SELECT setval('page_generation_clock_seq', GREATEST( + 1, + COALESCE((SELECT last_value FROM page_generation_clock_seq), 0), + COALESCE((SELECT value FROM page_generation_clock WHERE id = 1), 0), + COALESCE((SELECT MAX(generation) FROM pages), 0) + ))`, + ); + expect(await clockValue()).toBeGreaterThanOrEqual(999999); + }); +}); diff --git a/test/query-cache-gate.test.ts b/test/query-cache-gate.test.ts index 4dcb0c624..b7db9ae22 100644 --- a/test/query-cache-gate.test.ts +++ b/test/query-cache-gate.test.ts @@ -252,13 +252,16 @@ describe('buildPageGenerationsSnapshot (PGLite-backed)', () => { }); describe('CACHE_GATE_WHERE_CLAUSE (SQL shape regression)', () => { - test('v0.41.19.0: Layer 1 reads page_generation_clock (not MAX(generation))', () => { - expect(CACHE_GATE_WHERE_CLAUSE).toContain('page_generation_clock'); + test('v0.42.x: Layer 1 reads page_generation_clock_seq.last_value (not the locked row, not MAX)', () => { + expect(CACHE_GATE_WHERE_CLAUSE).toContain('page_generation_clock_seq'); + expect(CACHE_GATE_WHERE_CLAUSE).toContain('last_value'); expect(CACHE_GATE_WHERE_CLAUSE).toContain('qc.max_generation_at_store'); // Negative regression guard: the old MAX(generation) read shape MUST // be gone (codex CDX-1/CDX-2: it silently served stale on // UPDATE-to-non-max and DELETE). expect(CACHE_GATE_WHERE_CLAUSE).not.toContain('MAX(generation) FROM pages'); + // The locked single-row read (the BUG 1 contention source) MUST be gone. + expect(CACHE_GATE_WHERE_CLAUSE).not.toContain('value FROM page_generation_clock WHERE id'); }); test('contains Layer 2 per-page snapshot (jsonb_each + LEFT JOIN)', () => { diff --git a/test/sync-break-lock-all.test.ts b/test/sync-break-lock-all.test.ts index b910495b9..d8d50e04d 100644 --- a/test/sync-break-lock-all.test.ts +++ b/test/sync-break-lock-all.test.ts @@ -246,3 +246,67 @@ describe('R6 regression: schema bootstrap includes last_refreshed_at column', () expect(rows[0].is_nullable).toBe('YES'); }); }); + +// ============================================================================ +// BUG 5 (v0.42.x) — honest --force-break-lock diagnostic when no lock is held. +// Previously --force-break-lock emitted the same terse "not held (nothing to +// break)" line and exited 0, sending operators down a dead end when a sync was +// wedged for a reason other than a held lock. +// ============================================================================ +describe('BUG 5 — --force-break-lock honest no-lock diagnostic', () => { + const LOCK = 'gbrain-sync:wiki'; + + test('force + no lock → wedge_hint in JSON, status absent, rc 0', async () => { + const { runBreakLock } = await import('../src/commands/sync.ts'); + const logs: string[] = []; + const orig = console.log; + console.log = (...a: unknown[]) => { logs.push(a.map(String).join(' ')); }; + let rc: number; + try { + rc = await runBreakLock(engine, LOCK, 'wiki', { force: true, json: true }); + } finally { + console.log = orig; + } + expect(rc).toBe(0); + const parsed = JSON.parse(logs[0]); + expect(parsed.status).toBe('absent'); + expect(parsed.lock).toBe(LOCK); + expect(typeof parsed.wedge_hint).toBe('string'); + expect(parsed.wedge_hint).toContain('not a held lock'); + }); + + test('force + no lock → human output carries the wedge hint, not the terse line', async () => { + const { runBreakLock } = await import('../src/commands/sync.ts'); + const logs: string[] = []; + const orig = console.log; + console.log = (...a: unknown[]) => { logs.push(a.map(String).join(' ')); }; + let rc: number; + try { + rc = await runBreakLock(engine, LOCK, 'wiki', { force: true, json: false }); + } finally { + console.log = orig; + } + expect(rc).toBe(0); + const out = logs.join('\n'); + expect(out).toContain('nothing to break'); + expect(out).toContain('gbrain doctor'); + expect(out).not.toBe(`Lock ${LOCK} is not held (nothing to break).`); + }); + + test('non-force + no lock → unchanged terse line, no wedge_hint', async () => { + const { runBreakLock } = await import('../src/commands/sync.ts'); + const logs: string[] = []; + const orig = console.log; + console.log = (...a: unknown[]) => { logs.push(a.map(String).join(' ')); }; + let rc: number; + try { + rc = await runBreakLock(engine, LOCK, 'wiki', { force: false, json: true }); + } finally { + console.log = orig; + } + expect(rc).toBe(0); + const parsed = JSON.parse(logs[0]); + expect(parsed.status).toBe('absent'); + expect(parsed.wedge_hint).toBeUndefined(); + }); +}); diff --git a/test/think-gateway-adapter.test.ts b/test/think-gateway-adapter.test.ts index d9788af04..dbe830ce6 100644 --- a/test/think-gateway-adapter.test.ts +++ b/test/think-gateway-adapter.test.ts @@ -18,7 +18,7 @@ import { describe, test, expect } from 'bun:test'; import { __thinkAdapter } from '../src/core/think/index.ts'; import { resetGateway } from '../src/core/ai/gateway.ts'; -import { withEnv } from './helpers/with-env.ts'; +import { withEnv, emptyHome } from './helpers/with-env.ts'; describe('think gateway adapter — response shape conversion', () => { test('chatResultToMessage maps ChatResult.text to Anthropic.Message content[0].text', () => { @@ -76,7 +76,7 @@ describe('think gateway adapter — model-id normalization', () => { }); test('tryBuildGatewayClient returns null when ANTHROPIC_API_KEY is absent (preserves legacy NO_ANTHROPIC_API_KEY signal)', async () => { - await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => { + await withEnv({ ANTHROPIC_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => { const client = await __thinkAdapter.tryBuildGatewayClient('claude-opus-4-7'); expect(client).toBeNull(); }); @@ -86,7 +86,7 @@ describe('think gateway adapter — model-id normalization', () => { await withEnv({ ANTHROPIC_API_KEY: 'sk-test-key' }, async () => { expect(__thinkAdapter.hasAnthropicKey()).toBe(true); }); - await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => { + await withEnv({ ANTHROPIC_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => { expect(__thinkAdapter.hasAnthropicKey()).toBe(false); }); }); @@ -115,7 +115,7 @@ describe('think gateway adapter — #1698 slash form + explicit-model fork', () }); test('explicit anthropic model with no key THROWS (unavailable)', async () => { - await withEnv({ ANTHROPIC_API_KEY: undefined }, async () => { + await withEnv({ ANTHROPIC_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => { await expect( __thinkAdapter.tryBuildGatewayClient('anthropic:claude-sonnet-4-6', { explicitModel: true }), ).rejects.toThrow(/not usable.*unavailable/); @@ -164,7 +164,7 @@ describe('think gateway adapter — #1698 slash form + explicit-model fork', () // 'no LLM available' stub). A future refactor that turns this into a graceful path fails here. test('D1 backstop: explicit non-anthropic model, no key → BUILDS then create() THROWS (never a stub)', async () => { await withEnv( - { ANTHROPIC_API_KEY: undefined, DEEPSEEK_API_KEY: undefined, OPENAI_API_KEY: undefined }, + { ANTHROPIC_API_KEY: undefined, DEEPSEEK_API_KEY: undefined, OPENAI_API_KEY: undefined, GBRAIN_HOME: emptyHome() }, async () => { resetGateway(); // unconfigured → gateway.chat() throws AIConfigError at create() // deepseek:deepseek-chat passes validateModelId (real recipe + chat touchpoint) — the