diff --git a/CHANGELOG.md b/CHANGELOG.md index 77bc408ae..a719a2b01 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,94 @@ All notable changes to GBrain will be documented in this file. +## [0.41.8.0] - 2026-05-24 + +**`gbrain search`, `gbrain query`, and `gbrain get` now actually exit when they finish on PGLite.** + +Until today, if you ran `gbrain search "fox"` on a PGLite brain, the results printed in under a second... and then the process sat there at ~95-98% CPU forever. You had to kill it with Ctrl-C. Scripted callers (`gbrain search "x" && echo ok`) never reached the `&& echo ok`. Cron jobs timed out. This was the #1 community pain reported since v0.37, with five open issues — three for the search-hang shape ([#1247](https://github.com/garrytan/gbrain/issues/1247), [#1269](https://github.com/garrytan/gbrain/issues/1269), [#1290](https://github.com/garrytan/gbrain/issues/1290)) plus a related WASM-init failure ([#1340](https://github.com/garrytan/gbrain/issues/1340)) and a single-reporter sync hang ([#1342](https://github.com/garrytan/gbrain/issues/1342)). + +The fix turns out to be one structural change in two parts. v0.37 added a stale-page-detection feature that fires a background `UPDATE pages SET last_retrieved_at = NOW()` after every search/query/get. The CLI's job is to print the results, close the database, and exit — but on PGLite the database is a WASM runtime that holds Bun's event loop alive while the background UPDATE is still in flight. Closing the database mid-write strands the write on a dead handle, and Bun never notices the process is supposed to exit. So now the CLI explicitly waits for the background write to finish before closing the database. On the rare pathological case where the wait itself takes more than 5 seconds (a future bug we haven't seen yet but want to defend against), we log a stderr warning naming the leak and force-exit cleanly. Daemons (`gbrain serve`, `gbrain serve --http`) are explicitly excluded from the force-exit so they stay running. + +For #1340 (PGLite WASM init failing on older macOS + Bun 1.3.x), the error message now correctly identifies the root cause as Bun's vfs read-only mount rather than the unrelated macOS 26.3 WASM bug: + +```text +PGLite failed to initialize its WASM runtime. + This looks like a Bun vfs issue: `/$$bunfs/root` is read-only on + your system, so PGLite cannot extract its pglite.data WASM payload. + Fix: `bun upgrade` (newer Bun mounts the vfs writable). If that + does not help, run via Node: `node src/cli.ts` or install gbrain + using the Node-based path. See #1340 for details. +``` + +### The numbers that matter + +| Scenario | Before v0.41.8.0 | After v0.41.8.0 | +|---|---|---| +| `gbrain search "x"` on PGLite, exit time | Never (hangs at ~95-98% CPU until SIGKILL) | <2s | +| `gbrain query "x" --no-expand` on PGLite | Never exits | <2s | +| `gbrain get ` on PGLite | Never exits | <1s | +| Scripted `gbrain search ... && echo OK` | `OK` never printed | `OK` printed | +| `gbrain init --pglite` on macOS 12.7 + Bun 1.3.14 | Failed with misleading macOS 26.3 hint | Failed with correct bunfs/`bun upgrade` hint | +| `gbrain sync` hang ([#1342](https://github.com/garrytan/gbrain/issues/1342)) | No diagnostic output before hang | Phase breadcrumbs name WHICH phase spun | +| `gbrain serve --http` (daemon) | Stayed alive | Still stays alive (regression-tested) | + +### Why we narrowed the force-exit + +Two community PRs proposed competing fixes. [PR #1259](https://github.com/garrytan/gbrain/pull/1259) (jehoon, validated by @eloe, @bcallender, and @61tH0b) added the structural "wait for the background write to finish" pattern — the right approach, mirroring an existing fix we shipped for #1090. [PR #1337](https://github.com/garrytan/gbrain/pull/1337) (matt-dean-git) took a different approach: force-exit the process after `main()` returns for every non-daemon command. PR #1337 also reordered the disconnect to release the file lock before closing the database, and added a snapshot+early-null pattern to the disconnect itself. + +We took the drain from #1259, took the snapshot+early-null pattern from #1337, and narrowed PR #1337's force-exit to fire ONLY when the drain timed out — not unconditionally for every command. Reason: an unconditional force-exit would mask every future fire-and-forget regression. The narrow version preserves the diagnostic stderr warn that names the leaking surface, AND guarantees the CLI exits even on the pathological path. Reviewed via [/plan-eng-review](https://github.com/garrytan/gstack) (9 decisions) + Codex outside-voice (13 findings, all folded in). We also kept the original close-then-release disconnect order; PR #1337's swap to release-then-close would have widened the window where a sibling process could connect to a still-closing brain. + +### Things to watch + +- The `[gbrain phase] sync.` breadcrumbs now print to stderr at four new boundaries (`sync.resolve_repo`, `sync.load_active_pack`, `sync.validate_repo_state`, `sync.detect_head`). If you parse gbrain stderr in a script, the new lines mirror the existing `[gbrain phase] sync.git_pull start/done` pattern from v0.28.1. +- If you ever see `[last-retrieved] drain timed out after 5000ms; N writes still pending` on stderr, that's a defense-in-depth signal — it means a tracked background write took longer than 5 seconds. It's safe to ignore (the CLI still exits cleanly), but please file an issue with the pending count and the command you ran. This is how we'd find the next bug class in this surface. +- #1342 (`gbrain sync` hang after schema v89→v92) is NOT fixed in this release. It's a single-reporter bug with a pure-JS infinite-loop shape (per `sample `) that doesn't match any of the hypotheses we ruled out. We've filed it as a follow-up investigation in [TODOS.md](TODOS.md) with concrete diagnostic next steps. If you hit it, please attach a `bun --inspect-brk` stack — the new breadcrumbs will name which phase to look at. + +## To take advantage of v0.41.8.0 + +`gbrain upgrade` should do this automatically. If you were hitting #1247/#1269/#1290 before, no manual action is required — the fix is in the binary. + +1. **Run upgrade:** + ```bash + gbrain upgrade + ``` +2. **Verify the CLI exits cleanly:** + ```bash + time gbrain search "test" --limit 3 + echo "EXIT=$?" + ``` + `EXIT=0` and a wall time under 2 seconds (after the first cold-start) means the fix landed. Pre-fix you would see no exit at all. +3. **If you hit #1340 on older macOS + Bun:** run `bun upgrade` and retry. If that doesn't help, install gbrain via Node instead of via the Bun-compiled binary. +4. **If any step fails or you see new hangs**, please file an issue: + https://github.com/garrytan/gbrain/issues with the command you ran, your platform (macOS / Linux + Bun version), and whether you see the `[last-retrieved] drain timed out` stderr line. + +### Itemized changes + +#### Search/query/get hang fix (#1247, #1269, #1290) + +- `src/core/last-retrieved.ts` — new `awaitPendingLastRetrievedWrites(timeoutMs?)` drain helper with bounded 5s `Promise.race` timeout. Tracks every `bumpLastRetrievedAt` IIFE promise in a module-scoped `Set>`; resolves once all settle. Returns `{outcome, pending}` so the caller can decide its fallback. Mirrors the existing v0.36.1.x `awaitPendingSearchCacheWrites` precedent from #1090, plus the timeout the original helper lacks (a symmetry retrofit is filed as a v0.41+ TODO). +- `src/cli.ts` — awaits `awaitPendingLastRetrievedWrites()` unconditionally in the op-dispatch finally block, right after the existing search-cache drain. On `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve`), calls `process.exit(0)` AFTER `engine.disconnect()` completes. Per-op-name gating was deliberately NOT applied — PR #1259's original literal-name check would have left `search` and `get_page` exposed. +- `src/core/pglite-engine.ts:disconnect` — snapshot+early-null pattern (snapshot `_db`/`_lock` refs and null instance fields up front so concurrent `connect()` cannot observe a partial mid-close state) wrapped in try/finally so the file lock releases even if `db.close()` throws. KEEPS the original close-then-release order; PR #1337's release-then-close swap was rejected (widens the window where a sibling process could connect to a still-closing brain). + +#### #1340 WASM init hint routing + +- `src/core/pglite-engine.ts` — new exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)`. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence; surfaces a paste-ready `bun upgrade` hint + Node fallback. `macos-26-3` keeps the existing #223 link. Regex tightened per Codex eng-review finding #9 so generic `pglite.data` mentions don't false-trip the bunfs verdict. + +#### #1342 diagnostic breadcrumbs + +- `src/commands/sync.ts:performSyncInner` — four new stderr breadcrumbs at major phase boundaries: `sync.resolve_repo`, `sync.load_active_pack`, `sync.validate_repo_state` (when sourceId is set), `sync.detect_head`. Mirrors the pre-existing `sync.git_pull start/done` pattern. Doesn't fix #1342 but converts "hung with no output" into actionable diagnostic data. + +#### Test coverage + +- `test/last-retrieved.test.ts` (NEW) — 6 unit cases covering empty drain, single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout within bound, empty pageIds not tracked. +- `test/pglite-engine-disconnect.serial.test.ts` (NEW) — 5 lifecycle invariants: close-before-release ordering, snapshot observable inside close, lock-still-releases on close-throw, double-disconnect idempotency, reconnect-after-disconnect clean state. +- `test/pglite-init-classifier.test.ts` (NEW) — 12 pure-function unit cases including the #1340 reporter's exact error string round-trip + a negative case asserting generic `pglite.data` mentions don't trip the bunfs verdict. +- `test/e2e/pglite-cli-exit.serial.test.ts` (NEW IRON-RULE regression) — real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir; asserts `search`, `query --no-expand`, `get` all exit 0 within 15s; daemon-survival case asserts `gbrain serve --http` stays alive past 3s (regression guard for the narrow force-exit not misclassifying 'serve' as a non-daemon). +- `test/fix-wave-structural.test.ts` (EXTENDED) — behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect()` in cli.ts (survives variable-rename refactors), plus structural pins for the classifier exports and the sync breadcrumbs. Per D8 in the eng review (Codex finding #5), explicitly did NOT add a drift-guard counting `bumpLastRetrievedAt(` callers. + +### Credit + +PR [#1259](https://github.com/garrytan/gbrain/pull/1259) by jehoon supplied the structural drain pattern. PR [#1337](https://github.com/garrytan/gbrain/pull/1337) by matt-dean-git supplied the snapshot+early-null disconnect pattern and the force-exit idea we narrowed to fire only on the drain-timeout path. @eloe, @bcallender, and @61tH0b independently validated PR #1259's fix against real reproducers. Closing PRs land with `Co-Authored-By:` trailers on the merge commit. ## [0.41.7.0] - 2026-05-24 **Your compact OpenClaw resolver actually works now.** If you've grown your agent past 200 skills and switched to the compact list-format resolver (`- **gift-advisor**: gift idea | birthday gift`) because the markdown-table version got unreadable, `gbrain doctor` used to silently report every single skill as unreachable. On a 306-skill agent, that was 238 FAIL errors on every doctor run, and the list-format resolver was effectively invisible to gbrain. v0.41.7.0 fixes that — the parser now reads both shapes natively, mixes them in one file, and `gbrain doctor` reports 0 errors on the same resolver that previously broke it. diff --git a/CLAUDE.md b/CLAUDE.md index 23aa97dfc..e5fee77b3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -51,7 +51,7 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). — PGLite-specific DDL (pgvector, pg_trgm, triggers) - `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. @@ -234,7 +234,7 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. +- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. diff --git a/TODOS.md b/TODOS.md index f424b3b17..020557085 100644 --- a/TODOS.md +++ b/TODOS.md @@ -456,6 +456,86 @@ cleanup can move each into the relevant area section. primitive exists (`embedding_columns` from v0.36.3); migration verb doesn't. (Embedding cluster.) +--- +## v0.41.8.0 PGLite hang follow-ups (v0.41+) + +These were filed when v0.41.8.0 shipped the search/query/get hang fix +(#1247/#1269/#1290) + WASM init classifier (#1340) + sync breadcrumbs. +Three items deferred: + +- [ ] **Investigate #1342 — `gbrain sync` hangs after schema v89→v92 + migration (PGLite, single reporter).** Repro shape: ~99% CPU in pure-JS + JIT loop per `sample `, zero stderr output, reproduces with + `--dry-run --no-pull`. Triggered after migrations 89→92 landed (v89 + facts_event_type_column, v90 contextual_retrieval_columns, v91 + pages_generation_trigger_and_bookmark, v92 sources_github_repo_index). + Stale lock recovery from a `brain.pglite.broken-20260523-120636` + rename suggests half-applied schema state. + + **Ruled out** (per v0.41.8.0 plan-eng-review): NOT the + `withRefreshingLock` heartbeat (user takes the legacy global-lock + path — no setInterval); NOT the v91 trigger function (only fires on + writes, user repros with `--dry-run`); NOT the two `while (true)` + loops in `src/commands/sync.ts` (parallel worker pool + watch mode, + neither in the user's invocation path). + + **Next diagnostic steps**: + 1. Seed a fresh PGLite brain at schema v88 (snapshot the embedded + schema blob at that version into a test fixture), apply migrations + v89→v92, then run `performSync` with the user's exact flags and + an 8s timeout. Repeat with a partial-v91 state (column landed, + index didn't) to match the `brain.pglite.broken-...` clue. + 2. Run the reproducer under `bun --inspect-brk` and grab the V8 + stack at the spin point. + 3. Scan for `contextual_retrieval_mode IS NULL` paths in sync / + `src/core/import-file.ts` — the v90 column may have an unbounded + iteration somewhere when the per-source backfill kicks in. + + **Reporter's config**: PGLite, `~/.gbrain/brain.pglite`, + `ollama:nomic-embed-text` @ 768d, macOS 15.5, single 'default' + source. + + **Mitigation in v0.41.8.0**: phase breadcrumbs added to + `performSyncInner` so the next #1342-shaped report names WHICH phase + spun (resolve_repo / load_active_pack / validate_repo_state / + detect_head). Doesn't fix; makes reports actionable. + +- [ ] **Concurrent disconnect-during-connect race on `PGLiteEngine` + (adversarial-review C6, v0.41.8.0).** The v0.41.8.0 snapshot+early-null + pattern in `disconnect()` improves the partial-state race for the + common case (single instance, sequential lifecycle), but a concurrent + `connect()` and `disconnect()` on the same engine instance can still + strand: `disconnect()` snapshots+nulls the lock and releases it while + `connect()` is still in-flight (lock already acquired, awaiting + `PGlite.create()`). When `connect()` resolves, `this._db` is assigned + to a fresh handle but `this._lock` is null — engine is "connected" + but holds no file lock; another process can acquire it concurrently. + Unusual caller pattern in production (one instance per process, + sequential lifecycle), but tests sometimes do this and the contract + is undefined. Fix: serialize connect/disconnect with an instance-level + mutex, or document the constraint and assert single-flight at the + call site. + +- [ ] **Retrofit `awaitPendingSearchCacheWrites` with the same bounded + timeout v0.41.8.0 added to `awaitPendingLastRetrievedWrites`.** The + v0.36.1.x #1090 fix at `src/core/search/hybrid.ts:36-45` shipped the + drain pattern without a timeout; v0.41.8.0 added the timeout + warn + pattern to the new `awaitPendingLastRetrievedWrites` helper. For + symmetry (and to close the same future-failure mode in the cache + drain), apply the same `Promise.race` + stderr warn pattern. ~15 LOC + + 2 unit cases. Pair this with the drain-helper extraction below. + +- [ ] **Extract a shared `createDrainHelper()` factory when a third + fire-and-forget surface appears.** Per D4 in the v0.41.8.0 eng + review: two surfaces is the threshold for noticing, three for + extracting. `src/core/search/hybrid.ts:awaitPendingSearchCacheWrites` + + `src/core/last-retrieved.ts:awaitPendingLastRetrievedWrites` are + the two surfaces today. When a third surface is added (or when the + timeout-symmetry retrofit above lands and the duplication becomes + load-bearing), extract a `src/core/drain-helper.ts` factory consumed + by both call sites. Pair with the symmetry retrofit so they fire + together as one focused refactor. + --- ## v0.41 Eval-loop wave follow-ups (v0.42+) diff --git a/VERSION b/VERSION index c9fd62888..f380c4476 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.41.7.0 \ No newline at end of file +0.41.8.0 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 23eb20afe..e4d6fd152 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -193,7 +193,7 @@ strict behavior when unset. - `src/commands/search.ts:gbrain search stats` extension (v0.40.4.0) — new `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property to existing stats; `_meta.metric_glossary` adds two new keys (`graph_signals.enabled`, `graph_signals.failures_by_reason`). Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts` (6 cases). - `src/commands/doctor.ts` extension (v0.40.4.0) — new `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded in the message. Pinned by 7 new cases in `test/doctor.test.ts`. - `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`) -- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. — PGLite-specific DDL (pgvector, pg_trgm, triggers) +- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all 40 BrainEngine methods. `addLinksBatch` / `addTimelineEntriesBatch` use multi-row `unnest()` with manual `$N` placeholders. As of v0.13.1, `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error naming the macOS 26.3 WASM bug (#223) and pointing at `gbrain doctor`; the lock is released on failure so the next process can retry cleanly. v0.22.0: `searchKeyword` and `searchKeywordChunks` multiply `ts_rank` by the source-factor CASE expression at the chunk-grain level; `searchVector` becomes a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`. Inner LIMIT scales with offset to preserve pagination contract. As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for the specific forward-referenced state the embedded schema blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target table) and adds only what's missing. Closes the upgrade-wedge bug class that bit users 10+ times across 6 schema versions over 2 years (#239/#243/#266/#357/#366/#374/#375/#378/#395/#396). No-op on fresh installs and modern brains. **v0.35.5.0:** probe set extended in parity with postgres-engine.ts — `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`. Bootstrap also threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope. Closes #1018, #974, #820. **v0.37.10.0:** `getBrainScore` now returns 100/100 with full breakdown components (35/25/15/15/10) when `pageCount === 0`. Vacuous truth — an empty brain has no coverage problem to penalize. Pre-fix, fresh `gbrain init --pglite` users saw "Brain score 0/100" on first `gbrain doctor` run, which was structurally surprising AND triggered the v0.37.8.0 `doctor-report-remote.test.ts` flake (test expected `health_score >= 70` on a freshly-initialized test brain). Mirrored in postgres-engine.ts. Pinned by `test/brain-score-breakdown.test.ts`'s updated empty-brain assertion and the hermetic `test/doctor-report-remote.serial.test.ts`. **v0.41.8.0 (#1247/#1269/#1290/#1340):** two changes. (1) `disconnect()` rewritten with the snapshot+early-null pattern — snapshot `_db`/`_lock` refs and null the instance fields BEFORE any `await` so a concurrent `connect()` cannot observe a partial mid-close state — wrapped in a `try/finally` that guarantees lock-release even if `db.close()` throws (Codex eng-review finding #7 lock-leak guard). KEEPS the original close-then-release order; PR #1337's swap to release-then-close was explicitly rejected because it would widen the window where a sibling process could connect to a still-closing brain. Pinned by `test/pglite-engine-disconnect.serial.test.ts` (5 cases: ordering, snapshot, lock-leak-on-throw, double-disconnect idempotency, reconnect-after-disconnect). (2) New exported `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` route the `PGlite.create()` catch-block hint by failure shape. The `bunfs` verdict matches the literal `$$bunfs` marker OR `ENOENT[\s\S]*pglite\.data` co-occurrence (tightened from a generic `pglite.data` substring per Codex eng-review finding #9); it surfaces a paste-ready `bun upgrade` hint + Node fallback. The `macos-26-3` verdict keeps the existing #223 link. `unknown` falls through to the prior generic hint. Closes the user-facing half of #1340 — pre-fix every `PGlite.create()` failure pointed at #223 regardless of root cause. Pinned by `test/pglite-init-classifier.test.ts` (12 cases including the #1340 reporter's exact error string round-trip). — PGLite-specific DDL (pgvector, pg_trgm, triggers) - `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch` / `addTimelineEntriesBatch` use `INSERT ... SELECT FROM unnest($1::text[], ...) JOIN pages ON CONFLICT DO NOTHING RETURNING 1` — 4-5 array params regardless of batch size, sidesteps the 65535-parameter cap. As of v0.12.3, `searchKeyword` / `searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection (contributed by @garagon). `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. v0.22.0: `searchKeyword`, `searchKeywordChunks`, and `searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude clause from `src/core/search/sql-ranking.ts`. `searchVector` switches to a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in the outer SELECT) and carries `p.source_id` through inner→outer for v0.18 multi-source callers. v0.22.1 (#406): `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures). `executeRaw` is a single-statement passthrough — no per-call retry (D3 dropped that as unsound for non-idempotent statements; recovery is supervisor-driven). v0.22.1 (#363, contributed by @orendi84): `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. v0.22.1 (#409, contributed by @atrevino47): `countStaleChunks()` + `listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale`, eliminating ~76 MB/call client-side pull on a fully-embedded brain; `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding (consistency). As of v0.22.6.1, `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same forward-reference probe set as the PGLite engine, so old Postgres brains pinned at v0.13/v0.18/v0.19 walk forward cleanly instead of wedging on `column "..." does not exist`. **v0.35.5.0:** probe set extended for the column-only forward-reference cases the original v0.22.6.1 sweep missed — `files.source_id`, `files.page_id` (pre-v0.18 brains where `idx_files_source_id` was the choke point), `oauth_clients.source_id`, `oauth_clients.federated_read` (pre-v0.34 brains where v60+v61+v65 chain failed), and `sources.archived` + `sources.archived_at` + `sources.archive_expires_at` (pre-v0.26.5 brains where `CREATE TABLE IF NOT EXISTS sources` was a no-op on existing tables so the archive lifecycle columns never landed). Also (Codex P1 from pre-landing review): the entire probe path now runs on the DDL connection threaded down from `initSchema` — previously probes ran through the instance pool while the advisory lock sat on a different connection, opening a concurrent-bootstrap race for Supabase pooler users. Closes #1018, #974, #820. **v0.28.1:** `disconnect()` is now idempotent. New `_connectionStyle` instance field tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than falling through to `db.disconnect()` and clobbering the singleton. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` (2 cases). Closes the bug class where any test sharing an engine across multiple `worker.start()` / `worker.stop()` cycles silently broke its own DB connectivity. **v0.37.10.0:** `getBrainScore` empty-brain parity with PGLite — returns 100/100 with breakdown components 35/25/15/15/10 when `pageCount === 0` instead of the pre-fix 0/100. Same vacuous-truth rationale as the PGLite side; both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic. - `src/core/cjk.ts` (v0.32.7 CJK wave) — Single source of truth for CJK detection across the codebase. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). Replaces the inline hasCJK regex previously duplicated at `expansion.ts:58`. BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables); widening to Unicode property escapes is a v0.33+ TODO. Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`. - `src/core/audit-slug-fallback.ts` (v0.32.7 CJK wave) — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()` from shell-audit.ts. Separate surface from `sync-failures.jsonl` per codex outside-voice review — that file carries bookmark-gating semantics that info events shouldn't trigger. @@ -376,7 +376,7 @@ strict behavior when unset. - `src/commands/whoknows.ts` (v0.33) — `gbrain whoknows [--explain] [--limit N] [--json]`: expertise + relationship-proximity routing. Mirrors v0.29 salience/anomalies shape (pure `rankCandidates()` + `findExperts()` orchestrator + `runWhoknows()` CLI dispatch + thin-client routing). MCP op = `find_experts` (scope: read, localOnly: false) per ENG-D5. Ranking formula (ENG-D1 locked): `score = log(1 + raw_match) × max(0.1, exp(-days/180)) × (0.5 + 0.5 × salience)` where `raw_match` is hybridSearch's RRF+source-boost score. Filters at SQL via the new `SearchOpts.types: ['person', 'company']` (no post-filter waste). hybridSearch's internal salience+recency boosts are intentionally disabled — the locked formula applies on a clean signal. Floors prevent multiplicative-zero edge cases (cold-start people stay visible); ties break alphabetically by slug for determinism. 16 unit tests in `test/whoknows.test.ts` pin the math. - `src/commands/eval-whoknows.ts` (v0.33, v0.33.1.3 thin-client wiring) — `gbrain eval whoknows [--json] [--skip-replay]`: two-layer eval gate (ENG-D2). Layer 1 quality (hand-labeled fixture, top-3 hit rate ≥ 0.8). Layer 2 regression (`eval_candidates` replay set-Jaccard@3 ≥ 0.4). Sparseness fallback: < 20 replay-eligible rows → Layer 2 auto-skips with stderr warning. Stable JSON envelope with `schema_version: 1`. Exit 0/1/2 for pass/fail/usage so CI can gate. Mirrors v0.27.x cross-modal + v0.28.1 longmemeval dispatch shape under `src/commands/eval.ts`. **v0.33.1.3:** `WhoknowsFn` callable abstraction lets the gates be impl-agnostic. `runEvalWhoknows(engine: BrainEngine | null, args)` picks the impl at entry — thin-client mode (`isThinClient(cfg)`) routes per-query through `callRemoteTool(cfg, 'find_experts', {topic, limit})` via the v0.31.1 seam; local mode calls `findExperts(engine, ...)` directly. cli.ts adds a thin-client bypass before `connectEngine` for `gbrain eval whoknows`, matching the longmemeval/cross-modal no-DB pattern. Regression gate auto-skips in thin-client mode (no DB access to `eval_candidates`). Public exports `jaccardAtK`, `topKHit`, `readFixture`, `WhoknowsFn`, threshold constants are pinned by `test/eval-whoknows.test.ts` (25 cases, +2 for the null-engine signature contract). - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. -- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. +- `src/core/brainstorm/{domain-bank,orchestrator,judges}.ts` + `src/commands/{brainstorm,lsd,eval-brainstorm}.ts` + `src/core/last-retrieved.ts` (v0.37.0 Open Collider wave) — bisociation-grounded idea generation pair: `gbrain brainstorm ` (defensible, cite-heavy, 4 close × 6 far, judge threshold 4.0/5, save by default) and `gbrain lsd ` (Lateral Synaptic Drift — inverted judge that rejects ideas with resistance >4.5 ("too obvious"), stale-page bias via `pages.last_retrieved_at`, 2 close × 12 far, axiomatic inversions required, ephemeral by default). The architecture corrects Open Collider's training-data-grounded approach: gbrain has the user's actual cross-domain knowledge already, so the "domain bank" is prefix-stratified sampling from the user's own brain (`SELECT DISTINCT substring(slug from '^[^/]+/[^/]+')` cached 1h-TTL in `config` table per source) tiebroken by `JOIN page_links` connection_count, with corpus-sampling fallback when fewer prefixes than M exist. Distance scores normalized to [0,1] via `1 - clamp(cosine_distance, 0, 2) / 2` (1=opposite, 0=identical). The judge is a single `judges.ts` with `runJudge(config, ideas)` + two exported configs (`BRAINSTORM_JUDGE_CONFIG` weighted originality/resistance/thesis_density/concrete_grounding/cognitive_load 0.25/0.20/0.20/0.20/0.15 vs `LSD_JUDGE_CONFIG` with cognitive_load 0.50 + inversion rule). Calibration cold-start fallback (D4 + codex #8): when `calibration_profiles.active_bias_tags` is empty, judge runs without anti-bias context AND stderr-warns. Op-layer write-back at `src/core/operations.ts` `search`/`query`/`get_page` handlers fires `bumpLastRetrievedAt(engine, pageIds)` (fire-and-forget, 5-min throttled via SQL clause, default-on with `search.track_retrieval` config escape hatch per D13) — internal callers (sync, migrations, dream cycle) bypass the op layer so the LSD stale signal stays clean. **v0.41.8.0 (#1247/#1269/#1290):** the IIFE is now tracked in a module-scoped `Set>` (mirrors the v0.36.1.x `awaitPendingSearchCacheWrites` precedent for #1090). New exported `awaitPendingLastRetrievedWrites(timeoutMs?: number): Promise<{outcome, pending}>` resolves once all tracked promises settle, bounded by a 5s `Promise.race` timeout that stderr-warns with the pending count if it fires. `src/cli.ts` awaits the drain unconditionally for every op in the op-dispatch finally block BEFORE `engine.disconnect()`, then narrows a fallback `process.exit(0)` to fire ONLY when `outcome === 'timeout'` AND `shouldForceExitAfterMain(argv)` (excludes `serve` so daemons stay alive). Closes the PGLite CLI search/query/get-hang class: pre-fix, the IIFE raced disconnect, PGLite's WASM kept Bun's event loop alive, CLI hung at ~95-98% CPU until SIGKILL. Three community-validated reports; PR #1259 (jehoon) supplied the structural drain pattern; PR #1337 (matt-dean-git) supplied the snapshot+early-null disconnect pattern AND the force-exit guard we narrowed to fire only on the timeout path. Pinned by `test/last-retrieved.test.ts` (6 unit cases: empty/single/multi-pending settle, throw-in-IIFE still settles, permanently-pending hits timeout, empty pageIds untracked), `test/e2e/pglite-cli-exit.serial.test.ts` (IRON-RULE behavioral regression: real `bun src/cli.ts` subprocess against a hermetic PGLite tempdir, asserts search/get/query exit 0 in <15s + daemon-survival case), `test/fix-wave-structural.test.ts` (behavioral-positioning assertion that the drain `await` appears textually BEFORE `engine.disconnect` in the local-engine path — survives variable-rename refactors). Migration v79 adds `pages.last_retrieved_at TIMESTAMPTZ NULL` + full B-tree index (NOT partial — covers both NULL and range branches per codex r2 #6); full forward-reference bootstrap probe on both engines. Frontmatter `mode: lsd` makes the dream-cycle synthesize phase skip LSD output (noise-by-design — `isLsdOutput()` check in `src/core/cycle/transcript-discovery.ts` short-circuits `isDreamOutput()`). `gbrain eval brainstorm ` is a three-axis evaluation gate (distance + usefulness + grounding, conjunctive) per codex r2 #11 — distance alone is gameable. `gbrain doctor` gains `brainstorm_health` check surfacing (a) migration v79 applied, (b) `search.track_retrieval` setting, (c) calibration cold-start status. 38 unit tests across `test/brainstorm/{distance,lsd-mode-skip,eval-brainstorm}.test.ts`. Plan: `~/.claude/plans/system-instruction-you-are-working-staged-coral.md`. Open Collider source: `github.com/CL-ML/open-collider`. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. - `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.36.3.0:** new `embedding_column_registry` check probes each declared column via Postgres `format_type(atttypid, atttypmod)` so a registry entry claiming 1024d Voyage against an actual 1536d OpenAI column surfaces with a paste-ready `gbrain config set embedding_columns '{...}'` ALTER hint instead of mysterious "vector dimension mismatch" errors at search time. On Postgres the check also probes HNSW index presence (`pg_indexes` lookup keyed by column name) and warns when missing (search will still work via seq scan but won't hit the index). The active default column's population coverage is computed via `COUNT(*) FILTER (WHERE IS NOT NULL) / COUNT(*)` and warns below 90% — except empty brains (chunk_count = 0) where the gate short-circuits to `ok` so fresh `gbrain init` runs don't see "Active column 'embedding' is 0.0% populated" (CDX-5 codex fix). PGLite parity via the same SQL through `executeRaw` — registry validation happens on both engines. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. **v0.37.1.0:** new `skill_brain_first` check. Walks every SKILL.md under the configured skills dir (`autoDetectSkillsDirReadOnly` so `cd ~ && gbrain doctor` finds the bundled skills via the install-path fallback), calls `analyzeSkillBrainFirst()` from `src/core/skill-brain-first.ts` per file, aggregates verdicts into a single check with structured `Check.issues[]` for JSON tooling. Warn states: `missing_brain_first` (external-lookup pattern present, no canonical callout, no `brain_first: exempt`), `brain_first_typo` (near-miss declaration like `brain-first` or `BrainFirst` — paste-ready hint surfaces the correct snake_case form). Ok states: `compliant_callout`, `compliant_phase`, `compliant_position`, `exempt_frontmatter`, `no_external`. `--fix` routes through `dry-fix.ts` MISSING_RULE_PATTERNS to auto-insert the canonical `> **Convention:** see [conventions/brain-first.md](...)` callout (D6 install-path safety gate enforced — `--fix` from `~` refuses to write to the bundled tree). Snapshot+diff audit at `~/.gbrain/audit/skill-brain-first-YYYY-Www.jsonl` records detected / resolved / fixed transitions only (stable brains: 0 lines/run). Motivated by the 2026-05-19 tweet-shield incident: cross-modal eval flagged Garry's Palantir tweet as risky because no model knew he built it, but the brain already had "designed the entire Finance product UI" and "150+ PSDs from April-December 2006." Static check catches the AUTHORSHIP miss class; v0.37+ runtime gate (filed in TODOS.md) closes the dispatch side. @@ -3830,345 +3830,6 @@ echo "Dream cycle complete at $(date)" --- -## docs/guides/minions-deployment.md - -Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md - -# Minions Worker Deployment Guide - -Keep `gbrain jobs work` running across crashes, reboots, and Postgres -connection blips. Written for agents to execute line-by-line. - -## The problem - -The persistent worker can die silently from: - -- Database connection drops (Supabase/Postgres maintenance or network blips). -- Lock-renewal failures → the stall detector eventually dead-letters jobs. -- Bun process crashes with no automatic restart. -- Internal event-loop death (PID alive, worker loop stopped). - -When the worker dies, submitted jobs sit in `waiting` forever. The -canonical answer is `gbrain jobs supervisor` — a first-class CLI that -spawns `gbrain jobs work` as a child and auto-restarts it on crash. - -## Worker supervision - -### The canonical pattern - -`gbrain jobs supervisor` is an auto-restarting wrapper around -`gbrain jobs work`. It writes a PID file, restarts the worker on crash -with exponential backoff (1s → 60s cap), emits lifecycle events to an -audit file, and drains gracefully on SIGTERM (35s worker-drain window -before SIGKILL). Exit codes are documented so agents can branch on them. - -**Typical commands:** - -```bash -# Start in the foreground (blocks; Ctrl-C to stop). -gbrain jobs supervisor --concurrency 4 - -# Start detached — returns {"event":"started","supervisor_pid":…} on stdout. -gbrain jobs supervisor start --detach --json - -# Check liveness without reading log files. -gbrain jobs supervisor status --json - -# Graceful stop (SIGTERM + drain wait + SIGKILL fallback). -gbrain jobs supervisor stop -``` - -**Exit codes:** - -| Code | Meaning | -|---|---| -| 0 | Clean shutdown (SIGTERM/SIGINT received, worker drained) | -| 1 | Max crashes exceeded (worker kept dying) | -| 2 | Another supervisor holds the PID lock | -| 3 | PID file unwritable (permission / path error) | - -An agent seeing exit=2 can safely treat it as "one is already running"; -exit=1 should page a human. - -### Which supervisor when? - -The supervisor solves in-process crash recovery. Platform-level -supervision (systemd, Fly, Render) handles host-level failures. You -usually want both. - -| Environment | Recommendation | -|---|---| -| **Container (Fly / Railway / Render / Heroku)** | `gbrain jobs supervisor` runs as PID 1. The platform restarts the container on OOM / host loss; supervisor restarts the worker on crash. See [Fly.io](#flyio) / [Render / Railway / Heroku](#render--railway--heroku). | -| **Linux VM with systemd** | Two-layer recommended: systemd supervises `gbrain jobs supervisor`, which in turn supervises `gbrain jobs work`. Buys you automatic restart on reboot (systemd) plus fast crash recovery (supervisor). See [systemd](#systemd). | -| **Dev laptop / macOS** | `gbrain jobs supervisor` in a terminal. Ctrl-C stops it. No system-level setup needed. | - -### Variables used in this guide - -Substitute these once before copy-pasting any snippet. - -| Variable | Meaning | Typical value | -|---|---|---| -| `$GBRAIN_BIN` | Absolute path to the `gbrain` binary | `$(command -v gbrain)` — often `/usr/local/bin/gbrain` or `~/.bun/bin/gbrain` | -| `$GBRAIN_WORKER_USER` | OS user that owns the worker process | the same user that ran `gbrain init`; never `root` | -| `$GBRAIN_WORKSPACE` | `cwd` for shell jobs submitted by this deployment | absolute path, e.g. `/srv/my-brain` | -| `$GBRAIN_ENV_FILE` | Secrets file sourced by systemd / shell | `/etc/gbrain.env` (mode 600) | - -### Preconditions - -Run these before any deployment step. - -```bash -# 1. gbrain is on PATH and resolves to an absolute location. -command -v gbrain || { echo "gbrain not on PATH. Install, then retry."; exit 1; } - -# 2. DATABASE_URL points at reachable Postgres. -# (Supervisor is Postgres-only. PGLite's exclusive file lock blocks the -# separate worker process. If `config.engine === 'pglite'` the CLI rejects -# with a clear error.) -gbrain doctor --fast --json | jq '.checks[] | select(.name=="db_connectivity")' - -# 3. Schema is up to date. If version=0 or status=="fail": -# gbrain apply-migrations --yes -gbrain doctor --fast --json | jq '.checks[] | select(.name=="schema_version")' - -# 4. If you plan to submit `shell` jobs, pass --allow-shell-jobs to the -# supervisor (or export GBRAIN_ALLOW_SHELL_JOBS=1 before starting). -# Without the flag, the shell handler is disabled at worker startup. -``` - -## Agent usage (OpenClaw / Hermes / Cursor / Codex) - -Three-command pattern an agent can drive without shell archaeology: - -```bash -# Start (returns PIDs + pid_file on stdout as JSON, then detaches) -gbrain jobs supervisor start --detach --json -# → {"event":"started","supervisor_pid":1234,"worker_pid":1235,"pid_file":"/Users/you/.gbrain/supervisor.pid"} - -# Check health (machine-parseable JSON, no log scraping) -gbrain jobs supervisor status --json -# → {"running":true,"supervisor_pid":1234,"last_start":"2026-04-23T15:30:22Z","crashes_24h":0, ...} - -# Stop cleanly (SIGTERM + 35s drain + SIGKILL fallback) -gbrain jobs supervisor stop -``` - -Every lifecycle event (spawn, crash, backoff, health warning, max-crashes, -shutdown) is also written to `${GBRAIN_AUDIT_DIR:-~/.gbrain/audit}/supervisor-YYYY-Www.jsonl` -for historical inspection. `gbrain doctor` reads that file and surfaces -a `supervisor` check in its health report. - -## Deployment: systemd - -For long-running Linux VMs with shell access. - -```bash -# Create the worker user if it doesn't exist. -sudo useradd --system --home "$GBRAIN_WORKSPACE" --shell /usr/sbin/nologin gbrain \ - 2>/dev/null || true -sudo mkdir -p "$GBRAIN_WORKSPACE" && sudo chown gbrain:gbrain "$GBRAIN_WORKSPACE" - -# Install the env file (secrets stay out of the unit file). -sudo install -m 600 -o gbrain -g gbrain \ - docs/guides/minions-deployment-snippets/gbrain.env.example /etc/gbrain.env -sudoedit /etc/gbrain.env -# Fill in DATABASE_URL, optional GBRAIN_ALLOW_SHELL_JOBS=1. - -# Install the unit file, substituting /srv/gbrain → your workspace path. -sudo install -m 644 docs/guides/minions-deployment-snippets/systemd.service \ - /etc/systemd/system/gbrain-worker.service -sudo sed -i "s|/srv/gbrain|$GBRAIN_WORKSPACE|g" \ - /etc/systemd/system/gbrain-worker.service - -sudo systemctl daemon-reload -sudo systemctl enable --now gbrain-worker -sudo systemctl status gbrain-worker -journalctl -u gbrain-worker -n 50 -``` - -The shipped unit file invokes `gbrain jobs supervisor` (not `gbrain jobs work` -directly) so you get two-layer supervision: systemd restarts the supervisor -on host reboot, supervisor restarts the worker on in-process crash. - -`Restart=always` + `RestartSec=10s` handle the supervisor-level recovery. -The unit runs as unprivileged `gbrain` with `PrivateTmp`, `ProtectSystem=strict`, -and `ReadWritePaths=$GBRAIN_WORKSPACE,$HOME/.gbrain` (for the PID file and -audit log). `LimitNOFILE=65535` covers Bun + Postgres pool + concurrent -LLM subagent calls without hitting the default 1024 cap. - -## Deployment: Fly.io - -```bash -# Merge the [processes] block from fly.toml.partial into your fly.toml. -cat docs/guides/minions-deployment-snippets/fly.toml.partial >> fly.toml -# Review + edit as needed. - -# Set secrets (Fly handles restart on crash). -fly secrets set DATABASE_URL='postgres://…' GBRAIN_ALLOW_SHELL_JOBS=1 -``` - -The `[processes]` block runs `gbrain jobs supervisor` as PID 1. Fly -restarts the container on host failure; the supervisor restarts the -worker on in-process crash. - -## Deployment: Render / Railway / Heroku - -Drop [`Procfile`](./minions-deployment-snippets/Procfile) at the repo -root. The shipped Procfile calls `gbrain jobs supervisor`. Set -`DATABASE_URL` + optional `GBRAIN_ALLOW_SHELL_JOBS=1` via the platform's -env UI or CLI. - -## Deployment: inline `--follow` (no persistent worker) - -For short deterministic scripts on a fixed schedule where you don't need -a persistent worker between runs. Each cron run brings its own temporary -worker. `--follow` starts one on the queue and blocks until the -just-submitted job reaches a terminal state (`completed` / `failed` / -`dead` / `cancelled`). 2-3 s startup overhead per job; negligible vs job -duration for scheduled work. - -```bash -GBRAIN_ALLOW_SHELL_JOBS=1 gbrain jobs submit shell \ - --queue nightly-enrich \ - --params "{\"cmd\":\"$GBRAIN_BIN embed --stale\",\"cwd\":\"$GBRAIN_WORKSPACE\"}" \ - --follow \ - --timeout-ms 600000 -``` - -Replace `gbrain embed --stale` with whichever gbrain subcommand you're -scheduling (`sync`, `extract`, `orphans`, `doctor`, `check-backlinks`, -`lint`, `autopilot`). For strict single-job semantics on shared queues, -use a dedicated queue name like `nightly-enrich` above. - -## Upgrading from an older deployment - -### From `minion-watchdog.sh` (pre-v0.20) - -Earlier versions of this guide shipped a 68-line bash watchdog -(`minion-watchdog.sh`). It's been replaced by `gbrain jobs supervisor` -which handles everything the script did, plus atomic PID locking, -structured audit events, queue-scoped health checks, and graceful -drain on SIGTERM. - -**Migration:** - -```bash -# 1. Stop and remove the old watchdog. -sudo kill $(head -n1 /tmp/gbrain-worker.pid) 2>/dev/null -sudo rm -f /usr/local/bin/minion-watchdog.sh /tmp/gbrain-worker.pid \ - /tmp/gbrain-worker.log -crontab -e # delete the "*/5 * * * * /usr/local/bin/minion-watchdog.sh" line - -# 2. Start the supervisor (systemd users: reinstall the unit from -# docs/guides/minions-deployment-snippets/systemd.service, which -# now calls `gbrain jobs supervisor`). -gbrain jobs supervisor start --detach --json -# Or: sudo systemctl restart gbrain-worker - -# 3. Verify. -gbrain jobs supervisor status --json -gbrain doctor # 'supervisor' check should report running=true -``` - -### Schema / migration hygiene - -Regardless of which deployment path you're upgrading from: - -1. **Stop the worker before upgrading.** `gbrain jobs supervisor stop` - (or `sudo systemctl stop gbrain-worker`). Skipping this risks an - in-flight job landing partial schema. -2. **Run `gbrain upgrade`**. Then `gbrain apply-migrations --yes` if - `gbrain doctor` reports any migration as `partial` or `pending`. -3. **If you run shell jobs:** from v0.14 onward, pass - `--allow-shell-jobs` to the supervisor (or keep - `GBRAIN_ALLOW_SHELL_JOBS=1` in `/etc/gbrain.env`). Submitters don't - need the flag; only the worker does. -4. **Verify.** `gbrain doctor` should report zero `pending` or `partial` - migrations plus a healthy `supervisor` check. `gbrain jobs stats` - should show no unexplained growth in `dead` between pre- and - post-upgrade. - -## Known issues - -### Supabase connection drops - -The worker uses a single Postgres connection. If Supabase drops it -(maintenance, connection limits, network blip), lock renewal fails -silently. The stall detector then dead-letters the job after -`max_stalled` misses. - -**Current defaults that make this worse:** - -- `lockDuration: 30000` (30 s) — too short for long jobs during - connection blips. -- `max_stalled: 5` (schema column default — see `src/schema.sql` and - `src/core/pglite-schema.ts`). Five missed heartbeats before dead-letter. -- `stalledInterval: 30000` (30 s) — checks too aggressively. - -**Tune per-job today.** `gbrain jobs submit` accepts `--max-stalled N`, -`--backoff-type fixed|exponential`, `--backoff-delay `, -`--backoff-jitter 0..1`, and `--timeout-ms N` as first-class flags -(since v0.13.1). These write onto the job row at submit time — which is -what `handleStalled()` reads — so per-job tuning is the real knob today. - -### DO NOT pass `maxStalledCount` to `MinionWorker` - -It's a no-op. The stall detector reads the row's `max_stalled` column -(set at submit time), not the worker opt in `src/core/minions/worker.ts:74`. -Use `gbrain jobs submit --max-stalled N` per-job instead. - -### Zombie shell children - -When the Bun worker crashes hard, child processes from shell jobs can -become zombies. The supervisor's SIGTERM → 35s drain → SIGKILL window -covers the shell handler's 5 s child-kill grace (`KILL_GRACE_MS`). For -long-running shell jobs, prefer timeouts via `--timeout-ms` on submit -over relying on hard kills. - -## Smoke test - -```bash -# Supervisor alive? -gbrain jobs supervisor status --json | jq .running - -# Aggregate queue health. -gbrain jobs stats - -# Jobs currently stalled (still `active` with expired lock_until, pre-requeue). -gbrain jobs list --status active --limit 10 - -# Dead-lettered jobs. -gbrain jobs list --status dead --limit 10 - -# Shell handler registered? (check supervisor audit log or worker stderr.) -gbrain jobs supervisor status --json | jq '.worker_config.allow_shell_jobs' -``` - -## Uninstall - -**`gbrain jobs supervisor`** (foreground or `--detach`): - -```bash -gbrain jobs supervisor stop -``` - -**systemd:** - -```bash -sudo systemctl disable --now gbrain-worker -sudo rm /etc/systemd/system/gbrain-worker.service /etc/gbrain.env -sudo systemctl daemon-reload -``` - -**Fly / Render / Railway:** delete the `worker` process from `fly.toml` -/ `Procfile` and redeploy. Secrets set via `fly secrets` persist until -`fly secrets unset`. - -**Inline `--follow`:** remove the cron entry. Nothing else to clean up -— temporary workers exit with their jobs. - ---- - ## docs/guides/quiet-hours.md Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md diff --git a/package.json b/package.json index 338b3ddb6..bbb93a44f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.41.7.0", + "version": "0.41.8.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/scripts/check-test-isolation.allowlist b/scripts/check-test-isolation.allowlist index e9fbec473..5b04e6ca1 100644 --- a/scripts/check-test-isolation.allowlist +++ b/scripts/check-test-isolation.allowlist @@ -63,7 +63,6 @@ test/resolvers.test.ts test/scenarios.test.ts test/schema-bootstrap-coverage.test.ts test/search-limit.test.ts -test/seed-pglite.test.ts test/skillpack-check.test.ts test/source-resolver.test.ts test/storage-sync.test.ts diff --git a/scripts/check-test-isolation.sh b/scripts/check-test-isolation.sh index fdd135fe3..558278781 100755 --- a/scripts/check-test-isolation.sh +++ b/scripts/check-test-isolation.sh @@ -52,8 +52,21 @@ fi is_allowlisted() { local f="$1" - [ -z "$ALLOWLIST" ] && return 1 - echo "$ALLOWLIST" | grep -qxF "$f" + if [ -z "$ALLOWLIST" ]; then + return 1 + fi + # Use a pure-bash `case` whole-line match against the newline-delimited + # allowlist instead of `echo | grep -qxF`. v0.41.8 CI flake (verify job + # 77771356276): the grep pipe form occasionally failed to match the + # first allowlist entry on Ubuntu 24.04 + bash 5 under + # `bun run` + GNU `timeout` (couldn't reproduce on macOS bash 3.2 with + # the same allowlist file content + lint script content + checkout + # state). Pure-bash case is locale-free, pipe-free, subshell-free, + # set-e-quirk-free, and ~100x faster on every call. + case $'\n'"$ALLOWLIST"$'\n' in + *$'\n'"$f"$'\n'*) return 0 ;; + esac + return 1 } # Find non-serial unit test files (excluding test/e2e). Portable across diff --git a/scripts/llms-config.ts b/scripts/llms-config.ts index 02e9454b1..c2972ee12 100644 --- a/scripts/llms-config.ts +++ b/scripts/llms-config.ts @@ -113,6 +113,12 @@ export const SECTIONS: DocSection[] = [ description: "Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.", path: "docs/guides/minions-deployment.md", + // v0.41.8.0: 13KB deployment runbook. Web index entry stays; + // single-fetch bundle drops it to keep under FULL_SIZE_BUDGET + // (CLAUDE.md grew past 600KB once master's v0.41.2-v0.41.6 + + // this wave's annotations landed). Operators read this once; + // agents rarely need it in context. + includeInFull: false, }, { title: "docs/guides/quiet-hours.md", diff --git a/src/cli.ts b/src/cli.ts index f53099b91..03bbc89e3 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -10,6 +10,8 @@ import type { AIGatewayConfig } from './core/ai/types.ts'; import type { BrainEngine } from './core/engine.ts'; import { operations, OperationError } from './core/operations.ts'; import type { Operation, OperationContext } from './core/operations.ts'; +import { awaitPendingLastRetrievedWrites, type DrainOutcome } from './core/last-retrieved.ts'; +import { shouldForceExitAfterMain } from './core/cli-force-exit.ts'; import { serializeMarkdown } from './core/markdown.ts'; import { parseGlobalFlags, setCliOptions, getCliOptions } from './core/cli-options.ts'; import type { CliOptions } from './core/cli-options.ts'; @@ -180,6 +182,35 @@ async function main() { // Local engine path (unchanged behavior for local installs). const engine = await connectEngine(); + // v0.41.8.0 (#1247, #1269, #1290): the search / query / get_page + // op handlers fire-and-forget `bumpLastRetrievedAt` after returning + // results. On PGLite that IIFE keeps Bun's event loop alive past + // engine.disconnect(), hanging the CLI at ~95-98% CPU until SIGKILL. + // Drain the fire-and-forget set BEFORE disconnect; force-exit only + // if the drain itself times out (preserves stderr diagnostic signal + // AND guarantees the CLI doesn't re-hang at the disconnect layer). + // + // Defense-in-depth (adversarial-review C13): `engine.disconnect()` itself + // can hang on PGLite (db.close() or releaseLock racing OS-level FS state). + // Install an unref'd setTimeout hard-exit fallback BEFORE entering the + // try/catch/finally so a hung disconnect cannot defeat the force-exit + // contract. Daemons (`serve`) are excluded so they stay alive. + const DISCONNECT_HARD_DEADLINE_MS = 10_000; + let forceExitTimer: ReturnType | undefined; + if (shouldForceExitAfterMain()) { + forceExitTimer = setTimeout(() => { + console.warn( + `[cli] engine.disconnect() did not return within ${DISCONNECT_HARD_DEADLINE_MS}ms — force-exiting`, + ); + process.exit(0); + }, DISCONNECT_HARD_DEADLINE_MS); + // unref so the timer itself doesn't keep the event loop alive — only + // the actual pending work (PGLite WASM handle) does. Without unref, + // we'd block a clean exit by 10s on every successful CLI run. + forceExitTimer.unref?.(); + } + + let drainResult: DrainOutcome = { outcome: 'drained', pending: 0 }; try { const ctx = await makeContext(engine, params); const rawResult = await op.handler(ctx, params); @@ -194,7 +225,16 @@ async function main() { const { awaitPendingSearchCacheWrites } = await import('./core/search/hybrid.ts'); await awaitPendingSearchCacheWrites(); } + // Drain unconditionally for every op — empty-set fast-path is a + // few microseconds. Not per-op-name gated: that was the original + // PR #1259 mistake that left search and get_page exposed. + drainResult = await awaitPendingLastRetrievedWrites(); } catch (e: unknown) { + // C9 fix: drain BEFORE process.exit so a successful op that throws + // during stdout/format still gets its bumpLastRetrievedAt UPDATE + // a chance to commit. Bounded by the drain's own 5s timeout; the + // outer hard-exit timer above bounds the disconnect path. + try { await awaitPendingLastRetrievedWrites(); } catch { /* best-effort */ } if (e instanceof OperationError) { console.error(`Error [${e.code}]: ${e.message}`); if (e.suggestion) console.error(` Fix: ${e.suggestion}`); @@ -204,9 +244,19 @@ async function main() { process.exit(1); } finally { await engine.disconnect(); + if (forceExitTimer) clearTimeout(forceExitTimer); + // Narrow force-exit: only when the drain timed out AND we are NOT + // running a daemon. The drain helper already stderr-warned with the + // pending count, so the diagnostic signal is preserved. Without + // this guard a hung underlying promise can still keep Bun's loop + // alive past disconnect — Codex outside-voice finding #1. + if (drainResult.outcome === 'timeout' && shouldForceExitAfterMain()) { + process.exit(0); + } } } + function hasHelpFlag(args: string[]): boolean { return args.includes('--help') || args.includes('-h'); } diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 176444004..ebe939e12 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -538,6 +538,13 @@ export async function performSync(engine: BrainEngine, opts: SyncOpts): Promise< } async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise { + // v0.41.8.0 (D9 / #1342): phase breadcrumbs. The #1342 reporter saw + // ZERO stderr output before their sync hang, which made the bug + // impossible to triage. Mirror the existing `[gbrain phase] sync.git_pull` + // pattern at the major phase boundaries so the next #1342-shaped + // report names WHICH phase spun. Doesn't fix #1342 but converts + // "hung with no output" into actionable diagnostic data. + serr(`[gbrain phase] sync.resolve_repo`); // Resolve repo path const repoPath = opts.repoPath || await readSyncAnchor(engine, opts.sourceId, 'repo_path'); if (!repoPath) { @@ -547,6 +554,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise( @@ -618,6 +627,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise = new Set(['serve']); + +export function shouldForceExitAfterMain( + argv: string[] = process.argv.slice(2), +): boolean { + const command = argv.find((arg) => !arg.startsWith('-')); + if (!command) return true; + return !DAEMON_COMMANDS.has(command); +} diff --git a/src/core/last-retrieved.ts b/src/core/last-retrieved.ts index ea2a61c55..34ceaa473 100644 --- a/src/core/last-retrieved.ts +++ b/src/core/last-retrieved.ts @@ -39,6 +39,92 @@ import { isUndefinedColumnError } from './utils.ts'; let _trackRetrievalCache: { ts: number; enabled: boolean } | null = null; const TRACK_RETRIEVAL_CACHE_TTL_MS = 30_000; +/** + * v0.41.8.0 — fire-and-forget tracking + bounded drain. + * + * Issues #1247, #1269, #1290: PGLite CLI commands printed search / + * query / get_page output then hung at ~95-98% CPU until SIGKILL. + * Root cause: the IIFE below races `engine.disconnect()`. PGLite's + * WASM runtime kept Bun's event loop alive while the dangling + * UPDATE settled, and disconnect closed the DB out from under it. + * + * Solution mirrors the v0.36.1.x #1090 fix at + * `src/core/search/hybrid.ts:awaitPendingSearchCacheWrites`: track + * every IIFE promise in this module-scoped Set, expose a drain that + * resolves once all settle. The CLI awaits the drain before + * `engine.disconnect()` so the WASM handle never closes mid-write. + * + * Bounded with a 5s timeout via Promise.race. If a future + * fire-and-forget bug produces a permanently-pending promise, the + * drain stderr-warns with the pending count and resolves so the CLI + * can disconnect rather than re-creating the hang at this layer. + * The cli.ts caller then falls back to a narrow `process.exit(0)` + * (only on timeout, only for non-daemon commands) to guarantee + * exit. The companion `awaitPendingSearchCacheWrites` retrofit is + * filed as a v0.41+ TODO to keep both helpers symmetric. + */ +const pendingLastRetrievedWrites = new Set>(); +const DRAIN_TIMEOUT_MS = 5_000; + +export type DrainOutcome = { outcome: 'drained' | 'timeout'; pending: number }; + +export async function awaitPendingLastRetrievedWrites( + timeoutMs: number = DRAIN_TIMEOUT_MS, +): Promise { + if (pendingLastRetrievedWrites.size === 0) { + return { outcome: 'drained', pending: 0 }; + } + // Snapshot up front: if a new write appears after we start draining, + // we deliberately don't await it. CLI flow guarantees op-dispatch + // is complete before this call, so the set is effectively frozen. + const snapshot = Array.from(pendingLastRetrievedWrites); + let timer: ReturnType | undefined; + const timeout = new Promise<'timeout'>((resolve) => { + timer = setTimeout(() => resolve('timeout'), timeoutMs); + }); + const drain = Promise.allSettled(snapshot).then(() => 'drained' as const); + const outcome = await Promise.race([drain, timeout]); + if (timer) clearTimeout(timer); + if (outcome === 'timeout') { + const pending = pendingLastRetrievedWrites.size; + console.warn( + `[last-retrieved] drain timed out after ${timeoutMs}ms; ` + + `${pending} writes still pending`, + ); + // Adversarial-review C1: in long-lived daemons (`gbrain serve`), a + // timed-out IIFE stays in the set forever because its `.finally` + // never fires. Repeated timeouts leak references without bound. + // Drop this snapshot's tracked promises explicitly so the next + // drain doesn't see ghosts. The IIFEs themselves keep running and + // their results are still discarded; we just stop accumulating + // references to forever-pending work. + for (const p of snapshot) { + pendingLastRetrievedWrites.delete(p); + } + return { outcome: 'timeout', pending }; + } + return { outcome: 'drained', pending: 0 }; +} + +function trackLastRetrievedWrite(promise: Promise): void { + pendingLastRetrievedWrites.add(promise); + promise + .finally(() => pendingLastRetrievedWrites.delete(promise)) + .catch(() => { + /* swallow — IIFE already logged; .finally already removed */ + }); +} + +/** Test seam — clears the pending set so each test starts clean. */ +export function _resetPendingLastRetrievedWritesForTests(): void { + pendingLastRetrievedWrites.clear(); +} + +/** Test seam — peek the current pending count. */ +export function _peekPendingLastRetrievedWritesForTests(): number { + return pendingLastRetrievedWrites.size; +} + /** * Resolve `search.track_retrieval` config with a 30s in-process cache so * hot-path callers don't pay a SELECT per search. Default-on: missing @@ -77,8 +163,11 @@ export function _resetTrackRetrievalCacheForTests(): void { */ export function bumpLastRetrievedAt(engine: BrainEngine, pageIds: number[]): void { if (pageIds.length === 0) return; - // Fire-and-forget on purpose. We deliberately do NOT return the promise. - void (async () => { + // Fire-and-forget on purpose for callers (MCP, internal). The CLI + // path awaits the drain helper below before disconnecting, which + // is how #1247/#1269/#1290 are fixed without exposing the IIFE + // promise to every caller. + const promise = (async () => { try { const enabled = await isTrackingEnabled(engine); if (!enabled) return; @@ -102,4 +191,5 @@ export function bumpLastRetrievedAt(engine: BrainEngine, pageIds: number[]): voi console.warn(`[last-retrieved] write-back failed (best-effort): ${msg}`); } })(); + trackLastRetrievedWrite(promise); } diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 9d168eb3f..84ae58ef8 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -127,6 +127,67 @@ export function computeSnapshotSchemaHash( return hash.digest('hex'); } +/** + * v0.41.8.0 (#1340) — classify PGLite.create() init failures so + * the user-visible hint points at the right next step. + * + * `bunfs` — Bun's vfs ENOENT on older macOS where `/$$bunfs/root` + * is read-only, so PGLite can't extract its `pglite.data` WASM + * payload. Fix: `bun upgrade` (newer Bun versions mount the vfs + * writable) or run via Node. + * + * `macos-26-3` — the pre-existing #223 hint signature (early macOS + * 26.3 builds shipped a broken WASM runtime). + * + * `unknown` — falls through to a generic hint that still names the + * doctor command and the most-common-cause link. + * + * Regex tightened per Codex eng-review finding #9: don't match + * generic `pglite.data` substring (could fire on unrelated PGLite + * errors). Match the literal `$$bunfs` marker OR ENOENT+pglite.data + * co-occurrence. + */ +export type PgliteInitFailure = 'bunfs' | 'macos-26-3' | 'unknown'; + +export function classifyPgliteInitError(message: string): PgliteInitFailure { + if (/\$\$bunfs|ENOENT[\s\S]*pglite\.data/i.test(message)) return 'bunfs'; + if (/abort.*runtime|macos.*26\.3|wasm.*runtime/i.test(message)) { + return 'macos-26-3'; + } + return 'unknown'; +} + +export function buildPgliteInitErrorMessage( + verdict: PgliteInitFailure, + original: string, +): string { + const header = 'PGLite failed to initialize its WASM runtime.'; + let hint: string; + switch (verdict) { + case 'bunfs': + hint = + ' This looks like a Bun vfs issue: `/$$bunfs/root` is read-only on\n' + + ' your system, so PGLite cannot extract its pglite.data WASM payload.\n' + + ' Fix: `bun upgrade` (newer Bun mounts the vfs writable). If that\n' + + ' does not help, run via Node: `node src/cli.ts` or install gbrain\n' + + ' using the Node-based path. See #1340 for details.'; + break; + case 'macos-26-3': + hint = + ' This is most commonly the macOS 26.3 WASM bug:\n' + + ' https://github.com/garrytan/gbrain/issues/223'; + break; + case 'unknown': + default: + hint = + ' Most common cause: the macOS 26.3 WASM bug\n' + + ' (https://github.com/garrytan/gbrain/issues/223).\n' + + ' Run `gbrain doctor` for a full diagnosis.'; + break; + } + return `${header}\n${hint}\n Original error: ${original}`; +} + export class PGLiteEngine implements BrainEngine { readonly kind = 'pglite' as const; private _db: PGLiteDB | null = null; @@ -173,18 +234,15 @@ export class PGLiteEngine implements BrainEngine { extensions: { vector, pg_trgm }, }); } catch (err) { - // v0.13.1: any PGLite.create() failure becomes actionable. Most commonly - // this is the macOS 26.3 WASM bug (#223). We deliberately do NOT suggest - // "missing migrations" as a cause — migrations run AFTER create(), so a - // create-time abort has nothing to do with them. Nest the original error - // message so debugging isn't erased. + // v0.13.1: any PGLite.create() failure becomes actionable. v0.41.8.0 + // (#1340): the previous error hint hardcoded the macOS 26.3 link, but + // the same crash shape can come from Bun's vfs (`/$$bunfs/root` is + // read-only on older macOS + Bun 1.3.x, so PGLite can't extract its + // pglite.data WASM payload). Route the hint by failure shape so + // users get the right next step. const original = err instanceof Error ? err.message : String(err); - const wrapped = new Error( - `PGLite failed to initialize its WASM runtime.\n` + - ` This is most commonly the macOS 26.3 WASM bug: https://github.com/garrytan/gbrain/issues/223\n` + - ` Run \`gbrain doctor\` for a full diagnosis.\n` + - ` Original error: ${original}` - ); + const verdict = classifyPgliteInitError(original); + const wrapped = new Error(buildPgliteInitErrorMessage(verdict, original)); // Release the lock so a fresh process can try again; leaking the lock // here turns a recoverable init error into a stuck-brain state. if (this._lock?.acquired) { @@ -196,13 +254,30 @@ export class PGLiteEngine implements BrainEngine { } async disconnect(): Promise { - if (this._db) { - await this._db.close(); - this._db = null; - } - if (this._lock?.acquired) { - await releaseLock(this._lock); - this._lock = null; + // v0.41.8.0: snapshot + early-null up front so a concurrent + // `connect()` cannot observe `_db` pointing at a handle that's + // mid-close (partial-state race). Closes the bug class PR #1337 + // originally surfaced. + // + // try/finally guarantees the file lock releases even if + // `db.close()` throws. Pre-fix, a close-throw would leak the + // lock and the next gbrain invocation would wedge waiting for it. + // The pre-fix code happened to work because the close branch + // ran first and the lock branch ran second only when close + // didn't throw — moving to the snapshot pattern made the + // try/finally explicitly necessary. + const db = this._db; + this._db = null; + const lock = this._lock; + this._lock = null; + try { + if (db) { + await db.close(); + } + } finally { + if (lock?.acquired) { + await releaseLock(lock); + } } } diff --git a/test/cli-should-force-exit.test.ts b/test/cli-should-force-exit.test.ts new file mode 100644 index 000000000..d060c7fde --- /dev/null +++ b/test/cli-should-force-exit.test.ts @@ -0,0 +1,73 @@ +/** + * v0.41.8.0 — shouldForceExitAfterMain argv parsing unit tests. + * + * The function is the safety guard that protects daemons from the + * narrow timeout-only force-exit in cli.ts. If it misclassifies + * `gbrain serve` as a non-daemon (or any other intentional long- + * runner that gets added later), the daemon dies after the first + * request. Pure function; testable in isolation; deserves its own + * unit cases beyond the e2e daemon-survival smoke. + */ + +import { describe, test, expect } from 'bun:test'; +import { shouldForceExitAfterMain } from '../src/core/cli-force-exit.ts'; + +describe('shouldForceExitAfterMain — daemon survival gate', () => { + test('returns false for bare `serve` (stdio daemon)', () => { + expect(shouldForceExitAfterMain(['serve'])).toBe(false); + }); + + test('returns false for `serve --http --port 3131`', () => { + expect(shouldForceExitAfterMain(['serve', '--http', '--port', '3131'])).toBe(false); + }); + + test('returns false even when global flags precede `serve`', () => { + // `--quiet`, `--progress-json`, `--progress-interval=Nms` are stripped + // by parseGlobalFlags BEFORE command dispatch — but shouldForceExitAfterMain + // may be called with the raw argv. The .find skips flags, so the first + // positional should resolve to the actual command regardless of global + // flag position. This is the load-bearing case for `gbrain --quiet serve`. + expect(shouldForceExitAfterMain(['--quiet', 'serve'])).toBe(false); + expect(shouldForceExitAfterMain(['--progress-json', 'serve', '--http'])).toBe(false); + expect(shouldForceExitAfterMain(['--progress-interval=500', '--quiet', 'serve'])).toBe(false); + }); + + test('returns true for op commands (search/query/get)', () => { + expect(shouldForceExitAfterMain(['search', 'foxtrot'])).toBe(true); + expect(shouldForceExitAfterMain(['query', 'where is foo'])).toBe(true); + expect(shouldForceExitAfterMain(['get', 'people/alice'])).toBe(true); + }); + + test('returns true for non-daemon CLI commands', () => { + expect(shouldForceExitAfterMain(['stats'])).toBe(true); + expect(shouldForceExitAfterMain(['doctor'])).toBe(true); + expect(shouldForceExitAfterMain(['sync', '--no-pull'])).toBe(true); + expect(shouldForceExitAfterMain(['embed', '--stale'])).toBe(true); + }); + + test('returns true for empty argv (no command)', () => { + // Defensive: with no positional, `--version` / `--help` would have already + // exited. If we somehow land here with empty args, force-exit is safe + // (no daemon is running). + expect(shouldForceExitAfterMain([])).toBe(true); + }); + + test('returns true for flag-only argv', () => { + expect(shouldForceExitAfterMain(['--help'])).toBe(true); + expect(shouldForceExitAfterMain(['-h'])).toBe(true); + expect(shouldForceExitAfterMain(['--version'])).toBe(true); + }); + + test('uses process.argv.slice(2) by default when called with no args', () => { + // The default is just for the cli.ts call site convenience; the test + // verifies the default works without crashing. + expect(typeof shouldForceExitAfterMain()).toBe('boolean'); + }); + + test('substring match avoidance: `serves` is NOT `serve`', () => { + // Future-proofing against a `gbrain serves-foo` subcommand being + // misclassified as a daemon. Strict equality, not startsWith. + expect(shouldForceExitAfterMain(['serves'])).toBe(true); + expect(shouldForceExitAfterMain(['serve-cluster'])).toBe(true); + }); +}); diff --git a/test/e2e/pglite-cli-exit.serial.test.ts b/test/e2e/pglite-cli-exit.serial.test.ts new file mode 100644 index 000000000..13851c7be --- /dev/null +++ b/test/e2e/pglite-cli-exit.serial.test.ts @@ -0,0 +1,278 @@ +/** + * v0.41.8.0 — IRON-RULE regression for #1247, #1269, #1290. + * + * Pre-fix: `gbrain search`, `gbrain query`, `gbrain get` on PGLite + * printed results then hung at ~95-98% CPU until SIGKILL. + * + * Post-fix: each command exits 0 within a few seconds. + * + * This test spawns the CLI as a real subprocess against a hermetic + * GBRAIN_HOME tempdir, seeds a brain with 2 pages, runs each verb + * with a hard timeout, and asserts exit 0. Without the drain helper + * in cli.ts, every variant would time out. + * + * Bonus assertion: `gbrain serve --http` (a daemon) MUST stay alive + * after the first request — the narrow force-exit guard added in + * v0.41.8.0 is supposed to fire ONLY on op-dispatch drain timeout, + * NEVER for daemons. This catches any future regression where the + * force-exit gets broadened or the guard mis-recognizes 'serve'. + * + * Marked .serial because each test spawns a real bun subprocess that + * cold-starts PGLite WASM (~5-10s wallclock). Running these in the + * parallel pool would starve siblings of WASM init time. + * + * The reproducibility preconditions (per Codex eng-review #4) are + * encoded explicitly: + * - Seeded pages have `last_retrieved_at NULL` (verified via + * fresh PGLite init — column starts NULL). + * - At least one page id returned from each verb (search hits the + * literal title; get hits the seeded slug). + * - `search.track_retrieval` left unset → default-on path fires the + * bumpLastRetrievedAt write that pre-fix would race disconnect. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { spawn, spawnSync } from 'child_process'; +import { + mkdirSync, + mkdtempSync, + rmSync, + writeFileSync, + chmodSync, +} from 'fs'; +import { tmpdir } from 'os'; +import { join, resolve } from 'path'; + +const REPO_ROOT = resolve(import.meta.dir, '..', '..'); +const BIN_CACHE = join(REPO_ROOT, 'test', '.cache'); +const SHIM_PATH = join(BIN_CACHE, 'gbrain-pglite-exit-shim.sh'); + +beforeAll(() => { + // Same shim pattern as claw-test e2e: bun --compile can't bundle + // PGLite's pglite.data, so we delegate to `bun run src/cli.ts`. + mkdirSync(BIN_CACHE, { recursive: true }); + const shim = `#!/bin/sh\nexec bun run "${join(REPO_ROOT, 'src', 'cli.ts')}" "$@"\n`; + writeFileSync(SHIM_PATH, shim, 'utf-8'); + chmodSync(SHIM_PATH, 0o755); +}, 10_000); + +// Set up a fresh hermetic PGLite brain once per file; each verb +// runs against the same brain to amortize cold-start. +let tmpHome: string; +let repoSourceDir: string; +let runEnv: NodeJS.ProcessEnv; + +beforeAll(() => { + tmpHome = mkdtempSync(join(tmpdir(), 'gbrain-pglite-exit-')); + repoSourceDir = mkdtempSync(join(tmpdir(), 'gbrain-pglite-exit-src-')); + + // Seed a tiny git repo with 2 markdown pages so `gbrain sync` has + // something to import. The pages contain the literal token 'foxtrot' + // so search has a deterministic keyword hit. + writeFileSync( + join(repoSourceDir, 'alpha.md'), + '---\ntitle: Alpha\n---\nThe quick brown foxtrot jumps over the lazy dog.\n', + ); + writeFileSync( + join(repoSourceDir, 'beta.md'), + '---\ntitle: Beta\n---\nFoxtrot is a NATO phonetic letter F.\n', + ); + + // git init + commit so sync has a HEAD to anchor against + spawnSync('git', ['init', '-q', '-b', 'main'], { cwd: repoSourceDir }); + spawnSync('git', ['config', 'user.email', 'test@example.com'], { cwd: repoSourceDir }); + spawnSync('git', ['config', 'user.name', 'Test'], { cwd: repoSourceDir }); + spawnSync('git', ['add', '-A'], { cwd: repoSourceDir }); + spawnSync('git', ['commit', '-q', '-m', 'seed'], { cwd: repoSourceDir }); + + // Strip embedding-provider env vars so init doesn't refuse on the + // multi-provider ambiguity check. We don't need embeddings — sync + // runs with --no-embed below and search/get are keyword-only paths. + runEnv = { ...process.env, GBRAIN_HOME: tmpHome }; + delete runEnv.VOYAGE_API_KEY; + delete runEnv.ZEROENTROPY_API_KEY; + delete runEnv.OPENAI_API_KEY; + delete runEnv.ANTHROPIC_API_KEY; + delete runEnv.GOOGLE_API_KEY; + + const initResult = spawnSync( + SHIM_PATH, + ['init', '--pglite', '--repo', repoSourceDir, '--no-embedding', '--yes'], + { + cwd: REPO_ROOT, + env: runEnv, + encoding: 'utf-8', + timeout: 60_000, + }, + ); + if (initResult.status !== 0) { + throw new Error( + `gbrain init failed (code=${initResult.status}):\n` + + `STDOUT:\n${initResult.stdout}\n` + + `STDERR:\n${initResult.stderr}`, + ); + } + + // Sync to import the pages (no-embed: skip the embedding step so + // the test doesn't need any provider key). + const syncResult = spawnSync( + SHIM_PATH, + ['sync', '--repo', repoSourceDir, '--no-pull', '--no-embed'], + { + cwd: REPO_ROOT, + env: runEnv, + encoding: 'utf-8', + timeout: 60_000, + }, + ); + if (syncResult.status !== 0) { + throw new Error( + `gbrain sync failed (code=${syncResult.status}):\n` + + `STDOUT:\n${syncResult.stdout}\n` + + `STDERR:\n${syncResult.stderr}`, + ); + } +}, 180_000); + +afterAll(() => { + try { + rmSync(tmpHome, { recursive: true, force: true }); + rmSync(repoSourceDir, { recursive: true, force: true }); + } catch { + /* best effort cleanup */ + } +}); + +/** + * Spawn the CLI with a wall-clock timeout. Returns exit code + output. + * Without the v0.41.8.0 fix, the subprocess hangs forever and the + * timeout would force-kill. The IRON-RULE assertion is "exit code 0 + * within timeoutMs." + */ +function runWithTimeout( + args: string[], + timeoutMs: number, +): Promise<{ code: number | null; stdout: string; stderr: string; durationMs: number }> { + return new Promise((resolveOut) => { + const t0 = Date.now(); + const child = spawn(SHIM_PATH, args, { + cwd: REPO_ROOT, + env: runEnv, + }); + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (d) => (stdout += d.toString())); + child.stderr.on('data', (d) => (stderr += d.toString())); + const timer = setTimeout(() => { + try { child.kill('SIGKILL'); } catch { /* ignore */ } + }, timeoutMs); + child.on('exit', (code) => { + clearTimeout(timer); + resolveOut({ code, stdout, stderr, durationMs: Date.now() - t0 }); + }); + }); +} + +describe('v0.41.8.0 — PGLite CLI read commands exit cleanly (#1247/#1269/#1290)', () => { + test('gbrain search "foxtrot" exits 0 within 15s', async () => { + const { code, stdout, stderr, durationMs } = await runWithTimeout( + ['search', 'foxtrot', '--limit', '3'], + 15_000, + ); + if (code !== 0) { + throw new Error( + `expected exit 0, got ${code}; duration=${durationMs}ms\n` + + `STDOUT:\n${stdout}\nSTDERR:\n${stderr}`, + ); + } + expect(code).toBe(0); + // Must have actually returned a hit — else bumpLastRetrievedAt + // would have early-returned on empty pageIds and the bug wouldn't + // have been exercised. + expect(stdout.length).toBeGreaterThan(0); + }, 30_000); + + test('gbrain get returns a page body and exits 0 within 15s', async () => { + const { code, stdout, stderr, durationMs } = await runWithTimeout( + ['get', 'alpha'], + 15_000, + ); + if (code !== 0) { + throw new Error( + `expected exit 0, got ${code}; duration=${durationMs}ms\n` + + `STDOUT:\n${stdout}\nSTDERR:\n${stderr}`, + ); + } + expect(code).toBe(0); + expect(stdout).toContain('foxtrot'); + }, 30_000); + + test('gbrain query without --no-expand exits 0 within 15s (no API key)', async () => { + // Without an API key, expansion + vector branches degrade + // gracefully. The op still runs the keyword path and returns + // results. The DRAIN is what we're testing, not query quality. + const { code, stderr, durationMs } = await runWithTimeout( + ['query', 'foxtrot', '--limit', '3', '--no-expand'], + 15_000, + ); + if (code !== 0) { + // Some test environments may fail query on missing embed key — + // we tolerate that, but ALL of the rapid exit invariants still + // apply: the process MUST exit, not hang. duration < 15s proves + // it didn't hang; non-zero is acceptable. + expect(durationMs).toBeLessThan(15_000); + return; + } + expect(code).toBe(0); + }, 30_000); +}); + +describe('v0.41.8.0 — daemon survival (regression guard for narrow force-exit)', () => { + test('gbrain serve --http stays alive past the timeout window', async () => { + // Pick a likely-free ephemeral port. We're testing "still alive + // 3 seconds after startup" — if the force-exit guard misfired + // on 'serve', the process would die immediately after binding. + const port = 31000 + Math.floor(Math.random() * 1000); + const child = spawn( + SHIM_PATH, + ['serve', '--http', '--port', String(port), '--token-ttl', '60'], + { + cwd: REPO_ROOT, + env: runEnv, + detached: false, + }, + ); + + let exitedEarly = false; + let earlyCode: number | null = null; + child.on('exit', (code) => { + exitedEarly = true; + earlyCode = code; + }); + + // Give the server 3 seconds. If the force-exit narrow guard is + // working, the daemon stays alive past this window. + await new Promise((r) => setTimeout(r, 3_000)); + + const wasAlive = !exitedEarly; + try { + child.kill('SIGTERM'); + // Give it a moment to clean up + await new Promise((r) => setTimeout(r, 1_000)); + if (!exitedEarly) { + try { child.kill('SIGKILL'); } catch { /* already dead */ } + } + } catch { + /* already dead */ + } + + if (!wasAlive) { + throw new Error( + `gbrain serve --http exited within 3s (code=${earlyCode}). ` + + `If the narrow force-exit guard misclassified 'serve' as a ` + + `non-daemon command, this is the regression.`, + ); + } + expect(wasAlive).toBe(true); + }, 15_000); +}); diff --git a/test/find-experts-op.test.ts b/test/find-experts-op.test.ts index 97272760f..a40efc39c 100644 --- a/test/find-experts-op.test.ts +++ b/test/find-experts-op.test.ts @@ -20,8 +20,9 @@ import type { OperationContext } from '../src/core/operations.ts'; import type { ChunkInput } from '../src/core/types.ts'; let engine: PGLiteEngine; +let schemaDim: number; -function basisEmbedding(idx: number, dim = 1536): Float32Array { +function basisEmbedding(idx: number, dim: number): Float32Array { const emb = new Float32Array(dim); emb[idx % dim] = 1.0; return emb; @@ -32,6 +33,18 @@ beforeAll(async () => { await engine.connect({}); await engine.initSchema(); + // v0.41.8.0: query the schema's actual embedding dim instead of + // hardcoding 1536. Pre-fix, the test hardcoded 1536 but master's + // v0.36.0 default changed to ZeroEntropy 1280d, AND a gateway- + // configured local env may resolve to OpenAI 1536d. The dim is + // resolved at initSchema() time from the configured gateway (with + // DEFAULT_EMBEDDING_DIMENSIONS=1280 fallback). Either way, the seed + // embedding's dim must match the column's dim, so we ask the column. + const dimRows = await engine.executeRaw<{ atttypmod: number }>( + "SELECT atttypmod FROM pg_attribute WHERE attrelid = 'content_chunks'::regclass AND attname = 'embedding'", + ); + schemaDim = dimRows[0]?.atttypmod ?? 1280; + await engine.putPage('wiki/people/expert', { type: 'person', title: 'Expert', @@ -42,7 +55,7 @@ beforeAll(async () => { chunk_index: 0, chunk_text: 'Expert is the authority on widgets.', chunk_source: 'compiled_truth', - embedding: basisEmbedding(7), + embedding: basisEmbedding(7, schemaDim), token_count: 10, } as ChunkInput, ]); diff --git a/test/fix-wave-structural.test.ts b/test/fix-wave-structural.test.ts index 5ce5c5499..af4d74151 100644 --- a/test/fix-wave-structural.test.ts +++ b/test/fix-wave-structural.test.ts @@ -117,3 +117,74 @@ describe('v0.36.1.x #1124 — query --no-expand actually negates expand', () => expect(src).toMatch(/params\[positiveKey\]\s*=\s*false/); }); }); + +describe('v0.41.8.0 #1247/#1269/#1290 — drain last-retrieved before CLI disconnect', () => { + test('cli.ts imports awaitPendingLastRetrievedWrites', () => { + const src = readFileSync('src/cli.ts', 'utf8'); + // Allow additional type-imports from the same module (e.g. `type DrainOutcome`) + expect(src).toMatch(/import\s+\{[^}]*\bawaitPendingLastRetrievedWrites\b[^}]*\}\s*from\s+['"]\.\/core\/last-retrieved\.ts['"]/); + }); + + test('last-retrieved.ts exports the drain + tracks promises in a module-scoped Set', () => { + const src = readFileSync('src/core/last-retrieved.ts', 'utf8'); + expect(src).toMatch(/export async function awaitPendingLastRetrievedWrites/); + expect(src).toMatch(/pendingLastRetrievedWrites\s*=\s*new\s+Set/); + expect(src).toMatch(/pendingLastRetrievedWrites\.add\(promise\)/); + // Per D5+D8: snapshot pattern (Codex finding #3) + bounded timeout + expect(src).toMatch(/Promise\.race/); + expect(src).toMatch(/drain timed out/); + }); + + test('cli.ts behavioral positioning: drain appears BEFORE engine.disconnect in op-dispatch', () => { + const src = readFileSync('src/cli.ts', 'utf8'); + // Per D5+D8: replaces the brittle literal-output regex from PR #1259 + // with a behavioral-positioning assertion. The drain CALL must appear + // textually before the disconnect CALL in the local-engine path. + // Match `await fn(` not bare names — bare names also appear in + // comments and would false-match the comment ordering. + const localPath = src.match(/\/\/ Local engine path \(unchanged behavior[\s\S]+?^\}/m); + expect(localPath).not.toBeNull(); + const block = localPath![0]; + const drainCallRe = /await\s+awaitPendingLastRetrievedWrites\s*\(/; + const disconnectCallRe = /await\s+engine\.disconnect\s*\(/; + expect(block).toMatch(drainCallRe); + expect(block).toMatch(disconnectCallRe); + const drainMatch = block.match(drainCallRe); + const disconnectMatch = block.match(disconnectCallRe); + const drainIdx = block.indexOf(drainMatch![0]); + const disconnectIdx = block.indexOf(disconnectMatch![0]); + expect(drainIdx).toBeGreaterThan(-1); + expect(disconnectIdx).toBeGreaterThan(-1); + expect(drainIdx).toBeLessThan(disconnectIdx); + }); + + test('cli.ts uses shouldForceExitAfterMain only on the timeout path', () => { + const src = readFileSync('src/cli.ts', 'utf8'); + expect(src).toMatch(/import\s+\{\s*shouldForceExitAfterMain\s*\}\s*from\s+['"]\.\/core\/cli-force-exit\.ts['"]/); + // The force-exit gate MUST be conditioned on drainResult.outcome ==='timeout' + expect(src).toMatch(/drainResult\.outcome\s*===\s*['"]timeout['"]/); + }); + + test('cli-force-exit.ts daemon guard excludes "serve"', () => { + const src = readFileSync('src/core/cli-force-exit.ts', 'utf8'); + expect(src).toMatch(/export function shouldForceExitAfterMain/); + expect(src).toMatch(/DAEMON_COMMANDS[\s\S]*serve/); + }); +}); + +describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => { + test('pglite-engine.ts exports classifyPgliteInitError + buildPgliteInitErrorMessage', () => { + const src = readFileSync('src/core/pglite-engine.ts', 'utf8'); + expect(src).toMatch(/export function classifyPgliteInitError/); + expect(src).toMatch(/export function buildPgliteInitErrorMessage/); + // Per Codex finding #9: regex tightened to $$bunfs OR ENOENT+pglite.data + expect(src).toMatch(/\$\$bunfs/); + expect(src).toMatch(/ENOENT/); + }); + + test('pglite-engine.ts connect catch block routes through the classifier', () => { + const src = readFileSync('src/core/pglite-engine.ts', 'utf8'); + expect(src).toMatch(/classifyPgliteInitError\(original\)/); + expect(src).toMatch(/buildPgliteInitErrorMessage\(verdict, original\)/); + }); +}); diff --git a/test/last-retrieved.test.ts b/test/last-retrieved.test.ts new file mode 100644 index 000000000..fcd5bc8e6 --- /dev/null +++ b/test/last-retrieved.test.ts @@ -0,0 +1,176 @@ +/** + * v0.41.8.0 — drain helper for fire-and-forget last_retrieved_at writes. + * + * Closes #1247, #1269, #1290: PGLite CLI commands printed search / + * query / get_page output then hung at ~95-98% CPU until SIGKILL. + * The drain helper here is the structural fix paired with the + * cli.ts narrow timeout-only force-exit guard. + */ + +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import type { BrainEngine } from '../src/core/engine.ts'; +import { + awaitPendingLastRetrievedWrites, + bumpLastRetrievedAt, + _resetTrackRetrievalCacheForTests, + _resetPendingLastRetrievedWritesForTests, + _peekPendingLastRetrievedWritesForTests, +} from '../src/core/last-retrieved.ts'; + +// Minimal BrainEngine stub. The drain helper itself doesn't touch +// the engine — only bumpLastRetrievedAt does — so we control behavior +// through what `executeRaw` does. +function makeStubEngine(opts?: { + executeRaw?: (sql: string, params?: unknown[]) => Promise; + getConfig?: (key: string) => Promise; +}): BrainEngine { + const engine = { + kind: 'pglite' as const, + executeRaw: + opts?.executeRaw ?? + (async () => { + return []; + }), + getConfig: + opts?.getConfig ?? + (async () => { + return null; + }), + }; + return engine as unknown as BrainEngine; +} + +describe('awaitPendingLastRetrievedWrites', () => { + beforeEach(() => { + _resetPendingLastRetrievedWritesForTests(); + _resetTrackRetrievalCacheForTests(); + }); + + afterEach(() => { + _resetPendingLastRetrievedWritesForTests(); + _resetTrackRetrievalCacheForTests(); + }); + + test('empty set returns drained:0 immediately (fast-path)', async () => { + const t0 = Date.now(); + const result = await awaitPendingLastRetrievedWrites(); + const dt = Date.now() - t0; + expect(result).toEqual({ outcome: 'drained', pending: 0 }); + expect(dt).toBeLessThan(50); + }); + + test('single tracked write completes, drain resolves cleanly', async () => { + let resolved = false; + const engine = makeStubEngine({ + executeRaw: async () => { + await new Promise((r) => setTimeout(r, 30)); + resolved = true; + return []; + }, + }); + bumpLastRetrievedAt(engine, [1, 2, 3]); + expect(_peekPendingLastRetrievedWritesForTests()).toBe(1); + const result = await awaitPendingLastRetrievedWrites(); + expect(result).toEqual({ outcome: 'drained', pending: 0 }); + expect(resolved).toBe(true); + // Promise removed from set on settle + expect(_peekPendingLastRetrievedWritesForTests()).toBe(0); + }); + + test('multiple tracked writes all settled via allSettled', async () => { + let count = 0; + const engine = makeStubEngine({ + executeRaw: async () => { + await new Promise((r) => setTimeout(r, 20)); + count++; + return []; + }, + }); + bumpLastRetrievedAt(engine, [1]); + bumpLastRetrievedAt(engine, [2]); + bumpLastRetrievedAt(engine, [3]); + expect(_peekPendingLastRetrievedWritesForTests()).toBe(3); + const result = await awaitPendingLastRetrievedWrites(); + expect(result.outcome).toBe('drained'); + expect(count).toBe(3); + expect(_peekPendingLastRetrievedWritesForTests()).toBe(0); + }); + + test('throw inside IIFE still settles the promise; drain completes', async () => { + const engine = makeStubEngine({ + executeRaw: async () => { + throw new Error('synthetic-failure'); + }, + }); + bumpLastRetrievedAt(engine, [1]); + // Brief tick so the IIFE has a chance to run and reject + await new Promise((r) => setTimeout(r, 10)); + const result = await awaitPendingLastRetrievedWrites(); + expect(result.outcome).toBe('drained'); + expect(_peekPendingLastRetrievedWritesForTests()).toBe(0); + }); + + test('permanently-pending promise returns timeout outcome with pending count', async () => { + // Stage a manually-tracked promise that never resolves so the + // drain hits the timeout path. We can't use bumpLastRetrievedAt + // for this (its IIFE always settles via try/catch) — we need + // the raw track API. Use the public API and a never-resolving + // executeRaw stub. + const neverEngine = makeStubEngine({ + executeRaw: () => new Promise(() => { + /* never */ + }), + }); + bumpLastRetrievedAt(neverEngine, [1]); + await new Promise((r) => setTimeout(r, 10)); + expect(_peekPendingLastRetrievedWritesForTests()).toBe(1); + + const t0 = Date.now(); + const result = await awaitPendingLastRetrievedWrites(100); // 100ms test timeout + const dt = Date.now() - t0; + + expect(result.outcome).toBe('timeout'); + expect(result.pending).toBe(1); + // Should return within timeout + small buffer; not block forever + expect(dt).toBeGreaterThanOrEqual(100); + expect(dt).toBeLessThan(300); + // C1 fix: snapshot's tracked promises ARE dropped from the set on + // timeout so the next drain doesn't see ghosts (daemon leak guard). + expect(_peekPendingLastRetrievedWritesForTests()).toBe(0); + }); + + test('bumpLastRetrievedAt with empty pageIds does not track a promise', async () => { + const engine = makeStubEngine(); + bumpLastRetrievedAt(engine, []); + expect(_peekPendingLastRetrievedWritesForTests()).toBe(0); + const result = await awaitPendingLastRetrievedWrites(); + expect(result).toEqual({ outcome: 'drained', pending: 0 }); + }); + + test('C1 fix: second drain after timeout is clean (daemon leak guard)', async () => { + // Adversarial-review C1: in `gbrain serve` (long-lived), a timed-out + // IIFE used to stay tracked forever because its `.finally` never + // fires. Repeated timeouts would leak references without bound. + // After the timeout, the next drain MUST see an empty set and + // return immediately rather than re-timing-out on the same ghost. + const neverEngine = makeStubEngine({ + executeRaw: () => new Promise(() => { + /* never */ + }), + }); + bumpLastRetrievedAt(neverEngine, [1]); + await new Promise((r) => setTimeout(r, 10)); + expect(_peekPendingLastRetrievedWritesForTests()).toBe(1); + + const first = await awaitPendingLastRetrievedWrites(100); + expect(first.outcome).toBe('timeout'); + expect(_peekPendingLastRetrievedWritesForTests()).toBe(0); + + // Second drain with no new writes returns immediately. + const t0 = Date.now(); + const second = await awaitPendingLastRetrievedWrites(100); + const dt = Date.now() - t0; + expect(second).toEqual({ outcome: 'drained', pending: 0 }); + expect(dt).toBeLessThan(50); + }); +}); diff --git a/test/pglite-engine-disconnect.serial.test.ts b/test/pglite-engine-disconnect.serial.test.ts new file mode 100644 index 000000000..3c2fd75c7 --- /dev/null +++ b/test/pglite-engine-disconnect.serial.test.ts @@ -0,0 +1,221 @@ +/** + * v0.41.8.0 — PGLiteEngine.disconnect() lifecycle regression tests. + * + * Pins the invariants the v0.41.8.0 hang fix wave depends on: + * + * 1. ORDERING: `db.close()` is called BEFORE the file lock is + * released. A sibling process must not be able to acquire the + * lock and try to connect to a still-closing brain. PR #1337's + * original diff swapped this to release-then-close — we + * explicitly REJECTED that ordering. This test fails if a + * future maintainer reads the PR and applies the swap. + * + * 2. SNAPSHOT + EARLY-NULL: `this._db` is nulled BEFORE awaiting + * `close()`, so a concurrent `connect()` cannot observe a + * partial mid-close state. PR #1337's load-bearing contribution + * that we DID take. + * + * 3. LOCK LEAK GUARD: if `db.close()` throws, the file lock STILL + * releases. Codex outside-voice finding #7 in the eng review: + * without try/finally, a close-throw would wedge every next + * gbrain invocation on the stale lock. + * + * 4. IDEMPOTENCY: calling disconnect() twice is a clean no-op on + * the second call (no throw, no double-close attempt). + * + * 5. DOUBLE-DISCONNECT THEN CONNECT: after disconnect, a fresh + * connect() sees clean state and succeeds. + * + * Marked .serial because PGLite WASM cold-start dominates wallclock + * for fresh-engine-per-test cases — running these in the parallel + * shard pool would starve other PGLite tests of cold-start time. + */ + +import { describe, test, expect } from 'bun:test'; +import { mkdtempSync, rmSync } from 'fs'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; + +function newTempDataDir(): string { + return mkdtempSync(join(tmpdir(), 'gbrain-disconnect-test-')); +} + +describe('PGLiteEngine.disconnect() — v0.41.8.0 lifecycle invariants', () => { + test('ORDERING: db.close() is called BEFORE releaseLock()', async () => { + const dataDir = newTempDataDir(); + try { + const engine = new PGLiteEngine(); + await engine.connect({ database_path: dataDir }); + await engine.initSchema(); + + // Record the actual call order. We spy by replacing the db + // handle's close + the lock handle's release with timestamped + // wrappers. + const calls: string[] = []; + const eng = engine as unknown as { + _db: { close: () => Promise } | null; + _lock: { lockDir: string; acquired: boolean } | null; + }; + + const realClose = eng._db!.close.bind(eng._db!); + eng._db!.close = async () => { + // Tiny delay so a flipped ordering would actually show up + // (release-before-close would beat us if we returned instantly). + await new Promise((r) => setTimeout(r, 10)); + calls.push('db.close'); + return realClose(); + }; + + // releaseLock is module-level in pglite-lock.ts — to spy we have + // to swap the lock object's `acquired` flag detection won't + // route through us. Easier: monkey-patch by replacing the lock + // ref with one whose presence forces releaseLock to no-op (so + // we just measure that the close ran during disconnect and that + // the no-op happened in the same call). + // + // For the ORDERING test specifically, we wrap close and + // measure that the lockDir mkdir is still present immediately + // before close runs and gone after disconnect returns. The + // lockDir's existence is observable on disk. + const { existsSync } = await import('fs'); + const lockDir = eng._lock!.lockDir; + expect(existsSync(lockDir)).toBe(true); + + // Spy on the lock-release moment by polling lockDir existence + // from another timer: when close completes, the lock should + // STILL be present (close-then-release contract). + let lockStillPresentAtCloseFinish = false; + const origClose = eng._db!.close; + eng._db!.close = async () => { + await origClose(); + // Right after close resolves, the lock has NOT yet been + // released (the finally branch hasn't run yet). Check + // synchronously before yielding the event loop again. + lockStillPresentAtCloseFinish = existsSync(lockDir); + }; + + await engine.disconnect(); + + expect(calls).toContain('db.close'); + expect(lockStillPresentAtCloseFinish).toBe(true); + expect(existsSync(lockDir)).toBe(false); + } finally { + rmSync(dataDir, { recursive: true, force: true }); + } + }); + + test('SNAPSHOT + EARLY-NULL: _db is nulled before await close', async () => { + const dataDir = newTempDataDir(); + try { + const engine = new PGLiteEngine(); + await engine.connect({ database_path: dataDir }); + await engine.initSchema(); + + const eng = engine as unknown as { + _db: { close: () => Promise } | null; + }; + + let dbWasNullWhenCloseRan = false; + const realClose = eng._db!.close.bind(eng._db!); + eng._db!.close = async () => { + // Inside close, the engine's _db field should ALREADY be null + // (snapshot pattern). If it's not, the partial-state race is + // back. + dbWasNullWhenCloseRan = eng._db === null; + return realClose(); + }; + + await engine.disconnect(); + expect(dbWasNullWhenCloseRan).toBe(true); + } finally { + rmSync(dataDir, { recursive: true, force: true }); + } + }); + + test('LOCK LEAK GUARD: if db.close() throws, lock still releases', async () => { + const dataDir = newTempDataDir(); + try { + const engine = new PGLiteEngine(); + await engine.connect({ database_path: dataDir }); + await engine.initSchema(); + + const eng = engine as unknown as { + _db: { close: () => Promise } | null; + _lock: { lockDir: string; acquired: boolean } | null; + }; + + const { existsSync } = await import('fs'); + const lockDir = eng._lock!.lockDir; + expect(existsSync(lockDir)).toBe(true); + + // Force close to throw. The lock MUST still release. + eng._db!.close = async () => { + throw new Error('synthetic close failure'); + }; + + // The throw will propagate out of disconnect — that's fine. + // The contract is "lock releases regardless." + let threw = false; + try { + await engine.disconnect(); + } catch (e) { + threw = true; + expect(e instanceof Error && e.message).toContain('synthetic close failure'); + } + expect(threw).toBe(true); + // CRITICAL: lock must be gone even though close threw. + expect(existsSync(lockDir)).toBe(false); + } finally { + rmSync(dataDir, { recursive: true, force: true }); + } + }); + + test('IDEMPOTENCY: double disconnect is a clean no-op on the second call', async () => { + const dataDir = newTempDataDir(); + try { + const engine = new PGLiteEngine(); + await engine.connect({ database_path: dataDir }); + await engine.initSchema(); + + let closeCallCount = 0; + const eng = engine as unknown as { + _db: { close: () => Promise } | null; + }; + const realClose = eng._db!.close.bind(eng._db!); + eng._db!.close = async () => { + closeCallCount++; + return realClose(); + }; + + await engine.disconnect(); + expect(closeCallCount).toBe(1); + + // Second call: no throw, no second close + await engine.disconnect(); + expect(closeCallCount).toBe(1); + } finally { + rmSync(dataDir, { recursive: true, force: true }); + } + }); + + test('RECONNECT after disconnect sees clean state', async () => { + const dataDir = newTempDataDir(); + try { + const engine = new PGLiteEngine(); + await engine.connect({ database_path: dataDir }); + await engine.initSchema(); + await engine.disconnect(); + + // Same dataDir, fresh connect. Must succeed without lock contention. + await engine.connect({ database_path: dataDir }); + await engine.initSchema(); + // Smoke: a SELECT 1 round-trip proves the new handle is alive. + const result = await engine.executeRaw<{ ok: number }>('SELECT 1 AS ok'); + expect(result[0].ok).toBe(1); + await engine.disconnect(); + } finally { + rmSync(dataDir, { recursive: true, force: true }); + } + }); +}); diff --git a/test/pglite-init-classifier.test.ts b/test/pglite-init-classifier.test.ts new file mode 100644 index 000000000..f83cfadd1 --- /dev/null +++ b/test/pglite-init-classifier.test.ts @@ -0,0 +1,105 @@ +/** + * v0.41.8.0 (#1340) — PGLite init-error classifier + hint routing. + * + * Pure-function tests over the classifier + message builder. No + * PGLite cold-start required. The classifier sits in front of the + * connect() catch block and routes the user-visible hint by failure + * shape so users on macOS 12.7.6 + Bun 1.3.14 (the actual #1340 + * environment) don't get pointed at the macOS 26.3 hint (#223) by + * mistake. + * + * Codex eng-review finding #9: the regex must NOT match generic + * `pglite.data` substrings — only the literal `$$bunfs` marker OR + * the ENOENT+pglite.data co-occurrence that bun's vfs failure shows. + */ + +import { describe, test, expect } from 'bun:test'; +import { + classifyPgliteInitError, + buildPgliteInitErrorMessage, +} from '../src/core/pglite-engine.ts'; + +describe('classifyPgliteInitError', () => { + test('bunfs verdict for the literal $$bunfs marker', () => { + const msg = "ENOENT: no such file or directory, open '/$$bunfs/root/pglite.data'."; + expect(classifyPgliteInitError(msg)).toBe('bunfs'); + }); + + test('bunfs verdict for ENOENT + pglite.data co-occurrence (no $$bunfs prefix)', () => { + const msg = 'ENOENT: cannot open pglite.data: read-only file system'; + expect(classifyPgliteInitError(msg)).toBe('bunfs'); + }); + + test('macos-26-3 verdict for the existing #223 signature', () => { + const msg = 'abort() called from wasm runtime on macOS 26.3 build'; + expect(classifyPgliteInitError(msg)).toBe('macos-26-3'); + }); + + test('unknown verdict for generic / unrecognized errors', () => { + const msg = 'TypeError: cannot read property of undefined at PGlite.create'; + expect(classifyPgliteInitError(msg)).toBe('unknown'); + }); + + test('NEGATIVE: generic "pglite.data" mention WITHOUT ENOENT does not trip bunfs', () => { + // Per Codex finding #9: the prior overbroad regex `/bunfs|pglite\.data/i` + // would have classified this as bunfs. The tightened regex requires + // the literal $$bunfs marker OR ENOENT+pglite.data co-occurrence. + const msg = 'Failed to parse pglite.data manifest: invalid magic byte'; + expect(classifyPgliteInitError(msg)).toBe('unknown'); + }); + + test('case-insensitive matching on bunfs marker', () => { + expect(classifyPgliteInitError('SYSCALL ENOENT on /$$BUNFS/root')).toBe('bunfs'); + }); +}); + +describe('buildPgliteInitErrorMessage — hint routing', () => { + const original = 'synthetic original error'; + + test('bunfs verdict surfaces bun upgrade hint AND original error', () => { + const msg = buildPgliteInitErrorMessage('bunfs', original); + expect(msg).toContain('bun upgrade'); + expect(msg).toContain('Bun vfs'); + expect(msg).toContain(original); + // Must NOT redirect to the wrong issue + expect(msg).not.toContain('issues/223'); + }); + + test('macos-26-3 verdict surfaces the #223 link AND original error', () => { + const msg = buildPgliteInitErrorMessage('macos-26-3', original); + expect(msg).toContain('https://github.com/garrytan/gbrain/issues/223'); + expect(msg).toContain('macOS 26.3'); + expect(msg).toContain(original); + expect(msg).not.toContain('Bun vfs'); + }); + + test('unknown verdict surfaces the doctor + #223 fallback AND original error', () => { + const msg = buildPgliteInitErrorMessage('unknown', original); + expect(msg).toContain('gbrain doctor'); + expect(msg).toContain('issues/223'); + expect(msg).toContain(original); + }); + + test('all verdicts produce the canonical header line', () => { + for (const v of ['bunfs', 'macos-26-3', 'unknown'] as const) { + const msg = buildPgliteInitErrorMessage(v, original); + expect(msg.startsWith('PGLite failed to initialize its WASM runtime.')).toBe(true); + } + }); +}); + +describe('#1340 reproducer — exact reporter error string maps to bunfs', () => { + // This is the literal error string from the issue body. + const reportError = `ENOENT: no such file or directory, open '/$$bunfs/root/pglite.data'.`; + + test('classifier routes the reporter\'s error to bunfs', () => { + expect(classifyPgliteInitError(reportError)).toBe('bunfs'); + }); + + test('user-visible message names bun upgrade, NOT macOS 26.3', () => { + const verdict = classifyPgliteInitError(reportError); + const msg = buildPgliteInitErrorMessage(verdict, reportError); + expect(msg).toContain('bun upgrade'); + expect(msg).not.toMatch(/most commonly the macOS 26\.3/); + }); +}); diff --git a/test/seed-pglite.test.ts b/test/seed-pglite.serial.test.ts similarity index 100% rename from test/seed-pglite.test.ts rename to test/seed-pglite.serial.test.ts