mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.42.57.0 fix(pglite): incident — never steal a live data-dir lock + corrupted-store recovery hint (#2348) (#2400)
* fix(pglite): never steal the data-dir lock from a live holder (#2348) A busy `gbrain dream`/`embed` holder whose 30s heartbeat lapsed (the JS event loop is blocked during long synchronous WASM imports/CHECKPOINTs) used to get its lock reaped past the steal-grace window. PGLite/WASM is strictly single-writer, so a second OS process then opened the same data dir and corrupted the catalog + pgvector extension state (58P01 / internal_load_library / "type vector does not exist"), recoverable only by wipe+restore. Reap ONLY a dead PID; a live holder is never stolen — a wedged-but-alive or PID-reused holder makes the acquire time out with a message naming the PID. Removes the GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS knob (no longer meaningful). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pglite): point a corrupted store at reinit-pglite recovery (#2348) classifyPgliteInitError gains a `corrupt` verdict for the 58P01 / internal_load_library / "vector does not exist" / "content_chunks does not exist" signature (beating the generic wasm-runtime match), so an already-damaged store gets actionable recovery (gbrain reinit-pglite / restore a backup) instead of the wrong macOS-WASM hint. Updates KEY_FILES.md to current lock behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.55.0 fix(pglite): incident — never steal a live lock + corrupted-store recovery hint (#2348) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * v0.42.56.0 chore: re-bump past #2399 version collision + refresh ownerToken comment #2399 (security wave) claimed 0.42.55.0; take the next slot. Also updates the LockHandle.ownerToken JSDoc to current #2348 behavior (live holders are never reaped, so reap-then-reacquire is dead-holder-only; token guard stays as defense-in-depth). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
646179047a
commit
058f448b9a
@@ -2,6 +2,17 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.57.0] - 2026-07-02
|
||||
|
||||
**PGLite incident fix: a busy `gbrain dream` (or `embed`) could have its data-directory lock stolen and get its brain corrupted beyond in-place repair. The lock will no longer be taken from a process that is alive, and an already-corrupted store now tells you exactly how to recover.**
|
||||
|
||||
### Fixed
|
||||
- **A live PGLite holder is never stolen.** The data-directory lock used to be reaped if the holder's heartbeat went stale past a grace window. But the heartbeat runs on the JS event loop, which is blocked during long synchronous WASM imports/checkpoints, so a genuinely working `gbrain dream`/`embed` could look stale while fully alive. Reaping it let a second process open the same store and corrupt the catalog + pgvector extension (surfacing later as `relation "content_chunks" does not exist` / `type "vector" does not exist`, only recoverable by wipe-and-restore). The lock is now reaped only when the holder process is actually dead; a wedged-but-alive or PID-reused holder makes the acquire time out with a clear message naming the PID, instead of risking corruption.
|
||||
- **A corrupted PGLite store now explains how to recover.** When the store's catalog or pgvector extension can no longer load, the error names the cause and points at `gbrain reinit-pglite --embedding-model <id> --embedding-dimensions <N>` (or restoring a backup), instead of the unrelated "macOS WASM bug" hint. It also notes that deleting the lock dir or `postmaster.pid` does not fix it.
|
||||
|
||||
### To take advantage of v0.42.57.0
|
||||
`gbrain upgrade`. No migration. New corruption is prevented going forward. A brain already corrupted by a prior concurrent open cannot be repaired in place; the upgraded error message walks you through `gbrain reinit-pglite` or restoring a backup.
|
||||
|
||||
## [0.42.56.0] - 2026-07-02
|
||||
|
||||
**Life Chronicle: gbrain gains a temporal spine. Meetings and transcripts project into a queryable timeline, entities carry a bi-temporal ontology (sourced, confidence-weighted properties that supersede over time), and a low-friction diary captures interiority — so an agent can reconstruct "what happened the week of X", answer "when did I last interact with Y", and see how an entity's role or stance changed, instead of re-deriving chronology from scratch every session.** Built entirely on existing primitives (pages, the `facts` table, `timeline_entries`) — no new datastore. Auto-emission is off by default; opt in per below.
|
||||
|
||||
@@ -28,7 +28,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/commands/doctor.ts` extension — `graph_signals_coverage` check wired into both `runDoctor` (local) and `doctorReportRemote` (HTTP/JSON thin-client path). Reads `search.graph_signals` config first, falls back to mode default; silent `ok` when disabled. Computes inbound link coverage on the page set; warns at <10% with `gbrain extract all` fix hint; `ok` at ≥30% ("fire on most queries") and 10-29% ("fire occasionally"), each with the percentage embedded. Pinned by cases in `test/doctor.test.ts`.
|
||||
- `src/core/engine-factory.ts` — Engine factory with dynamic imports (`'pglite'` | `'postgres'`).
|
||||
- `src/core/pglite-engine.ts` — PGLite (embedded Postgres 17.5 via WASM) implementation, all BrainEngine methods. `listLinkSources({sourceId?, sourceIds?})` returns distinct `link_source` provenances + counts (`ORDER BY count DESC, link_source ASC NULLS LAST`; scalar + federated scoped; parity with postgres-engine.ts) powering `gbrain link-sources`. `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the whole batch as one JSONB document via `jsonb_to_recordset(($1::jsonb)->'rows')` (bound through `executeRawJsonb` with a `{ rows }` wrapper; rows built by the shared `src/core/batch-rows.ts` helpers, NUL-stripped), and are `batchRetry`-wrapped. `connect()` wraps `PGlite.create()` in a try/catch that emits an actionable error (macOS 26.3 WASM bug #223, points at `gbrain doctor`); the lock is released on failure so the next process can retry cleanly. `searchKeyword`/`searchKeywordChunks` multiply `ts_rank` by the source-factor CASE at chunk grain; `searchVector` is a two-stage CTE — inner CTE keeps `ORDER BY cc.embedding <=> vec` so HNSW stays usable, outer SELECT re-ranks by `raw_score * source_factor`, inner LIMIT scales with offset to preserve pagination. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL — probes for forward-referenced state the embedded blob needs (`pages.source_id`, `links.link_source`, `links.origin_page_id`, `content_chunks.symbol_name`, `content_chunks.language`, `sources` FK target, plus `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`, `sources.archived_at`, `sources.archive_expires_at`) and adds only what's missing; threads the DDL connection from `initSchema` so probes run inside the advisory-lock scope; no-op on fresh installs and modern brains (closes the upgrade-wedge bug class #239/#243/#266/#357/#366/#374/#375/#378/#395/#396/#1018/#974/#820). `getBrainScore` returns 100/100 with full breakdown (35/25/15/15/10) when `pageCount === 0` (vacuous truth — empty brain has no coverage problem); Pinned by `test/brain-score-breakdown.test.ts` empty-brain assertion + `test/doctor-report-remote.serial.test.ts`. `disconnect()` uses snapshot+early-null (snapshot `_db`/`_lock`, null instance fields BEFORE any `await` so a concurrent `connect()` can't see a partial mid-close state) wrapped in try/finally guaranteeing lock-release even if `db.close()` throws; KEEPS close-then-release order (release-then-close was rejected: it would widen the window where a sibling process connects to a still-closing brain); Pinned by `test/pglite-engine-disconnect.serial.test.ts`. `PGlite.create()` runs inside `preservingProcessExitCode` (#2084): PGLite's Emscripten runtime writes its own status into `process.exitCode` (99 at create; in-memory brains run initdb whose status lands on a later tick; the exit status at close — and assigning `undefined` cannot clear a prior value), which would otherwise silently clobber an errored op's exit 1 back to 0. The wrapper keeps the global tidy for external readers; `db.close()` stays unwrapped (its 0-write is baseline behavior test runners depend on). The CLI's exit verdict is immune either way — it lives in the gbrain-owned channel in `cli-force-exit.ts` and never reads `process.exitCode` back. Exports `classifyPgliteInitError(message): 'bunfs' | 'macos-26-3' | 'unknown'` + `buildPgliteInitErrorMessage(verdict, original)` routing the catch-block hint by failure shape (`bunfs` matches literal `$$bunfs` OR `ENOENT[\s\S]*pglite\.data` co-occurrence, surfaces a paste-ready `bun upgrade` + Node fallback; `macos-26-3` keeps the #223 link; `unknown` falls through); Pinned by `test/pglite-init-classifier.test.ts`. Implements `deletePages(slugs, {sourceId})` + `resolveSlugsByPaths(paths, {sourceId})` via `slug = ANY($1::text[])` array-param binding, caller-chunking primitive throwing when input exceeds `DELETE_BATCH_SIZE`, `deletePages` returns `RETURNING slug` rows so callers filter `pagesAffected` to confirmed deletes. Implements the embedding-signature stale-detection quartet — `sumStaleChunkChars({sourceId?, signature?})`, `setPageEmbeddingSignature(slug, {sourceId?, signature})`, `invalidateStaleSignatureEmbeddings({signature, sourceId?})`, widened `countStaleChunks({sourceId?, signature?})` (the `signature` opt widens via `JOIN pages p ... WHERE cc.embedding IS NULL OR (p.embedding_signature IS NOT NULL AND p.embedding_signature <> $signature)`, NULL grandfathered); parity SQL with postgres-engine.ts. PGLite-specific DDL (pgvector, pg_trgm, triggers).
|
||||
- `src/core/pglite-lock.ts` (#2058) — advisory data-dir lock so two processes never open the same embedded PGLite (single-connection WASM Postgres) at once. Atomic `mkdir` of `.gbrain-lock/` + a `lock` file carrying `{pid, acquired_at, refreshed_at}`. A held lock HEARTBEATS its `refreshed_at` every 30s (`.unref()`ed timer) so a long-running but LIVE holder (an `embed` job can run for minutes) is never mistaken for stale. A waiting acquirer reaps the holder only when it is dead (PID gone) OR has stopped refreshing past the steal grace (`GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS`, default 600s) — pairing PID liveness with heartbeat freshness defeats BOTH the WAL-corruption bug (stealing a live writer) and the PID-reuse false positive (a recycled PID reading as alive). Each holder carries an ownership token (`<pid>:<acquired_at>`); the heartbeat and `releaseLock` verify the on-disk lock is STILL theirs before touching it, so a stalled-then-resumed holder that was already reaped + replaced can't clobber the new owner. In-memory engines take no lock (no file, no concurrent access). Pinned by `test/pglite-lock.test.ts`.
|
||||
- `src/core/pglite-lock.ts` — advisory data-dir lock so two processes never open the same embedded PGLite (single-connection WASM Postgres) at once. Atomic `mkdir` of `.gbrain-lock/` + a `lock` file carrying `{pid, acquired_at, refreshed_at}`. A held lock HEARTBEATS its `refreshed_at` every 30s (`.unref()`ed timer; informational). A waiting acquirer reaps a holder ONLY when its PID is dead — a LIVE holder is NEVER stolen, regardless of how stale its heartbeat is (#2348). The heartbeat runs on the JS event loop, which is blocked during long synchronous WASM imports/CHECKPOINTs, so a working `dream`/`embed` holder can look stale while alive; the old steal-on-stale-heartbeat grace let a second OS process open the same data dir and corrupt the catalog + pgvector extension (58P01 / `internal_load_library` / `type "vector" does not exist`), recoverable only by wipe+restore. A wedged-but-alive or PID-reused holder now makes the acquire TIME OUT with a message naming the PID (the user removes the lock explicitly) rather than risk corruption. Each holder carries an ownership token (`<pid>:<acquired_at>`); the heartbeat and `releaseLock` verify the on-disk lock is STILL theirs before touching it. In-memory engines take no lock. Pinned by `test/pglite-lock.test.ts`. A corrupted store surfaces a `reinit-pglite` recovery hint via `classifyPgliteInitError`'s `corrupt` verdict in `pglite-engine.ts`.
|
||||
- `src/core/postgres-engine.ts` — Postgres + pgvector implementation (Supabase / self-hosted). `addLinksBatch`/`addTimelineEntriesBatch`/`addTakesBatch` pass the batch as one JSONB document — `INSERT ... SELECT FROM jsonb_to_recordset(($1::jsonb)->'rows') AS v(...) JOIN pages ...` bound through `executeRawJsonb({ rows })` — which encodes arbitrary free text safely (the old `unnest(${arr}::text[])` array-literal path crashed Postgres with "malformed array literal" on calendar/Zoom context, gbrain#1861) and sidesteps the 65535-parameter cap; takes declares native recordset column types (`page_id int, weight real, active boolean, …`) so no per-element casts; all three are `batchRetry`-wrapped. `searchKeyword`/`searchVector` scope `statement_timeout` via `sql.begin` + `SET LOCAL` so the GUC dies with the transaction instead of leaking across the pooled postgres.js connection. `getEmbeddingsByChunkIds` uses `tryParseEmbedding` so one corrupt row skips+warns instead of killing the query. `searchKeyword`/`searchKeywordChunks`/`searchVector` apply source-aware ranking by inlining the source-factor CASE and `NOT (col LIKE …)` hard-exclude from `src/core/search/sql-ranking.ts`; `searchVector` is a two-stage CTE (HNSW-safe inner ORDER BY, source-boost re-rank in outer SELECT) carrying `p.source_id` inner→outer. `_savedConfig` retains the connect config; `reconnect()` tears down + recreates the pool from saved config (called by supervisor watchdog after 3 consecutive health-check failures, and by `batchRetry` on a retryable connection error). Concurrent callers share one in-flight `_reconnectPromise` (they await the single reconnect rather than racing a half-rebuilt pool); ownership re-samples through the atomic `db.connect()` token on the connect leg. `reconnect(ctx?)` accepts the triggering error and records a pool-recovery audit event (`reap_detected`/`reconnect_other`/`reconnect_succeeded`/`reconnect_failed`) for the `pool_reap_health` doctor check. `executeRaw` is a single-statement passthrough — no per-call retry (unsound for non-idempotent statements; recovery is supervisor-driven). `connect()` applies `resolveSessionTimeouts()` from `db.ts` as connection-time startup parameters (`statement_timeout`, `idle_in_transaction_session_timeout`) so orphan pgbouncer backends can't hold locks for hours. `countStaleChunks()`+`listStaleChunks()` server-side-filter on `embedding IS NULL` for `embed --stale` (eliminates ~76 MB/call client-side pull); `upsertChunks()` resets both `embedding` AND `embedded_at` to NULL when chunk_text changes without a new embedding. `initSchema()` calls `applyForwardReferenceBootstrap()` BEFORE replaying SCHEMA_SQL on the same probe set as PGLite (extended for column-only forward-reference cases: `files.source_id`, `files.page_id`, `oauth_clients.source_id`, `oauth_clients.federated_read`, `sources.archived`/`archived_at`/`archive_expires_at`); the entire probe path runs on the DDL connection threaded from `initSchema` (closing a concurrent-bootstrap race for Supabase pooler users); closes #1018/#974/#820. `disconnect()` is idempotent — `_connectionStyle` tracks whether the engine owns its pool (worker engines) or shares the module-level singleton; second call on an instance-pool engine is a no-op rather than clobbering the singleton; and a module-style engine only calls `db.disconnect()` when it owns the singleton (`_ownsModuleSingleton`, set from the `db.connect()` creation token), so a borrower probe engine's teardown leaves the cycle owner's connection intact. Pinned by `test/e2e/postgres-engine-disconnect-idempotency.test.ts` + `test/postgres-engine-singleton-ownership.test.ts`. `getBrainScore` empty-brain parity with PGLite — 100/100 with breakdown 35/25/15/15/10 when `pageCount === 0` (both engines must agree to keep `doctor-report-remote.serial.test.ts` deterministic). Implements `deletePages(slugs, {sourceId}): Promise<string[]>` via `DELETE FROM pages WHERE slug = ANY($1::text[]) AND source_id = $2 RETURNING slug` (single round-trip; caller chunks); `resolveSlugsByPaths` does `SELECT slug, source_path FROM pages WHERE source_path = ANY($1::text[]) AND source_id = $2`; FK cascades through `content_chunks`/`links`/`tags`/`raw_data`/`timeline_entries`/`page_versions`, `files.page_id`+`links.origin_page_id` go SET NULL; throws when input exceeds `DELETE_BATCH_SIZE` (from `src/core/engine-constants.ts`); both short-circuit on empty input. Implements the embedding-signature stale-detection quartet (`sumStaleChunkChars`, `setPageEmbeddingSignature`, `invalidateStaleSignatureEmbeddings`, widened `countStaleChunks`, all accept optional `signature` extending "stale" to model/dims-swap drift via the `pages.embedding_signature` JOIN, NULL grandfathered; the `embedding IS NULL` server-side filter is preserved as the no-signature fast path); Pinned by `test/e2e/engine-parity.test.ts`.
|
||||
- `src/core/cjk.ts` — Single source of truth for CJK detection. Exports `CJK_RANGES_REGEX`, `CJK_SLUG_CHARS` (character-class fragment for embedding inside other regexes), `CJK_SENTENCE_DELIMITERS` (`。！？`), `CJK_CLAUSE_DELIMITERS` (`；：，、`), `CJK_DENSITY_THRESHOLD = 0.30`, `hasCJK(s)`, `countCJKAwareWords(s)` (30% density threshold — English docs with one Japanese term stay whitespace-tokenized; Chinese-dominant docs get char-counted), and `escapeLikePattern(s)` (escapes `%`, `_`, `\\` for `ILIKE ... ESCAPE '\\'`). BMP-only ranges (Han / Hiragana / Katakana / Hangul Syllables). Consumers: `expansion.ts`, `sync.ts:slugifySegment`, `operations.ts:validatePageSlug + validateFilename`, `chunkers/recursive.ts:countWords + DELIMITERS`, `pglite-engine.ts:searchKeyword + searchKeywordChunks`.
|
||||
- `src/core/audit-slug-fallback.ts` — Weekly ISO-week-rotated audit JSONL at `~/.gbrain/audit/slug-fallback-YYYY-Www.jsonl`. `logSlugFallback(slug, sourcePath)` fires when `importFromFile` falls back to a frontmatter slug because `slugifyPath` returned empty (emoji / Thai / Arabic / non-CJK exotic-script filenames). `readRecentSlugFallbacks(days)` reads the last N days for `gbrain doctor`'s `slug_fallback_audit` check. Honors `GBRAIN_AUDIT_DIR` via the shared `resolveAuditDir()`. Separate surface from `sync-failures.jsonl` — that file carries bookmark-gating semantics that info events shouldn't trigger.
|
||||
|
||||
+1
-1
@@ -144,5 +144,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.56.0"
|
||||
"version": "0.42.57.0"
|
||||
}
|
||||
|
||||
@@ -158,10 +158,18 @@ export function computeSnapshotSchemaHash(
|
||||
* errors). Match the literal `$$bunfs` marker OR ENOENT+pglite.data
|
||||
* co-occurrence.
|
||||
*/
|
||||
export type PgliteInitFailure = 'bunfs' | 'macos-26-3' | 'unknown';
|
||||
export type PgliteInitFailure = 'bunfs' | 'macos-26-3' | 'corrupt' | 'unknown';
|
||||
|
||||
export function classifyPgliteInitError(message: string): PgliteInitFailure {
|
||||
if (/\$\$bunfs|ENOENT[\s\S]*pglite\.data/i.test(message)) return 'bunfs';
|
||||
// #2348: a corrupted PGLite data dir (two OS processes opened it concurrently
|
||||
// and trashed the catalog/extension state) surfaces as a 58P01 internal error
|
||||
// loading the pgvector library, or the vector type / a core relation gone
|
||||
// missing. Distinct, actionable cause — must beat the generic wasm-runtime
|
||||
// match below so the user is pointed at recovery, not the macOS WASM bug.
|
||||
if (/58P01|internal_load_library|type "?vector"? does not exist|relation "?content_chunks"? does not exist/i.test(message)) {
|
||||
return 'corrupt';
|
||||
}
|
||||
if (/abort.*runtime|macos.*26\.3|wasm.*runtime/i.test(message)) {
|
||||
return 'macos-26-3';
|
||||
}
|
||||
@@ -188,6 +196,18 @@ export function buildPgliteInitErrorMessage(
|
||||
' This is most commonly the macOS 26.3 WASM bug:\n' +
|
||||
' https://github.com/garrytan/gbrain/issues/223';
|
||||
break;
|
||||
case 'corrupt':
|
||||
hint =
|
||||
' Your PGLite store looks corrupted (the catalog or the pgvector\n' +
|
||||
' extension cannot load). This happens when two processes opened the\n' +
|
||||
' same brain at once — now prevented (#2348), but an already-damaged\n' +
|
||||
' store cannot be repaired in place. Recover:\n' +
|
||||
' 1. Restore a backup of the brain.pglite directory if you have one, OR\n' +
|
||||
' 2. Rebuild from your brain repo:\n' +
|
||||
' gbrain reinit-pglite --embedding-model <id> --embedding-dimensions <N>\n' +
|
||||
' (wipes + re-inits + re-syncs; DB-only state is re-derived).\n' +
|
||||
' Deleting .gbrain-lock/ or postmaster.pid does NOT fix this.';
|
||||
break;
|
||||
case 'unknown':
|
||||
default:
|
||||
hint =
|
||||
|
||||
+25
-30
@@ -24,17 +24,17 @@ const LOCK_FILE = 'lock';
|
||||
// LIVE holder (embed jobs run for many minutes) is never mistaken for stale.
|
||||
const HEARTBEAT_INTERVAL_MS = 30_000;
|
||||
|
||||
// #2058: a holder whose heartbeat refreshed within this window is ALIVE and is
|
||||
// NEVER stolen, regardless of how old the lock is. Only a holder that STOPPED
|
||||
// refreshing past this grace (hung, crashed without cleanup, or a PID since
|
||||
// reused by an unrelated process) is reaped. Pairing heartbeat-age with PID
|
||||
// liveness is what defeats both the WAL-corruption bug (stealing a live
|
||||
// writer) AND the PID-reuse false-positive (a recycled PID reading as "alive").
|
||||
// Env-overridable as an incident escape hatch, matching the sync-lock knobs.
|
||||
function stealGraceMs(): number {
|
||||
const env = parseInt(process.env.GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS ?? '', 10);
|
||||
return Number.isFinite(env) && env > 0 ? env * 1000 : 10 * 60 * 1000; // default 600s
|
||||
}
|
||||
// #2348: there is NO steal-on-stale-heartbeat anymore. A holder whose PID is
|
||||
// alive is NEVER reaped, regardless of how long its heartbeat has been stale.
|
||||
// PGLite/WASM is strictly single-writer; the heartbeat runs on the JS event
|
||||
// loop, which is BLOCKED during long synchronous imports/CHECKPOINTs, so a
|
||||
// genuinely working `gbrain dream`/embed holder can look stale while alive.
|
||||
// Reaping it (the old #2058 grace window) let a second OS process open the same
|
||||
// data dir and corrupt the catalog + pgvector extension state (58P01 /
|
||||
// internal_load_library / `type "vector" does not exist`), recoverable only by
|
||||
// wipe+restore. Only a DEAD PID is reaped now; a wedged-but-alive or PID-reused
|
||||
// holder makes the acquire time out with a message naming the PID (the user
|
||||
// removes the lock explicitly) rather than risk corruption.
|
||||
|
||||
export interface LockHandle {
|
||||
lockDir: string;
|
||||
@@ -47,11 +47,12 @@ export interface LockHandle {
|
||||
heartbeat?: ReturnType<typeof setInterval>;
|
||||
lockPath?: string;
|
||||
/**
|
||||
* #2058 (codex): our ownership token (`<pid>:<acquired_at>`). If we stall
|
||||
* past the steal grace, another process can reap + re-acquire. When we
|
||||
* resume, the heartbeat and release MUST verify the on-disk lock is STILL
|
||||
* ours before touching it — otherwise a resumed stale holder would refresh
|
||||
* or delete the NEW owner's live lock, re-opening the concurrent-writer hole.
|
||||
* Our ownership token (`<pid>:<acquired_at>`). Since #2348 a LIVE holder is
|
||||
* never reaped, so reap-then-reacquire happens only after the original holder
|
||||
* is dead — but the heartbeat and release STILL verify the on-disk lock is
|
||||
* ours before touching it (defense-in-depth: a crash-then-restart on a reused
|
||||
* PID, or a misclassification, must never let a stale handle refresh or delete
|
||||
* the NEW owner's live lock and re-open the concurrent-writer hole).
|
||||
*/
|
||||
ownerToken?: string;
|
||||
}
|
||||
@@ -134,25 +135,19 @@ export async function acquireLock(dataDir: string | undefined, opts?: { timeoutM
|
||||
try {
|
||||
const lockData = JSON.parse(readFileSync(lockPath, 'utf-8'));
|
||||
const lockPid = lockData.pid as number;
|
||||
const lockTime = lockData.acquired_at as number;
|
||||
|
||||
// #2058: classify by PID liveness AND heartbeat freshness. A holder
|
||||
// that is alive AND refreshed its heartbeat within the steal grace is
|
||||
// genuinely working (e.g. a multi-minute embed) and is NEVER reaped —
|
||||
// force-removing it here is what corrupted the single-writer WAL.
|
||||
// #2348: classify ONLY by PID liveness. A live holder is NEVER reaped
|
||||
// (stealing a live single-writer is what corrupted the catalog/extension
|
||||
// state). A long synchronous import blocks the heartbeat, so "stale
|
||||
// heartbeat" is NOT evidence of death — only a dead PID is.
|
||||
const alive = isProcessAlive(lockPid);
|
||||
const lastRefresh = (lockData.refreshed_at as number | undefined) ?? lockTime;
|
||||
const sinceRefresh = Date.now() - lastRefresh;
|
||||
if (!alive) {
|
||||
// Holder process is gone — reap.
|
||||
// Holder process is gone — reap and try to acquire.
|
||||
try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition, try again */ }
|
||||
} else if (sinceRefresh > stealGraceMs()) {
|
||||
// PID is alive but the heartbeat stopped past the grace window:
|
||||
// either the holder hung, or this PID was reused by an unrelated
|
||||
// process (the real holder died and stopped refreshing). Reap.
|
||||
try { rmSync(lockDir, { recursive: true, force: true }); } catch { /* race condition */ }
|
||||
} else {
|
||||
// Live holder refreshing within grace — wait and retry.
|
||||
// Live holder — wait and retry. If it is genuinely wedged (or its PID
|
||||
// was reused by an unrelated process), the acquire times out below
|
||||
// with a message naming the PID; we never force-steal a live holder.
|
||||
await new Promise(r => setTimeout(r, 1000));
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -51,6 +51,22 @@ describe('classifyPgliteInitError', () => {
|
||||
test('case-insensitive matching on bunfs marker', () => {
|
||||
expect(classifyPgliteInitError('SYSCALL ENOENT on /$$BUNFS/root')).toBe('bunfs');
|
||||
});
|
||||
|
||||
// #2348 — corrupted PGLite data dir (concurrent open trashed catalog/extension).
|
||||
test('corrupt verdict for the 58P01 internal_load_library signature', () => {
|
||||
const msg = 'error: relation "content_chunks" does not exist\n code: 58P01\n file: "dfmgr.c"\n routine: "internal_load_library"';
|
||||
expect(classifyPgliteInitError(msg)).toBe('corrupt');
|
||||
});
|
||||
|
||||
test('corrupt verdict when the vector type can no longer load', () => {
|
||||
expect(classifyPgliteInitError('type "vector" does not exist')).toBe('corrupt');
|
||||
});
|
||||
|
||||
test('corrupt verdict beats the wasm-runtime match (58P01 wins over "wasm runtime")', () => {
|
||||
// A message mentioning both must classify as corrupt, not macos-26-3 —
|
||||
// recovery guidance, not the wrong macOS-WASM hint.
|
||||
expect(classifyPgliteInitError('wasm runtime: 58P01 internal_load_library')).toBe('corrupt');
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildPgliteInitErrorMessage — hint routing', () => {
|
||||
@@ -80,8 +96,16 @@ describe('buildPgliteInitErrorMessage — hint routing', () => {
|
||||
expect(msg).toContain(original);
|
||||
});
|
||||
|
||||
test('corrupt verdict surfaces the reinit-pglite recovery, NOT the macOS hint', () => {
|
||||
const msg = buildPgliteInitErrorMessage('corrupt', original);
|
||||
expect(msg).toContain('gbrain reinit-pglite');
|
||||
expect(msg).toContain('corrupted');
|
||||
expect(msg).toContain(original);
|
||||
expect(msg).not.toContain('issues/223');
|
||||
});
|
||||
|
||||
test('all verdicts produce the canonical header line', () => {
|
||||
for (const v of ['bunfs', 'macos-26-3', 'unknown'] as const) {
|
||||
for (const v of ['bunfs', 'macos-26-3', 'corrupt', 'unknown'] as const) {
|
||||
const msg = buildPgliteInitErrorMessage(v, original);
|
||||
expect(msg.startsWith('PGLite failed to initialize its WASM runtime.')).toBe(true);
|
||||
}
|
||||
|
||||
+10
-18
@@ -3,7 +3,6 @@ import { mkdirSync, rmSync, existsSync, readFileSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import { acquireLock, releaseLock, type LockHandle } from '../src/core/pglite-lock';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
const TEST_DIR = join(tmpdir(), 'gbrain-lock-test-' + process.pid);
|
||||
|
||||
@@ -133,25 +132,18 @@ describe('pglite-lock #2058 heartbeat + steal-grace', () => {
|
||||
expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(true);
|
||||
});
|
||||
|
||||
test('a LIVE PID whose heartbeat went stale past the grace window IS reaped', async () => {
|
||||
// PID is alive (our own) but hasn't refreshed in 20min (> 600s grace):
|
||||
// hung holder, or a reused PID whose real holder is gone. Reap + acquire.
|
||||
test('[REGRESSION #2348] a LIVE PID with a STALE heartbeat is NOT stolen', async () => {
|
||||
// The #2348 corruption: a live `gbrain dream`/embed holder whose heartbeat
|
||||
// lapsed (the JS event loop is blocked during a long synchronous WASM
|
||||
// import) used to get its lock reaped past the grace window — letting a
|
||||
// second OS process open the same data dir and corrupt the catalog +
|
||||
// pgvector extension state. A live PID is now NEVER stolen, regardless of
|
||||
// how stale its heartbeat is. Acquire must time out, not steal.
|
||||
writeHolder({ pid: process.pid, acquiredAgoMs: 25 * 60_000, refreshedAgoMs: 20 * 60_000 });
|
||||
|
||||
const lock = await acquireLock(TEST_DIR, { timeoutMs: 2000 });
|
||||
expect(lock.acquired).toBe(true);
|
||||
await releaseLock(lock);
|
||||
});
|
||||
|
||||
test('GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS tunes the grace window', async () => {
|
||||
// withEnv keeps the process-global mutation isolated across shard files.
|
||||
await withEnv({ GBRAIN_PGLITE_LOCK_STEAL_GRACE_SECONDS: '5' }, async () => {
|
||||
// Refreshed 30s ago — fresh under the 600s default, STALE under 5s.
|
||||
writeHolder({ pid: process.pid, acquiredAgoMs: 60_000, refreshedAgoMs: 30_000 });
|
||||
const lock = await acquireLock(TEST_DIR, { timeoutMs: 2000 });
|
||||
expect(lock.acquired).toBe(true);
|
||||
await releaseLock(lock);
|
||||
});
|
||||
await expect(acquireLock(TEST_DIR, { timeoutMs: 1200 })).rejects.toThrow(/Timed out/);
|
||||
// The live holder's lock is still present — never force-removed.
|
||||
expect(existsSync(join(TEST_DIR, '.gbrain-lock'))).toBe(true);
|
||||
});
|
||||
|
||||
test('[REGRESSION] releaseLock does NOT remove a lock that was stolen + re-acquired by another process', async () => {
|
||||
|
||||
Reference in New Issue
Block a user