* fix(jobs): reap stale dead-holder cycle/sync locks (#1972)
A crashed sync (OOM, recycle, SIGKILL) stranded its gbrain_cycle_locks row
until something contended for it — reclaim was on-contention only. Add a
host-scoped background reaper: reapDeadHolderLocks deletes locks whose holder
PID is provably dead on this host, scoped to the gbrain-sync:*/gbrain-cycle*
namespaces only (never elections/supervisor/reindex), with a snapshot-matched
delete (date_trunc on acquired_at) that is TOCTOU-safe against PID reuse.
Reuses isHolderDeadLocally (same-host + ESRCH + 60s grace). doctor --fix now
auto-reaps for no-autopilot brains. DRY: selectLockRows + shared mapper now
back inspectLock + listStaleLocks (killed the triplication).
* fix(db): bound pool disconnect so teardown can't eat CLI output (#1972)
pool.end() against PgBouncer transaction-mode never drained, so disconnect
blocked until the CLI's 10s force-exit fired and process.exit()'d mid-write,
truncating stdout (e.g. #1959's relational query returned empty). Add a
gbrain-owned endPoolBounded(pool): Promise.race of pool.end({timeout}) against
a hard timer, so teardown is bounded regardless of what postgres.js does and is
testable. connection-manager ends its direct + read pools concurrently so the
per-pool bounds don't stack. PGLite disconnect is unaffected.
* fix(cycle): complete cooperative-abort coverage + wire lock reaper (#1972)
v0.42.29 made only the embed phase honor the abort signal; a 24h pull still
showed force-evicts from a long non-embed phase ignoring it. Thread the signal
into every cycle-reachable long loop: extract (extractForSlugs + the full-walk
extractLinksFromDir/extractTimelineFromDir), extract_facts (per-page loop +
embed signal + the phantom-redirect 30s lock-retry), and consolidate's bucket
loop. Add a terminal abort check so an aborted cycle never stamps
last_full_cycle_at as a completed run (Codex #9). lint now yields + checks
abort every 200 pages (it's synchronous; the yield is what lets the signal
land). New phase-duration force-evict attribution log names any phase that
crosses the 30s deadline. Wire reapDeadHolderLocks at cycle start.
* chore: bump version and changelog (v0.42.38.0)
#1972 — stale-lock reaper, bounded pool disconnect, and complete
cooperative-abort coverage across cycle phases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(key-files): current-state for the #1972 reaper, bounded disconnect, abort coverage
document-release: update db-lock.ts (reapDeadHolderLocks + selectLockRows DRY),
db.ts (endPoolBounded), and abort-check.ts (coverage now spans extract/
extract_facts/consolidate/lint + terminal guard) entries to current truth.
* test(isolation): fix shard-order flakes exposed by #1972's new test files
Adding 3 new test files reshuffled the hash-based shards, exposing two
pre-existing test-isolation bugs:
- cycle-consolidate.test.ts assumed the global legacy-embedding preload's
1536-d gateway config still held at initSchema, but a co-sharded test that
calls resetGateway() in teardown nulls it, so initSchema fell back to the
1280-d default and built a halfvec(1280) facts column its 1536-d fixtures
can't fill. Re-pin the legacy OpenAI/1536 config in beforeAll (the pattern
legacy-embedding-preload.ts documents for 1536-d fixture tests).
- db-lock-heartbeat-takeover.test.ts (merged from master's #1794) mutated
process.env.GBRAIN_LOCK_STEAL_GRACE_SECONDS raw, tripping check:test-isolation
rule R1. Convert to withEnv().
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(postgres): module-singleton ownership — borrower disconnect no longer nulls the cycle's connection (#1404/#1471/#1619)
gbrain dream on Postgres failed every DB phase with "No database connection:
connect() has not been called": a short-lived borrower probe engine (lint/doctor
config-lift, no poolSize) called db.disconnect() in its own disconnect(), nulling
the shared module singleton the long-lived cycle owner was still using. The module
`sql` is only ever nulled by db.disconnect() (postgres.js auto-reconnects its own
pool), so the failure was always a borrower-disconnect, never an idle-pooler drop.
Fix: db.connect() returns whether THIS call created the singleton (atomic — no
await between the null-check and the sql=postgres() assignment), PostgresEngine
stores it as _ownsModuleSingleton, and disconnect() only calls db.disconnect()
when it owns the connection. Borrowers no-op. Hardening: db.disconnect() snapshots
+nulls sql before awaiting end(); reconnect() shares one in-flight _reconnectPromise.
Tests: new postgres-engine-singleton-ownership.test.ts; expanded DB-gated e2e
matrix (owner/borrower, creation-not-role, symmetric CLI-exit, owner-reconnect-
with-live-borrower); module-style getter asymmetry; #1570 shared-recovery
regression updated to assert the fixed contract.
Co-Authored-By: nullhex-io <noreply@github.com>
Co-Authored-By: joelwp <noreply@github.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.42.15.0)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: nullhex-io <noreply@github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
