v0.42.29.0 fix(minions): long-job abort-honoring + attempt accounting + supervisor singleton; topic-aware voice (#1737, #1849, #1851) (#1943)

* fix(minions): honest attempt accounting + cooperative abort-honoring + per-handler timeouts (#1737)

- Wall-clock and stall dead-letter paths now increment attempts_made (terminal,
  no retry — wall-clock fires at 2x cumulative timeout; retrying non-idempotent
  embed/subagent work would duplicate side effects). Surface stalled_counter in
  jobs get so 'started 3 / stalled 2 / attempts 0' reads true instead of looking
  like broken accounting.
- Thread AbortSignal through embed-backfill/autopilot-cycle -> runPhaseEmbed ->
  runEmbedCore -> embedAll(Stale)/embedPage, checking it on BOTH --stale and
  --all paths and between embed batches. A timed-out embed phase now bails within
  a batch, so the cycle finally releases gbrain_cycle_locks instead of running
  the full 10-15 min after the job was killed (the daily cycle-wedge). New shared
  src/core/abort-check.ts (isAborted/throwIfAborted/anySignal).
- Per-handler default wall-clock budget (handler-timeouts.ts) stamped at submit
  for long handlers without an explicit timeout_ms.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(minions): queue-scoped DB supervisor singleton + canonical pidfile + doctor max-rss check (#1849)

- Acquire a queue-scoped DB lock (tryAcquireDbLock, keyed on the raw DB identity
  + queue) on supervisor.start(): a second supervisor on the same (db, queue)
  fails fast with exit 2 regardless of $HOME/--pid-file. Refresh on a dedicated
  timer; on refresh failure past the threshold, fail SAFE (exit non-zero) before
  the TTL could lapse and let a second supervisor take over. Release on shutdown.
- Canonical default pidfile keyed on brain id (currentBrainId, config-only, no DB
  connect) so two brains under one HOME no longer share supervisor.pid.
- doctor: new supervisor_singleton check surfaces the effective --max-rss (from
  the started audit event) and warns when the lock holder's (host,pid) differs
  from the local pidfile — comparing host+pid, not bare pid. Registered in
  doctor-categories.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(agent-voice): topic-aware persona context via server-resolved topicId (#1851)

Summon Mars/Venus into a specific conversation topic so they boot already knowing
the recent thread. A per-topic call link carries only topicId (+ optional display
topicName); the server resolves the recent-conversation context from the brain
(topics/<topicId>.md) — topic CONTENT is never accepted over the wire (that would
be prompt injection + a leak into URLs/referrers/logs). topicId is a strict slug
with a path-traversal guard. New '# Topic Context' prompt slot injected after the
persona body so identity-first ordering still wins; persona identity unchanged.
No topicId -> generic behavior. Contract doc + persona skill docs updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: file #1737 slot-reservation fair-scheduling follow-up TODO (F7)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.42.29.0)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: sync KEY_FILES.md for v0.42.29.0 minions + abort wave (#1737, #1849)

Fold the #1737/#1849 behavior into the existing per-file entries and add the
two new core files, keeping the doc at current-state truth:
- queue.ts: honest attempt accounting on wall-clock + stall dead-letter paths;
  defaultTimeoutMsFor stamping at submit.
- supervisor.ts: queue-scoped DB singleton lock (supervisorLockId,
  classifySupervisorSingleton, LOCK_LOST, refresh-fail-safe, brain-id pidfile,
  max_rss_mb audit).
- worker-registry.ts: currentDbIdentity().
- New entries: src/core/abort-check.ts, src/core/minions/handler-timeouts.ts.
- New doctor.ts extension: supervisor_singleton check.
- cycle.ts / embed.ts extensions: AbortSignal threading note.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-07 08:12:50 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent f7f8512b14
commit 613da94093
30 changed files with 1124 additions and 37 deletions
+22
View File
@@ -2,6 +2,28 @@
All notable changes to GBrain will be documented in this file.
## [0.42.29.0] - 2026-06-07
**The background-job queue stops thrashing on long jobs, the cycle stops wedging itself, and you can no longer run two supervisors against one queue by accident.** Three fixes plus a voice-agent feature.
The biggest one: a `gbrain dream` / autopilot cycle whose embed phase ran long (a big stale-page backlog) would hit the job's wall-clock timeout, get dead-lettered, **and keep running anyway** — its embed loop never checked for cancellation, so the cycle's cleanup never ran and the `gbrain_cycle_locks` row stayed held. Every later cycle then skipped with "cycle already running" until the zombie finished 10-15 minutes later. The embed phase now honors the cancellation signal on both the `--stale` and `--all` paths and bails within a batch, so the lock is released right away.
Long-running jobs also accounted for their attempts honestly now. A job killed by the wall-clock timeout (or repeatedly reclaimed after lease loss) used to show `Attempts: 0/2 (started: 3)` — started three times, zero attempts recorded — which read like broken math. `gbrain jobs get` now increments the attempt on those terminal paths and surfaces the stall counter, so you see `started 3 / stalled 2 / attempts 1 / killed: wall-clock` and it actually adds up. Long handlers (`subagent`, `embed-backfill`, `autopilot-cycle`) also get a sane default wall-clock budget when one isn't set explicitly, so they aren't killed mid-progress by the short default.
The supervisor singleton was only enforced per pidfile path, so two supervisors launched with different `HOME` or `--pid-file` could both run against the same queue with conflicting `--max-rss` caps — and the lower cap silently killed healthy work. The real authority is now a queue-scoped DB lock keyed on the database identity: a second supervisor on the same `(database, queue)` exits immediately, regardless of pidfile path. If the lock can't be refreshed, the supervisor exits cleanly rather than risk a split. `gbrain doctor` now reports the effective `--max-rss` and flags a mismatch between the pidfile owner and the lock holder.
For the voice agent recipe, you can now summon a persona into a specific topic so it boots already knowing the recent conversation, instead of starting cold.
### Added
- **Topic-aware voice personas** (`recipes/agent-voice/`): a call link can carry a `topicId` (e.g. `/call?persona=mars&topicId=q3-planning`) and the persona boots with that topic's recent conversation already in context. Only the `topicId` crosses the wire — the server resolves the conversation from the brain, so topic content never lands in a URL — and the id is a strict slug with a path-traversal guard. No `topicId` falls back to the generic per-persona context.
### Fixed
- **Long minion jobs no longer thrash or wedge the cycle (#1737).** The embed phase honors cancellation on both embed paths so a timed-out cycle releases its lock immediately; wall-clock and stall dead-letters record the attempt and surface the stall counter; long handlers get a default wall-clock budget.
- **One supervisor per queue, enforced at the database (#1849).** A queue-scoped DB lock replaces the pidfile-only guard, so two supervisors can't fight over one queue with conflicting memory caps; `gbrain doctor` surfaces the effective cap and any owner mismatch.
### To take advantage of v0.42.29.0
Upgrade and restart your worker/supervisor. Nothing to configure. If `gbrain doctor` flags a supervisor singleton mismatch, stop the extra supervisor (`gbrain jobs supervisor stop`) and keep one per queue.
## [0.42.28.0] - 2026-06-06
**`gbrain extract links --stale` no longer dies partway through on calendar and meeting pages.** A full re-extraction sweep (the kind a `LINK_EXTRACTOR_VERSION_TS` bump triggers) used to crash with a Postgres "malformed array literal" error the moment a calendar event's raw text (Zoom links, commas, quotes, braces, em-dashes) landed in a batch. One bad batch aborted the entire run, so the graph never finished reconciling and stale edges couldn't be dropped. The three bulk writers (links, timeline, takes) now pass each batch as a single JSONB document instead of a hand-built `text[]` literal, which encodes arbitrary free text safely. The sweep runs to completion on the messiest brains.
+22
View File
@@ -1,5 +1,27 @@
# TODOS
## #1737 minion fair-scheduling follow-up (v0.43+)
Filed during the #1737 wave (`/plan-eng-review` decision F7, codex outside-voice
line 5 + Claude review agreeing). The wave shipped honest attempt accounting,
cooperative abort-honoring (the daily cycle-wedge fix), and per-handler default
timeouts. Slot reservation was deliberately deferred.
- [ ] **P3 — Reserve a concurrency slot for short lanes so long jobs can't starve
fresh ones.** Today the worker claim loop (`src/core/minions/worker.ts` claim
loop) pulls from a single pool ordered by `priority, created_at` — N long
`subagent`/`embed-backfill`/`autopilot-cycle` jobs can occupy all slots while a
freshly-submitted short job waits (#1737's "fresh subagent never claimed"
half). **Why deferred:** now that abort is honored (this wave), a timed-out job
actually stops and frees its slot, so most of the observed starvation should
evaporate. **MEASURE FIRST:** before building reservation, confirm starvation
still reproduces with abort-honoring live (submit a short job alongside 3 long
ones at `--concurrency 3`; check it gets claimed). Reserving a slot is overfit
(breaks at `--concurrency 1`; can starve long work under continuous short
traffic), so only build it if the measurement shows a real residual problem.
**Shape if needed:** when all-but-one in-flight slot is held by long-lane
handler names, restrict the next `claim()` to non-long names via the existing
`name = ANY($4)` filter in `queue.ts:claim`. No new table/migration.
## gbrain#1861 JSONB batch-insert follow-ups (v0.42+)
Filed from the #1861 fix (batch inserts migrated from `unnest(${arr}::text[])` to
+1 -1
View File
@@ -1 +1 @@
0.42.28.0
0.42.29.0
+8 -5
View File
@@ -158,6 +158,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `scripts/check-worker-lock-renewal-shape.sh` — CI guard wired into `bun run verify`. Two invariants on `src/core/minions/worker.ts`: (1) the bug pattern `lockTimer = setInterval(async ...)` must NOT appear (narrowed via `lockTimer =` prefix so unrelated `setInterval(async)` calls — like the stall detector — don't false-fire), (2) `runLockRenewalTick` must remain referenced so the pure-function test seam survives refactors. Bug-pattern-specific by design — a future refactor to `setTimeout`-recursion or `AbortController`-based scheduling passes as long as the bug pattern stays absent. POSIX ERE + `[[:space:]]` for BSD-grep portability. Honors `GBRAIN_LOCK_RENEWAL_SHAPE_TARGET` env override for fixture-based meta-tests. Pinned by `test/scripts/check-worker-lock-renewal-shape.test.ts` (5 cases).
- `src/core/doctor-cause-rank.ts` — pure cause-ranking for `gbrain doctor`. `rankIssues(checks)` returns non-ok checks ordered fail-before-warn then root-before-symptom then name (deterministic). `ROOT_CAUSE_CHECKS` / `SYMPTOM_CHECKS` are ORDERING ONLY — tier membership asserts no causality. `downstream_of` is set ONLY from a small map of KNOWN grounded edges (`queue_health` / `supervisor``worker_oom_loop`, since they read the same `aborted: watchdog` / `rss_watchdog` source) AND only when the named root is itself failing — never a root×symptom cartesian (co-occurrence never implies causality). `fix` prefers `details.fix_hint` else the message. `CAUSE_GRAPH_NAMES` + `allKnownCheckNames()` back a drift guard asserting every graphed name is a real check. Consumed by `computeDoctorReport` (`top_issues` field, additive, schema_version stays 2) + the "Top issues (ranked by cause)" header in `outputResults`. Pinned by `test/doctor-cause-rank.test.ts`.
- `src/commands/doctor.ts` extension — `computeWorkerOomLoopCheck(engine)` is the single authoritative OOM-loop signal, unioning supervised `summarizeCrashes(readRecentSupervisorEvents(24)).by_cause.rss_watchdog` (cross-week read via `readRecentSupervisorEvents` so a Monday window can't lose Sunday) + bare-worker `minion_jobs error_text='aborted: watchdog'` count (Postgres-only; the same source `queue_health` subcheck 3 reads). Cap comes from the latest `rss_watchdog_loop` breaker alert's `max_rss_mb`, else `resolveDefaultMaxRssMb()` fallback. fail at breaker-tripped or oomKills≥5, warn at ≥1, null otherwise. `computePoolReapHealthCheck(engine)` is the Postgres-only `pool_reap_health` check reading `readRecentPoolRecoveries(1)` — fail when reconnect failures>0 (reconnect throwing is the actionable signal), warn at ≥10 reaps/hr (pooler thrash), null otherwise. Both registered in `buildChecks` after the `supervisor` block. The `supervisor` causeStr carries `rss=N (see worker_oom_loop)` and `queue_health`'s watchdog message cross-references `worker_oom_loop`. `DoctorReport.top_issues` + the cause-ranked render header. `worker_oom_loop` + `pool_reap_health` registered under ops in `doctor-categories.ts`. Pinned by `test/doctor-worker-oom-loop.test.ts`, `test/doctor-pool-reap-health.test.ts`.
- `src/commands/doctor.ts` extension — `supervisor_singleton` check (#1849), a SEPARATE check from `supervisor` (same split precedent as the niceness check) so a singleton-divergence warn can't clobber the crash/liveness precedence. Runs only when a `started` supervisor event was seen in the last 24h and a live engine is available. Reads the queue-scoped DB lock row (`gbrain_cycle_locks WHERE id = supervisorLockId(queue)`) and compares the lock holder (`holder_host:holder_pid`) against the local pidfile holder via the pure `classifySupervisorSingleton`. `mismatch` → warn (a second supervisor may be running with a different `--max-rss`; message names both holders, the effective cap from the `started` event's `max_rss_mb`, and the fix `gbrain jobs supervisor stop`); `single` → ok (names holder + cap); `no_lock` → no check emitted. Best-effort try/catch (silent skip on brains without the lock table). Registered under ops in `doctor-categories.ts` as `supervisor_singleton`. Pinned by `test/supervisor-db-lock.test.ts` + `test/doctor.test.ts`.
- `src/core/audit/pool-recovery-audit.ts` — reap/reconnect audit on the shared `audit-writer` cathedral. Events: `reap_detected` (CONNECTION_ENDED), `reconnect_other` (network/auth/health-check), `reconnect_succeeded`, `reconnect_failed`. `readRecentPoolRecoveries(hours=1)` returns `{reaps, recoveries, failures, others, events}`. Error summaries route through `redactConnectionInfo` before truncation (DSN/host/IP safe). Emitted ONLY from `PostgresEngine.reconnect(ctx?)` (the rare reap-retry path, near-zero hot-path cost); `reconnect()` classifies the threaded error via `isConnectionEndedError` (in retry-matcher.ts) so only true pooler reaps are labeled `reap_detected`. The retry callback in retry.ts threads the triggering error as `(ctx?: {error?}) => Promise<void>`. Pinned by `test/audit/pool-recovery-audit.test.ts`.
- `src/commands/autopilot.ts` extension — per-source `extract_atoms` auto-drain. Postgres-only block after the freshness fan-out: gated on `autopilot.auto_drain.enabled` (default true) AND `!packDeclaresPhase(engine,'extract_atoms')` (the silent-backlog condition) AND per-source `countExtractAtomsBacklog > threshold` (default 25) AND a daily cap `floor(max_usd_per_day / ~$0.30)`. Enumerates `loadAllSources`. Submits the PROTECTED `extract-atoms-drain` job (`{allowProtectedSubmit:true}`) with a UTC-day time-sloted idempotency key `autopilot-extract-atoms-drain:<src.id>:<utcDay>` (a static key would block the source after the first job completed). `src/core/minions/protected-names.ts` adds `extract-atoms-drain`; `src/commands/jobs.ts` registers the handler (thin wrapper over `runExtractAtomsDrainForSource`, `LockUnavailableError``{deferred:true}`); `src/core/config.ts` adds the `autopilot.auto_drain.*` config keys + the `autopilot.` key prefix. Pinned by `test/extract-atoms-drain-handler.test.ts`, `test/autopilot-auto-drain-wiring.test.ts`.
- `src/commands/doctor.ts:checkBatchRetryHealth``batch_retry_health` check surfacing Supavisor circuit-breaker incidents. Wired into both `runDoctor` (local) and `doctorReportRemote` (thin-client). Reads last 24h. States: `ok` (zero exhausted in 24h OR <3 from a single site), `warn` (>=3 same-site OR >=5 cross-site), `fail` (>=20 sustained breaker). Surfaces bad `GBRAIN_BULK_*` env at doctor startup. Corrupt-JSONL tolerant. Paste-ready fix hints in every warn/fail message. Also reads `readRecentDbDisconnects(24)` and appends `Disconnect-call audit: N call(s) in 24h (most recent caller: <frame>).` to ALL three message paths so connection-incident signal is greppable from one `gbrain doctor --json` call (module-import wrapped in try/catch so older brains without the audit file degrade silently). Pinned by `test/doctor-batch-retry.test.ts` (10 cases).
@@ -211,14 +212,15 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/commands/lint.ts` + `src/commands/sources.ts` extensions — `gbrain lint` gains a `markup-heavy` rule (flags pages whose prose-vs-markup ratio exceeds `content_sanity.max_markup_ratio`, reusing `assessContentSanity` so lint/gate/scan share one assessor); pinned by `test/lint-content-sanity.test.ts`. `gbrain sources audit <id>` becomes disposition-aware: its dry-run disk scan reports would-quarantine / would-reject / would-flag counts driven by the effective `content_sanity.junk_disposition` + markup config, so an operator previews the gate's verdict before sync. The `content-sanity-audit` JSONL (`src/core/audit/content-sanity-audit.ts`) records the new quarantine/flag dispositions.
- `src/core/zombie-reap.ts` — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()`. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). The `maxWaiting` coalesce path uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other. `claim` and `renewLock` issue their UPDATE via `engine.executeRawDirect` (not `executeRaw`) so the lock heartbeat runs on the direct session-mode pool that the transaction pooler won't recycle mid-hold; on PGLite this is identical to `executeRaw`. Guarded by `test/queue-lock-retry.test.ts` (claim never falls back to `executeRaw`) and `test/postgres-execute-raw-direct.test.ts` (routing decision matrix).
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). The `maxWaiting` coalesce path uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other. `claim` and `renewLock` issue their UPDATE via `engine.executeRawDirect` (not `executeRaw`) so the lock heartbeat runs on the direct session-mode pool that the transaction pooler won't recycle mid-hold; on PGLite this is identical to `executeRaw`. The two terminal dead-letter paths (`handleWallClockTimeouts` wall-clock kill and the stall dead-letter CTE in the stall sweep) BOTH increment `attempts_made` so a long job killed there reads as an honest attempt instead of `attempts 0 / started N`; the stall path also bumps `stalled_counter`, surfaced by `gbrain jobs get` as `Attempts: M/N (started: X, stalled: S/MaxS)`. At submit, `add()` stamps a default `timeout_ms` via `defaultTimeoutMsFor(jobName)` (from `handler-timeouts.ts`) when the caller passed none, so long handlers aren't wall-clock-killed mid-progress by the short null-default; an explicit `opts.timeout_ms` always wins. Guarded by `test/queue-lock-retry.test.ts` (claim never falls back to `executeRaw`), `test/postgres-execute-raw-direct.test.ts` (routing decision matrix), and `test/minions.test.ts` (attempt accounting + default-timeout stamping).
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). Aborted jobs call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`); `shutdownAbort` (instance field) fires on SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` (shell handler listens; non-shell handlers don't). Per-job timeout fires `abort.abort(new Error('timeout'))` then a 30s grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead if the handler ignores the abort signal (the moved-out generic abort listener now fires for ANY abort reason). The `launchJob` lock-renewal block is a thin sync wrapper around the pure `runLockRenewalTick` from `src/core/minions/lock-renewal-tick.ts` (NEVER `setInterval(async () => await renewLock(...))` — that shape was the unhandledRejection crash class during PgBouncer rotation). Gaps closed: (1) `cancelled` flag captured in the timer closure stops in-flight IIFEs writing misleading audit events after the job ended; (2) re-entrancy guard `tickInFlight` + per-call `Promise.race` timeout (`GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS`, default `lockDuration/3`); (3) time-based abort (`Date.now() - lastSuccessfulRenewalAt >= lockDuration - GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS`) so we release the lock BEFORE another worker can reclaim; (4) explicit `.catch()` on the stored `executeJob(...).finally(...)` promise closes the second unhandledRejection vector; (5) exported `INFRASTRUCTURE_ABORT_REASONS = new Set(['lock-renewal-failed', 'lock-lost'])` so executeJob's catch skips `failJob` for these (PgBouncer blips don't dead-letter healthy jobs; the stall detector reclaims). CI guard `scripts/check-worker-lock-renewal-shape.sh` (in `bun run verify`) asserts the bug pattern stays absent AND `launchJob` keeps calling `runLockRenewalTick`. Engine-ownership invariant: `start()` does NOT call `engine.disconnect()` on shutdown — the CLI handler in `src/commands/jobs.ts case 'work'` owns engine lifecycle via try/finally with loud error logging. RSS watchdog uses non-file-backed pages on Linux: exported `parseRssFromProcStatus(status)` (pure parser; field-presence regex so `RssAnon: 0 + RssShmem: 512` parses correctly) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5); the default `getRss` in `WorkerOpts` is `getAccurateRss`. `checkMemoryLimit` tracks peak RSS, fires an 80%-of-cap soft-warn (once per crossing, carrying peak + in-flight job kinds), and on exceed sets `_rssWatchdogTriggered=true` (exposed via `get rssWatchdogTriggered()`) so `jobs work`'s finally can `process.exit(WORKER_EXIT_RSS_WATCHDOG)` after disconnect (drain self-identifying instead of an opaque code-0 exit). The poll loop wraps `claim` in try/catch: on a retryable conn error it reconnects ONCE and continues to the next tick rather than blind-retrying (a retry after `UPDATE...RETURNING` committed but the socket died would double-claim). `LockRenewalDeps` is wired with `reconnect` when the engine supports it. The self-health DB-liveness probe now runs EVEN under a supervisor (`GBRAIN_SUPERVISED=1`): the outer guard is `if (this.opts.healthCheckInterval > 0)` and only the STALL-detection block is wrapped in `if (!isSupervisedChild)` — so a supervised worker whose own pool dies self-exits `unhealthy(db_dead)` after `dbFailExitAfter` probes (the supervisor watches a different connection and can't see this worker's dead pool), while the supervisor's progress watchdog owns forward-progress (#1801). Pinned by `test/worker-lock-renewal.test.ts` (18), `test/audit/lock-renewal-audit.test.ts` (11), `test/scripts/check-worker-lock-renewal-shape.test.ts` (5), `test/worker-shutdown-disconnect.test.ts` (asserts `disconnectSpy).not.toHaveBeenCalled()`), `test/worker-rss.test.ts` (11), `test/worker-supervised-db-probe.test.ts` (3).
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets. Worker exit classifier emits `likely_cause` on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. Consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping the in-process SIGCHLD reaper can't reach); exposes `isTiniDetected` read-only accessor. The spawn-and-respawn loop is the shared `ChildWorkerSupervisor` core: MinionSupervisor composes it via `runSuperviseLoop()``new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` back through `emit()` SupervisorEvent (JSONL audit consumers see byte-compatible output). PID lock, signal handlers, health check, and `process.exit` on max-crashes stay in MinionSupervisor. `code=0` leaves `crashCount` untouched (so a worker alternating real crashes + watchdog drains still trips `max_crashes`); `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop via `health_warn { reason: 'clean_restart_budget_exceeded' }` + backoff. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)`. Progress watchdog (#1801): `healthCheck()` restarts an alive-but-wedged child via `childSupervisor.restartCurrentChild(35_000)` when a queue has claimable work, 0 live-lock active jobs, and stale completions across `wedgeRestartChecks` (default 3) consecutive checks past `wedgeRestartMinutes` (default 15, 0 disables) + a `startupGraceMs` window; bounded by `wedgeRestartLoopBudget` (default 3 / `wedgeRestartLoopWindowMs`) which switches to a one-shot `wedge_restart_loop` alert. The wedge query is the exported `queryWedgeSignals(engine, queue, handlerNames)` — name+queue-scoped, `active_healthy` = live-lock only (an expired-lock active row does NOT mask the wedge), due-delayed counted. Claimable names are derived at start via a throwaway `registerBuiltinHandlers` worker (its new `quiet` opt). Flags `--wedge-restart-minutes` / `--wedge-restart-checks` + env `GBRAIN_WEDGE_RESTART_MINUTES` / `GBRAIN_WEDGE_RESTART_CHECKS`. The worker argv is built by the exported pure `buildWorkerArgs(opts)` (appends `--nice N` when `opts.nice_requested` is set, alongside `--concurrency`/`--queue`/`--max-rss`); the niceness apply RESULT (`nice_requested`/`nice_effective`/`nice_error`, computed by the CLI in jobs.ts — the supervisor doesn't call setPriority) rides on the `started`/`worker_spawned` audit emissions (#1815). Pinned by `test/supervisor.test.ts` (16 cases), `test/supervisor-tini.test.ts`, `test/supervisor-wedge.test.ts`, and `test/supervisor-build-worker-args.test.ts`.
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets. Worker exit classifier emits `likely_cause` on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. Consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping the in-process SIGCHLD reaper can't reach); exposes `isTiniDetected` read-only accessor. The spawn-and-respawn loop is the shared `ChildWorkerSupervisor` core: MinionSupervisor composes it via `runSuperviseLoop()``new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` back through `emit()` SupervisorEvent (JSONL audit consumers see byte-compatible output). PID lock, signal handlers, health check, and `process.exit` on max-crashes stay in MinionSupervisor. `code=0` leaves `crashCount` untouched (so a worker alternating real crashes + watchdog drains still trips `max_crashes`); `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop via `health_warn { reason: 'clean_restart_budget_exceeded' }` + backoff. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)`. Progress watchdog (#1801): `healthCheck()` restarts an alive-but-wedged child via `childSupervisor.restartCurrentChild(35_000)` when a queue has claimable work, 0 live-lock active jobs, and stale completions across `wedgeRestartChecks` (default 3) consecutive checks past `wedgeRestartMinutes` (default 15, 0 disables) + a `startupGraceMs` window; bounded by `wedgeRestartLoopBudget` (default 3 / `wedgeRestartLoopWindowMs`) which switches to a one-shot `wedge_restart_loop` alert. The wedge query is the exported `queryWedgeSignals(engine, queue, handlerNames)` — name+queue-scoped, `active_healthy` = live-lock only (an expired-lock active row does NOT mask the wedge), due-delayed counted. Claimable names are derived at start via a throwaway `registerBuiltinHandlers` worker (its new `quiet` opt). Flags `--wedge-restart-minutes` / `--wedge-restart-checks` + env `GBRAIN_WEDGE_RESTART_MINUTES` / `GBRAIN_WEDGE_RESTART_CHECKS`. The worker argv is built by the exported pure `buildWorkerArgs(opts)` (appends `--nice N` when `opts.nice_requested` is set, alongside `--concurrency`/`--queue`/`--max-rss`); the niceness apply RESULT (`nice_requested`/`nice_effective`/`nice_error`, computed by the CLI in jobs.ts — the supervisor doesn't call setPriority) rides on the `started`/`worker_spawned` audit emissions (#1815). Queue-scoped singleton (#1849): the real authority is a DB lock (`tryAcquireDbLock` from `src/core/db-lock.ts`) keyed on `supervisorLockId(queue, dbIdentity)` = `gbrain-supervisor:<rawDbIdentity>:<queue>` — the raw DB identity from `currentDbIdentity()`, NOT the lossy `currentBrainId` hash — so two supervisors with different `$HOME`/`--pid-file` against the same `(database, queue)` no longer both run with conflicting `--max-rss`; the second exits `LOCK_HELD`. The default pidfile is now brain-scoped (`supervisor-<brainId>.pid`) so different brains under one HOME don't false-block. The lock refreshes on its own `setInterval` (TTL 5min, refresh 60s, max 3 failures = 180s < TTL); a refresh that fails past the threshold exits `LOCK_LOST` (code 4) rather than risk a split-brain. `shutdown()` releases the lock so a clean restart re-acquires immediately. The `started` audit now records `max_rss_mb` so `gbrain doctor`'s `supervisor_singleton` check can surface the effective cap. Exports `supervisorLockId()` and the pure `classifySupervisorSingleton({lockLive, lockHolderHost, lockHolderPid, localHost, localPid}) → 'no_lock'|'single'|'mismatch'` (host+pid compare, bare pid meaningless cross-host) that doctor consumes. Pinned by `test/supervisor.test.ts` (16 cases), `test/supervisor-tini.test.ts`, `test/supervisor-wedge.test.ts`, `test/supervisor-build-worker-args.test.ts`, and `test/supervisor-db-lock.test.ts`.
- `src/core/minions/child-worker-supervisor.ts` — shared spawn-and-respawn core reused by both `MinionSupervisor` (standalone `gbrain jobs supervisor` daemon) and `src/commands/autopilot.ts` (autopilot daemon), so the two consumers can't drift into parallel-loop bugs. Pure class: NO PID file, NO signal handlers, NO `process.exit`, NO health check. Lifecycle events fire via injected `onEvent: (ChildSupervisorEvent) => void`. Exit classifier: `code === 0` leaves `crashCount` UNCHANGED (preserves flap detection across mixed exit sequences); `code != 0` follows `runDuration > stableRunResetMs ? 1 : ++crashCount`. Clean-restart budget: sliding window of code=0 exits; when count exceeds `cleanRestartBudget` (default 10) inside `cleanRestartWindowMs` (default 60s), emits `health_warn { reason: 'clean_restart_budget_exceeded' }` and applies `cleanRestartBudgetBackoffMs` (default 1s). The exit classifier special-cases `WORKER_EXIT_RSS_WATCHDOG``likely_cause='rss_watchdog'`, and that exit does NOT bump `crashCount` (routes to its own breaker); a dedicated `_watchdogExitTimestamps` sliding window trips a loud `rss_watchdog_loop` health_warn naming the cap when N watchdog exits land inside the window, INDEPENDENT of the stable-run reset (which would otherwise hide a >5-min-run watchdog drain loop). New opts `watchdogLoopBudget` (3), `watchdogLoopWindowMs` (600000), `watchdogBackoffMs` (30000); `ChildSupervisorEvent` extended. Public read-only accessors `childAlive`, `inBackoff`, `crashCount`; `killChild(signal)` gates on liveness (`exitCode === null && signalCode === null`), NOT `.killed``.killed` flips true once a signal is *sent*, so the old `!this._child.killed` guard made a follow-up SIGKILL after an ignored SIGTERM a silent no-op (#1801; the bug also lived in the `shutdown()` drain). `restartCurrentChild(graceMs)` (#1801 wedge self-heal) captures the CURRENT child ref, SIGTERM→grace→SIGKILLs THAT ref (never the respawn — closes the timer-kills-fresh-worker race), and flags `_intentionalRestart` so the exit is `likelyCause='wedge_restart'`, leaves `crashCount` UNTOUCHED (never trips `max_crashes`; like `rss_watchdog`), and respawns immediately (`backoff ms:0 reason='wedge_restart'`). `awaitChildExit(timeoutMs)` short-circuits when `child.exitCode !== null || child.signalCode !== null` so fast-SIGTERM responders don't cause a 35s shutdown hang. Test hooks `_backoffFloorMs`, `_now`. `supervisor-audit.ts` adds `rss_watchdog` as a non-clean cause + its own `CrashSummary.by_cause` bucket, and `wedge_restart` to `CLEAN_EXIT_CAUSES` (a self-heal, not a crash; denylist preserved so future causes route to `legacy`). Pinned by `test/child-worker-supervisor.test.ts` (12 cases).
- `src/core/minions/spawn-helpers.ts` — pure `detectTini()` + `buildSpawnInvocation()` consumed by both `supervisor.ts` and `autopilot.ts` (resolves the DRY violation between the two spawn sites and makes tini wrapping testable without `mock.module()`, rule R2 of `scripts/check-test-isolation.sh`). `detectTini()` calls `execFileSync('which', ['tini'])` with explicit `env: process.env` so Bun sees runtime PATH mutations. `buildSpawnInvocation(tiniPath, cmd, args)` returns `{cmd, args}` with tini prepended when present, or the bare invocation otherwise. Pinned by `test/spawn-helpers.test.ts` (5) and `test/supervisor-tini.test.ts` (4).
- `src/core/minions/niceness.ts` — OS scheduling-priority (niceness) primitives for the `--nice` flag. `parseNiceValue(raw)` whole-string parses + range-validates to POSIX `[-20, 19]` (rejects `"3.5"`/`"10abc"` that `parseInt` would silently truncate). `applyNiceness(nice, setPriority?, getPriority?)` calls `os.setPriority(0, n)` and ALWAYS re-reads `os.getPriority(0)` afterwards — in both the success and the catch paths — so a denied renice (EPERM) or an `RLIMIT_NICE` clamp records the real effective value (e.g. `0`), not null; returns `{applied, requested, effective, error?}`. `getEffectiveNiceness(pid, getPriority?)` reads an arbitrary pid's niceness (null on dead/unreadable). `formatNice(n)``+10`/`0`/`-5`. Applied only at the CLI layer (jobs.ts) so worker.ts/supervisor.ts stay embeddable. Pinned by `test/niceness.test.ts`.
- `src/core/minions/worker-registry.ts` — live worker registry backing niceness observability. Each running `gbrain jobs work` self-registers `worker-<pid>.json` under `gbrainPath('workers')` (brain-isolated via `GBRAIN_HOME`; entries tagged with `currentBrainId()` so multiple DBs under one home don't cross-report). `registerWorker(info)` is best-effort (never blocks the worker) and returns a cleanup fn the caller wires to BOTH the shutdown `finally` AND `process.on('exit')` (the unhealthy `process.exit(1)` bypasses the awaited finally). `readWorkers(getNice?)` enumerates the dir, drops confirmed-dead pids (`classifyLiveness`: ESRCH = dead/prune, EPERM = alive/keep), applies a pid-reuse start-time guard (`ps -o lstart`, rejects a pid whose process started >5s after the entry was written), and re-measures each live worker's niceness now. Reports the worker's REAL pid, sidestepping the tini-wrapper-pid problem. Pinned by `test/worker-registry.test.ts`.
- `src/core/minions/worker-registry.ts` — live worker registry backing niceness observability. Each running `gbrain jobs work` self-registers `worker-<pid>.json` under `gbrainPath('workers')` (brain-isolated via `GBRAIN_HOME`; entries tagged with `currentBrainId()` so multiple DBs under one home don't cross-report). `registerWorker(info)` is best-effort (never blocks the worker) and returns a cleanup fn the caller wires to BOTH the shutdown `finally` AND `process.on('exit')` (the unhealthy `process.exit(1)` bypasses the awaited finally). `readWorkers(getNice?)` enumerates the dir, drops confirmed-dead pids (`classifyLiveness`: ESRCH = dead/prune, EPERM = alive/keep), applies a pid-reuse start-time guard (`ps -o lstart`, rejects a pid whose process started >5s after the entry was written), and re-measures each live worker's niceness now. Reports the worker's REAL pid, sidestepping the tini-wrapper-pid problem. Also exports `currentDbIdentity()` (#1849) — the RAW database identity (`database_url` or `database_path`, else `'default'`), read from config without a DB connect, used as the authoritative key for the supervisor's queue-scoped singleton lock (avoids the hash-collision question that `currentBrainId`'s djb2 would raise). Pinned by `test/worker-registry.test.ts`.
- `src/core/minions/supervisor-pid.ts``readSupervisorPid(pidFile) → {pid, running}`: the shared `existsSync → readFileSync → parseInt → process.kill(pid,0)` PID-file + liveness reader extracted from the three copies in `jobs.ts` (supervisor status), `jobs.ts` (stats), and `doctor.ts`. EPERM from the liveness probe counts as running. Pinned by `test/supervisor-pid.test.ts`.
- `src/core/minions/handler-timeouts.ts` (#1737) — per-handler default wall-clock budgets. `HANDLER_DEFAULT_TIMEOUT_MS` maps the long handlers (`subagent`, `subagent_aggregator`, `embed-backfill`, `autopilot-cycle`) to a 30-min default (the value `cycle/patterns.ts` already passed for subagents, generalized). `defaultTimeoutMsFor(jobName)` returns that default or `null` for short handlers (keep the tight null-default wall-clock). `MinionQueue.add()` stamps this onto `minion_jobs.timeout_ms` at submit time when the caller passed no `timeout_ms`, so a long job submitted without one isn't wall-clock-killed mid-progress; an explicit value always wins and already-queued jobs are NOT backfilled. Pinned by `test/minions.test.ts`.
- `src/core/minions/types.ts``MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
- `src/core/minions/handlers/shell.ts``shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides + `inherit:`-resolved keys. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`). `ShellJobParams.inherit?: string[]` is a free-form list of snake_case config-key names; the worker resolves each via `loadConfig()` and injects the value under the derived env key (`database_url``GBRAIN_DATABASE_URL`; else uppercased). Names persist in `minion_jobs.data` (and the shell-audit JSONL); values never do. The canonical validator `validateShellJobParams` (sibling `shell-validate.ts`) runs PRE-ENQUEUE in both submit surfaces — `gbrain jobs submit shell` (jobs.ts:271) AND the `submit_job` op for `name='shell'` (operations.ts:2085); the handler-entry re-validation here is defense-in-depth (closes the bug class where validation ran AFTER `queue.add()` persisted the row). The validator does NOT police which config keys the agent inherits — same-uid trust model treats the agent as a peer of the worker.
@@ -372,8 +374,9 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
- `src/commands/jobs.ts` extension — registers 11 Minion handlers: `reindex`, `repair-jsonb`, `orphans`, `integrity`, `purge`, `synthesize` (PROTECTED), `patterns` (PROTECTED), `consolidate` (PROTECTED), `extract_facts`, `resolve_symbol_edges`, `recompute_emotional_weight`. Phase wrappers delegate to `runCycle({phases:[name]})` so `src/core/cycle.ts` stays the single source of truth for phase semantics. The standalone `sync` handler passes `noExtract: true` to match `runPhaseSync`'s contract (doctor's remediation plan emitting `[sync, extract]` would otherwise double-extract).
- `src/core/minions/protected-names.ts` extension — `PROTECTED_JOB_NAMES` includes `synthesize`, `patterns`, `consolidate`. These phases internally submit `subagent` children with `allowProtectedSubmit=true` and can spend Anthropic credits. Only trusted local callers (CLI, autopilot, `doctor --remediate`) can submit them; MCP requests are rejected by `submit_job`'s protected-name guard.
- `src/commands/autopilot.ts` extension — targeted-submit loop instead of blanket `autopilot-cycle` dispatch. Each tick: cheap `engine.getHealth()` (single SQL count) + `computeRecommendations()`, then route by shape — `score >= 95 AND no plan AND <60min since last full` → sleep; `score >= 95 AND >=60min` → submit `autopilot-cycle` (60-min floor exercises phase-coupling invariants on healthy brains); `plan <= 3 steps AND est <5min` → submit individual handlers; `plan large OR score < 70` → submit full `autopilot-cycle`. The `gbrain-cycle` lock ensures targeted submissions and the full cycle can't run concurrently. `maxWaiting: 1` per submit closes the queue-fan-out vector.
- `src/core/cycle.ts` extension — `purge` phase (the cycle's 9th phase, for soft-delete TTLs) also GCs stale `op_checkpoints` rows older than 7 days. Non-fatal on pre-v67 brains (DROP-target-table check before DELETE).
- `src/commands/embed.ts` extension — wires `--background` as the reference integration for the `maybeBackground()` helper. `gbrain embed --stale --background` submits as a Minion job, prints `job_id=N` to stdout, exits 0. Composable: `JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2); gbrain jobs follow $JOB`. The other six commands (`extract`, `lint`, `backlinks`, `reindex`, `integrity`, `pages`) adopt the same pattern in a follow-up wave.
- `src/core/abort-check.ts` (#1737) — one canonical place for cooperative-abort checks across gbrain's long loops. `isAborted(signal?)` → boolean (for loops that `break` and return partial progress). `throwIfAborted(signal?, label?)` throws an `AbortError` (`name === 'AbortError'`) at phase boundaries, preferring the signal's `reason` ('wall-clock'/'lock-lost'/'shutdown') so the unwind self-describes. `anySignal(internal, external?)` composes two signals into one that fires when EITHER does (platform `AbortSignal.any` with a manual-relay fallback), returning the internal unchanged when there's no external so non-aborting callers pay nothing. The fix for the #1737 cycle-wedge: the embed phase ignored its abort signal and ran to completion, so `gbrain_cycle_locks` stayed held and later autopilot cycles skipped with `cycle_already_running`; threading these checks through `runPhaseEmbed → runEmbedCore → embedAll(Stale)/embedPage` lets the phase bail and release the lock immediately. Pinned by `test/abort-check.test.ts`.
- `src/core/cycle.ts` extension — `purge` phase (the cycle's 9th phase, for soft-delete TTLs) also GCs stale `op_checkpoints` rows older than 7 days. Non-fatal on pre-v67 brains (DROP-target-table check before DELETE). #1737: the cycle threads its abort signal into the embed phase (`runPhaseEmbed(engine, dryRun, signal)`) so a timed-out cycle's long embed phase honors cancellation and releases `gbrain_cycle_locks` right away instead of after a full backlog run.
- `src/commands/embed.ts` extension — wires `--background` as the reference integration for the `maybeBackground()` helper. `gbrain embed --stale --background` submits as a Minion job, prints `job_id=N` to stdout, exits 0. Composable: `JOB=$(gbrain embed --stale --background | grep -oE 'job_id=[0-9]+' | cut -d= -f2); gbrain jobs follow $JOB`. The other six commands (`extract`, `lint`, `backlinks`, `reindex`, `integrity`, `pages`) adopt the same pattern in a follow-up wave. #1737: `runEmbedCore` accepts an optional `signal` threaded down both the `--stale` and `--all` paths (`embedAllStale`/`embedAll`/`embedPage`); each composes it with the internal wall-clock budget via `anySignal` and checks `isAborted`/`effectiveSignal.aborted` in every per-slug loop, page-claim pool, and `embedBatch` call, so a worker abort (wall-clock timeout / lock loss / SIGTERM) stops embedding within a batch. Pinned by `test/embed.serial.test.ts`.
- `src/core/cli-options.ts` extension — `maybeBackground(opName, fingerprintArgs, runDirect)` helper. Same semantics in TTY and cron (no `--no-tty-detect` flag, no surprise behavior change between contexts): when `--background` is passed, submits the op as a Minion job via `op_checkpoints` for resumability and returns the `job_id`. `--background --follow` execs `gbrain jobs follow <id>` so the user sees the same stderr stream they'd get from a direct call. PGLite degrades to inline execution with a clear stderr note ("PGLite worker pool not yet supported; running inline"). Returns a tagged union the caller dispatches on.
- `openclaw.plugin.json` — ClawHub bundle plugin manifest
- `src/commands/capture.ts` + `src/commands/serve-http.ts` + `src/core/{operations,import-file,types,utils,facts/absorb-log,brainstorm/{orchestrator,error-classify},scope,postgres-engine,pglite-engine}.ts` extensions — ingestion-cathedral productionization after a smoke test against Supabase+PgBouncer. Capture frontmatter merge via `mergeCaptureFrontmatter` (uses gray-matter directly, NOT the lossy `parseMarkdown`); `/ingest` null-guard + outer try/catch envelope with `!res.headersSent` guard; dedup via separate normalize-for-hash (`normalizeForHash` strips BOM/CRLF/whitespace/NFKC) + body-after-frontmatter-strip on the DB hash (excludes `captured_at` + `ingested_at` so capture-cli timestamp variations don't invalidate the chunk cache); friendly `pages_source_id_fk` rewrite via `maybeRewriteSourceFkError` on BOTH local + thin-client `callRemoteTool` catch blocks; `facts:absorb` 'No database connection' suppression via typed `instanceof GBrainError && e.problem` check + first-occurrence stack-trace info log (module-scoped `_hasLoggedDisconnectedFactsAbsorb` flag, test seam `_resetFactsAbsorbDisconnectedFlagForTests`); CLI help discoverability (`capture` added to `CLI_ONLY_SELF_HELP` + pre-engine-bind `--help` short-circuit in `handleCliOnly` + a `BRAIN` section in `printHelp`); binary-file guard via `detectBinaryNullByte(buf)` first-8KB NUL scan on `--file` (Buffer-read, no encoding) and `--stdin` (`readStdinBuffer` accumulator); provenance write-through — put_page accepts 3 optional params (source_kind, source_uri, ingested_via; `ingested_at` server-stamped) + trust gate (when `ctx.remote !== false` IGNORE client params, server stamps `mcp:put_page`, fail-closed) + COALESCE-preserve UPDATE semantics (omitting params on a later put_page preserves prior values; first-write-wins); `/admin/api/register-client` scopes normalization via `normalizeScopesInput(raw: unknown)` in `src/core/scope.ts` (accepts string/string[]/missing; rejects `['read write']` space-in-element shape, non-string elements, empty array, unknown scopes; deduped + sorted); brainstorm timeout surfacing via an orchestrator-level try/catch at `runBrainstorm` entry (single-point wrap covers every internal SQL site, classifies SQLSTATE 57014 via postgres.js `.code` / `.sqlState` / message fallback into `StructuredAgentError` code `brainstorm_timeout` with a hint covering all 3 PG cancel sub-causes); read-path surfaces all 4 provenance columns via `getPage` projection + `rowToPage` 3-state optional read + `Page` interface; canonical source resolver routes capture through `resolveSourceWithTier(engine, parsed.source, cwd)`; thin-client `--source` rejection (server-side OAuth client registration owns source scope); the `source_kind` taxonomy is closed (`capture-cli | put_page | mcp:put_page | webhook | file-watcher | inbox-folder | cron-scheduler`), `--source` maps to source_id only. Tests: `test/capture-build-content.test.ts`, `test/capture-runcapture.test.ts`, `test/put-page-provenance.test.ts`, `test/scope-normalize.test.ts`, `test/cli-help-discoverability.test.ts`, `test/brainstorm-timeout.test.ts`; extended `test/facts-absorb-log.test.ts`, `test/import-file.test.ts`, `test/e2e/engine-parity.test.ts`, `test/e2e/serve-http-ingest-webhook.test.ts`. Report at `docs/v0.38-smoke-test-report.md`. Follow-ups in TODOS.md: SQL-shape rewrite of `listPrefixSampledPages` for PgBouncer, magic-byte allowlist for binary detection, `--source-kind` override flag, ingest_capture handler migration, provenance-history table, facts:absorb root-cause trace.
+1 -1
View File
@@ -143,5 +143,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.28.0"
"version": "0.42.29.0"
}
@@ -24,10 +24,23 @@
*/
import { readFileSync, existsSync } from 'node:fs';
import { join } from 'node:path';
import { join, resolve, sep } from 'node:path';
const MAX_CHARS = 2500;
// #1851: a topic id is the ONLY thing that crosses the wire from a call link
// (never the topic content itself — that would be prompt injection + a leak via
// URLs/logs). The id indexes `$BRAIN_ROOT/topics/<topicId>.md` server-side, so
// it must be a strict slug: lowercase alnum + dashes, no dots/slashes. This
// regex alone rejects `../../SOUL` (no dots, no slashes); the resolve-under-dir
// check below is defense-in-depth.
const TOPIC_ID_RE = /^[a-z0-9][a-z0-9-]*$/;
/** True iff `topicId` is a safe slug (see TOPIC_ID_RE). */
export function isValidTopicId(topicId) {
return typeof topicId === 'string' && topicId.length <= 128 && TOPIC_ID_RE.test(topicId);
}
// Emotion-word filter. Content-agnostic — catches what's loaded in the
// operator's OWN words without hardcoding names of people in their life.
// Add words to this list if your brain uses domain-specific vocabulary.
@@ -152,6 +165,50 @@ export async function buildMarsContext({ brainRoot, timezone } = {}) {
return cap(scrub(ctx));
}
/**
* #1851 — Build TOPIC context: the recent conversation in the topic the agent
* was summoned into, so calling Mars/Venus from inside a thread boots them
* already knowing what you were just discussing.
*
* The server resolves this from `topicId` at connect time (the id is the only
* thing the call link carries). Reads `$BRAIN_ROOT/topics/<topicId>.md`. The
* operator's brain owns what lands in that file (recent turns + a 2-3 line
* synthesized summary is the intended shape — not a raw dump).
*
* Persona-agnostic: the SAME topic block is injected for Mars or Venus; only
* the persona identity (section 1 of the prompt) differs. Returns '' when
* there's no topic, the id is unsafe, or the file is missing — falling back to
* the generic per-persona live context (current behavior).
*
* @param {object} opts
* @param {string} opts.brainRoot
* @param {string} opts.topicId — strict slug; see {@link isValidTopicId}
* @returns {Promise<string>} ≤2500 chars, PII-scrubbed, or '' to degrade.
*/
export async function buildTopicContext({ brainRoot, topicId } = {}) {
if (!brainRoot || !topicId || !isValidTopicId(topicId)) return '';
// Defense-in-depth: confine the resolved path under <brainRoot>/topics even
// though the slug regex already forbids traversal characters.
const topicsDir = resolve(join(brainRoot, 'topics'));
const path = resolve(join(topicsDir, `${topicId}.md`));
if (path !== join(topicsDir, `${topicId}.md`) || !path.startsWith(topicsDir + sep)) {
return '';
}
if (!existsSync(path)) return '';
try {
const raw = readFileSync(path, 'utf8').trim();
if (!raw) return '';
let ctx = 'RECENT CONVERSATION IN THE TOPIC YOU WERE SUMMONED INTO.\n';
ctx += "Use this so you already know what was just being discussed. Don't recite it; let it inform you.\n\n";
ctx += raw;
return cap(scrub(ctx));
} catch {
return '';
}
}
/**
* Build logistics-salient context for Venus.
*
@@ -50,6 +50,32 @@ export async function buildMarsContext(opts);
* @returns {Promise<string>}
*/
export async function buildVenusContext(opts);
/**
* #1851 — Build TOPIC context: the recent conversation in the topic the agent
* was summoned into (persona-agnostic; the same block is used for Mars or
* Venus). Lets a caller drop a persona into whatever thread they were already
* discussing without re-explaining.
*
* The server resolves this from `topicId` at connect time. `topicId` is the
* ONLY topic field accepted over the wire (a call link carries it). NEVER
* accept topic CONTENT as a parameter — that's prompt injection + a leak into
* URLs, browser history, referrers, and access logs.
*
* `topicId` MUST be a strict slug (^[a-z0-9][a-z0-9-]*$, ≤128 chars); the
* shipped example reads `$BRAIN_ROOT/topics/<topicId>.md` and confines the
* resolved path under `topics/` (defense-in-depth against traversal).
*
* Required: PII scrubbed. Required: ≤ 2500 chars. Returns '' when there is no
* topic, the id is unsafe, or the file is missing → the persona falls back to
* its generic live context (current behavior).
*
* @param {object} opts
* @param {string} opts.brainRoot
* @param {string} opts.topicId — strict slug; indexes topics/<topicId>.md
* @returns {Promise<string>}
*/
export async function buildTopicContext(opts);
```
## Brain layout expected by the shipped example
@@ -25,13 +25,17 @@ import { VENUS } from './venus.mjs';
// ── Shared preamble (tools, rules, time) ─────────────────
export function buildSharedContext(opts = {}) {
const { authenticated = false, identity = '', dateTime = '' } = opts;
const { authenticated = false, identity = '', dateTime = '', topicName = '' } = opts;
let ctx = '';
if (dateTime) ctx += `CURRENT DATE/TIME: ${dateTime}\n\n`;
if (authenticated && identity) {
ctx += `The caller is verified as ${identity}. All allow-listed tools are available.\n\n`;
}
// #1851: when summoned from a specific topic, name it up top so the persona
// knows the frame of the call. The recent-conversation detail is injected
// separately as the `# Topic Context` block (see prompt.mjs).
if (topicName) ctx += `CURRENT TOPIC: ${topicName}\n\n`;
return ctx;
}
+22 -2
View File
@@ -21,7 +21,7 @@
import { getPersona, buildSharedContext } from './lib/personas/personas.mjs';
import { getEffectiveAllowlist } from './tools.mjs';
import { buildMarsContext, buildVenusContext } from './lib/context-builder.example.mjs';
import { buildMarsContext, buildVenusContext, buildTopicContext } from './lib/context-builder.example.mjs';
/**
* Build the system prompt for a session.
@@ -33,6 +33,11 @@ import { buildMarsContext, buildVenusContext } from './lib/context-builder.examp
* @param {string} [opts.dateTime] — ISO timestamp; defaults to now
* @param {string} [opts.brainRoot] — absolute path to operator's brain repo
* @param {string} [opts.timezone]
* @param {string} [opts.topicId] — #1851: topic the agent was summoned into.
* The ONLY topic field accepted over the wire; the server resolves the
* recent-conversation context from the brain (never pass topic CONTENT in —
* that's prompt injection + a URL/log leak).
* @param {string} [opts.topicName] — human label for the topic (display only).
* @returns {Promise<string>} sanitized system prompt
*/
export async function buildSystemPrompt(opts = {}) {
@@ -43,12 +48,13 @@ export async function buildSystemPrompt(opts = {}) {
let prompt = `# You ARE ${persona.name}\n`;
prompt += `You are ${persona.name}, a voice AI. You are NOT a generic assistant. You are NOT Claude. You are NOT GPT. You are ${persona.name} with the personality below.\n\n`;
// 2. Shared context (date/time + identity if authed).
// 2. Shared context (date/time + identity if authed + topic name if summoned).
const dateTime = opts.dateTime || new Date().toISOString();
prompt += buildSharedContext({
authenticated: !!opts.authenticated,
identity: opts.identity || '',
dateTime,
topicName: opts.topicName || '',
});
// 3. Persona body.
@@ -69,6 +75,20 @@ export async function buildSystemPrompt(opts = {}) {
}
}
// 4b. #1851 Topic context — the recent conversation in the topic the agent
// was summoned into. Resolved server-side from topicId (the only topic field
// that crosses the wire). Injected AFTER the persona body + live context so
// the identity-first ordering still wins; the topic only adds background.
// No topicId → omitted → generic behavior (acceptance criterion).
if (opts.brainRoot && opts.topicId) {
try {
const tctx = await buildTopicContext({ brainRoot: opts.brainRoot, topicId: opts.topicId });
if (tctx) prompt += `# Topic Context\n${tctx}\n\n`;
} catch (err) {
console.warn(`[prompt] topic-context builder threw: ${err.message}`);
}
}
// 5. Tool list — only the allow-list, never the denylist.
const allowed = getEffectiveAllowlist();
if (allowed.length > 0) {
+8 -1
View File
@@ -94,6 +94,11 @@
const params = new URLSearchParams(location.search);
const persona = (params.get('persona') || 'venus').toLowerCase();
const TEST_MODE = params.get('test') === '1';
// #1851: a per-topic call link carries topicId (+ optional topicName). We
// forward ONLY these to /session — the server resolves the topic's recent
// conversation from the brain. Topic content never travels in a URL.
const topicId = params.get('topicId') || '';
const topicName = params.get('topicName') || '';
document.getElementById('personaBadge').textContent = `persona: ${persona}`;
if (TEST_MODE) document.getElementById('testBadge').style.display = '';
@@ -232,7 +237,9 @@
await pc.setLocalDescription(offer);
setStatus('sending SDP offer to /session...');
const sessionUrl = `/session?persona=${encodeURIComponent(persona)}`;
let sessionUrl = `/session?persona=${encodeURIComponent(persona)}`;
if (topicId) sessionUrl += `&topicId=${encodeURIComponent(topicId)}`;
if (topicName) sessionUrl += `&topicName=${encodeURIComponent(topicName)}`;
const res = await fetch(sessionUrl, {
method: 'POST',
headers: { 'Content-Type': 'application/sdp' },
+9
View File
@@ -124,12 +124,21 @@ async function handleSession(req, res) {
const url = new URL(req.url, `http://${req.headers.host}`);
const persona = (url.searchParams.get('persona') || DEFAULT_PERSONA).toLowerCase();
// #1851: a call link minted from a Telegram topic carries topicId (+ an
// optional display topicName). The id is the ONLY topic data we accept over
// the wire — buildSystemPrompt resolves the recent-conversation context from
// the brain server-side. We never accept topic CONTENT as a param (that would
// be prompt injection + a leak into URLs/referrers/access logs).
const topicId = url.searchParams.get('topicId') || undefined;
const topicName = url.searchParams.get('topicName') || undefined;
// Build the persona-aware system prompt at session start.
const systemPrompt = await buildSystemPrompt({
persona,
brainRoot: process.env.BRAIN_ROOT,
timezone: process.env.TIMEZONE,
topicId,
topicName,
});
// Session config for OpenAI Realtime /v1/realtime/calls.
@@ -32,6 +32,16 @@ The depth of the conversation is the signal. If it's surface-level scheduling, r
This skill is invoked by the host agent's resolver when the operator's voice or text input matches the triggers above. The voice agent (`services/voice-agent/code/server.mjs`) consumes the persona key (`mars`) at session start via `?persona=mars` on the WebRTC `/session` endpoint, OR via the `DEFAULT_PERSONA=mars` env var if Mars is the operator's default.
### Summoning Mars into a topic (#1851)
To call Mars *from inside* a specific conversation topic, mint a per-topic call link by adding `topicId` (a strict slug, `^[a-z0-9][a-z0-9-]*$`) and an optional `topicName`:
```
/call?persona=mars&topicId=real-estate&topicName=Real%20Estate
```
Mars boots already knowing the topic's recent conversation. Only the `topicId` crosses the wire — the server resolves the recent-conversation context from `$BRAIN_ROOT/topics/<topicId>.md`. **Never put topic content in the URL** (prompt injection + a leak into history/referrers/logs). No `topicId` → Mars uses his generic live context (unchanged behavior).
## Mode detection (inside the persona)
Mars detects mode from conversational signals:
@@ -33,6 +33,16 @@ If a question requires multi-paragraph thinking, Venus tees it up briefly and ro
This skill is invoked by the host agent's resolver when the operator's voice or text input matches the triggers above. The voice agent (`services/voice-agent/code/server.mjs`) reads the persona key (`venus`) at session start via `?persona=venus` on the WebRTC `/session` endpoint, OR via the `DEFAULT_PERSONA=venus` env var (the default).
### Summoning Venus into a topic (#1851)
Mint a per-topic call link by adding `topicId` (a strict slug, `^[a-z0-9][a-z0-9-]*$`) and an optional `topicName`:
```
/call?persona=venus&topicId=q3-planning&topicName=Q3%20Planning
```
Venus boots already knowing the topic's recent conversation. Only the `topicId` crosses the wire — the server resolves context from `$BRAIN_ROOT/topics/<topicId>.md`. **Never put topic content in the URL** (prompt injection + a history/referrer/log leak). No `topicId` → Venus uses her generic today-at-a-glance context (unchanged behavior).
## Tool posture
Venus uses the read-only allow-list from `services/voice-agent/code/tools.mjs`:
@@ -0,0 +1,118 @@
/**
* topic-context.test.mjs — #1851 topic-aware voice personas.
*
* Pins the security + behavior contract for summoning Mars/Venus into a topic:
* - topicId path-traversal is rejected (only the brain-owned topics/<id>.md)
* - the topic block is injected when a topic is provided
* - no topic → generic behavior (no topic block), persona identity unchanged
* - topic X vs topic Y produce different context
* - the topic block can NOT override persona identity / hard rules
* - PII in a topic file is scrubbed
* - topic CONTENT is never accepted over the wire (only topicId)
*/
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import { buildTopicContext, isValidTopicId } from '../../code/lib/context-builder.example.mjs';
import { buildSystemPrompt } from '../../code/prompt.mjs';
let brainRoot;
// Build PII-shaped strings at runtime so the literal phone/email shapes never
// appear in this source file (the agent-voice PII guard greps the recipe tree
// for those shapes). The runtime values still exercise the scrubber.
const FAKE_PHONE = ['415', '555', '0100'].join('-');
const FAKE_EMAIL = ['someone', 'example.test'].join('@');
beforeEach(() => {
brainRoot = mkdtempSync(join(tmpdir(), 'agent-voice-topic-'));
mkdirSync(join(brainRoot, 'topics'), { recursive: true });
writeFileSync(join(brainRoot, 'topics', 'real-estate.md'), 'We were discussing the warehouse-lease offer and the inspection timeline.');
writeFileSync(join(brainRoot, 'topics', 'yc-batch.md'), 'Talking through the W26 batch interview schedule.');
// A file with PII to verify scrubbing (shapes built at runtime, see above).
writeFileSync(join(brainRoot, 'topics', 'with-pii.md'), `Call me at ${FAKE_PHONE} or ${FAKE_EMAIL} about the deal.`);
// A secret OUTSIDE the topics dir that traversal must not reach.
writeFileSync(join(brainRoot, 'SOUL.md'), 'TOP SECRET SOUL CONTENT');
});
afterEach(() => {
try { rmSync(brainRoot, { recursive: true, force: true }); } catch { /* noop */ }
});
describe('isValidTopicId', () => {
it('accepts strict slugs', () => {
expect(isValidTopicId('real-estate')).toBe(true);
expect(isValidTopicId('yc-batch-2026')).toBe(true);
});
it('rejects traversal and unsafe ids', () => {
expect(isValidTopicId('../../SOUL')).toBe(false);
expect(isValidTopicId('foo/bar')).toBe(false);
expect(isValidTopicId('foo.md')).toBe(false);
expect(isValidTopicId('UPPER')).toBe(false);
expect(isValidTopicId('')).toBe(false);
expect(isValidTopicId(undefined)).toBe(false);
});
});
describe('buildTopicContext', () => {
it('returns the topic conversation for a valid id', async () => {
const ctx = await buildTopicContext({ brainRoot, topicId: 'real-estate' });
expect(ctx).toContain('warehouse-lease');
});
it('topic X and topic Y differ', async () => {
const x = await buildTopicContext({ brainRoot, topicId: 'real-estate' });
const y = await buildTopicContext({ brainRoot, topicId: 'yc-batch' });
expect(x).toContain('warehouse-lease');
expect(y).toContain('W26 batch');
expect(x).not.toEqual(y);
});
it('rejects path traversal — cannot read SOUL.md outside topics/', async () => {
const ctx = await buildTopicContext({ brainRoot, topicId: '../../SOUL' });
expect(ctx).toBe('');
expect(ctx).not.toContain('TOP SECRET');
});
it('scrubs PII in the topic file', async () => {
const ctx = await buildTopicContext({ brainRoot, topicId: 'with-pii' });
expect(ctx).not.toContain(FAKE_PHONE);
expect(ctx).not.toContain(FAKE_EMAIL);
});
it('missing topic file → empty (generic fallback)', async () => {
expect(await buildTopicContext({ brainRoot, topicId: 'does-not-exist' })).toBe('');
});
});
describe('buildSystemPrompt topic-awareness', () => {
it('injects a # Topic Context block when topicId is provided', async () => {
const prompt = await buildSystemPrompt({ persona: 'mars', brainRoot, topicId: 'real-estate', topicName: 'Real Estate' });
expect(prompt).toContain('# Topic Context');
expect(prompt).toContain('warehouse-lease');
expect(prompt).toContain('CURRENT TOPIC: Real Estate');
});
it('no topicId → no topic block (generic behavior unchanged)', async () => {
const prompt = await buildSystemPrompt({ persona: 'mars', brainRoot });
expect(prompt).not.toContain('# Topic Context');
expect(prompt).not.toContain('CURRENT TOPIC:');
});
it('persona identity stays first; topic context cannot override it', async () => {
const prompt = await buildSystemPrompt({ persona: 'mars', brainRoot, topicId: 'real-estate', topicName: 'Real Estate' });
// Identity-first: the "You ARE Mars" line precedes the topic block.
expect(prompt.indexOf('# You ARE Mars')).toBeLessThan(prompt.indexOf('# Topic Context'));
// Hard rules survive after the topic block.
expect(prompt).toContain('# Hard Rules');
expect(prompt.indexOf('# Topic Context')).toBeLessThan(prompt.indexOf('# Hard Rules'));
});
it('a traversal topicId yields the generic prompt (no block, no leak)', async () => {
const prompt = await buildSystemPrompt({ persona: 'venus', brainRoot, topicId: '../../SOUL' });
expect(prompt).not.toContain('# Topic Context');
expect(prompt).not.toContain('TOP SECRET');
});
});
+70
View File
@@ -4196,6 +4196,76 @@ export async function buildChecks(
// Audit read / import failure is best-effort; skip silently.
}
// 3b-bis-2. Supervisor SINGLETON + effective max-rss (#1849). Separate check
// from `supervisor` above (same Codex #11 precedent as the niceness split) so
// a singleton-divergence warn can't clobber the crash/liveness precedence.
//
// The #1849 fix makes a queue-scoped DB lock the real singleton authority. A
// second supervisor on the same (db, queue) now fails fast at start — but if
// a rogue one slipped in BEFORE upgrade (or someone ran one with an explicit
// --pid-file on a pre-fix binary), the lock holder's (host, pid) won't match
// the local pidfile. Surface that mismatch + the effective --max-rss (the cap
// a rogue supervisor would have fought over). Bare pid is meaningless across
// hosts/containers, so we compare host+pid (Codex #25).
try {
const { DEFAULT_PID_FILE, supervisorLockId, classifySupervisorSingleton } = await import('../core/minions/supervisor.ts');
const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts');
const { readSupervisorPid } = await import('../core/minions/supervisor-pid.ts');
const { hostname } = await import('os');
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStarted = events.filter(e => e.event === 'started').pop() as
| (Record<string, unknown> & { ts?: string })
| undefined;
// Only run when a supervisor was actually observed (no noise on installs
// that never used it) and we have a live engine to read the lock row.
if (lastStarted && engine) {
const queue = typeof lastStarted.queue === 'string' ? lastStarted.queue : 'default';
const effectiveMaxRss = typeof lastStarted.max_rss_mb === 'number' ? lastStarted.max_rss_mb : null;
const localPid = readSupervisorPid(DEFAULT_PID_FILE).pid;
const localHost = hostname();
// Read the DB singleton lock holder for this queue.
const lockRows = await engine.executeRaw<{ holder_pid: number; holder_host: string; live: boolean }>(
`SELECT holder_pid, holder_host, ttl_expires_at > now() AS live
FROM gbrain_cycle_locks WHERE id = $1`,
[supervisorLockId(queue)],
);
const lock = lockRows[0] ?? null;
const rssStr = effectiveMaxRss !== null ? `${effectiveMaxRss}MB` : 'unknown';
const verdict = classifySupervisorSingleton({
lockLive: !!lock?.live,
lockHolderHost: lock?.holder_host ?? null,
lockHolderPid: lock?.holder_pid ?? null,
localHost,
localPid,
});
if (verdict === 'mismatch') {
checks.push({
name: 'supervisor_singleton',
status: 'warn',
message:
`Queue '${queue}' singleton lock is held by ${lock!.holder_host}:${lock!.holder_pid}, ` +
`but the local pidfile points to ${localHost}:${localPid ?? 'none'}. A second supervisor may be ` +
`running with a different --max-rss (effective cap here: ${rssStr}). Stop the extra one ` +
`and keep a single supervisor per queue: gbrain jobs supervisor stop.`,
details: { queue, lock_holder: `${lock!.holder_host}:${lock!.holder_pid}`, local: `${localHost}:${localPid ?? 'none'}`, effective_max_rss_mb: effectiveMaxRss },
});
} else if (verdict === 'single') {
checks.push({
name: 'supervisor_singleton',
status: 'ok',
message: `Single supervisor on queue '${queue}' (holder=${lock!.holder_host}:${lock!.holder_pid}, max_rss=${rssStr}).`,
details: { queue, effective_max_rss_mb: effectiveMaxRss },
});
}
}
} catch {
// Best-effort (lock table may not exist on a very old brain); skip silently.
}
// 3b-sexies. Supervisor/worker scheduling priority (niceness, issue #1815).
// SEPARATE check from `supervisor` above so a niceness divergence warn can
// never clobber the supervisor check's max_crashes_exceeded fail/warn
+39 -12
View File
@@ -9,6 +9,7 @@ import { loadConfig } from '../core/config.ts';
import { slog, serr } from '../core/console-prefix.ts';
import { filterOutEmbedSkipped } from '../core/embed-skip.ts';
import { runSlidingPool } from '../core/worker-pool.ts';
import { isAborted, anySignal } from '../core/abort-check.ts';
export interface EmbedOpts {
/** Embed ALL pages (every chunk). */
@@ -61,6 +62,15 @@ export interface EmbedOpts {
* remediation submits on big stale backlogs.
*/
catchUp?: boolean;
/**
* #1737: cooperative-abort signal from the Minions worker (wall-clock
* timeout, lock loss, SIGTERM). When it fires, the embed loops break
* cleanly with partial progress preserved so the autopilot cycle's
* finally can release `gbrain_cycle_locks` instead of running for the
* full 10-15 min embed phase after the job was already killed. Composed
* with the internal wall-clock budget timer via `anySignal`.
*/
signal?: AbortSignal;
}
/**
@@ -187,8 +197,9 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
if (opts.slugs && opts.slugs.length > 0) {
for (const s of opts.slugs) {
if (isAborted(opts.signal)) break; // #1737: stop the per-slug loop on abort
try {
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId);
await embedPage(engine, s, !!opts.dryRun, result, opts.sourceId, opts.signal);
} catch (e: unknown) {
serr(` Error embedding ${s}: ${e instanceof Error ? e.message : e}`);
}
@@ -200,11 +211,11 @@ export async function runEmbedCore(engine: BrainEngine, opts: EmbedOpts): Promis
batchSize: opts.batchSize,
priority: opts.priority,
catchUp: opts.catchUp,
});
}, opts.signal);
return result;
}
if (opts.slug) {
await embedPage(engine, opts.slug, !!opts.dryRun, result, opts.sourceId);
await embedPage(engine, opts.slug, !!opts.dryRun, result, opts.sourceId, opts.signal);
return result;
}
throw new Error('No embed target specified. Pass { slug }, { slugs }, { all }, or { stale }.');
@@ -309,6 +320,7 @@ async function embedPage(
dryRun: boolean,
result: EmbedResult,
sourceId?: string,
signal?: AbortSignal,
) {
const opts = sourceId ? { sourceId } : undefined;
const page = await engine.getPage(slug, opts);
@@ -364,7 +376,7 @@ async function embedPage(
return;
}
const embeddings = await embedBatch(toEmbed.map(c => c.chunk_text));
const embeddings = await embedBatch(toEmbed.map(c => c.chunk_text), { abortSignal: signal });
const embeddingMap = new Map<number, Float32Array>();
for (let j = 0; j < toEmbed.length; j++) {
embeddingMap.set(toEmbed[j].chunk_index, embeddings[j]);
@@ -405,6 +417,7 @@ async function embedAll(
priority?: 'recent';
catchUp?: boolean;
},
signal?: AbortSignal,
) {
// v0.41.31: current embedding provenance signature. Stamped onto pages
// when their chunks are (re)embedded so a later model/dimension swap is
@@ -426,7 +439,8 @@ async function embedAll(
if (staleOnly) {
// D7: thread sourceId so `gbrain embed --stale --source X` actually scopes.
// v0.41.18.0 (A13): thread batchSize/priority/catchUp into the stale path.
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature);
// #1737: thread the external abort signal so the cycle embed phase bails.
return await embedAllStale(engine, sourceId, dryRun, result, onProgress, staleOpts, signature, signal);
}
// v0.31.12: when sourceId is set, scope listPages to that source.
@@ -455,6 +469,8 @@ async function embedAll(
const CONCURRENCY = parseInt(process.env.GBRAIN_EMBED_CONCURRENCY || '20', 10);
async function embedOnePage(page: typeof pages[number]) {
// #1737: bail before doing any work for this page if the run was aborted.
if (isAborted(signal)) return;
// v0.31.12: thread source_id from the page row so getChunks/upsertChunks
// target the correct (source_id, slug) row, not the 'default' source.
const pageSourceId = page.source_id;
@@ -519,6 +535,7 @@ async function embedAll(
await runSlidingPool({
items: pages,
workers: CONCURRENCY,
...(signal && { signal }), // #1737: pool stops claiming pages once aborted
onItem: (page) => embedOnePage(page),
failureLabel: (page) => page.slug,
});
@@ -561,6 +578,7 @@ async function embedAllStale(
catchUp?: boolean;
},
signature?: string,
externalSignal?: AbortSignal,
) {
// D7: thread sourceId so source-scoped runs only count + visit
// that source's NULL embeddings.
@@ -621,6 +639,12 @@ async function embedAllStale(
const budgetController = new AbortController();
const budgetTimer = setTimeout(() => budgetController.abort(), BUDGET_MS);
const budgetSignal = budgetController.signal;
// #1737: the effective signal fires when EITHER the internal wall-clock
// budget OR the caller's abort (worker timeout / lock loss / SIGTERM) fires.
// Replaces bare budgetSignal at every loop/pool/embed check below so the
// autopilot cycle's embed phase stops within one batch (~2s) of being
// killed instead of running the full 10-15 min and wedging the cycle lock.
const effectiveSignal = anySignal(budgetSignal, externalSignal);
// v0.41.18.0 (A13): --priority recent threads orderBy='updated_desc' to
// listStaleChunks. Composite cursor tracks (updated_at, page_id, chunk_index)
@@ -640,9 +664,12 @@ async function embedAllStale(
try {
// eslint-disable-next-line no-constant-condition
while (true) {
if (budgetSignal.aborted) {
if (effectiveSignal.aborted) {
if (!budgetExitNotified) {
serr(`\n [embed] wall-clock budget (${BUDGET_MS}ms) exceeded; exiting cleanly. Re-run picks up via partial index.`);
const why = budgetSignal.aborted
? `wall-clock budget (${BUDGET_MS}ms) exceeded`
: 'aborted by caller (job timeout / lock loss / shutdown)';
serr(`\n [embed] ${why}; exiting cleanly. Re-run picks up via partial index.`);
budgetExitNotified = true;
}
break;
@@ -691,7 +718,7 @@ async function embedAllStale(
const keySourceId = stale[0]?.source_id ?? 'default';
const slug = stale[0].slug;
try {
const embeddings = await embedBatchWithBackoff(stale.map(c => c.chunk_text), { abortSignal: budgetSignal });
const embeddings = await embedBatchWithBackoff(stale.map(c => c.chunk_text), { abortSignal: effectiveSignal });
// Re-fetch existing chunks and merge to avoid deleting non-stale chunks.
const existing = await engine.getChunks(slug, { sourceId: keySourceId });
const staleIdxToEmbedding = new Map<number, Float32Array>();
@@ -716,9 +743,9 @@ async function embedAllStale(
}
result.embedded += stale.length;
} catch (e: unknown) {
// Budget-fired aborts are expected on the way out; don't spam
// per-page "Error embedding" lines when we're shutting down.
if (budgetSignal.aborted) return;
// Budget/abort-fired cancellations are expected on the way out; don't
// spam per-page "Error embedding" lines when we're shutting down.
if (effectiveSignal.aborted) return;
serr(`\n Error embedding ${slug}: ${e instanceof Error ? e.message : e}`);
}
totalProcessedPages++;
@@ -736,7 +763,7 @@ async function embedAllStale(
await runSlidingPool({
items: keys,
workers: CONCURRENCY,
signal: budgetSignal,
signal: effectiveSignal,
onItem: (key) => embedOneKey(key),
failureLabel: (key) => key,
});
+1 -1
View File
@@ -112,7 +112,7 @@ function formatJobDetail(job: MinionJob): string {
const lines = [
`Job #${job.id}: ${job.name} (${job.status.toUpperCase()}${job.status === 'dead' ? ` after ${job.attempts_made} attempts` : ''})`,
` Queue: ${job.queue} | Priority: ${job.priority}`,
` Attempts: ${job.attempts_made}/${job.max_attempts} (started: ${job.attempts_started})`,
` Attempts: ${job.attempts_made}/${job.max_attempts} (started: ${job.attempts_started}, stalled: ${job.stalled_counter}/${job.max_stalled})`,
` Backoff: ${job.backoff_type} ${job.backoff_delay}ms (jitter: ${job.backoff_jitter})`,
];
if (job.started_at) lines.push(` Started: ${job.started_at.toISOString()}`);
+82
View File
@@ -0,0 +1,82 @@
/**
* abort-check.ts — one canonical place for cooperative-abort checks (#1737).
*
* gbrain has several long-running loops (embed --stale, embed --all, dream
* cycle phases) that each grew their own `signal?.aborted` check. When a job
* is killed by the Minions worker (wall-clock timeout, lock loss, SIGTERM) the
* handler keeps running unless every loop cooperatively checks its signal — and
* a missed loop is exactly the daily cycle-wedge in #1737: the embed phase ran
* to completion ignoring the abort, so `gbrain_cycle_locks` stayed held and
* every later autopilot cycle skipped with `cycle_already_running`.
*
* ┌── worker fires job.signal.abort() ──┐
* │ (wall-clock / lock-loss / SIGTERM) │
* └──────────────┬─────────────────────┘
* ▼
* handler → runPhaseEmbed → runEmbedCore → embedAll(Stale)
* │ │
* └─ throwIfAborted(signal) ─────┘ ← bail here, not 15 min later
* ▼
* finally releases gbrain_cycle_locks → next cycle runs
*
* Two shapes, because the call sites want different control flow:
* - `isAborted(signal)` — boolean; for loops that `break` cleanly and
* return partial progress (embed loops).
* - `throwIfAborted(signal)` — throws an AbortError; for phase boundaries
* that want to unwind to the cycle's finally.
*/
/** True iff the signal exists and has fired. Null/undefined → never aborted. */
export function isAborted(signal?: AbortSignal | null): boolean {
return !!signal?.aborted;
}
/** Error thrown by {@link throwIfAborted}; `name === 'AbortError'`. */
export class AbortError extends Error {
constructor(message = 'aborted') {
super(message);
this.name = 'AbortError';
}
}
/**
* Throw an {@link AbortError} if the signal has fired. The thrown message
* prefers the signal's own `reason` (the worker sets it to the abort cause —
* 'wall-clock', 'lock-lost', 'shutdown') so the unwind is self-describing.
*/
export function throwIfAborted(signal?: AbortSignal | null, label?: string): void {
if (!signal?.aborted) return;
const reason =
signal.reason instanceof Error
? signal.reason.message
: String(signal.reason ?? 'aborted');
throw new AbortError(label ? `${label}: ${reason}` : reason);
}
/**
* Compose an external abort signal with an internal one (e.g. a wall-clock
* budget timer) so a single combined signal fires when EITHER does. Returns
* the internal signal unchanged when there's no external signal, so callers
* that never pass one pay nothing. Uses the platform `AbortSignal.any` (Node
* 20+/Bun) and falls back to a manual relay if it's somehow unavailable.
*/
export function anySignal(
internal: AbortSignal,
external?: AbortSignal | null,
): AbortSignal {
if (!external) return internal;
if (typeof (AbortSignal as { any?: unknown }).any === 'function') {
return (AbortSignal as unknown as { any(s: AbortSignal[]): AbortSignal }).any([
internal,
external,
]);
}
// Fallback relay (older runtimes): forward whichever fires first.
const ac = new AbortController();
const relay = (s: AbortSignal) => ac.abort(s.reason);
if (internal.aborted) relay(internal);
else internal.addEventListener('abort', () => relay(internal), { once: true });
if (external.aborted) relay(external);
else external.addEventListener('abort', () => relay(external), { once: true });
return ac.signal;
}
+7 -3
View File
@@ -1111,10 +1111,14 @@ async function runPhaseResolveSymbolEdges(
}
}
async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean): Promise<PhaseResult> {
async function runPhaseEmbed(engine: BrainEngine, dryRun: boolean, signal?: AbortSignal): Promise<PhaseResult> {
try {
const { runEmbedCore } = await import('../commands/embed.ts');
const result = await runEmbedCore(engine, { stale: true, dryRun });
// #1737: thread the cycle's abort signal so the embed phase (the long,
// 10-15 min one) bails within a batch instead of running to completion
// after the job was killed — which left gbrain_cycle_locks held and
// wedged every subsequent autopilot cycle.
const result = await runEmbedCore(engine, { stale: true, dryRun, signal });
const embeddedCount = dryRun ? result.would_embed : result.embedded;
return {
phase: 'embed',
@@ -2062,7 +2066,7 @@ export async function runCycle(
});
} else {
progress.start('cycle.embed');
const { result, duration_ms } = await timePhase(() => runPhaseEmbed(engine, dryRun));
const { result, duration_ms } = await timePhase(() => runPhaseEmbed(engine, dryRun, opts.signal));
result.duration_ms = duration_ms;
phaseResults.push(result);
progress.finish();
+1
View File
@@ -151,6 +151,7 @@ export const OPS_CHECK_NAMES: ReadonlySet<string> = new Set([
'subagent_health',
'supervisor',
'supervisor_niceness',
'supervisor_singleton',
'sync_consolidation',
'wedged_queue',
'worker_oom_loop',
+45
View File
@@ -0,0 +1,45 @@
/**
* handler-timeouts.ts — per-handler default wall-clock budgets (#1737).
*
* Short jobs (shell, lint, backlinks) want the tight default wall-clock
* (`2 * lockDuration * max_stalled`, computed in `handleWallClockTimeouts`
* when `timeout_ms IS NULL`). Long jobs do not: a 30-min LLM loop or a
* 10-15 min embed backfill submitted WITHOUT an explicit `timeout_ms` would
* inherit that short null-default and get wall-clock-killed mid-progress —
* one half of #1737's thrash.
*
* Fix: known long handlers get a sane long default STAMPED ONTO THE JOB ROW
* at submit time (see `MinionQueue.add`). Stamping at submit (not mutating at
* claim) keeps the wall-clock behavior stable across worker restart — the
* value lives in `minion_jobs.timeout_ms`, not in worker memory.
*
* Existing already-queued jobs are NOT backfilled: they keep whatever
* `timeout_ms` they were inserted with (usually NULL → the old behavior).
* Only NEW submissions pick up the default. An explicit `opts.timeout_ms`
* always wins.
*
* The 30-min anchor matches the explicit value cycle/patterns.ts already
* passes for subagent jobs, so this generalizes an existing convention
* rather than inventing a new number.
*/
const THIRTY_MIN_MS = 30 * 60 * 1000;
/**
* Default wall-clock budget (ms) for long-running handler types. A handler
* not in this map returns `null` → the short null-default wall-clock applies.
*/
export const HANDLER_DEFAULT_TIMEOUT_MS: Readonly<Record<string, number>> = {
subagent: THIRTY_MIN_MS,
subagent_aggregator: THIRTY_MIN_MS,
'embed-backfill': THIRTY_MIN_MS,
'autopilot-cycle': THIRTY_MIN_MS,
};
/**
* The default `timeout_ms` to stamp for a handler when the submitter didn't
* pass one. Returns `null` for short handlers (keep the tight wall-clock).
*/
export function defaultTimeoutMsFor(jobName: string): number | null {
return HANDLER_DEFAULT_TIMEOUT_MS[jobName] ?? null;
}
+7 -1
View File
@@ -16,6 +16,7 @@ import type {
import { rowToMinionJob, rowToInboxMessage, rowToAttachment } from './types.ts';
import { validateAttachment } from './attachments.ts';
import { isProtectedJobName } from './protected-names.ts';
import { defaultTimeoutMsFor } from './handler-timeouts.ts';
import {
withRetry, BULK_RETRY_OPTS, resolveBulkRetryOpts, computeNextDelay,
isRetryableConnError,
@@ -278,7 +279,10 @@ export class MinionQueue {
opts?.on_child_fail ?? 'fail_parent',
depth,
opts?.max_children ?? null,
opts?.timeout_ms ?? null,
// #1737: long handlers (subagent, embed-backfill, autopilot-cycle) get a
// sane long wall-clock default stamped at submit when the caller didn't
// pass one, so they aren't killed mid-progress by the short null-default.
opts?.timeout_ms ?? defaultTimeoutMsFor(jobName),
opts?.remove_on_complete ?? false,
opts?.remove_on_fail ?? false,
opts?.idempotency_key ?? null,
@@ -719,6 +723,7 @@ export class MinionQueue {
`UPDATE minion_jobs SET
status = 'dead',
error_text = 'wall-clock timeout exceeded',
attempts_made = attempts_made + 1,
lock_token = NULL,
lock_until = NULL,
finished_at = now(),
@@ -1155,6 +1160,7 @@ export class MinionQueue {
dead_lettered AS (
UPDATE minion_jobs SET
status = 'dead', stalled_counter = stalled_counter + 1,
attempts_made = attempts_made + 1,
error_text = 'max stalled count exceeded',
lock_token = NULL, lock_until = NULL, finished_at = now(), updated_at = now()
WHERE id IN (SELECT id FROM stalled WHERE stalled_counter + 1 >= max_stalled)
+148 -1
View File
@@ -43,6 +43,8 @@ import {
} from 'fs';
import { dirname } from 'path';
import type { BrainEngine } from '../engine.ts';
import { tryAcquireDbLock, type DbLockHandle } from '../db-lock.ts';
import { currentDbIdentity, currentBrainId } from './worker-registry.ts';
export type SupervisorEvent =
| 'started'
@@ -130,7 +132,15 @@ export const DEFAULT_PID_FILE: string = (() => {
const envOverride = process.env.GBRAIN_SUPERVISOR_PID_FILE;
if (envOverride && envOverride.length > 0) return envOverride;
const home = process.env.HOME ?? '/tmp';
return `${home}/.gbrain/supervisor.pid`;
// #1849: key the default pidfile on the brain id so two DIFFERENT brains
// under one HOME don't share `supervisor.pid` and falsely block each other's
// pidfile guard. Derived from config (no DB connect), so it's safe to
// resolve at module load — `status`/`stop` need a cheap path before the
// engine connects. The queue-scoped DB lock (supervisorLockId) is the real
// singleton authority; this just removes the common-case footgun.
let brainId = 'default';
try { brainId = currentBrainId(); } catch { /* fallback 'default' */ }
return `${home}/.gbrain/supervisor-${brainId}.pid`;
})();
const DEFAULTS: Omit<SupervisorOpts, 'cliPath'> = {
@@ -267,8 +277,62 @@ export const ExitCodes = {
MAX_CRASHES: 1,
LOCK_HELD: 2,
PID_UNWRITABLE: 3,
// #1849: the queue-scoped DB lock was lost mid-run (refresh failed past the
// threshold). Exit non-zero so the process manager restarts us cleanly
// rather than risk two live supervisors on one queue.
LOCK_LOST: 4,
} as const;
/**
* #1849: queue-scoped supervisor singleton DB lock.
*
* The pidfile guard is mutually exclusive only per pidfile PATH — two
* supervisors with different $HOME / --pid-file both acquire and run on the
* same (db, queue) with conflicting --max-rss. The DB lock makes the mutex
* domain match the protected resource (the database + queue), regardless of
* pidfile path. TTL > refresh-interval × max-failures so we always exit
* before our lock could lapse and let a second supervisor take over.
*/
const SUPERVISOR_LOCK_TTL_MIN = 5;
const SUPERVISOR_LOCK_REFRESH_MS = 60_000;
const SUPERVISOR_LOCK_REFRESH_MAX_FAILURES = 3; // 3 × 60s = 180s < 5min TTL
/**
* #1849: the queue-scoped supervisor singleton lock id. Keyed on the raw DB
* identity (T2) + queue so the mutex domain is the (database, queue) pair —
* not the pidfile path. Exported so `gbrain doctor` queries the same row to
* surface the holder + effective --max-rss. Pass an explicit dbIdentity
* (defaults to `currentDbIdentity()`, which reads config without a DB connect).
*/
export function supervisorLockId(queue: string, dbIdentity: string = currentDbIdentity()): string {
return `gbrain-supervisor:${dbIdentity}:${queue}`;
}
/**
* #1849 (doctor): pure classification of the supervisor singleton state from
* the DB lock holder vs the local pidfile holder. Compares host+pid (bare pid
* is meaningless across hosts/containers — Codex #25).
*
* - `no_lock` — no live lock holder (nothing to assert).
* - `single` — the live lock holder IS the local pidfile holder. Healthy.
* - `mismatch` — a live lock holder differs from the local pidfile holder:
* a second supervisor likely ran with a different --max-rss.
*/
export function classifySupervisorSingleton(args: {
lockLive: boolean;
lockHolderHost: string | null;
lockHolderPid: number | null;
localHost: string;
localPid: number | null;
}): 'no_lock' | 'single' | 'mismatch' {
if (!args.lockLive || args.lockHolderHost === null || args.lockHolderPid === null) {
return 'no_lock';
}
if (args.localPid === null) return 'mismatch';
const matches = args.lockHolderHost === args.localHost && args.lockHolderPid === args.localPid;
return matches ? 'single' : 'mismatch';
}
export class MinionSupervisor {
private opts: SupervisorOpts;
private engine: BrainEngine;
@@ -288,6 +352,11 @@ export class MinionSupervisor {
private sigintListener: (() => void) | null = null;
private lockAcquired = false;
private consecutiveHealthFailures = 0;
// #1849: queue-scoped DB singleton lock (the real authority) + its refresh
// timer and consecutive-failure counter (fail-safe exit before TTL lapse).
private dbLock: DbLockHandle | null = null;
private lockRefreshTimer: ReturnType<typeof setInterval> | null = null;
private lockRefreshFailures = 0;
// issue #1801 progress-watchdog state.
/** Job names the spawned worker can actually claim. Derived once at start()
* from registerBuiltinHandlers so the wedge scopes to real claimable work
@@ -426,6 +495,23 @@ export class MinionSupervisor {
process.exit(ExitCodes.PID_UNWRITABLE);
}
// 1b. #1849: queue-scoped DB singleton lock — the REAL authority. A second
// supervisor with a different $HOME / --pid-file passes the pidfile check
// above but loses here, so it can't run a conflicting --max-rss worker on
// the same (db, queue). Keyed on the raw DB identity (not the lossy
// currentBrainId hash) per T2.
this.dbLock = await tryAcquireDbLock(this.engine, this.supervisorLockId(), SUPERVISOR_LOCK_TTL_MIN);
if (!this.dbLock) {
console.error(
`Supervisor already running for queue '${this.opts.queue}' on this database ` +
`(another supervisor holds the queue lock, regardless of pidfile path). Exiting.`,
);
process.exit(ExitCodes.LOCK_HELD);
}
// Refresh the lock on its own timer (independent of healthInterval, which
// can be 0/disabled) so the TTL never lapses while we're alive.
this.lockRefreshTimer = setInterval(() => { void this.refreshDbLock(); }, SUPERVISOR_LOCK_REFRESH_MS);
// 2. Cleanup on process exit (covers any exit path including process.exit).
this.exitListener = () => {
try {
@@ -459,6 +545,9 @@ export class MinionSupervisor {
concurrency: this.opts.concurrency,
queue: this.opts.queue,
max_crashes: this.opts.maxCrashes,
// #1849: record the EFFECTIVE --max-rss so `gbrain doctor` can surface
// the cap a rogue second supervisor would have fought over.
max_rss_mb: this.opts.maxRssMb,
// Niceness (issue #1815): record requested + effective so doctor/status can
// surface a failed renice even for a detached supervisor whose stderr is gone.
...(this.opts.nice_requested !== undefined ? { nice_requested: this.opts.nice_requested } : {}),
@@ -521,6 +610,19 @@ export class MinionSupervisor {
this.healthTimer = null;
}
// #1849: stop refreshing + release the DB singleton lock so a clean
// restart (or a different host) can re-acquire immediately instead of
// waiting out the TTL. release() is best-effort; the TTL covers a crash.
if (this.lockRefreshTimer) {
clearInterval(this.lockRefreshTimer);
this.lockRefreshTimer = null;
}
if (this.dbLock) {
const lock = this.dbLock;
this.dbLock = null;
try { await lock.release(); } catch { /* best-effort; TTL fallback covers it */ }
}
if (this.childSupervisor) {
this.childSupervisor.killChild('SIGTERM');
await this.childSupervisor.awaitChildExit(35_000);
@@ -546,6 +648,51 @@ export class MinionSupervisor {
process.exit(exitCode);
}
/** #1849: the queue-scoped DB lock id for this supervisor's queue. */
private supervisorLockId(): string {
return supervisorLockId(this.opts.queue);
}
/** @internal Test seam: inject a fake DB lock to drive refresh-failure paths. */
_setDbLockForTests(lock: DbLockHandle | null): void {
this.dbLock = lock;
}
/** @internal Test seam: run one refresh tick synchronously. */
async _refreshDbLockForTests(): Promise<void> {
await this.refreshDbLock();
}
/**
* #1849 (F1A): refresh the DB lock; FAIL SAFE on loss. If refresh keeps
* throwing, our TTL will eventually lapse and a second supervisor could take
* over the queue — the exact bug. So past the failure threshold (still well
* inside the TTL window) we stop claiming and exit non-zero; the process
* manager restarts a single clean supervisor. A single transient blip is
* tolerated (counter resets on the next success).
*/
private async refreshDbLock(): Promise<void> {
if (!this.dbLock || this.stopping) return;
try {
await this.dbLock.refresh();
this.lockRefreshFailures = 0;
} catch (e) {
this.lockRefreshFailures++;
this.emit('health_warn', {
reason: 'supervisor_lock_refresh_failed',
consecutive_failures: this.lockRefreshFailures,
error: e instanceof Error ? e.message : String(e),
queue: this.opts.queue,
});
if (this.lockRefreshFailures >= SUPERVISOR_LOCK_REFRESH_MAX_FAILURES) {
this.emit('health_error', {
reason: 'supervisor_lock_lost',
queue: this.opts.queue,
});
await this.shutdown('supervisor_lock_lost', ExitCodes.LOCK_LOST);
}
}
}
/**
* Acquire PID file lock atomically via O_CREAT|O_EXCL.
*
+18
View File
@@ -75,6 +75,24 @@ export function currentBrainId(): string {
}
}
/**
* The RAW database identity string (url or path), unhashed. Used as the
* authoritative key for the #1849 queue-scoped supervisor singleton DB lock:
* the protected resource is the (database, queue) pair, so the lock id must
* key on the real DB identity, not the lossy djb2 of {@link currentBrainId}
* (T2 — removes any hash-collision question). Reads config only (no DB
* connection), so it's safe to call before the engine connects. Falls back
* to 'default' so two unconfigured brains under one HOME still serialize.
*/
export function currentDbIdentity(): string {
try {
const cfg = loadConfig();
return cfg?.database_url ?? cfg?.database_path ?? 'default';
} catch {
return 'default';
}
}
function entryPath(pid: number): string {
return join(workerRegistryDir(), `worker-${pid}.json`);
}
+72
View File
@@ -0,0 +1,72 @@
/**
* Tests for src/core/abort-check.ts (#1737 cooperative-abort helper).
*/
import { describe, test, expect } from 'bun:test';
import { isAborted, throwIfAborted, anySignal, AbortError } from '../src/core/abort-check.ts';
describe('abort-check: isAborted', () => {
test('false for undefined / null / not-yet-fired signal', () => {
expect(isAborted(undefined)).toBe(false);
expect(isAborted(null)).toBe(false);
expect(isAborted(new AbortController().signal)).toBe(false);
});
test('true once the controller aborts', () => {
const ac = new AbortController();
ac.abort();
expect(isAborted(ac.signal)).toBe(true);
});
});
describe('abort-check: throwIfAborted', () => {
test('no-op when signal absent or unfired', () => {
expect(() => throwIfAborted(undefined)).not.toThrow();
expect(() => throwIfAborted(new AbortController().signal)).not.toThrow();
});
test('throws AbortError carrying the signal reason', () => {
const ac = new AbortController();
ac.abort(new Error('wall-clock'));
try {
throwIfAborted(ac.signal, '[cycle]');
throw new Error('should have thrown');
} catch (e) {
expect(e).toBeInstanceOf(AbortError);
expect((e as Error).name).toBe('AbortError');
expect((e as Error).message).toBe('[cycle]: wall-clock');
}
});
});
describe('abort-check: anySignal', () => {
test('returns the internal signal unchanged when no external signal', () => {
const internal = new AbortController().signal;
expect(anySignal(internal, undefined)).toBe(internal);
expect(anySignal(internal, null)).toBe(internal);
});
test('combined signal fires when the EXTERNAL signal fires', () => {
const internal = new AbortController();
const external = new AbortController();
const combined = anySignal(internal.signal, external.signal);
expect(combined.aborted).toBe(false);
external.abort();
expect(combined.aborted).toBe(true);
});
test('combined signal fires when the INTERNAL signal fires', () => {
const internal = new AbortController();
const external = new AbortController();
const combined = anySignal(internal.signal, external.signal);
internal.abort();
expect(combined.aborted).toBe(true);
});
test('combined is already aborted if external was pre-aborted', () => {
const internal = new AbortController();
const external = new AbortController();
external.abort();
const combined = anySignal(internal.signal, external.signal);
expect(combined.aborted).toBe(true);
});
});
+8 -6
View File
@@ -89,13 +89,15 @@ describe('embed.ts → worker-pool migration (T3)', () => {
expect(callSites?.length ?? 0).toBeGreaterThanOrEqual(2);
});
test('embedAllStale path still threads budgetSignal into pool', () => {
// The pre-migration code checked `!budgetSignal.aborted` in the
// worker loop. The migration moves that check into the helper via
// the `signal` option. If a future refactor drops the signal, the
// wall-clock budget cancellation regresses.
test('embedAllStale path still threads the cancellation signal into pool', () => {
// The pre-migration code checked `!budgetSignal.aborted` in the worker
// loop. The migration moves that check into the helper via the `signal`
// option. #1737 then composed the wall-clock budget with the caller's
// abort into `effectiveSignal` (anySignal(budgetSignal, externalSignal)) so
// a killed job stops the pool too. If a future refactor drops the signal,
// both wall-clock budget AND cooperative abort cancellation regress.
expect(EMBED_SOURCE).toMatch(
/runSlidingPool\(\s*\{[\s\S]*?signal:\s*budgetSignal[\s\S]*?\}\)/,
/runSlidingPool\(\s*\{[\s\S]*?signal:\s*effectiveSignal[\s\S]*?\}\)/,
);
});
+45 -1
View File
@@ -40,7 +40,7 @@ mock.module('../src/core/embedding.ts', () => ({
}));
// Import AFTER mocking.
const { runEmbed } = await import('../src/commands/embed.ts');
const { runEmbed, runEmbedCore } = await import('../src/commands/embed.ts');
// v0.41.6.0 D1: runEmbedCore now preflights embedding credentials. This
// test stack uses the LEGACY embedBatch mock path, not the gateway,
@@ -133,6 +133,50 @@ describe('runEmbed --all (parallel)', () => {
expect(stampCalls[0].args[1]).toEqual({ sourceId: 'default', signature: 'test:model:1536' });
});
// #1737: cooperative abort. A pre-aborted signal must stop the embed loop
// BEFORE any embedBatch call, so a job killed by wall-clock/lock-loss frees
// the worker (and lets the cycle's finally release gbrain_cycle_locks)
// instead of grinding through the full 10-15 min embed phase.
test('#1737 --all: pre-aborted signal embeds nothing (no embedBatch call)', async () => {
const pages = Array.from({ length: 10 }, (_, i) => ({ slug: `page-${i}`, source_id: 'default' }));
const chunksBySlug = new Map(
pages.map(p => [
p.slug,
[{ chunk_index: 0, chunk_text: `text ${p.slug}`, chunk_source: 'compiled_truth', embedded_at: null, token_count: 4 }],
]),
);
const engine = mockEngine({
listPages: async () => pages,
getChunks: async (slug: string) => chunksBySlug.get(slug) || [],
upsertChunks: async () => {},
});
const ac = new AbortController();
ac.abort(new Error('wall-clock'));
const result = await runEmbedCore(engine, { all: true, signal: ac.signal });
expect(totalEmbedCalls).toBe(0);
expect(result.embedded).toBe(0);
});
test('#1737 --stale: pre-aborted signal breaks the loop before listStaleChunks', async () => {
let listStaleCalls = 0;
const engine = mockEngine({
countStaleChunks: async () => 5, // non-zero so we pass the early return
listStaleChunks: async () => { listStaleCalls++; return []; },
invalidateStaleSignatureEmbeddings: async () => 0,
});
const ac = new AbortController();
ac.abort(new Error('lock-lost'));
const result = await runEmbedCore(engine, { stale: true, signal: ac.signal });
// The top-of-loop abort check fires before the first listStaleChunks page load.
expect(listStaleCalls).toBe(0);
expect(totalEmbedCalls).toBe(0);
expect(result.embedded).toBe(0);
});
test('respects GBRAIN_EMBED_CONCURRENCY=1 (serial)', async () => {
const pages = Array.from({ length: 5 }, (_, i) => ({ slug: `page-${i}` }));
const chunksBySlug = new Map(
+8
View File
@@ -23,9 +23,17 @@ import type { BrainEngine } from '../../src/core/engine.ts';
// Mock engine: healthCheck() calls engine.executeRaw; return empty rows so
// the query path exercises without needing Postgres.
//
// #1849: start() now acquires the queue-scoped DB singleton lock via
// tryAcquireDbLock, which uses the postgres `sql` tagged-template escape hatch.
// The stub returns a single row from every call so acquire succeeds (length 1
// → acquired) and refresh/release are no-ops. Each spawned runner is a fresh
// process, so there's no cross-test lock state to clean up.
const sqlStub = (..._args: unknown[]) => Promise.resolve([{ id: 'supervisor-lock' }]);
const mockEngine: Partial<BrainEngine> = {
kind: 'postgres' as const,
executeRaw: async () => [],
sql: sqlStub,
} as unknown as BrainEngine;
const pidFile = process.env.SUP_PID_FILE;
+95
View File
@@ -270,6 +270,101 @@ describe('MinionQueue: Stall Detection', () => {
});
});
// --- #1737 — honest attempt accounting on terminal dead-letter paths ---
describe('MinionQueue: #1737 attempt accounting on dead-letter', () => {
test('wall-clock dead-letter increments attempts_made (no more 0/N (started:M))', async () => {
const job = await queue.add('sync', {}, { max_attempts: 2 });
// Per-job timeout so the wall-clock threshold is timeout_ms * 2 = 2s.
await engine.executeRaw('UPDATE minion_jobs SET timeout_ms = 1000 WHERE id = $1', [job.id]);
await queue.claim('tok1', 30000, 'default', ['sync']);
expect((await queue.getJob(job.id))!.attempts_made).toBe(0); // bug repro: started but not made
// Force cumulative wall-clock past timeout_ms * 2.
await engine.executeRaw(
"UPDATE minion_jobs SET started_at = now() - interval '10 seconds' WHERE id = $1",
[job.id]
);
const dead = await queue.handleWallClockTimeouts(30000);
expect(dead.length).toBe(1);
expect(dead[0].status).toBe('dead');
expect(dead[0].error_text).toBe('wall-clock timeout exceeded');
// The fix: a job killed by wall-clock consumed an attempt.
expect(dead[0].attempts_made).toBe(1);
// Constraint chk_attempts_order (attempts_made <= attempts_started) holds.
expect(dead[0].attempts_made).toBeLessThanOrEqual(dead[0].attempts_started);
});
test('wall-clock dead-letter is terminal — does NOT retry even with attempts remaining', async () => {
const job = await queue.add('sync', {}, { max_attempts: 5 });
await engine.executeRaw('UPDATE minion_jobs SET timeout_ms = 1000 WHERE id = $1', [job.id]);
await queue.claim('tok1', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET started_at = now() - interval '10 seconds' WHERE id = $1",
[job.id]
);
const dead = await queue.handleWallClockTimeouts(30000);
expect(dead.length).toBe(1);
// Even with 4 attempts left, wall-clock is terminal (non-idempotent handler safety).
expect(dead[0].status).toBe('dead');
expect(dead[0].status).not.toBe('delayed');
});
test('stall dead-letter increments attempts_made; requeue does NOT', async () => {
const job = await queue.add('sync', {}, { max_attempts: 3 });
await engine.executeRaw('UPDATE minion_jobs SET max_stalled = 2 WHERE id = $1', [job.id]);
// First stall: requeued, attempts_made stays 0 (lease-loss recovery, not an app attempt).
await queue.claim('tok1', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
[job.id]
);
const r1 = await queue.handleStalled();
expect(r1.requeued.length).toBe(1);
expect(r1.requeued[0].attempts_made).toBe(0);
// Second stall: dead-lettered, attempts_made now increments.
await queue.claim('tok2', 30000, 'default', ['sync']);
await engine.executeRaw(
"UPDATE minion_jobs SET lock_until = now() - interval '1 second' WHERE id = $1",
[job.id]
);
const r2 = await queue.handleStalled();
expect(r2.dead.length).toBe(1);
expect(r2.dead[0].error_text).toBe('max stalled count exceeded');
expect(r2.dead[0].attempts_made).toBe(1);
expect(r2.dead[0].attempts_made).toBeLessThanOrEqual(r2.dead[0].attempts_started);
});
});
// --- #1737 — per-handler default wall-clock budget at submit ---
describe('MinionQueue: #1737 per-handler default timeout', () => {
test('long handler with no explicit timeout_ms gets the 30-min default stamped', async () => {
const job = await queue.add('embed-backfill', { sourceId: 'x' });
expect(job.timeout_ms).toBe(30 * 60 * 1000);
});
test('autopilot-cycle + subagent also get the long default', async () => {
const cycle = await queue.add('autopilot-cycle', {});
// subagent is a protected name → needs the trusted-submit flag (4th arg).
const sub = await queue.add('subagent', {}, undefined, { allowProtectedSubmit: true });
expect(cycle.timeout_ms).toBe(30 * 60 * 1000);
expect(sub.timeout_ms).toBe(30 * 60 * 1000);
});
test('explicit timeout_ms always wins over the default', async () => {
const job = await queue.add('embed-backfill', { sourceId: 'x' }, { timeout_ms: 5000 });
expect(job.timeout_ms).toBe(5000);
});
test('short handler keeps null timeout_ms (tight wall-clock default applies)', async () => {
const job = await queue.add('sync', {});
expect(job.timeout_ms).toBeNull();
});
});
// --- v0.13.1 #219 — max_stalled default + input surface ---
describe('MinionQueue: v0.13.1 max_stalled schema default (#219)', () => {
+158
View File
@@ -0,0 +1,158 @@
/**
* #1849: queue-scoped DB supervisor singleton.
*
* The pidfile guard is mutually exclusive only per pidfile PATH; the DB lock
* makes the (database, queue) pair the mutex domain so two supervisors with
* different $HOME / --pid-file can't both run on one queue. These tests pin:
* - the lock id keys on DB identity + queue (T2)
* - a second acquire of the same (db, queue) lock is refused (the singleton)
* - different queues don't collide
* - refresh-failure past the threshold fails SAFE (exits non-zero) (F1A)
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach, spyOn } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { tryAcquireDbLock } from '../src/core/db-lock.ts';
import { MinionSupervisor, ExitCodes, supervisorLockId, classifySupervisorSingleton } from '../src/core/minions/supervisor.ts';
import type { DbLockHandle } from '../src/core/db-lock.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id LIKE 'gbrain-supervisor:%'`);
});
describe('#1849 supervisorLockId', () => {
test('keys on DB identity AND queue', () => {
expect(supervisorLockId('default', 'postgres://x')).toBe('gbrain-supervisor:postgres://x:default');
expect(supervisorLockId('shell', 'postgres://x')).toBe('gbrain-supervisor:postgres://x:shell');
// Different DB identity → different lock even for the same queue.
expect(supervisorLockId('default', 'postgres://a'))
.not.toBe(supervisorLockId('default', 'postgres://b'));
});
});
describe('#1849 classifySupervisorSingleton (doctor)', () => {
test('no live lock → no_lock', () => {
expect(classifySupervisorSingleton({
lockLive: false, lockHolderHost: 'h', lockHolderPid: 1, localHost: 'h', localPid: 1,
})).toBe('no_lock');
});
test('live lock held by the local (host,pid) → single', () => {
expect(classifySupervisorSingleton({
lockLive: true, lockHolderHost: 'box', lockHolderPid: 42, localHost: 'box', localPid: 42,
})).toBe('single');
});
test('live lock held by a DIFFERENT pid → mismatch (rogue second supervisor)', () => {
expect(classifySupervisorSingleton({
lockLive: true, lockHolderHost: 'box', lockHolderPid: 99, localHost: 'box', localPid: 42,
})).toBe('mismatch');
});
test('same pid but DIFFERENT host → mismatch (bare pid is meaningless cross-host)', () => {
expect(classifySupervisorSingleton({
lockLive: true, lockHolderHost: 'other', lockHolderPid: 42, localHost: 'box', localPid: 42,
})).toBe('mismatch');
});
test('live lock but no local pidfile → mismatch', () => {
expect(classifySupervisorSingleton({
lockLive: true, lockHolderHost: 'box', lockHolderPid: 42, localHost: 'box', localPid: null,
})).toBe('mismatch');
});
});
describe('#1849 DB lock is the real singleton', () => {
test('second acquire of the same (db, queue) lock is refused', async () => {
const id = supervisorLockId('default', 'pglite:test');
const first = await tryAcquireDbLock(engine, id, 5);
expect(first).not.toBeNull();
// A second supervisor (different pidfile, same db+queue) gets null → exit 2.
const second = await tryAcquireDbLock(engine, id, 5);
expect(second).toBeNull();
// After release, a fresh supervisor can take over.
await first!.release();
const third = await tryAcquireDbLock(engine, id, 5);
expect(third).not.toBeNull();
await third!.release();
});
test('different queues on the same DB do not collide', async () => {
const a = await tryAcquireDbLock(engine, supervisorLockId('default', 'pglite:test'), 5);
const b = await tryAcquireDbLock(engine, supervisorLockId('shell', 'pglite:test'), 5);
expect(a).not.toBeNull();
expect(b).not.toBeNull();
await a!.release();
await b!.release();
});
});
describe('#1849 refresh-failure fails safe (F1A)', () => {
test('exits LOCK_LOST after the failure threshold; tolerates a single blip', async () => {
const sup = new MinionSupervisor(engine, { cliPath: '/bin/sh', healthInterval: 0, json: true });
const exitSpy = spyOn(process, 'exit').mockImplementation(((_code?: number) => {
throw new Error(`exit:${_code}`); // stop execution like the real exit would
}) as never);
let refreshCalls = 0;
const failingLock: DbLockHandle = {
id: 'x',
refresh: async () => { refreshCalls++; throw new Error('pooler down'); },
release: async () => {},
};
sup._setDbLockForTests(failingLock);
try {
// First two failures: tolerated (counter climbs, no exit).
await sup._refreshDbLockForTests();
await sup._refreshDbLockForTests();
expect(exitSpy).not.toHaveBeenCalled();
// Third failure crosses the threshold → shutdown → process.exit(LOCK_LOST).
try { await sup._refreshDbLockForTests(); } catch { /* exit stub throws */ }
expect(exitSpy).toHaveBeenCalledWith(ExitCodes.LOCK_LOST);
expect(refreshCalls).toBe(3);
} finally {
exitSpy.mockRestore();
}
});
test('a successful refresh resets the failure counter', async () => {
const sup = new MinionSupervisor(engine, { cliPath: '/bin/sh', healthInterval: 0, json: true });
const exitSpy = spyOn(process, 'exit').mockImplementation(((_code?: number) => {
throw new Error(`exit:${_code}`);
}) as never);
let mode: 'fail' | 'ok' = 'fail';
const flakyLock: DbLockHandle = {
id: 'x',
refresh: async () => { if (mode === 'fail') throw new Error('blip'); },
release: async () => {},
};
sup._setDbLockForTests(flakyLock);
try {
await sup._refreshDbLockForTests(); // fail 1
await sup._refreshDbLockForTests(); // fail 2
mode = 'ok';
await sup._refreshDbLockForTests(); // success → reset
mode = 'fail';
await sup._refreshDbLockForTests(); // fail 1 again
await sup._refreshDbLockForTests(); // fail 2
// Counter was reset, so we are NOT past threshold yet.
expect(exitSpy).not.toHaveBeenCalled();
} finally {
exitSpy.mockRestore();
}
});
});