mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
v0.42.22.0 fix(minions): supervisor progress watchdog + worker DB self-defense — alive-but-wedged worker self-heals (#1801) (#1824)
* fix(minions): supervisor progress watchdog + worker DB self-defense under supervision (#1801) Alive-but-wedged worker (dead DB pool, process still up) now self-heals in minutes instead of a silent 15h halt. - supervisor: progress watchdog restarts a child that makes no forward progress on claimable work (name+queue-scoped, active_healthy/due-delayed aware, startup-grace + loop-budget bounded); runtime handler-name derivation. - child-worker-supervisor: killChild gates on liveness not .killed (also fixes the existing shutdown SIGKILL no-op); restartCurrentChild kills the captured child ref; intentional restart doesn't count toward max_crashes. - worker: DB-liveness probe runs under supervision (db_dead self-exit), stall detection stays supervised-off. - doctor: standalone per-queue wedged_queue check + state->status fix in the remote queue_health check. - jobs/queue: queue-scoped getStats wedge fields + jobs stats WEDGED line. * fix(minions): wedge_restart_loop one-shot + supervised-probe comment + jobs-stats threshold (review) Pre-landing adversarial review findings: - wedge_restart_loop warn now fires once per exhausted window via a re-arming flag, not every health tick (was flooding the audit log for the full window). - Correct the stale GBRAIN_SUPERVISED comment: the DB probe runs under supervision now; only stall detection is skipped. - jobs stats WEDGED line reads GBRAIN_WEDGED_QUEUE_WARN_MINUTES so it agrees with the doctor wedged_queue threshold. * chore: bump version and changelog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: queue-ops runbook + KEY_FILES for the #1801 wedge watchdog (v0.42.22.0) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
f3ade6c0c3
commit
f4959348c2
@@ -2,6 +2,69 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.42.22.0] - 2026-06-03
|
||||
|
||||
**A background worker whose database connection quietly dies no longer sits there alive-but-doing-nothing for hours. Your brain keeps processing jobs instead of silently stalling overnight.**
|
||||
|
||||
Here's the failure this fixes. You run the job supervisor (`gbrain jobs supervisor`), which babysits a worker process that chews through your queue: syncs, embeds, the nightly cycle. Behind a connection pooler (the common Supabase setup), the worker's database connection can get dropped and never come back. The worker process stays *running* the whole time. It just can't talk to the database anymore, so it claims no jobs and finishes nothing. Jobs pile up. Nothing crashes, nothing alerts. One brain sat like this for about 15 hours: 57 jobs waiting, zero being worked, the supervisor logging the same "no recent completions" line once a minute and doing nothing about it. The only fix was noticing by hand and killing the whole process tree so a fresh one could start with a working connection.
|
||||
|
||||
The reason every safety net missed it: they all check whether the process is *alive*, and it was. A worker that's running but wedged passes every liveness check, every `ps`, every container health probe. What nobody was checking was whether it's making *forward progress*.
|
||||
|
||||
This release adds that check, in two independent layers so one covers the other:
|
||||
|
||||
- **The worker now notices its own dead connection.** It already had a "can I reach the database?" heartbeat, but that heartbeat was switched off whenever a supervisor was watching (on the theory the supervisor had it covered). It didn't. Now the heartbeat runs under supervision too: a worker whose pool is dead exits on its own within about three minutes, and the supervisor restarts it with a fresh connection.
|
||||
- **The supervisor now watches forward progress, not just liveness.** If a queue has work the worker can handle, nothing is actively being worked, and nothing has completed for 15 minutes while the worker claims to be alive, the supervisor treats the worker as wedged and restarts it. This catches every cause of a stall, not just dead connections (a genuinely stuck job handler, a deadlock, anything).
|
||||
|
||||
And the stall is now loud instead of buried. `gbrain jobs stats` prints a `WEDGED QUEUE` line, and `gbrain doctor` reports a `wedged_queue` health error with the fix command, so you catch it in the daily check instead of 15 hours later.
|
||||
|
||||
### How to use it
|
||||
|
||||
Nothing to configure. `gbrain upgrade`, restart your supervisor, done. The watchdog is on by default with conservative thresholds. If you want to tune it:
|
||||
|
||||
```
|
||||
# minutes of no-forward-progress before the supervisor restarts a wedged worker (default 15; 0 disables)
|
||||
gbrain jobs supervisor --wedge-restart-minutes 15
|
||||
# consecutive checks that must agree before acting (default 3)
|
||||
gbrain jobs supervisor --wedge-restart-checks 3
|
||||
```
|
||||
|
||||
### What you'd see
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| Worker pool dies, process stays up | sits idle forever (15h observed) | self-exits in ~3 min, supervisor respawns with a fresh pool |
|
||||
| A job handler genuinely hangs | invisible to the supervisor | restarted after 15 min of no progress |
|
||||
| You check `gbrain jobs stats` | "57 waiting, 0 active" with no flag | loud `WEDGED QUEUE` line + fix command |
|
||||
| `gbrain doctor` | silent (the remote check was even querying the wrong column) | `wedged_queue` health error, grouped per queue |
|
||||
|
||||
### What we caught before shipping
|
||||
|
||||
An independent review of the plan found real bugs in the first draft that would have let the fix itself fail in the same way it was meant to prevent:
|
||||
|
||||
- The supervisor's existing "force kill" path was a silent no-op. It guarded on "did we already send a signal" instead of "is the process still alive," so the follow-up `SIGKILL` after an ignored `SIGTERM` never actually fired. That bug was already present in the existing shutdown path; it's fixed here too.
|
||||
- A worker that died mid-job leaves a stale "active" row behind. The first draft would have read that stale row as "something's being worked" and suppressed the restart forever. The watchdog now only counts jobs holding a live lock.
|
||||
- The restart is now accounted as a deliberate self-heal, so a recurring wedge can't slowly burn through the crash budget and take the whole supervisor down. After a few futile restarts in a window it stops restarting and just alerts, because at that point a restart clearly isn't the fix.
|
||||
|
||||
This release coexists with the v0.42.16.0 doctor self-heal work (OOM-loop detection, pool-reap health); the two cover different failure modes and run side by side.
|
||||
|
||||
## To take advantage of v0.42.22.0
|
||||
|
||||
`gbrain upgrade`, then restart your job supervisor so the new watchdog is in effect:
|
||||
|
||||
```
|
||||
gbrain jobs supervisor stop && gbrain jobs supervisor start
|
||||
```
|
||||
|
||||
Verify: `gbrain doctor` should show a `wedged_queue` check (it reads `ok` on a healthy queue). If you ever see it go to a health error, the fix is the same stop/start above, plus `gbrain jobs retry <id>` on any dead-lettered jobs.
|
||||
|
||||
### For contributors
|
||||
- `child-worker-supervisor.ts`: `killChild` now gates on `exitCode/signalCode === null` (liveness) instead of `.killed` (the v0.42.16-era no-op bug, also fixing the `shutdown()` drain); new `restartCurrentChild(graceMs)` captures the child ref and SIGTERM→grace→SIGKILLs *that* ref (never the respawn); a new `wedge_restart` cause flows through the exit classifier and `supervisor-audit.ts` `CLEAN_EXIT_CAUSES` so it's not counted as a crash.
|
||||
- `supervisor.ts`: progress watchdog in `healthCheck()` + new exported `queryWedgeSignals(engine, queue, handlerNames)` (name+queue-scoped, `active_healthy` = live-lock only, due-delayed counted); runtime handler-name derivation via a throwaway `registerBuiltinHandlers` worker (new `quiet` opt) so the wedge scopes to actually-claimable names with zero duplicated constant; startup-grace + `wedgeRestartLoopBudget` knobs.
|
||||
- `worker.ts`: DB-liveness probe un-gated under `GBRAIN_SUPERVISED`; stall detection stays supervised-off.
|
||||
- `doctor.ts`: standalone per-queue `wedged_queue` check (local + remote) + `state`→`status` fix on the remote `queue_health` SQL (it errored every run and silently returned "No queue activity").
|
||||
- `queue.ts`/`jobs.ts`: queue-scoped `getStats` wedge block + `jobs stats` WEDGED line.
|
||||
- Tests: `supervisor-wedge`, `worker-supervised-db-probe`, `doctor-wedged-queue`, `queue-getstats-wedge`, plus `child-worker-supervisor` additions (behavioral restart coverage + PGLite SQL semantics for every reviewed edge case + structural regression guards). Plan went through eng review + a Codex outside-voice pass; all 16 findings folded in.
|
||||
|
||||
## [0.42.21.0] - 2026-06-02
|
||||
|
||||
**Your nightly `gbrain dream` stops silently losing every database phase.** If you run gbrain on Postgres (local or Supabase), the dream cycle has been quietly failing: the `lint` and `backlinks` phases work, then `sync`, `synthesize`, `embed`, and the rest all blow up with `No database connection: connect() has not been called`. The extract phase reports "created 0 links" while actually dropping every row. Run the same phases one at a time in separate commands and they all work — only the full cycle breaks. The result: your brain quietly stops staying up to date, and the cycle leaves a stuck lock behind.
|
||||
|
||||
@@ -210,9 +210,9 @@ per-release `**vX.Y.Z:**` narration — CI enforces this
|
||||
- `src/core/zombie-reap.ts` — idempotent `installSigchldHandler()` so JS-spawned children get reaped via Bun's internal `waitpid()`. Bun (like Node) only auto-reaps when a SIGCHLD listener is registered; without it, every child the worker spawns (shell jobs, embed batches, sub-agents) becomes a zombie on exit and holds connection slots. Called once at module load from `src/cli.ts` (Windows platform guard — SIGCHLD doesn't exist on Windows). Cross-file leak guard via `_uninstallSigchldHandlerForTests()`. Layer 1 of the three-layer zombie defense; Layer 2 is tini-as-PID-1 wrapping the worker subtree (via `src/core/minions/spawn-helpers.ts`); Layer 3 is the container's own tini for hard Bun crashes.
|
||||
- `src/core/minions/` — Minions job queue: BullMQ-inspired, Postgres-native (queue, worker, backoff, types, protected-names, quiet-hours, stagger, handlers/shell).
|
||||
- `src/core/minions/queue.ts` — MinionQueue class (submit, claim, complete, fail, stall detection, parent-child, depth/child-cap, per-job timeouts, cascade-kill, attachments, idempotency keys, child_done inbox, removeOnComplete/Fail). `add()` takes a 4th `trusted` arg (separate from `opts` to prevent spread leakage); protected names in `PROTECTED_JOB_NAMES` require `{allowProtectedSubmit: true}` and the check runs trim-normalized (whitespace-bypass safe). `add()` plumbs `max_stalled` through with a `[1, 100]` clamp; omitted values let the schema DEFAULT (5) kick in. `handleWallClockTimeouts(lockDurationMs)` is Layer 3 kill shot for jobs where `FOR UPDATE SKIP LOCKED` stall detection and the timeout sweep both fail to evict (wedged worker holding a row lock via a pending transaction). The `maxWaiting` coalesce path uses `pg_advisory_xact_lock` keyed on `(name, queue)` to serialize concurrent submits for the same key, and filters on `queue` in addition to `name` so cross-queue same-name jobs don't suppress each other.
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). Aborted jobs call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`); `shutdownAbort` (instance field) fires on SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` (shell handler listens; non-shell handlers don't). Per-job timeout fires `abort.abort(new Error('timeout'))` then a 30s grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead if the handler ignores the abort signal (the moved-out generic abort listener now fires for ANY abort reason). The `launchJob` lock-renewal block is a thin sync wrapper around the pure `runLockRenewalTick` from `src/core/minions/lock-renewal-tick.ts` (NEVER `setInterval(async () => await renewLock(...))` — that shape was the unhandledRejection crash class during PgBouncer rotation). Gaps closed: (1) `cancelled` flag captured in the timer closure stops in-flight IIFEs writing misleading audit events after the job ended; (2) re-entrancy guard `tickInFlight` + per-call `Promise.race` timeout (`GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS`, default `lockDuration/3`); (3) time-based abort (`Date.now() - lastSuccessfulRenewalAt >= lockDuration - GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS`) so we release the lock BEFORE another worker can reclaim; (4) explicit `.catch()` on the stored `executeJob(...).finally(...)` promise closes the second unhandledRejection vector; (5) exported `INFRASTRUCTURE_ABORT_REASONS = new Set(['lock-renewal-failed', 'lock-lost'])` so executeJob's catch skips `failJob` for these (PgBouncer blips don't dead-letter healthy jobs; the stall detector reclaims). CI guard `scripts/check-worker-lock-renewal-shape.sh` (in `bun run verify`) asserts the bug pattern stays absent AND `launchJob` keeps calling `runLockRenewalTick`. Engine-ownership invariant: `start()` does NOT call `engine.disconnect()` on shutdown — the CLI handler in `src/commands/jobs.ts case 'work'` owns engine lifecycle via try/finally with loud error logging. RSS watchdog uses non-file-backed pages on Linux: exported `parseRssFromProcStatus(status)` (pure parser; field-presence regex so `RssAnon: 0 + RssShmem: 512` parses correctly) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5); the default `getRss` in `WorkerOpts` is `getAccurateRss`. `checkMemoryLimit` tracks peak RSS, fires an 80%-of-cap soft-warn (once per crossing, carrying peak + in-flight job kinds), and on exceed sets `_rssWatchdogTriggered=true` (exposed via `get rssWatchdogTriggered()`) so `jobs work`'s finally can `process.exit(WORKER_EXIT_RSS_WATCHDOG)` after disconnect (drain self-identifying instead of an opaque code-0 exit). The poll loop wraps `claim` in try/catch: on a retryable conn error it reconnects ONCE and continues to the next tick rather than blind-retrying (a retry after `UPDATE...RETURNING` committed but the socket died would double-claim). `LockRenewalDeps` is wired with `reconnect` when the engine supports it. Pinned by `test/worker-lock-renewal.test.ts` (18), `test/audit/lock-renewal-audit.test.ts` (11), `test/scripts/check-worker-lock-renewal-shape.test.ts` (5), `test/worker-shutdown-disconnect.test.ts` (asserts `disconnectSpy).not.toHaveBeenCalled()`), `test/worker-rss.test.ts` (11).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets. Worker exit classifier emits `likely_cause` on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. Consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping the in-process SIGCHLD reaper can't reach); exposes `isTiniDetected` read-only accessor. The spawn-and-respawn loop is the shared `ChildWorkerSupervisor` core: MinionSupervisor composes it via `runSuperviseLoop()` → `new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` back through `emit()` SupervisorEvent (JSONL audit consumers see byte-compatible output). PID lock, signal handlers, health check, and `process.exit` on max-crashes stay in MinionSupervisor. `code=0` leaves `crashCount` untouched (so a worker alternating real crashes + watchdog drains still trips `max_crashes`); `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop via `health_warn { reason: 'clean_restart_budget_exceeded' }` + backoff. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)`. Pinned by `test/supervisor.test.ts` (16 cases) and `test/supervisor-tini.test.ts`.
|
||||
- `src/core/minions/child-worker-supervisor.ts` — shared spawn-and-respawn core reused by both `MinionSupervisor` (standalone `gbrain jobs supervisor` daemon) and `src/commands/autopilot.ts` (autopilot daemon), so the two consumers can't drift into parallel-loop bugs. Pure class: NO PID file, NO signal handlers, NO `process.exit`, NO health check. Lifecycle events fire via injected `onEvent: (ChildSupervisorEvent) => void`. Exit classifier: `code === 0` leaves `crashCount` UNCHANGED (preserves flap detection across mixed exit sequences); `code != 0` follows `runDuration > stableRunResetMs ? 1 : ++crashCount`. Clean-restart budget: sliding window of code=0 exits; when count exceeds `cleanRestartBudget` (default 10) inside `cleanRestartWindowMs` (default 60s), emits `health_warn { reason: 'clean_restart_budget_exceeded' }` and applies `cleanRestartBudgetBackoffMs` (default 1s). The exit classifier special-cases `WORKER_EXIT_RSS_WATCHDOG` — `likely_cause='rss_watchdog'`, and that exit does NOT bump `crashCount` (routes to its own breaker); a dedicated `_watchdogExitTimestamps` sliding window trips a loud `rss_watchdog_loop` health_warn naming the cap when N watchdog exits land inside the window, INDEPENDENT of the stable-run reset (which would otherwise hide a >5-min-run watchdog drain loop). New opts `watchdogLoopBudget` (3), `watchdogLoopWindowMs` (600000), `watchdogBackoffMs` (30000); `ChildSupervisorEvent` extended. Public read-only accessors `childAlive`, `inBackoff`, `crashCount`; `killChild(signal)` + `awaitChildExit(timeoutMs)` for shutdown (the latter short-circuits when `child.exitCode !== null || child.signalCode !== null` so fast-SIGTERM responders don't cause a 35s shutdown hang). Test hooks `_backoffFloorMs`, `_now`. `supervisor-audit.ts` adds `rss_watchdog` as a non-clean cause + its own `CrashSummary.by_cause` bucket (denylist preserved so future causes route to `legacy`). Pinned by `test/child-worker-supervisor.test.ts` (7 cases).
|
||||
- `src/core/minions/worker.ts` — MinionWorker class (handler registry, lock renewal, graceful shutdown, timeout safety net). Aborted jobs call `failJob` with reason (`timeout`/`cancel`/`lock-lost`/`shutdown`); `shutdownAbort` (instance field) fires on SIGTERM/SIGINT and propagates to `ctx.shutdownSignal` (shell handler listens; non-shell handlers don't). Per-job timeout fires `abort.abort(new Error('timeout'))` then a 30s grace-then-evict safety net force-evicts the job from `inFlight` and marks it dead if the handler ignores the abort signal (the moved-out generic abort listener now fires for ANY abort reason). The `launchJob` lock-renewal block is a thin sync wrapper around the pure `runLockRenewalTick` from `src/core/minions/lock-renewal-tick.ts` (NEVER `setInterval(async () => await renewLock(...))` — that shape was the unhandledRejection crash class during PgBouncer rotation). Gaps closed: (1) `cancelled` flag captured in the timer closure stops in-flight IIFEs writing misleading audit events after the job ended; (2) re-entrancy guard `tickInFlight` + per-call `Promise.race` timeout (`GBRAIN_LOCK_RENEWAL_CALL_TIMEOUT_MS`, default `lockDuration/3`); (3) time-based abort (`Date.now() - lastSuccessfulRenewalAt >= lockDuration - GBRAIN_LOCK_RENEWAL_SAFETY_MARGIN_MS`) so we release the lock BEFORE another worker can reclaim; (4) explicit `.catch()` on the stored `executeJob(...).finally(...)` promise closes the second unhandledRejection vector; (5) exported `INFRASTRUCTURE_ABORT_REASONS = new Set(['lock-renewal-failed', 'lock-lost'])` so executeJob's catch skips `failJob` for these (PgBouncer blips don't dead-letter healthy jobs; the stall detector reclaims). CI guard `scripts/check-worker-lock-renewal-shape.sh` (in `bun run verify`) asserts the bug pattern stays absent AND `launchJob` keeps calling `runLockRenewalTick`. Engine-ownership invariant: `start()` does NOT call `engine.disconnect()` on shutdown — the CLI handler in `src/commands/jobs.ts case 'work'` owns engine lifecycle via try/finally with loud error logging. RSS watchdog uses non-file-backed pages on Linux: exported `parseRssFromProcStatus(status)` (pure parser; field-presence regex so `RssAnon: 0 + RssShmem: 512` parses correctly) and `getAccurateRss(readStatus?)` (reads `/proc/self/status` for `RssAnon + RssShmem`, falls back to `process.memoryUsage().rss` on macOS / restricted containers / kernel <4.5); the default `getRss` in `WorkerOpts` is `getAccurateRss`. `checkMemoryLimit` tracks peak RSS, fires an 80%-of-cap soft-warn (once per crossing, carrying peak + in-flight job kinds), and on exceed sets `_rssWatchdogTriggered=true` (exposed via `get rssWatchdogTriggered()`) so `jobs work`'s finally can `process.exit(WORKER_EXIT_RSS_WATCHDOG)` after disconnect (drain self-identifying instead of an opaque code-0 exit). The poll loop wraps `claim` in try/catch: on a retryable conn error it reconnects ONCE and continues to the next tick rather than blind-retrying (a retry after `UPDATE...RETURNING` committed but the socket died would double-claim). `LockRenewalDeps` is wired with `reconnect` when the engine supports it. The self-health DB-liveness probe now runs EVEN under a supervisor (`GBRAIN_SUPERVISED=1`): the outer guard is `if (this.opts.healthCheckInterval > 0)` and only the STALL-detection block is wrapped in `if (!isSupervisedChild)` — so a supervised worker whose own pool dies self-exits `unhealthy(db_dead)` after `dbFailExitAfter` probes (the supervisor watches a different connection and can't see this worker's dead pool), while the supervisor's progress watchdog owns forward-progress (#1801). Pinned by `test/worker-lock-renewal.test.ts` (18), `test/audit/lock-renewal-audit.test.ts` (11), `test/scripts/check-worker-lock-renewal-shape.test.ts` (5), `test/worker-shutdown-disconnect.test.ts` (asserts `disconnectSpy).not.toHaveBeenCalled()`), `test/worker-rss.test.ts` (11), `test/worker-supervised-db-probe.test.ts` (3).
|
||||
- `src/core/minions/supervisor.ts` — MinionSupervisor process manager. Spawns `gbrain jobs work` as a child, restarts on crash with exponential backoff, periodic health check. `consecutiveHealthFailures` counter; on 3 consecutive failures emits `health_warn` with `reason: 'db_connection_degraded'` and calls `engine.reconnect()` to swap in a fresh pool, then resets. Worker exit classifier emits `likely_cause` on `worker_exited` events: `oom_or_external_kill` (SIGKILL), `graceful_shutdown` (SIGTERM), `runtime_error` (code 1), `clean_exit` (code 0), `unknown`. Consumes `detectTini()` + `buildSpawnInvocation()` from `src/core/minions/spawn-helpers.ts` to wrap the worker subtree in tini-as-PID-1 when tini is on `PATH` (handles native-addon zombie reaping the in-process SIGCHLD reaper can't reach); exposes `isTiniDetected` read-only accessor. The spawn-and-respawn loop is the shared `ChildWorkerSupervisor` core: MinionSupervisor composes it via `runSuperviseLoop()` → `new ChildWorkerSupervisor({...})` and maps `ChildSupervisorEvent` back through `emit()` SupervisorEvent (JSONL audit consumers see byte-compatible output). PID lock, signal handlers, health check, and `process.exit` on max-crashes stay in MinionSupervisor. `code=0` leaves `crashCount` untouched (so a worker alternating real crashes + watchdog drains still trips `max_crashes`); `cleanRestartBudget` (default 10 restarts per 60s) caps the macOS/non-Linux-fallback tight-loop via `health_warn { reason: 'clean_restart_budget_exceeded' }` + backoff. `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)`. Progress watchdog (#1801): `healthCheck()` restarts an alive-but-wedged child via `childSupervisor.restartCurrentChild(35_000)` when a queue has claimable work, 0 live-lock active jobs, and stale completions across `wedgeRestartChecks` (default 3) consecutive checks past `wedgeRestartMinutes` (default 15, 0 disables) + a `startupGraceMs` window; bounded by `wedgeRestartLoopBudget` (default 3 / `wedgeRestartLoopWindowMs`) which switches to a one-shot `wedge_restart_loop` alert. The wedge query is the exported `queryWedgeSignals(engine, queue, handlerNames)` — name+queue-scoped, `active_healthy` = live-lock only (an expired-lock active row does NOT mask the wedge), due-delayed counted. Claimable names are derived at start via a throwaway `registerBuiltinHandlers` worker (its new `quiet` opt). Flags `--wedge-restart-minutes` / `--wedge-restart-checks` + env `GBRAIN_WEDGE_RESTART_MINUTES` / `GBRAIN_WEDGE_RESTART_CHECKS`. Pinned by `test/supervisor.test.ts` (16 cases), `test/supervisor-tini.test.ts`, and `test/supervisor-wedge.test.ts`.
|
||||
- `src/core/minions/child-worker-supervisor.ts` — shared spawn-and-respawn core reused by both `MinionSupervisor` (standalone `gbrain jobs supervisor` daemon) and `src/commands/autopilot.ts` (autopilot daemon), so the two consumers can't drift into parallel-loop bugs. Pure class: NO PID file, NO signal handlers, NO `process.exit`, NO health check. Lifecycle events fire via injected `onEvent: (ChildSupervisorEvent) => void`. Exit classifier: `code === 0` leaves `crashCount` UNCHANGED (preserves flap detection across mixed exit sequences); `code != 0` follows `runDuration > stableRunResetMs ? 1 : ++crashCount`. Clean-restart budget: sliding window of code=0 exits; when count exceeds `cleanRestartBudget` (default 10) inside `cleanRestartWindowMs` (default 60s), emits `health_warn { reason: 'clean_restart_budget_exceeded' }` and applies `cleanRestartBudgetBackoffMs` (default 1s). The exit classifier special-cases `WORKER_EXIT_RSS_WATCHDOG` — `likely_cause='rss_watchdog'`, and that exit does NOT bump `crashCount` (routes to its own breaker); a dedicated `_watchdogExitTimestamps` sliding window trips a loud `rss_watchdog_loop` health_warn naming the cap when N watchdog exits land inside the window, INDEPENDENT of the stable-run reset (which would otherwise hide a >5-min-run watchdog drain loop). New opts `watchdogLoopBudget` (3), `watchdogLoopWindowMs` (600000), `watchdogBackoffMs` (30000); `ChildSupervisorEvent` extended. Public read-only accessors `childAlive`, `inBackoff`, `crashCount`; `killChild(signal)` gates on liveness (`exitCode === null && signalCode === null`), NOT `.killed` — `.killed` flips true once a signal is *sent*, so the old `!this._child.killed` guard made a follow-up SIGKILL after an ignored SIGTERM a silent no-op (#1801; the bug also lived in the `shutdown()` drain). `restartCurrentChild(graceMs)` (#1801 wedge self-heal) captures the CURRENT child ref, SIGTERM→grace→SIGKILLs THAT ref (never the respawn — closes the timer-kills-fresh-worker race), and flags `_intentionalRestart` so the exit is `likelyCause='wedge_restart'`, leaves `crashCount` UNTOUCHED (never trips `max_crashes`; like `rss_watchdog`), and respawns immediately (`backoff ms:0 reason='wedge_restart'`). `awaitChildExit(timeoutMs)` short-circuits when `child.exitCode !== null || child.signalCode !== null` so fast-SIGTERM responders don't cause a 35s shutdown hang. Test hooks `_backoffFloorMs`, `_now`. `supervisor-audit.ts` adds `rss_watchdog` as a non-clean cause + its own `CrashSummary.by_cause` bucket, and `wedge_restart` to `CLEAN_EXIT_CAUSES` (a self-heal, not a crash; denylist preserved so future causes route to `legacy`). Pinned by `test/child-worker-supervisor.test.ts` (12 cases).
|
||||
- `src/core/minions/spawn-helpers.ts` — pure `detectTini()` + `buildSpawnInvocation()` consumed by both `supervisor.ts` and `autopilot.ts` (resolves the DRY violation between the two spawn sites and makes tini wrapping testable without `mock.module()`, rule R2 of `scripts/check-test-isolation.sh`). `detectTini()` calls `execFileSync('which', ['tini'])` with explicit `env: process.env` so Bun sees runtime PATH mutations. `buildSpawnInvocation(tiniPath, cmd, args)` returns `{cmd, args}` with tini prepended when present, or the bare invocation otherwise. Pinned by `test/spawn-helpers.test.ts` (5) and `test/supervisor-tini.test.ts` (4).
|
||||
- `src/core/minions/types.ts` — `MinionJobInput` + `MinionJobStatus` + handler context types. `MinionJobInput.max_stalled` is optional; omitted values let the schema DEFAULT (5) kick in, provided values are clamped to `[1, 100]`.
|
||||
- `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules.
|
||||
|
||||
@@ -16,6 +16,39 @@ gbrain doctor --json | jq '.checks[] | select(.name == "queue_health")'
|
||||
- **waiting-depth**: any per-name queue deeper than 10 (override via
|
||||
`GBRAIN_QUEUE_WAITING_THRESHOLD`). Signals a missing `maxWaiting`.
|
||||
|
||||
## The worker is alive but wedged (dead pool)
|
||||
|
||||
The nastiest stall: the worker process is *running* (passes `ps` / `kill -0` /
|
||||
container health), but its DB connection died (common behind a transaction
|
||||
pooler) and never came back, so it claims no jobs and finishes nothing. Jobs
|
||||
pile up with **0 active**. Liveness checks all pass; nothing crashes.
|
||||
|
||||
As of v0.42.22.0 this self-heals — you usually won't have to do anything:
|
||||
|
||||
- **The worker exits on its own dead pool.** Under a supervisor, the worker's
|
||||
DB-liveness probe runs and self-exits (`db_dead`) after ~3 minutes; the
|
||||
supervisor respawns it with a fresh pool.
|
||||
- **The supervisor restarts a worker that stops making progress.** If a queue
|
||||
has claimable work, **0 live-lock active jobs**, and no completions for 15
|
||||
minutes while the child is alive, the supervisor restarts it (covers stuck
|
||||
handlers too, not just dead pools). Tune with `--wedge-restart-minutes` /
|
||||
`--wedge-restart-checks` on `gbrain jobs supervisor` (0 disables).
|
||||
|
||||
The signal is loud now — check either:
|
||||
|
||||
```bash
|
||||
gbrain jobs stats --queue default # prints a WEDGED QUEUE line
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "wedged_queue")'
|
||||
```
|
||||
|
||||
`wedged_queue` is a per-queue health **error** (0 active_healthy + waiting > 0 +
|
||||
stale completions). Manual fix if you ever need it:
|
||||
|
||||
```bash
|
||||
gbrain jobs supervisor stop && gbrain jobs supervisor start # fresh pool
|
||||
gbrain jobs retry <id> # dead-lettered jobs
|
||||
```
|
||||
|
||||
## Triage commands
|
||||
|
||||
```bash
|
||||
|
||||
+1
-1
@@ -143,5 +143,5 @@
|
||||
"bun": ">=1.3.10"
|
||||
},
|
||||
"license": "MIT",
|
||||
"version": "0.42.21.0"
|
||||
"version": "0.42.22.0"
|
||||
}
|
||||
|
||||
+83
-1
@@ -642,9 +642,12 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
// shape; skip the check there with an informational message.
|
||||
if (engine.kind === 'postgres') {
|
||||
try {
|
||||
// issue #1801: column is `status`, not `state` (schema.sql:780). The
|
||||
// pre-fix query errored every run and the catch silently returned "No
|
||||
// queue activity," so this remote/thin-client check was a no-op.
|
||||
const rows = await engine.executeRaw<{ stalled: string | number }>(
|
||||
`SELECT COUNT(*) AS stalled FROM minion_jobs
|
||||
WHERE state = 'active'
|
||||
WHERE status = 'active'
|
||||
AND started_at IS NOT NULL
|
||||
AND started_at < NOW() - INTERVAL '1 hour'`,
|
||||
);
|
||||
@@ -663,6 +666,9 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
|
||||
checks.push({ name: 'queue_health', status: 'ok', message: 'PGLite — no queue to check' });
|
||||
}
|
||||
|
||||
// issue #1801 — wedged_queue (cross-surface parity with buildChecks).
|
||||
checks.push(await computeWedgedQueueCheck(engine));
|
||||
|
||||
// v0.41 Bug 2 / Eng D8 — subagent_health surfaces rate-lease pressure to the operator.
|
||||
checks.push(await checkSubagentHealth(engine));
|
||||
|
||||
@@ -1459,6 +1465,78 @@ export async function checkRerankerHealth(engine: BrainEngine): Promise<Check> {
|
||||
* Also surfaces (codex M-10): runs resolveBulkRetryOpts(process.env) at
|
||||
* startup so bad GBRAIN_BULK_* config fails at doctor time, not first-retry.
|
||||
*/
|
||||
/**
|
||||
* issue #1801 — `wedged_queue` check. Surfaces the alive-but-wedged-worker
|
||||
* signature (a queue with claimable work waiting, zero live-lock active jobs,
|
||||
* and stale completions) as a health ERROR, so an operator / the daily doctor
|
||||
* catches a silent processing halt in minutes, not 15 hours.
|
||||
*
|
||||
* Postgres-only (PGLite has no multi-process worker surface). Grouped BY queue
|
||||
* (Codex #15) so a healthy worker on one queue can't mask a wedged one.
|
||||
* `active_healthy` counts only live-lock active rows, so an expired-lock active
|
||||
* row (a worker that died mid-job) does NOT mask the wedge (Codex #6). The
|
||||
* check is conservative for the advisory surface: it fails only on stale-after-
|
||||
* progress (mins_since_completion > threshold, non-null); a queue that never
|
||||
* completed anything is left to the supervisor's startup-grace-aware watchdog
|
||||
* to avoid crying wolf on a freshly-submitted queue with no worker yet.
|
||||
*
|
||||
* Exported so `test/doctor.test.ts` drives it directly. Reads
|
||||
* GBRAIN_WEDGED_QUEUE_WARN_MINUTES (default 15).
|
||||
*/
|
||||
export async function computeWedgedQueueCheck(engine: BrainEngine): Promise<Check> {
|
||||
if (engine.kind !== 'postgres') {
|
||||
return { name: 'wedged_queue', status: 'ok', message: 'PGLite — no queue to check' };
|
||||
}
|
||||
const thresholdMin = _resolveEnvNumber('GBRAIN_WEDGED_QUEUE_WARN_MINUTES', 15);
|
||||
try {
|
||||
const rows = await engine.executeRaw<{
|
||||
queue: string;
|
||||
active_healthy: string | number;
|
||||
waiting: string | number;
|
||||
mins_since_completion: string | number | null;
|
||||
}>(
|
||||
`SELECT queue,
|
||||
count(*) FILTER (WHERE status = 'active' AND lock_until > now()) AS active_healthy,
|
||||
count(*) FILTER (WHERE status = 'waiting') AS waiting,
|
||||
EXTRACT(EPOCH FROM (now() - max(updated_at) FILTER (WHERE status = 'completed'))) / 60
|
||||
AS mins_since_completion
|
||||
FROM minion_jobs
|
||||
GROUP BY queue`,
|
||||
);
|
||||
const wedged: string[] = [];
|
||||
for (const r of rows) {
|
||||
const activeHealthy = Number(r.active_healthy ?? 0);
|
||||
const waiting = Number(r.waiting ?? 0);
|
||||
const mins = r.mins_since_completion === null ? null : Number(r.mins_since_completion);
|
||||
// Conservative: only flag stale-after-progress (non-null mins past
|
||||
// threshold). The null-completions case is the supervisor's job.
|
||||
if (activeHealthy === 0 && waiting > 0 && mins !== null && mins > thresholdMin) {
|
||||
wedged.push(`'${r.queue}' (${waiting} waiting, 0 active, ${Math.round(mins)}m since last completion)`);
|
||||
}
|
||||
}
|
||||
if (wedged.length === 0) {
|
||||
return { name: 'wedged_queue', status: 'ok', message: 'No wedged queues' };
|
||||
}
|
||||
return {
|
||||
name: 'wedged_queue',
|
||||
status: 'fail',
|
||||
message:
|
||||
`Wedged queue(s) — worker alive but not claiming work: ${wedged.join('; ')}. ` +
|
||||
`Restart the worker so it rebuilds a fresh DB pool: ` +
|
||||
`\`gbrain jobs supervisor stop && gbrain jobs supervisor start\`, ` +
|
||||
`then \`gbrain jobs retry <id>\` on any dead-lettered jobs.`,
|
||||
details: { wedged_queues: wedged.length, threshold_minutes: thresholdMin },
|
||||
};
|
||||
} catch (e) {
|
||||
// Pre-migration brains / transient errors: advisory check stays ok.
|
||||
return {
|
||||
name: 'wedged_queue',
|
||||
status: 'ok',
|
||||
message: `Skipped (${e instanceof Error ? e.message : String(e)})`,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export async function checkBatchRetryHealth(_engine: BrainEngine): Promise<Check> {
|
||||
try {
|
||||
// Codex M-10: surface bad env config at doctor time.
|
||||
@@ -6709,6 +6787,10 @@ export async function buildChecks(
|
||||
// surfacing via the batch-retry audit JSONL. Codex H-9 thresholds.
|
||||
progress.heartbeat('batch_retry_health');
|
||||
checks.push(await checkBatchRetryHealth(engine));
|
||||
// issue #1801 wedged_queue — alive-but-wedged worker (claimable work
|
||||
// waiting, zero live-lock active, stale completions) as a health error.
|
||||
progress.heartbeat('wedged_queue');
|
||||
checks.push(await computeWedgedQueueCheck(engine));
|
||||
// v0.40.4 graph_signals_coverage — global inbound-link density when
|
||||
// graph_signals is enabled in the active mode bundle.
|
||||
progress.heartbeat('graph_signals_coverage');
|
||||
|
||||
+50
-8
@@ -533,7 +533,8 @@ HANDLER TYPES (built in)
|
||||
try { await queue.ensureSchema(); }
|
||||
catch (e) { console.error(e instanceof Error ? e.message : String(e)); process.exit(1); }
|
||||
|
||||
const stats = await queue.getStats();
|
||||
const statsQueue = parseFlag(args, '--queue') ?? 'default';
|
||||
const stats = await queue.getStats({ queue: statsQueue });
|
||||
|
||||
console.log('Job Stats (last 24h):');
|
||||
if (stats.by_type.length > 0) {
|
||||
@@ -547,6 +548,31 @@ HANDLER TYPES (built in)
|
||||
}
|
||||
console.log(`\n Queue health: ${stats.queue_health.waiting} waiting, ${stats.queue_health.active} active, ${stats.queue_health.stalled} stalled`);
|
||||
|
||||
// issue #1801 — wedged-queue signature (queue-scoped): a worker is alive
|
||||
// but claiming nothing while work waits. `active_healthy` (live-lock only)
|
||||
// means an expired-lock active row doesn't mask it. Loud line so the
|
||||
// operator/agent catches a silent halt in `jobs stats`, not 15h later.
|
||||
{
|
||||
const w = stats.wedge;
|
||||
const mins = w.minutes_since_completion;
|
||||
// Same threshold the doctor `wedged_queue` check uses, so the two
|
||||
// advisory surfaces agree (issue #1801).
|
||||
const wedgeMins = (() => {
|
||||
const raw = parseInt(process.env.GBRAIN_WEDGED_QUEUE_WARN_MINUTES ?? '', 10);
|
||||
return Number.isFinite(raw) && raw > 0 ? raw : 15;
|
||||
})();
|
||||
const wedged = w.active_healthy === 0 && w.waiting > 0 && (mins === null || mins > wedgeMins);
|
||||
if (wedged) {
|
||||
const since = mins === null ? 'no completions on record' : `${mins}m since last completion`;
|
||||
console.log(
|
||||
`\n ⚠ WEDGED QUEUE '${w.queue}': ${w.waiting} waiting, 0 active (live-lock), ${since}.\n` +
|
||||
` A worker may be alive but stuck (dead DB pool / stuck handler). Fix:\n` +
|
||||
` gbrain jobs supervisor stop && gbrain jobs supervisor start # rebuild a fresh pool\n` +
|
||||
` gbrain jobs retry <id> # for dead-lettered jobs`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// v0.41 Bug 2 / Eng D8 — surface lease pressure to the operator.
|
||||
// Reads minion_lease_pressure_log windowed at 1h. Best-effort: pre-v93
|
||||
// brains (no table) silently skip; the queue_health line above is the
|
||||
@@ -849,8 +875,12 @@ HANDLER TYPES (built in)
|
||||
watchdogNote = `, watchdog: ${maxRssMb}MB (auto-sized from ${Math.round(d.basisMb / 1024)}GB ${d.source} RAM)`;
|
||||
}
|
||||
}
|
||||
const healthNote = !isSupervisedChild && healthCheckInterval > 0
|
||||
? `, health-check: ${Math.round(healthCheckInterval / 1000)}s`
|
||||
// issue #1801 (fix #2): the DB-liveness probe runs under supervision too;
|
||||
// only stall detection is supervised-off. Report accordingly.
|
||||
const healthNote = healthCheckInterval > 0
|
||||
? (isSupervisedChild
|
||||
? `, db-probe: ${Math.round(healthCheckInterval / 1000)}s`
|
||||
: `, health-check: ${Math.round(healthCheckInterval / 1000)}s`)
|
||||
: '';
|
||||
console.log(`Minion worker started (queue: ${queueName}, concurrency: ${concurrency}${watchdogNote}${healthNote})`);
|
||||
console.log(`Registered handlers: ${worker.registeredNames.join(', ')}`);
|
||||
@@ -1130,7 +1160,17 @@ HANDLER TYPES (built in)
|
||||
*
|
||||
* Per the v0.11.1 plan (Codex architecture #5 — tension 3).
|
||||
*/
|
||||
export async function registerBuiltinHandlers(worker: MinionWorker, engine: BrainEngine): Promise<void> {
|
||||
export async function registerBuiltinHandlers(
|
||||
worker: MinionWorker,
|
||||
engine: BrainEngine,
|
||||
opts?: { quiet?: boolean },
|
||||
): Promise<void> {
|
||||
// `quiet` suppresses the informational startup stderr lines. The supervisor
|
||||
// (issue #1801) runs this against a throwaway worker purely to read
|
||||
// `registeredNames` for wedge name-scoping — it must not spam the operator's
|
||||
// terminal with "shell handler registered…" lines. The real `jobs work` path
|
||||
// omits opts and prints as before.
|
||||
const quiet = opts?.quiet === true;
|
||||
worker.register('sync', async (job) => {
|
||||
const { performSync } = await import('./sync.ts');
|
||||
const repoPath = typeof job.data.repoPath === 'string' ? job.data.repoPath : undefined;
|
||||
@@ -1512,10 +1552,12 @@ export async function registerBuiltinHandlers(worker: MinionWorker, engine: Brai
|
||||
{
|
||||
const { shellHandler } = await import('../core/minions/handlers/shell.ts');
|
||||
worker.register('shell', shellHandler);
|
||||
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
|
||||
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
|
||||
} else {
|
||||
process.stderr.write('[minion worker] shell handler registered in guarded mode (set GBRAIN_ALLOW_SHELL_JOBS=1 to execute shell jobs)\n');
|
||||
if (!quiet) {
|
||||
if (process.env.GBRAIN_ALLOW_SHELL_JOBS === '1') {
|
||||
process.stderr.write('[minion worker] shell handler enabled (GBRAIN_ALLOW_SHELL_JOBS=1)\n');
|
||||
} else {
|
||||
process.stderr.write('[minion worker] shell handler registered in guarded mode (set GBRAIN_ALLOW_SHELL_JOBS=1 to execute shell jobs)\n');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -151,6 +151,7 @@ export const OPS_CHECK_NAMES: ReadonlySet<string> = new Set([
|
||||
'subagent_health',
|
||||
'supervisor',
|
||||
'sync_consolidation',
|
||||
'wedged_queue',
|
||||
'worker_oom_loop',
|
||||
'ze_embedding_health',
|
||||
]);
|
||||
|
||||
@@ -57,7 +57,7 @@ export type ChildSupervisorEvent =
|
||||
kind: 'backoff';
|
||||
ms: number;
|
||||
crashCount: number;
|
||||
reason: 'clean_exit' | 'crash' | 'budget_exceeded' | 'rss_watchdog';
|
||||
reason: 'clean_exit' | 'crash' | 'budget_exceeded' | 'rss_watchdog' | 'wedge_restart';
|
||||
}
|
||||
| {
|
||||
kind: 'health_warn';
|
||||
@@ -152,6 +152,12 @@ export class ChildWorkerSupervisor {
|
||||
private _child: ChildProcess | null = null;
|
||||
private _inBackoff = false;
|
||||
private _lastStartTime = 0;
|
||||
/** issue #1801: set by restartCurrentChild() before a deliberate wedge
|
||||
* SIGTERM so the exit handler skips crash accounting. Cleared on exit. */
|
||||
private _intentionalRestart = false;
|
||||
/** issue #1801: carried from the exit handler to applyBackoff so a wedge
|
||||
* self-heal respawns immediately (ms:0) instead of paying crash backoff. */
|
||||
private _lastWasIntentionalRestart = false;
|
||||
|
||||
constructor(opts: ChildWorkerSupervisorOpts) {
|
||||
this.opts = opts;
|
||||
@@ -178,9 +184,17 @@ export class ChildWorkerSupervisor {
|
||||
* shutdown paths. Idempotent — `kill('SIGTERM')` on a dead child is a no-op.
|
||||
*/
|
||||
killChild(signal: NodeJS.Signals): void {
|
||||
if (this._child && !this._child.killed) {
|
||||
// Gate on LIVENESS, not `.killed`. Node's `child.killed` flips true the
|
||||
// moment a signal has been *sent*, not when the process exits — so a guard
|
||||
// of `!this._child.killed` makes a follow-up SIGKILL (after an ignored
|
||||
// SIGTERM) a silent no-op. That bug bit both the wedge escalation AND the
|
||||
// existing shutdown() drain (issue #1801, Codex #1). Checking exitCode /
|
||||
// signalCode === null means "still running," so SIGKILL actually lands on a
|
||||
// process that ignored SIGTERM.
|
||||
const child = this._child;
|
||||
if (child && child.exitCode === null && child.signalCode === null) {
|
||||
try {
|
||||
this._child.kill(signal);
|
||||
child.kill(signal);
|
||||
} catch {
|
||||
/* already dead */
|
||||
}
|
||||
@@ -200,7 +214,17 @@ export class ChildWorkerSupervisor {
|
||||
*/
|
||||
awaitChildExit(timeoutMs: number): Promise<void> {
|
||||
if (!this._child) return Promise.resolve();
|
||||
const child = this._child;
|
||||
return this._awaitExit(this._child, timeoutMs);
|
||||
}
|
||||
|
||||
/**
|
||||
* awaitChildExit, bound to a SPECIFIC captured child reference rather than
|
||||
* `this._child`. The wedge-restart path (issue #1801) must wait on the child
|
||||
* it SIGTERMed, not whatever `this._child` points at later — by the time the
|
||||
* grace window elapses, `run()` may have respawned a fresh child, and a
|
||||
* `this._child`-based wait/kill would hit the replacement (Codex #2).
|
||||
*/
|
||||
private _awaitExit(child: ChildProcess, timeoutMs: number): Promise<void> {
|
||||
// Already exited? `exitCode` becomes non-null once Node has seen the
|
||||
// child terminate. `signalCode` is the symmetric flag for kill-signal
|
||||
// termination — checked too so a SIGKILLed child also short-circuits.
|
||||
@@ -224,6 +248,42 @@ export class ChildWorkerSupervisor {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Intentional restart of the CURRENT child — the supervisor's wedge watchdog
|
||||
* (issue #1801) calls this when a worker is alive but making no forward
|
||||
* progress (dead pool, stuck handler). Captures the live child reference,
|
||||
* SIGTERMs it, waits up to `graceMs` on THAT captured ref, then SIGKILLs THAT
|
||||
* ref if it ignored SIGTERM. Operating on the captured reference (never
|
||||
* `this._child`) closes the race where `run()` respawns a fresh child during
|
||||
* the grace window (Codex #2). The exit is flagged intentional so it does NOT
|
||||
* count toward crashCount / max_crashes (Codex #3) — a deliberate self-heal,
|
||||
* not a worker defect; the caller bounds repetition via its wedge-restart
|
||||
* loop budget. `run()` respawns a fresh worker (and a fresh DB pool) on exit.
|
||||
*/
|
||||
async restartCurrentChild(graceMs: number): Promise<void> {
|
||||
const child = this._child;
|
||||
if (!child) return;
|
||||
this._intentionalRestart = true;
|
||||
// SIGTERM the captured child (liveness-gated, same fix as killChild).
|
||||
if (child.exitCode === null && child.signalCode === null) {
|
||||
try {
|
||||
child.kill('SIGTERM');
|
||||
} catch {
|
||||
/* already dead */
|
||||
}
|
||||
}
|
||||
await this._awaitExit(child, graceMs);
|
||||
// Escalate to SIGKILL only if the CAPTURED child is still alive — never the
|
||||
// respawned one.
|
||||
if (child.exitCode === null && child.signalCode === null) {
|
||||
try {
|
||||
child.kill('SIGKILL');
|
||||
} catch {
|
||||
/* already dead */
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the spawn-and-respawn loop. Resolves when:
|
||||
* 1. composer.isStopping() returns true, OR
|
||||
@@ -318,7 +378,16 @@ export class ChildWorkerSupervisor {
|
||||
// through the shared `classifyWorkerExit` helper so doctor.ts and
|
||||
// jobs.ts (audit-log consumers) read the same rule.
|
||||
this._lastExitCode = code;
|
||||
if (code === WORKER_EXIT_RSS_WATCHDOG) {
|
||||
if (this._intentionalRestart) {
|
||||
// issue #1801: deliberate wedge self-heal (restartCurrentChild). Leave
|
||||
// crashCount UNTOUCHED — like the RSS watchdog — so a recurring wedge
|
||||
// never trips max_crashes and kills the daemon. A SIGTERM exit has
|
||||
// code=null, which classifyWorkerExit() (correctly) treats as a crash,
|
||||
// so without this guard the wedge restart would increment crashCount
|
||||
// (Codex #3). The supervisor's wedgeRestartLoopBudget bounds repetition.
|
||||
this._intentionalRestart = false;
|
||||
this._lastWasIntentionalRestart = true;
|
||||
} else if (code === WORKER_EXIT_RSS_WATCHDOG) {
|
||||
// issue #1678: RSS-watchdog drain. NOT a code defect — leave
|
||||
// crashCount untouched so it never trips max_crashes (which would
|
||||
// stop ALL job processing). Tracked in its own window so the
|
||||
@@ -351,7 +420,13 @@ export class ChildWorkerSupervisor {
|
||||
|
||||
// Likely-cause heuristic, kept verbatim from MinionSupervisor.
|
||||
let likelyCause: string;
|
||||
if (signal === 'SIGKILL') {
|
||||
if (this._lastWasIntentionalRestart) {
|
||||
// issue #1801: a deliberate wedge restart. Label it as such even
|
||||
// though the kill signal was SIGTERM/SIGKILL, so the audit summary
|
||||
// (supervisor-audit.ts CLEAN_EXIT_CAUSES) counts it as a self-heal,
|
||||
// not a crash.
|
||||
likelyCause = 'wedge_restart';
|
||||
} else if (signal === 'SIGKILL') {
|
||||
likelyCause = 'oom_or_external_kill';
|
||||
} else if (signal === 'SIGTERM') {
|
||||
likelyCause = 'graceful_shutdown';
|
||||
@@ -381,6 +456,20 @@ export class ChildWorkerSupervisor {
|
||||
|
||||
/** Compute and apply backoff based on the most recent exit classifier. */
|
||||
private async applyBackoff(): Promise<void> {
|
||||
if (this._lastWasIntentionalRestart) {
|
||||
// issue #1801: a deliberate wedge restart. Respawn immediately (a fresh
|
||||
// pool is the whole point of the self-heal) — no crash backoff, no
|
||||
// budget accounting (the supervisor's wedgeRestartLoopBudget owns that).
|
||||
this._lastWasIntentionalRestart = false;
|
||||
this.opts.onEvent({
|
||||
kind: 'backoff',
|
||||
ms: 0,
|
||||
crashCount: this._crashCount,
|
||||
reason: 'wedge_restart',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (this._lastExitCode === 0) {
|
||||
// D2: check the clean-restart budget. If exceeded, emit health_warn
|
||||
// and apply a fixed cooldown so the next spawn isn't instant. This
|
||||
|
||||
@@ -148,13 +148,17 @@ export function readRecentSupervisorEvents(
|
||||
* Denylist of clean-exit `likely_cause` values. Anything not in this set —
|
||||
* including future unrecognized values — counts as a crash. Matches the
|
||||
* domain asymmetry: clean exits are explicit (the worker exited because we
|
||||
* asked it to); crashes are an open catch-all. If a future maintainer adds a
|
||||
* asked it to); crashes are an open catch-all. `wedge_restart` (issue #1801)
|
||||
* is a deliberate supervisor self-heal of an alive-but-wedged worker — counted
|
||||
* as clean here so doctor / `jobs supervisor status` don't inflate the crash
|
||||
* count on self-heals; the wedge itself surfaces via the
|
||||
* `restarting_wedged_worker` / `wedge_restart_loop` health_warn instead. If a future maintainer adds a
|
||||
* new `likely_cause` upstream in `child-worker-supervisor.ts` (e.g.
|
||||
* `lock_lost`, `panic`), the doctor surfaces it by default instead of
|
||||
* silently underreporting — denylist semantics close the bug class this
|
||||
* helper was added to fix.
|
||||
*/
|
||||
const CLEAN_EXIT_CAUSES = new Set(['clean_exit', 'graceful_shutdown']);
|
||||
const CLEAN_EXIT_CAUSES = new Set(['clean_exit', 'graceful_shutdown', 'wedge_restart']);
|
||||
|
||||
/**
|
||||
* Per-cause crash bucket shape returned by `summarizeCrashes()`. Bucket names
|
||||
|
||||
@@ -498,12 +498,28 @@ export class MinionQueue {
|
||||
}
|
||||
|
||||
/** Get job statistics. */
|
||||
async getStats(opts?: { since?: Date }): Promise<{
|
||||
async getStats(opts?: { since?: Date; queue?: string }): Promise<{
|
||||
by_status: Record<string, number>;
|
||||
by_type: Array<{ name: string; total: number; completed: number; failed: number; dead: number; avg_duration_ms: number | null }>;
|
||||
queue_health: { waiting: number; active: number; stalled: number };
|
||||
/**
|
||||
* issue #1801 — QUEUE-SCOPED wedge signature for the `jobs stats` WEDGED
|
||||
* line. by_status/by_type/queue_health above stay GLOBAL (dashboard
|
||||
* overview); this block is scoped to one queue (default 'default') because
|
||||
* a wedge is per-queue — a healthy worker on one queue must not mask a
|
||||
* wedged one (Codex #14/#15). `active_healthy` counts only live-lock active
|
||||
* rows so an expired-lock row (worker died mid-job) does NOT mask the wedge.
|
||||
*/
|
||||
wedge: {
|
||||
queue: string;
|
||||
active_healthy: number;
|
||||
waiting: number;
|
||||
last_completed_at: string | null;
|
||||
minutes_since_completion: number | null;
|
||||
};
|
||||
}> {
|
||||
const since = opts?.since ?? new Date(Date.now() - 86400000);
|
||||
const wedgeQueue = opts?.queue ?? 'default';
|
||||
|
||||
// Status counts
|
||||
const statusRows = await this.engine.executeRaw<{ status: string; count: string }>(
|
||||
@@ -539,6 +555,23 @@ export class MinionQueue {
|
||||
);
|
||||
const stalled = parseInt(stalledRows[0]?.count ?? '0', 10);
|
||||
|
||||
// issue #1801 — queue-scoped wedge signature (one query, one queue).
|
||||
const wedgeRows = await this.engine.executeRaw<{
|
||||
active_healthy: string;
|
||||
waiting: string;
|
||||
last_completed: string | null;
|
||||
}>(
|
||||
`SELECT
|
||||
count(*) FILTER (WHERE status = 'active' AND lock_until > now())::text AS active_healthy,
|
||||
count(*) FILTER (WHERE status = 'waiting')::text AS waiting,
|
||||
max(updated_at) FILTER (WHERE status = 'completed')::text AS last_completed
|
||||
FROM minion_jobs
|
||||
WHERE queue = $1`,
|
||||
[wedgeQueue],
|
||||
);
|
||||
const wr = wedgeRows[0] ?? { active_healthy: '0', waiting: '0', last_completed: null };
|
||||
const wedgeLastCompleted = wr.last_completed ? new Date(wr.last_completed) : null;
|
||||
|
||||
return {
|
||||
by_status,
|
||||
by_type,
|
||||
@@ -547,6 +580,15 @@ export class MinionQueue {
|
||||
active: by_status['active'] ?? 0,
|
||||
stalled,
|
||||
},
|
||||
wedge: {
|
||||
queue: wedgeQueue,
|
||||
active_healthy: parseInt(wr.active_healthy ?? '0', 10),
|
||||
waiting: parseInt(wr.waiting ?? '0', 10),
|
||||
last_completed_at: wr.last_completed,
|
||||
minutes_since_completion: wedgeLastCompleted
|
||||
? Math.round((Date.now() - wedgeLastCompleted.getTime()) / 60_000)
|
||||
: null,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+310
-27
@@ -84,6 +84,26 @@ export interface SupervisorOpts {
|
||||
* resolveDefaultMaxRssMb() (issue #1678) instead of a flat default.
|
||||
* Set to 0 to spawn the worker without a watchdog. */
|
||||
maxRssMb: number;
|
||||
/**
|
||||
* issue #1801 — progress watchdog. When a child is alive but makes no forward
|
||||
* progress on claimable work for this many minutes (waiting_claimable > 0 and
|
||||
* active_healthy == 0 across `wedgeRestartChecks` consecutive health checks),
|
||||
* the supervisor forcibly restarts it so the respawn rebuilds a fresh DB pool.
|
||||
* Set to 0 to DISABLE the wedge watchdog. Default: 15.
|
||||
*/
|
||||
wedgeRestartMinutes: number;
|
||||
/** Consecutive wedged health checks required before a restart fires (hysteresis). Default: 3. */
|
||||
wedgeRestartChecks: number;
|
||||
/** Max wedge restarts inside `wedgeRestartLoopWindowMs` before the supervisor
|
||||
* stops restarting and emits `wedge_restart_loop` (alert-only — a dead-pool
|
||||
* wedge resolves in one restart; a loop means restart isn't the fix). Default: 3. */
|
||||
wedgeRestartLoopBudget: number;
|
||||
/** Sliding window for the wedge-restart loop breaker, ms. Default: 30 min. */
|
||||
wedgeRestartLoopWindowMs: number;
|
||||
/** After a (re)spawn, suppress wedge evaluation for this long so a fresh
|
||||
* worker gets a fair claim window before the wedge clock applies (the DB's
|
||||
* last_completed is still stale right after a restart). Default: 2× healthInterval. */
|
||||
startupGraceMs: number;
|
||||
/** Optional event sink (Lane C audit writer). Called for every lifecycle event. */
|
||||
onEvent?: (event: SupervisorEmission) => void;
|
||||
/**
|
||||
@@ -111,8 +131,21 @@ const DEFAULTS: Omit<SupervisorOpts, 'cliPath'> = {
|
||||
allowShellJobs: false,
|
||||
json: false,
|
||||
maxRssMb: 2048,
|
||||
// issue #1801 progress-watchdog defaults. Conservative: a dead-pool wedge is
|
||||
// caught faster by the worker's own DB probe (fix #2, ~3 min); this 15-min
|
||||
// watchdog is the cause-agnostic backstop for non-DB wedges (stuck handler,
|
||||
// deadlock) so it deliberately fires slower to avoid racing fix #2.
|
||||
wedgeRestartMinutes: 15,
|
||||
wedgeRestartChecks: 3,
|
||||
wedgeRestartLoopBudget: 3,
|
||||
wedgeRestartLoopWindowMs: 30 * 60_000,
|
||||
startupGraceMs: 120_000, // overridden to 2× healthInterval in the constructor
|
||||
};
|
||||
|
||||
/** Grace before SIGKILL when restarting a wedged child — reuses the 35s
|
||||
* shutdown() drain window (issue #1801, D3). */
|
||||
const WEDGE_RESTART_GRACE_MS = 35_000;
|
||||
|
||||
/** Calculate backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s cap. */
|
||||
export function calculateBackoffMs(crashCount: number): number {
|
||||
const base = Math.min(1000 * Math.pow(2, Math.max(crashCount, 0)), 60_000);
|
||||
@@ -130,6 +163,69 @@ function isProcessAlive(pid: number): boolean {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #1801 — queue + name-scoped wedge signals. Exported (with the SQL) so
|
||||
* the FILTER semantics are testable against a real engine (PGLite) without
|
||||
* spinning a supervisor process:
|
||||
* - `activeHealthy` counts only LIVE-lock active rows, so an expired-lock
|
||||
* active row (a worker that died mid-job) does NOT mask the wedge (Codex #6).
|
||||
* - `waitingClaimable` counts waiting OR due-delayed (delay_until <= now) rows
|
||||
* whose name the worker can claim — due-delayed because a dead pool means
|
||||
* promoteDelayed never ran (Codex #5/#7).
|
||||
* - `lastCompletedClaimable` is freshness of progress on claimable names only.
|
||||
*/
|
||||
export interface WedgeSignals {
|
||||
stalled: number;
|
||||
activeHealthy: number;
|
||||
waiting: number;
|
||||
waitingClaimable: number;
|
||||
lastCompleted: Date | null;
|
||||
lastCompletedClaimable: Date | null;
|
||||
}
|
||||
|
||||
export async function queryWedgeSignals(
|
||||
engine: BrainEngine,
|
||||
queue: string,
|
||||
handlerNames: string[],
|
||||
): Promise<WedgeSignals> {
|
||||
const rows = await engine.executeRaw<{
|
||||
stalled: string;
|
||||
active_healthy: string;
|
||||
waiting: string;
|
||||
waiting_claimable: string;
|
||||
last_completed: string | null;
|
||||
last_completed_claimable: string | null;
|
||||
}>(
|
||||
`SELECT
|
||||
count(*) FILTER (WHERE status = 'active' AND lock_until < now())::text AS stalled,
|
||||
count(*) FILTER (WHERE status = 'active' AND lock_until > now())::text AS active_healthy,
|
||||
count(*) FILTER (WHERE status = 'waiting')::text AS waiting,
|
||||
count(*) FILTER (WHERE (status = 'waiting'
|
||||
OR (status = 'delayed' AND delay_until <= now()))
|
||||
AND name = ANY($2::text[]))::text AS waiting_claimable,
|
||||
max(updated_at) FILTER (WHERE status = 'completed')::text AS last_completed,
|
||||
max(updated_at) FILTER (WHERE status = 'completed'
|
||||
AND name = ANY($2::text[]))::text AS last_completed_claimable
|
||||
FROM minion_jobs
|
||||
WHERE queue = $1`,
|
||||
[queue, handlerNames],
|
||||
);
|
||||
const row = rows[0] ?? {
|
||||
stalled: '0', active_healthy: '0', waiting: '0',
|
||||
waiting_claimable: '0', last_completed: null, last_completed_claimable: null,
|
||||
};
|
||||
return {
|
||||
stalled: parseInt(row.stalled ?? '0', 10),
|
||||
activeHealthy: parseInt(row.active_healthy ?? '0', 10),
|
||||
waiting: parseInt(row.waiting ?? '0', 10),
|
||||
waitingClaimable: parseInt(row.waiting_claimable ?? '0', 10),
|
||||
lastCompleted: row.last_completed ? new Date(row.last_completed) : null,
|
||||
lastCompletedClaimable: row.last_completed_claimable
|
||||
? new Date(row.last_completed_claimable)
|
||||
: null,
|
||||
};
|
||||
}
|
||||
|
||||
/** Exit codes for documented agent branching. */
|
||||
export const ExitCodes = {
|
||||
CLEAN: 0,
|
||||
@@ -157,6 +253,23 @@ export class MinionSupervisor {
|
||||
private sigintListener: (() => void) | null = null;
|
||||
private lockAcquired = false;
|
||||
private consecutiveHealthFailures = 0;
|
||||
// issue #1801 progress-watchdog state.
|
||||
/** Job names the spawned worker can actually claim. Derived once at start()
|
||||
* from registerBuiltinHandlers so the wedge scopes to real claimable work
|
||||
* (no false-positive on unhandled names). Empty = watchdog inert. */
|
||||
private handlerNames: string[] = [];
|
||||
/** Consecutive health checks that saw the wedge condition. */
|
||||
private consecutiveWedgedChecks = 0;
|
||||
/** Timestamps of recent wedge restarts (loop-breaker window). */
|
||||
private wedgeRestartTimestamps: number[] = [];
|
||||
/** Whether the `wedge_restart_loop` give-up alert has already fired for the
|
||||
* current exhausted window — so it emits once, not every health tick. Re-arms
|
||||
* when a real restart fires again (window drained below budget). */
|
||||
private wedgeLoopAlerted = false;
|
||||
/** True while a restartCurrentChild() is in flight (suppresses re-escalation). */
|
||||
private escalationInFlight = false;
|
||||
/** Wall-clock of the most recent child spawn (startup-grace anchor). */
|
||||
private childStartedAt: number | null = null;
|
||||
|
||||
constructor(engine: BrainEngine, opts: Partial<SupervisorOpts> & { cliPath: string }) {
|
||||
this.engine = engine;
|
||||
@@ -171,6 +284,13 @@ export class MinionSupervisor {
|
||||
this.opts.maxRssMb = resolveDefaultMaxRssMb();
|
||||
}
|
||||
|
||||
// issue #1801: default the startup grace to 2× the health interval so a
|
||||
// freshly (re)spawned worker gets at least two health ticks to claim work
|
||||
// before the wedge clock can fire. Honors an explicit override.
|
||||
if (opts.startupGraceMs === undefined) {
|
||||
this.opts.startupGraceMs = this.opts.healthInterval * 2;
|
||||
}
|
||||
|
||||
// Detect tini for zombie reaping. Resolved once at construction so we
|
||||
// don't shell out on every respawn. Belt-and-suspenders with the
|
||||
// SIGCHLD handler in cli.ts — tini catches children spawned by native
|
||||
@@ -189,6 +309,35 @@ export class MinionSupervisor {
|
||||
return this.tiniPath !== '';
|
||||
}
|
||||
|
||||
/**
|
||||
* @internal Test seams for the issue #1801 wedge watchdog. The escalation
|
||||
* state machine (counter / thresholds / startup grace / loop budget) is hard
|
||||
* to exercise via the real spawn loop, so tests inject a fake child supervisor
|
||||
* + wedge state and drive a single health check directly.
|
||||
*/
|
||||
_setChildSupervisorForTests(
|
||||
cs: Pick<ChildWorkerSupervisor, 'childAlive' | 'inBackoff' | 'restartCurrentChild'>,
|
||||
): void {
|
||||
this.childSupervisor = cs as unknown as ChildWorkerSupervisor;
|
||||
}
|
||||
/** @internal */
|
||||
_setWedgeStateForTests(s: { handlerNames?: string[]; childStartedAt?: number | null }): void {
|
||||
if (s.handlerNames !== undefined) this.handlerNames = s.handlerNames;
|
||||
if (s.childStartedAt !== undefined) this.childStartedAt = s.childStartedAt;
|
||||
}
|
||||
/** @internal Run one health check (the timer body) synchronously. */
|
||||
async _healthCheckOnceForTests(): Promise<void> {
|
||||
await this.healthCheck();
|
||||
}
|
||||
/** @internal */
|
||||
get _consecutiveWedgedChecksForTests(): number {
|
||||
return this.consecutiveWedgedChecks;
|
||||
}
|
||||
/** @internal */
|
||||
get _wedgeRestartCountForTests(): number {
|
||||
return this.wedgeRestartTimestamps.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Emit a lifecycle event. In JSON mode, writes a JSONL record to stderr.
|
||||
* In human mode, writes a human-readable log line to stdout (info) or
|
||||
@@ -277,10 +426,49 @@ export class MinionSupervisor {
|
||||
max_crashes: this.opts.maxCrashes,
|
||||
});
|
||||
|
||||
// 6. Run the supervise loop (respawn on crash, bounded by maxCrashes).
|
||||
// 6. Derive the claimable job-name set for the wedge watchdog (issue #1801).
|
||||
// Done before the loop so the first health check can scope correctly.
|
||||
await this.deriveHandlerNames();
|
||||
|
||||
// 7. Run the supervise loop (respawn on crash, bounded by maxCrashes).
|
||||
await this.runSuperviseLoop();
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #1801: derive the exact set of job names the spawned worker can claim,
|
||||
* by running the real `registerBuiltinHandlers` against a throwaway worker (never
|
||||
* started — no timers, no DB) and reading `registeredNames`. Zero duplication vs
|
||||
* a hand-maintained constant, and it auto-tracks every future handler (Codex
|
||||
* #16). Lazy-imported to avoid the supervisor.ts ↔ jobs.ts / worker.ts import
|
||||
* cycle. On any failure the watchdog stays inert (empty names → waiting_claimable
|
||||
* is always 0 → no restart) rather than risk a misscoped kill — the worker's own
|
||||
* DB probe (fix #2) still covers the dead-pool case.
|
||||
*/
|
||||
private async deriveHandlerNames(): Promise<void> {
|
||||
try {
|
||||
const [{ MinionWorker }, { registerBuiltinHandlers }] = await Promise.all([
|
||||
import('./worker.ts'),
|
||||
import('../../commands/jobs.ts'),
|
||||
]);
|
||||
const probe = new MinionWorker(this.engine, {
|
||||
queue: this.opts.queue,
|
||||
healthCheckInterval: 0,
|
||||
});
|
||||
// `shell` is always registered regardless of allowShellJobs (the env only
|
||||
// gates execution), so registeredNames is a static set — `quiet` just
|
||||
// suppresses the informational startup lines during derivation.
|
||||
await registerBuiltinHandlers(probe, this.engine, { quiet: true });
|
||||
this.handlerNames = [...probe.registeredNames];
|
||||
} catch (e) {
|
||||
this.handlerNames = [];
|
||||
this.emit('health_warn', {
|
||||
reason: 'wedge_watchdog_inert',
|
||||
error: e instanceof Error ? e.message : String(e),
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/** Unified shutdown path. Reason becomes the audit event name; exitCode is process exit. */
|
||||
private async shutdown(reason: string, exitCode: number): Promise<void> {
|
||||
if (this.stopping) return;
|
||||
@@ -433,9 +621,10 @@ export class MinionSupervisor {
|
||||
delete env.GBRAIN_ALLOW_SHELL_JOBS;
|
||||
}
|
||||
// Signal to the child worker that it's running under a supervisor.
|
||||
// The worker's self-health-check (DB probes, stall detection) is
|
||||
// redundant when the supervisor already provides these — setting
|
||||
// this env var causes the worker to skip its own health timer.
|
||||
// issue #1801: the worker's DB-liveness probe STILL runs under supervision
|
||||
// (it's the only "is MY pool dead" signal; the supervisor watches a
|
||||
// different connection). This env var only makes the worker skip its STALL
|
||||
// detection — the supervisor's progress watchdog owns forward-progress.
|
||||
env.GBRAIN_SUPERVISED = '1';
|
||||
|
||||
this.childSupervisor = new ChildWorkerSupervisor({
|
||||
@@ -468,6 +657,11 @@ export class MinionSupervisor {
|
||||
private relayChildEvent(event: ChildSupervisorEvent): void {
|
||||
switch (event.kind) {
|
||||
case 'worker_spawned':
|
||||
// issue #1801: anchor the startup grace + reset the wedge counter so a
|
||||
// fresh child is judged on its own forward progress, not the prior
|
||||
// (possibly wedged) one's stale DB state.
|
||||
this.childStartedAt = Date.now();
|
||||
this.consecutiveWedgedChecks = 0;
|
||||
this.emit('worker_spawned', {
|
||||
pid: event.pid >= 0 ? event.pid : undefined,
|
||||
cli_path: this.opts.cliPath,
|
||||
@@ -539,35 +733,22 @@ export class MinionSupervisor {
|
||||
this.healthInFlight = true;
|
||||
|
||||
try {
|
||||
// Blocker 2+3+6: single FILTER query scoped to this.opts.queue.
|
||||
// 'stalled' = active jobs whose lock_until has passed (matches
|
||||
// queue.ts:848 handleStalled() definition — same set that the queue
|
||||
// itself will requeue/dead-letter on next tick).
|
||||
const rows = await this.engine.executeRaw<{
|
||||
stalled: string;
|
||||
waiting: string;
|
||||
last_completed: string | null;
|
||||
}>(
|
||||
`SELECT
|
||||
count(*) FILTER (WHERE status = 'active' AND lock_until < now())::text AS stalled,
|
||||
count(*) FILTER (WHERE status = 'waiting')::text AS waiting,
|
||||
max(updated_at) FILTER (WHERE status = 'completed')::text AS last_completed
|
||||
FROM minion_jobs
|
||||
WHERE queue = $1`,
|
||||
[this.opts.queue],
|
||||
);
|
||||
const sig = await queryWedgeSignals(this.engine, this.opts.queue, this.handlerNames);
|
||||
|
||||
// Reset consecutive failure counter on successful health check
|
||||
this.consecutiveHealthFailures = 0;
|
||||
|
||||
const row = rows[0] ?? { stalled: '0', waiting: '0', last_completed: null };
|
||||
const stalledCount = parseInt(row.stalled ?? '0', 10);
|
||||
const waitingCount = parseInt(row.waiting ?? '0', 10);
|
||||
const lastCompleted = row.last_completed ? new Date(row.last_completed) : null;
|
||||
const stalledCount = sig.stalled;
|
||||
const activeHealthyCount = sig.activeHealthy;
|
||||
const waitingCount = sig.waiting;
|
||||
const waitingClaimableCount = sig.waitingClaimable;
|
||||
|
||||
const now = Date.now();
|
||||
const minutesSinceCompletion = lastCompleted
|
||||
? Math.round((now - lastCompleted.getTime()) / 60_000)
|
||||
const minutesSinceCompletion = sig.lastCompleted
|
||||
? Math.round((now - sig.lastCompleted.getTime()) / 60_000)
|
||||
: null;
|
||||
const minutesSinceClaimable = sig.lastCompletedClaimable
|
||||
? Math.round((now - sig.lastCompletedClaimable.getTime()) / 60_000)
|
||||
: null;
|
||||
|
||||
// F2 (per-threshold warns) — each is a distinct health_warn with reason.
|
||||
@@ -599,6 +780,43 @@ export class MinionSupervisor {
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
}
|
||||
|
||||
// issue #1801 — progress watchdog. A child that is ALIVE but makes no
|
||||
// forward progress on claimable work is invisible to liveness checks
|
||||
// (the dead-pool zombie that caused the 15h halt). Restart it so the
|
||||
// respawn rebuilds a fresh DB pool.
|
||||
//
|
||||
// - active_healthy === 0 is the real guard: a worker mid-job holds a
|
||||
// live lock (active_healthy > 0) and resets the counter; an expired-
|
||||
// lock active row does NOT suppress (Codex #6).
|
||||
// - waiting_claimable > 0: there is work THIS worker could claim
|
||||
// (name-scoped; incl. due-delayed) (Codex #5/#7).
|
||||
// - claimable progress is stale (or never happened) past the window.
|
||||
// - startup grace: a freshly (re)spawned worker gets a fair claim
|
||||
// window before the wedge clock applies (Codex #9/#10).
|
||||
const childAgeMs = this.childStartedAt !== null ? now - this.childStartedAt : 0;
|
||||
const pastStartupGrace = childAgeMs > this.opts.startupGraceMs;
|
||||
const claimableStale =
|
||||
minutesSinceClaimable === null || minutesSinceClaimable > this.opts.wedgeRestartMinutes;
|
||||
const wedged =
|
||||
this.opts.wedgeRestartMinutes > 0 &&
|
||||
waitingClaimableCount > 0 &&
|
||||
activeHealthyCount === 0 &&
|
||||
claimableStale &&
|
||||
workerAlive &&
|
||||
!inBackoff &&
|
||||
!this.stopping &&
|
||||
!this.escalationInFlight &&
|
||||
pastStartupGrace;
|
||||
|
||||
if (wedged) {
|
||||
this.consecutiveWedgedChecks++;
|
||||
if (this.consecutiveWedgedChecks >= this.opts.wedgeRestartChecks) {
|
||||
await this.escalateWedgedWorker(waitingClaimableCount, minutesSinceClaimable);
|
||||
}
|
||||
} else {
|
||||
this.consecutiveWedgedChecks = 0;
|
||||
}
|
||||
} catch (e) {
|
||||
this.consecutiveHealthFailures++;
|
||||
const errMsg = e instanceof Error ? e.message : String(e);
|
||||
@@ -639,4 +857,69 @@ export class MinionSupervisor {
|
||||
this.healthInFlight = false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* issue #1801: forcibly restart an alive-but-wedged child. Bounded by a
|
||||
* sliding-window loop budget — a dead-pool wedge resolves in exactly one
|
||||
* restart (fresh process ⇒ fresh pool), so repeated restarts inside the
|
||||
* window mean restart isn't the fix (e.g. a deterministically-failing
|
||||
* handler); switch to alert-only via `wedge_restart_loop` instead of
|
||||
* thrashing. The kill mechanics live in ChildWorkerSupervisor.restartCurrentChild
|
||||
* (owns child identity + the intentional-restart crash accounting).
|
||||
*
|
||||
* Awaited inside healthCheck()'s try, so `healthInFlight` keeps the next
|
||||
* health tick from overlapping the ~grace-bounded restart, and
|
||||
* `escalationInFlight` keeps the wedge predicate from re-firing.
|
||||
*/
|
||||
private async escalateWedgedWorker(
|
||||
waitingClaimable: number,
|
||||
minutesSinceCompletion: number | null,
|
||||
): Promise<void> {
|
||||
const cs = this.childSupervisor;
|
||||
if (!cs) return;
|
||||
|
||||
const now = Date.now();
|
||||
this.wedgeRestartTimestamps = this.wedgeRestartTimestamps.filter(
|
||||
(t) => t > now - this.opts.wedgeRestartLoopWindowMs,
|
||||
);
|
||||
// `>=` so the budget is a real ceiling (the Nth restart is the last allowed,
|
||||
// not the N+1th) — issue #1801 Codex #13.
|
||||
if (this.wedgeRestartTimestamps.length >= this.opts.wedgeRestartLoopBudget) {
|
||||
// Give up restarting (restart isn't fixing it). Alert ONCE on entry to
|
||||
// the exhausted state — without this flag the wedge predicate re-fires
|
||||
// every health tick for the whole window and floods the audit log with
|
||||
// identical `wedge_restart_loop` warns. Reset the counter so the predicate
|
||||
// doesn't busy-spin; the flag re-arms below once a real restart fires
|
||||
// again (window drained below budget).
|
||||
if (!this.wedgeLoopAlerted) {
|
||||
this.wedgeLoopAlerted = true;
|
||||
this.emit('health_warn', {
|
||||
reason: 'wedge_restart_loop',
|
||||
count: this.wedgeRestartTimestamps.length,
|
||||
window_ms: this.opts.wedgeRestartLoopWindowMs,
|
||||
waiting_claimable: waitingClaimable,
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
}
|
||||
this.consecutiveWedgedChecks = 0;
|
||||
return;
|
||||
}
|
||||
|
||||
this.wedgeLoopAlerted = false; // re-arm: window has room, we're restarting
|
||||
this.wedgeRestartTimestamps.push(now);
|
||||
this.consecutiveWedgedChecks = 0;
|
||||
this.escalationInFlight = true;
|
||||
this.emit('health_warn', {
|
||||
reason: 'restarting_wedged_worker',
|
||||
waiting_claimable: waitingClaimable,
|
||||
minutes_since_completion: minutesSinceCompletion,
|
||||
consecutive_wedged_checks: this.opts.wedgeRestartChecks,
|
||||
queue: this.opts.queue,
|
||||
});
|
||||
try {
|
||||
await cs.restartCurrentChild(WEDGE_RESTART_GRACE_MS);
|
||||
} finally {
|
||||
this.escalationInFlight = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -329,12 +329,15 @@ export class MinionWorker extends EventEmitter {
|
||||
}, this.opts.rssCheckInterval);
|
||||
}
|
||||
|
||||
// Self-health-check — provides supervisor-grade monitoring for bare workers.
|
||||
// Disabled when running under a supervisor (GBRAIN_SUPERVISED=1) or when
|
||||
// healthCheckInterval is 0. Catches two failure modes that leave the process
|
||||
// alive but non-functional:
|
||||
// 1. DB connection death (Supabase/PgBouncer drops, network blip)
|
||||
// 2. Worker stall (event loop alive but not claiming/completing jobs)
|
||||
// Self-health-check. Catches two failure modes that leave the process alive
|
||||
// but non-functional:
|
||||
// 1. DB connection death (Supabase/PgBouncer drops, network blip) — runs
|
||||
// ALWAYS (incl. under a supervisor), because it's the only signal for
|
||||
// "MY pool is dead" and the supervisor watches a different connection
|
||||
// (issue #1801, fix #2). Disabled only when healthCheckInterval is 0.
|
||||
// 2. Worker stall (event loop alive but not claiming/completing jobs) —
|
||||
// runs only when NOT supervised (GBRAIN_SUPERVISED=1); under a
|
||||
// supervisor the progress watchdog owns forward-progress detection.
|
||||
//
|
||||
// On failure, emits an `'unhealthy'` event with a structured reason. The
|
||||
// CLI layer (`src/commands/jobs.ts:work`) subscribes and decides whether to
|
||||
@@ -347,7 +350,17 @@ export class MinionWorker extends EventEmitter {
|
||||
// `consecutiveDbFailures`. The recursive pattern guarantees one tick at a time.
|
||||
const isSupervisedChild = process.env.GBRAIN_SUPERVISED === '1';
|
||||
let healthTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
if (!isSupervisedChild && this.opts.healthCheckInterval > 0) {
|
||||
// issue #1801 (fix #2): the DB-liveness probe (part 1) runs EVEN under a
|
||||
// supervisor — it's the worker's own "is MY pool dead" signal, and the
|
||||
// supervisor watches a DIFFERENT connection, so it cannot see this worker's
|
||||
// dead pool. A supervised worker whose pool dies now self-exits (db_dead →
|
||||
// process.exit(1) via the jobs.ts listener) and the supervisor respawns it
|
||||
// with a fresh pool in ~3 min — faster than, and orthogonal to, the
|
||||
// supervisor's 15-min progress watchdog (which backstops NON-DB wedges).
|
||||
// Stall detection (part 2) STAYS gated to non-supervised: the supervisor's
|
||||
// progress watchdog now owns forward-progress, so the worker's own stall
|
||||
// detector would double-act.
|
||||
if (this.opts.healthCheckInterval > 0) {
|
||||
let consecutiveDbFailures = 0;
|
||||
let lastKnownCompleted = this.jobsCompleted;
|
||||
let lastCompletionTime = Date.now();
|
||||
@@ -408,7 +421,12 @@ export class MinionWorker extends EventEmitter {
|
||||
return; // Skip stall check when DB is flaky
|
||||
}
|
||||
|
||||
// --- 2. Stall detection ---
|
||||
// --- 2. Stall detection (NON-supervised only) ---
|
||||
// Under a supervisor, the supervisor's progress watchdog (issue #1801)
|
||||
// owns forward-progress detection; running the worker's own stall
|
||||
// detector too would double-act (and both emitting 'unhealthy' +
|
||||
// supervisor SIGTERM race). Bare `gbrain jobs work` keeps it.
|
||||
if (!isSupervisedChild) {
|
||||
if (this.jobsCompleted > lastKnownCompleted) {
|
||||
lastKnownCompleted = this.jobsCompleted;
|
||||
lastCompletionTime = Date.now();
|
||||
@@ -469,6 +487,7 @@ export class MinionWorker extends EventEmitter {
|
||||
} else {
|
||||
stallWarningSince = null;
|
||||
}
|
||||
} // end stall detection (NON-supervised only)
|
||||
} finally {
|
||||
healthRunning = false;
|
||||
if (this.running && !healthExited) {
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
*/
|
||||
|
||||
import { describe, it, expect, afterEach } from 'bun:test';
|
||||
import { chmodSync, mkdirSync, rmSync, writeFileSync } from 'fs';
|
||||
import { chmodSync, mkdirSync, readFileSync, rmSync, writeFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
import {
|
||||
@@ -470,4 +470,143 @@ esac
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('issue #1801 — restartCurrentChild + killChild liveness fix', () => {
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
// ESRCH = no such process (dead). EPERM = process exists but we can't
|
||||
// signal it (alive) — under `bun test` a spawned child can land in a state
|
||||
// where kill(pid,0) reports EPERM, so treat anything-but-ESRCH as alive.
|
||||
const isAlive = (pid: number): boolean => {
|
||||
try { process.kill(pid, 0); return true; }
|
||||
catch (e) { return (e as NodeJS.ErrnoException)?.code !== 'ESRCH'; }
|
||||
};
|
||||
|
||||
// Worker that IGNORES SIGTERM and sleeps, so only SIGKILL can stop it.
|
||||
function makeSigtermIgnorer(name: string): Harness {
|
||||
return makeHarness(name, "trap '' TERM\nsleep 30");
|
||||
}
|
||||
|
||||
async function startInBackground(h: Harness): Promise<{
|
||||
sup: ChildWorkerSupervisor;
|
||||
events: ChildSupervisorEvent[];
|
||||
firstPid: number;
|
||||
stop: () => Promise<void>;
|
||||
}> {
|
||||
const events: ChildSupervisorEvent[] = [];
|
||||
let stopping = false;
|
||||
let resolveSpawn: (pid: number) => void;
|
||||
const firstSpawn = new Promise<number>((r) => { resolveSpawn = r; });
|
||||
const sup = new ChildWorkerSupervisor({
|
||||
cliPath: h.workerScript,
|
||||
args: [],
|
||||
maxCrashes: 100,
|
||||
_backoffFloorMs: 5,
|
||||
isStopping: () => stopping,
|
||||
onMaxCrashesExceeded: () => { stopping = true; },
|
||||
onEvent: (e) => {
|
||||
events.push(e);
|
||||
if (e.kind === 'worker_spawned') resolveSpawn(e.pid);
|
||||
},
|
||||
});
|
||||
const runPromise = sup.run();
|
||||
const firstPid = await firstSpawn;
|
||||
const stop = async () => {
|
||||
stopping = true;
|
||||
// SIGKILL-retry until run() returns (children ignore SIGTERM).
|
||||
for (let i = 0; i < 60; i++) {
|
||||
sup.killChild('SIGKILL');
|
||||
const done = await Promise.race([
|
||||
runPromise.then(() => true),
|
||||
sleep(50).then(() => false),
|
||||
]);
|
||||
if (done) return;
|
||||
}
|
||||
await runPromise;
|
||||
};
|
||||
return { sup, events, firstPid, stop };
|
||||
}
|
||||
|
||||
// Structural regression for Codex #1: killChild MUST gate on liveness
|
||||
// (exitCode/signalCode === null), NOT on `.killed`. `.killed` flips true the
|
||||
// moment a signal is *sent*, so a `!this._child.killed` guard makes a
|
||||
// follow-up SIGKILL (after an ignored SIGTERM) a silent no-op — the bug that
|
||||
// left the existing shutdown() drain unable to force-kill a stuck worker.
|
||||
// (The SIGTERM→SIGKILL behavior is exercised end-to-end by the
|
||||
// restartCurrentChild test below + standalone repros; a live-process
|
||||
// assertion that a SIGTERM-ignoring child survives is unreliable under the
|
||||
// `bun test` runtime, so the no-regression contract is pinned structurally.)
|
||||
it('killChild gates on liveness, not .killed (Codex #1 regression)', () => {
|
||||
const src = readFileSync(
|
||||
join(import.meta.dir, '..', 'src', 'core', 'minions', 'child-worker-supervisor.ts'),
|
||||
'utf8',
|
||||
);
|
||||
const killChildBody = src.slice(
|
||||
src.indexOf('killChild(signal: NodeJS.Signals)'),
|
||||
src.indexOf('awaitChildExit('),
|
||||
);
|
||||
// Strip comment lines so the doc note explaining the OLD bug (which names
|
||||
// `.killed`) doesn't trip the negative assertion — we check the CODE.
|
||||
const code = killChildBody
|
||||
.split('\n')
|
||||
.filter((l) => !l.trim().startsWith('//') && !l.trim().startsWith('*'))
|
||||
.join('\n');
|
||||
expect(code).toContain('exitCode === null');
|
||||
expect(code).toContain('signalCode === null');
|
||||
// The buggy `.killed` guard must be gone from the code.
|
||||
expect(code).not.toContain('.killed');
|
||||
});
|
||||
|
||||
it('restartCurrentChild SIGKILLs the captured child, respawns, labels wedge_restart, leaves crashCount=0', async () => {
|
||||
const h = makeSigtermIgnorer('restart-current');
|
||||
const ctx = await startInBackground(h);
|
||||
try {
|
||||
const oldPid = ctx.firstPid;
|
||||
await ctx.sup.restartCurrentChild(150); // SIGTERM ignored → SIGKILL after 150ms
|
||||
await sleep(400); // let the old child exit + run() respawn (ms:0 wedge backoff)
|
||||
|
||||
expect(isAlive(oldPid)).toBe(false); // captured child killed
|
||||
|
||||
const spawns = ctx.events.filter((e) => e.kind === 'worker_spawned');
|
||||
expect(spawns.length).toBeGreaterThanOrEqual(2); // respawned
|
||||
|
||||
const wedgeExit = ctx.events.find(
|
||||
(e) => e.kind === 'worker_exited' && e.likelyCause === 'wedge_restart',
|
||||
);
|
||||
expect(wedgeExit).toBeDefined();
|
||||
if (wedgeExit && wedgeExit.kind === 'worker_exited') {
|
||||
expect(wedgeExit.crashCount).toBe(0); // Codex #3 — not counted as a crash
|
||||
}
|
||||
|
||||
const wedgeBackoff = ctx.events.find(
|
||||
(e) => e.kind === 'backoff' && e.reason === 'wedge_restart',
|
||||
);
|
||||
expect(wedgeBackoff).toBeDefined();
|
||||
if (wedgeBackoff && wedgeBackoff.kind === 'backoff') {
|
||||
expect(wedgeBackoff.ms).toBe(0); // immediate respawn
|
||||
}
|
||||
|
||||
// Codex #2 — the respawned child is alive and was NOT killed by a stale
|
||||
// timer aimed at the old child.
|
||||
expect(ctx.sup.childAlive).toBe(true);
|
||||
} finally {
|
||||
await ctx.stop();
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
|
||||
it('repeated wedge restarts never trip max_crashes (crashCount stays 0)', async () => {
|
||||
const h = makeSigtermIgnorer('restart-no-crash');
|
||||
const ctx = await startInBackground(h);
|
||||
try {
|
||||
for (let i = 0; i < 3; i++) {
|
||||
await ctx.sup.restartCurrentChild(120);
|
||||
await sleep(300);
|
||||
}
|
||||
expect(ctx.sup.crashCount).toBe(0); // three self-heals, zero crashes
|
||||
} finally {
|
||||
await ctx.stop();
|
||||
h.cleanup();
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
/**
|
||||
* issue #1801 fix #3 — doctor surfaces the wedged-queue signature as a health
|
||||
* ERROR (and the latent `state`→`status` regression in the remote queue_health
|
||||
* check is gone).
|
||||
*
|
||||
* computeWedgedQueueCheck is Postgres-only (short-circuits to ok on PGLite), so
|
||||
* we run its grouped SQL on a real PGLite engine behind a `kind: 'postgres'`
|
||||
* stub. Seeds verify: wedged → fail; live-lock → ok; null-completions →
|
||||
* conservative ok (the supervisor's startup-grace-aware watchdog owns that case);
|
||||
* per-queue grouping so a healthy queue doesn't mask a wedged one (Codex #15).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { computeWedgedQueueCheck } from '../src/commands/doctor.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
|
||||
let base: PGLiteEngine;
|
||||
let pgLike: BrainEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
base = new PGLiteEngine();
|
||||
await base.connect({});
|
||||
await base.initSchema();
|
||||
// computeWedgedQueueCheck only reads .kind + .executeRaw.
|
||||
pgLike = {
|
||||
kind: 'postgres',
|
||||
executeRaw: base.executeRaw.bind(base),
|
||||
} as unknown as BrainEngine;
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await base.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Wipe only minion_jobs (preserve config/version that ensureSchema reads).
|
||||
await base.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
async function seed(
|
||||
queue: string,
|
||||
name: string,
|
||||
status: string,
|
||||
extra: { lockUntilSql?: string; updatedAtSql?: string } = {},
|
||||
): Promise<void> {
|
||||
await base.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, queue, status, lock_until, updated_at)
|
||||
VALUES ($1, $2, $3, ${extra.lockUntilSql ?? 'NULL'}, ${extra.updatedAtSql ?? 'now()'})`,
|
||||
[name, queue, status],
|
||||
);
|
||||
}
|
||||
|
||||
describe('issue #1801 fix #3 — computeWedgedQueueCheck', () => {
|
||||
it('flags a wedged queue (waiting, 0 active_healthy, stale completion) as fail', async () => {
|
||||
await seed('default', 'cycle', 'waiting');
|
||||
await seed('default', 'cycle', 'completed', { updatedAtSql: "now() - interval '20 min'" });
|
||||
const check = await computeWedgedQueueCheck(pgLike);
|
||||
expect(check.status).toBe('fail');
|
||||
expect(check.message).toContain("'default'");
|
||||
});
|
||||
|
||||
it('does NOT flag when a job holds a live lock (active_healthy > 0)', async () => {
|
||||
await seed('default', 'cycle', 'waiting');
|
||||
await seed('default', 'cycle', 'active', { lockUntilSql: "now() + interval '5 min'" });
|
||||
await seed('default', 'cycle', 'completed', { updatedAtSql: "now() - interval '20 min'" });
|
||||
const check = await computeWedgedQueueCheck(pgLike);
|
||||
expect(check.status).toBe('ok');
|
||||
});
|
||||
|
||||
it('does NOT flag an expired-lock active row as healthy — still wedged (Codex #6)', async () => {
|
||||
await seed('default', 'cycle', 'waiting');
|
||||
await seed('default', 'cycle', 'active', { lockUntilSql: "now() - interval '2 min'" }); // expired
|
||||
await seed('default', 'cycle', 'completed', { updatedAtSql: "now() - interval '20 min'" });
|
||||
const check = await computeWedgedQueueCheck(pgLike);
|
||||
expect(check.status).toBe('fail'); // expired lock does not count as active_healthy
|
||||
});
|
||||
|
||||
it('is conservative on never-completed queues (null mins → ok)', async () => {
|
||||
await seed('default', 'cycle', 'waiting'); // no completed row → mins null
|
||||
const check = await computeWedgedQueueCheck(pgLike);
|
||||
expect(check.status).toBe('ok');
|
||||
});
|
||||
|
||||
it('groups by queue — a healthy queue does not mask a wedged one (Codex #15)', async () => {
|
||||
// healthy queue
|
||||
await seed('q-healthy', 'cycle', 'active', { lockUntilSql: "now() + interval '5 min'" });
|
||||
await seed('q-healthy', 'cycle', 'completed', { updatedAtSql: 'now()' });
|
||||
// wedged queue
|
||||
await seed('q-wedged', 'cycle', 'waiting');
|
||||
await seed('q-wedged', 'cycle', 'completed', { updatedAtSql: "now() - interval '30 min'" });
|
||||
const check = await computeWedgedQueueCheck(pgLike);
|
||||
expect(check.status).toBe('fail');
|
||||
expect(check.message).toContain("'q-wedged'");
|
||||
expect(check.message).not.toContain("'q-healthy'");
|
||||
});
|
||||
|
||||
it('returns ok on PGLite (no multi-process worker surface)', async () => {
|
||||
const check = await computeWedgedQueueCheck(base as unknown as BrainEngine);
|
||||
expect(check.status).toBe('ok');
|
||||
expect(check.message).toContain('PGLite');
|
||||
});
|
||||
});
|
||||
|
||||
describe('issue #1801 fix #3 — remote queue_health state→status regression', () => {
|
||||
it('doctor.ts no longer queries the non-existent `state` column', () => {
|
||||
const src = readFileSync(
|
||||
join(import.meta.dir, '..', 'src', 'commands', 'doctor.ts'),
|
||||
'utf8',
|
||||
);
|
||||
// The column is `status`; the pre-fix `WHERE state = 'active'` errored every
|
||||
// run and the catch silently returned "No queue activity".
|
||||
expect(src).not.toContain("state = 'active'");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,75 @@
|
||||
/**
|
||||
* issue #1801 fix #3 — MinionQueue.getStats() exposes a QUEUE-SCOPED wedge
|
||||
* block (the data behind the `jobs stats` WEDGED line). active_healthy counts
|
||||
* only live-lock active rows; the block is scoped to one queue so a healthy
|
||||
* worker on another queue can't mask a wedged one (Codex #14).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionQueue } from '../src/core/minions/queue.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
let queue: MinionQueue;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
queue = new MinionQueue(engine);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await engine.executeRaw('DELETE FROM minion_jobs');
|
||||
});
|
||||
|
||||
async function seed(
|
||||
q: string,
|
||||
name: string,
|
||||
status: string,
|
||||
extra: { lockUntilSql?: string; updatedAtSql?: string } = {},
|
||||
): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, queue, status, lock_until, updated_at)
|
||||
VALUES ($1, $2, $3, ${extra.lockUntilSql ?? 'NULL'}, ${extra.updatedAtSql ?? 'now()'})`,
|
||||
[name, q, status],
|
||||
);
|
||||
}
|
||||
|
||||
describe('issue #1801 — getStats wedge block', () => {
|
||||
it('reports the wedge signature for the requested queue', async () => {
|
||||
await seed('default', 'cycle', 'waiting');
|
||||
await seed('default', 'cycle', 'active', { lockUntilSql: "now() - interval '1 min'" }); // expired
|
||||
await seed('default', 'cycle', 'completed', { updatedAtSql: "now() - interval '20 min'" });
|
||||
|
||||
const stats = await queue.getStats({ queue: 'default' });
|
||||
expect(stats.wedge.queue).toBe('default');
|
||||
expect(stats.wedge.active_healthy).toBe(0); // expired lock not counted
|
||||
expect(stats.wedge.waiting).toBe(1);
|
||||
expect(stats.wedge.minutes_since_completion).not.toBeNull();
|
||||
expect(stats.wedge.minutes_since_completion!).toBeGreaterThanOrEqual(15);
|
||||
});
|
||||
|
||||
it('counts a live-lock active row as healthy', async () => {
|
||||
await seed('default', 'cycle', 'active', { lockUntilSql: "now() + interval '5 min'" });
|
||||
const stats = await queue.getStats({ queue: 'default' });
|
||||
expect(stats.wedge.active_healthy).toBe(1);
|
||||
});
|
||||
|
||||
it('is queue-scoped — other queues do not bleed into the wedge block', async () => {
|
||||
await seed('other', 'cycle', 'waiting');
|
||||
const stats = await queue.getStats({ queue: 'default' });
|
||||
expect(stats.wedge.waiting).toBe(0); // nothing in 'default'
|
||||
});
|
||||
|
||||
it('defaults the wedge queue to "default" when unspecified', async () => {
|
||||
await seed('default', 'cycle', 'waiting');
|
||||
const stats = await queue.getStats();
|
||||
expect(stats.wedge.queue).toBe('default');
|
||||
expect(stats.wedge.waiting).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,250 @@
|
||||
/**
|
||||
* issue #1801 — supervisor progress watchdog.
|
||||
*
|
||||
* Two layers:
|
||||
* 1. Decision logic (counter / thresholds / startup grace / loop budget) via
|
||||
* the MinionSupervisor test seams + a stub engine — no DB, no spawn.
|
||||
* 2. SQL semantics of `queryWedgeSignals` against a real PGLite engine —
|
||||
* pins Codex #6 (expired-lock active row does NOT count as active_healthy),
|
||||
* #7 (due-delayed counts as claimable), #5 (name-scoping).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { MinionSupervisor, queryWedgeSignals } from '../src/core/minions/supervisor.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Layer 1: decision logic (stub engine + fake child supervisor)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
interface WedgeRow {
|
||||
stalled?: string;
|
||||
active_healthy?: string;
|
||||
waiting?: string;
|
||||
waiting_claimable?: string;
|
||||
last_completed?: string | null;
|
||||
last_completed_claimable?: string | null;
|
||||
}
|
||||
|
||||
function minutesAgoIso(min: number): string {
|
||||
return new Date(Date.now() - min * 60_000).toISOString();
|
||||
}
|
||||
|
||||
function makeSup(opts?: {
|
||||
row?: WedgeRow;
|
||||
childAlive?: boolean;
|
||||
inBackoff?: boolean;
|
||||
wedgeRestartMinutes?: number;
|
||||
wedgeRestartChecks?: number;
|
||||
wedgeRestartLoopBudget?: number;
|
||||
childStartedAtMsAgo?: number;
|
||||
}) {
|
||||
const events: Array<{ event: string; reason?: string; [k: string]: unknown }> = [];
|
||||
let restartCalls = 0;
|
||||
const rowRef: { current: WedgeRow } = {
|
||||
current: opts?.row ?? {},
|
||||
};
|
||||
|
||||
const stubEngine = {
|
||||
kind: 'postgres',
|
||||
async executeRaw() {
|
||||
const r = rowRef.current;
|
||||
return [{
|
||||
stalled: r.stalled ?? '0',
|
||||
active_healthy: r.active_healthy ?? '0',
|
||||
waiting: r.waiting ?? '0',
|
||||
waiting_claimable: r.waiting_claimable ?? '0',
|
||||
last_completed: r.last_completed ?? null,
|
||||
last_completed_claimable: r.last_completed_claimable ?? null,
|
||||
}];
|
||||
},
|
||||
} as unknown as BrainEngine;
|
||||
|
||||
const sup = new MinionSupervisor(stubEngine, {
|
||||
cliPath: '/bin/true',
|
||||
maxRssMb: 0, // skip cgroup auto-size
|
||||
healthInterval: 60_000,
|
||||
wedgeRestartMinutes: opts?.wedgeRestartMinutes ?? 15,
|
||||
wedgeRestartChecks: opts?.wedgeRestartChecks ?? 3,
|
||||
wedgeRestartLoopBudget: opts?.wedgeRestartLoopBudget ?? 3,
|
||||
startupGraceMs: 120_000,
|
||||
onEvent: (e) => events.push(e as { event: string; reason?: string }),
|
||||
});
|
||||
|
||||
sup._setChildSupervisorForTests({
|
||||
childAlive: opts?.childAlive ?? true,
|
||||
inBackoff: opts?.inBackoff ?? false,
|
||||
restartCurrentChild: async () => { restartCalls++; },
|
||||
} as never);
|
||||
|
||||
sup._setWedgeStateForTests({
|
||||
handlerNames: ['cycle'],
|
||||
childStartedAt: Date.now() - (opts?.childStartedAtMsAgo ?? 600_000), // 10 min ago
|
||||
});
|
||||
|
||||
return {
|
||||
sup,
|
||||
events,
|
||||
getRestartCalls: () => restartCalls,
|
||||
setRow: (r: WedgeRow) => { rowRef.current = r; },
|
||||
};
|
||||
}
|
||||
|
||||
const WEDGE_ROW: WedgeRow = {
|
||||
active_healthy: '0',
|
||||
waiting: '0',
|
||||
waiting_claimable: '5',
|
||||
last_completed_claimable: minutesAgoIso(20), // stale > 15
|
||||
};
|
||||
|
||||
describe('issue #1801 — supervisor wedge decision logic', () => {
|
||||
it('restarts a wedged worker after wedgeRestartChecks consecutive checks', async () => {
|
||||
const h = makeSup({ row: WEDGE_ROW });
|
||||
await h.sup._healthCheckOnceForTests();
|
||||
await h.sup._healthCheckOnceForTests();
|
||||
expect(h.getRestartCalls()).toBe(0); // not yet (default 3 checks)
|
||||
await h.sup._healthCheckOnceForTests();
|
||||
expect(h.getRestartCalls()).toBe(1);
|
||||
expect(h.events.some((e) => e.event === 'health_warn' && e.reason === 'restarting_wedged_worker')).toBe(true);
|
||||
});
|
||||
|
||||
it('never escalates when a job holds a live lock (active_healthy > 0)', async () => {
|
||||
const h = makeSup({ row: { ...WEDGE_ROW, active_healthy: '2' } });
|
||||
for (let i = 0; i < 5; i++) await h.sup._healthCheckOnceForTests();
|
||||
expect(h.getRestartCalls()).toBe(0);
|
||||
});
|
||||
|
||||
it('suppresses restart during the startup grace window (Codex #9/#10)', async () => {
|
||||
const h = makeSup({ row: WEDGE_ROW, childStartedAtMsAgo: 1_000 }); // just spawned
|
||||
for (let i = 0; i < 4; i++) await h.sup._healthCheckOnceForTests();
|
||||
expect(h.getRestartCalls()).toBe(0);
|
||||
});
|
||||
|
||||
it('wedgeRestartMinutes <= 0 disables the watchdog', async () => {
|
||||
const h = makeSup({ row: WEDGE_ROW, wedgeRestartMinutes: 0 });
|
||||
for (let i = 0; i < 4; i++) await h.sup._healthCheckOnceForTests();
|
||||
expect(h.getRestartCalls()).toBe(0);
|
||||
});
|
||||
|
||||
it('inBackoff suppresses escalation (respawn window)', async () => {
|
||||
const h = makeSup({ row: WEDGE_ROW, inBackoff: true });
|
||||
for (let i = 0; i < 4; i++) await h.sup._healthCheckOnceForTests();
|
||||
expect(h.getRestartCalls()).toBe(0);
|
||||
});
|
||||
|
||||
it('resets the counter when the wedge clears', async () => {
|
||||
const h = makeSup({ row: WEDGE_ROW, wedgeRestartChecks: 3 });
|
||||
await h.sup._healthCheckOnceForTests(); // wedged 1
|
||||
await h.sup._healthCheckOnceForTests(); // wedged 2
|
||||
h.setRow({ ...WEDGE_ROW, active_healthy: '1' }); // healthy → reset
|
||||
await h.sup._healthCheckOnceForTests();
|
||||
expect(h.sup._consecutiveWedgedChecksForTests).toBe(0);
|
||||
h.setRow(WEDGE_ROW); // wedged again
|
||||
await h.sup._healthCheckOnceForTests(); // 1
|
||||
await h.sup._healthCheckOnceForTests(); // 2
|
||||
expect(h.getRestartCalls()).toBe(0); // still under 3 after the reset
|
||||
await h.sup._healthCheckOnceForTests(); // 3
|
||||
expect(h.getRestartCalls()).toBe(1);
|
||||
});
|
||||
|
||||
it('loop budget (>=) stops restarting and emits wedge_restart_loop ONCE (Codex #13 + loop-spam fix)', async () => {
|
||||
const h = makeSup({ row: WEDGE_ROW, wedgeRestartChecks: 1, wedgeRestartLoopBudget: 2 });
|
||||
await h.sup._healthCheckOnceForTests(); // restart 1 (timestamps: 0 < 2 → push)
|
||||
await h.sup._healthCheckOnceForTests(); // restart 2 (timestamps: 1 < 2 → push)
|
||||
await h.sup._healthCheckOnceForTests(); // budget: 2 >= 2 → no restart, loop warn
|
||||
await h.sup._healthCheckOnceForTests(); // still exhausted → MUST NOT re-warn
|
||||
await h.sup._healthCheckOnceForTests(); // still exhausted → MUST NOT re-warn
|
||||
expect(h.getRestartCalls()).toBe(2);
|
||||
const loopWarns = h.events.filter((e) => e.event === 'health_warn' && e.reason === 'wedge_restart_loop');
|
||||
expect(loopWarns.length).toBe(1); // fired once on entry, not every tick (no audit flood)
|
||||
});
|
||||
|
||||
it('null last_completed_claimable wedges only after the startup grace', async () => {
|
||||
// Never-completed claimable work, child past grace → treated as stale.
|
||||
const h = makeSup({
|
||||
row: { active_healthy: '0', waiting_claimable: '3', last_completed_claimable: null },
|
||||
wedgeRestartChecks: 3,
|
||||
});
|
||||
for (let i = 0; i < 3; i++) await h.sup._healthCheckOnceForTests();
|
||||
expect(h.getRestartCalls()).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Layer 2: SQL semantics of queryWedgeSignals (real PGLite)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
async function seed(
|
||||
queue: string,
|
||||
name: string,
|
||||
status: string,
|
||||
extra: { lockUntilSql?: string; delayUntilSql?: string; updatedAtSql?: string } = {},
|
||||
): Promise<void> {
|
||||
await engine.executeRaw(
|
||||
`INSERT INTO minion_jobs (name, queue, status, lock_until, delay_until, updated_at)
|
||||
VALUES ($1, $2, $3, ${extra.lockUntilSql ?? 'NULL'}, ${extra.delayUntilSql ?? 'NULL'}, ${extra.updatedAtSql ?? 'now()'})`,
|
||||
[name, queue, status],
|
||||
);
|
||||
}
|
||||
|
||||
describe('issue #1801 — queryWedgeSignals SQL semantics (PGLite)', () => {
|
||||
it('scopes claimable by name, counts due-delayed, excludes not-due-delayed (Codex #5/#7)', async () => {
|
||||
const q = 'sql-a';
|
||||
await seed(q, 'cycle', 'waiting'); // claimable
|
||||
await seed(q, 'cycle', 'delayed', { delayUntilSql: "now() - interval '1 min'" }); // due → claimable
|
||||
await seed(q, 'cycle', 'delayed', { delayUntilSql: "now() + interval '1 hour'" }); // not due → not
|
||||
await seed(q, 'other', 'waiting'); // wrong name → not claimable
|
||||
await seed(q, 'cycle', 'completed', { updatedAtSql: 'now()' });
|
||||
|
||||
const sig = await queryWedgeSignals(engine, q, ['cycle']);
|
||||
expect(sig.waitingClaimable).toBe(2); // waiting + due-delayed
|
||||
expect(sig.waiting).toBe(2); // total waiting (cycle + other)
|
||||
expect(sig.lastCompletedClaimable).not.toBeNull();
|
||||
});
|
||||
|
||||
it('expired-lock active row counts as stalled, NOT active_healthy (Codex #6)', async () => {
|
||||
const q = 'sql-b';
|
||||
await seed(q, 'cycle', 'active', { lockUntilSql: "now() - interval '1 min'" }); // expired
|
||||
await seed(q, 'cycle', 'waiting');
|
||||
|
||||
const sig = await queryWedgeSignals(engine, q, ['cycle']);
|
||||
expect(sig.activeHealthy).toBe(0); // expired lock does NOT suppress the wedge
|
||||
expect(sig.stalled).toBe(1);
|
||||
expect(sig.waitingClaimable).toBe(1);
|
||||
});
|
||||
|
||||
it('live-lock active row counts as active_healthy', async () => {
|
||||
const q = 'sql-c';
|
||||
await seed(q, 'cycle', 'active', { lockUntilSql: "now() + interval '5 min'" }); // live
|
||||
await seed(q, 'cycle', 'waiting');
|
||||
|
||||
const sig = await queryWedgeSignals(engine, q, ['cycle']);
|
||||
expect(sig.activeHealthy).toBe(1);
|
||||
expect(sig.stalled).toBe(0);
|
||||
});
|
||||
|
||||
it('is queue-scoped — other queues do not bleed in', async () => {
|
||||
await seed('q1', 'cycle', 'waiting');
|
||||
await seed('q2', 'cycle', 'waiting');
|
||||
const sig = await queryWedgeSignals(engine, 'q1', ['cycle']);
|
||||
expect(sig.waitingClaimable).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,109 @@
|
||||
/**
|
||||
* issue #1801 fix #2 — the worker's DB-liveness probe runs EVEN under a
|
||||
* supervisor (GBRAIN_SUPERVISED=1), so a supervised worker whose own pool dies
|
||||
* self-exits (db_dead → CLI process.exit(1) → supervisor respawns a fresh pool)
|
||||
* instead of sitting alive-but-wedged forever. Stall detection STAYS gated to
|
||||
* non-supervised (the supervisor's progress watchdog owns forward-progress).
|
||||
*
|
||||
* Behavioral: real PGLite engine with `executeRaw('SELECT 1')` monkeypatched to
|
||||
* throw, so the probe fails and emitUnhealthy('db_dead') fires after
|
||||
* dbFailExitAfter probes. Structural: pins the gating split in worker.ts so a
|
||||
* refactor can't silently re-disable the probe under supervision.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from 'bun:test';
|
||||
import { readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { MinionWorker, type UnhealthyReason } from '../src/core/minions/worker.ts';
|
||||
import { withEnv } from './helpers/with-env.ts';
|
||||
|
||||
// No resetPgliteState/beforeEach: these tests never insert jobs (empty queue),
|
||||
// and resetPgliteState TRUNCATEs the `config` table that carries the minion
|
||||
// schema `version` key the worker's ensureSchema() reads.
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
/** Make the DB-liveness probe (`SELECT 1`) throw; delegate every other query.
|
||||
* Returns a restore fn that removes the instance override (falls back to the
|
||||
* prototype method). */
|
||||
function breakLivenessProbe(eng: PGLiteEngine): () => void {
|
||||
const real = eng.executeRaw.bind(eng);
|
||||
(eng as { executeRaw: unknown }).executeRaw = async (sql: string, params?: unknown[]) => {
|
||||
if (typeof sql === 'string' && sql.trim() === 'SELECT 1') {
|
||||
throw new Error('probe boom (simulated dead pool)');
|
||||
}
|
||||
return real(sql, params as never);
|
||||
};
|
||||
return () => { delete (eng as { executeRaw?: unknown }).executeRaw; };
|
||||
}
|
||||
|
||||
async function runUntilUnhealthy(supervised: boolean): Promise<UnhealthyReason | null> {
|
||||
const restore = breakLivenessProbe(engine);
|
||||
try {
|
||||
return await withEnv(
|
||||
supervised ? { GBRAIN_SUPERVISED: '1' } : { GBRAIN_SUPERVISED: undefined },
|
||||
async () => {
|
||||
const worker = new MinionWorker(engine, {
|
||||
queue: 'default',
|
||||
concurrency: 1,
|
||||
pollInterval: 25,
|
||||
maxRssMb: 0,
|
||||
healthCheckInterval: 20,
|
||||
dbFailExitAfter: 3,
|
||||
dbProbeTimeoutMs: 200,
|
||||
});
|
||||
worker.register('noop', async () => {});
|
||||
|
||||
const got = new Promise<UnhealthyReason>((resolve) => {
|
||||
worker.on('unhealthy', (i) => resolve(i));
|
||||
});
|
||||
const runPromise = worker.start();
|
||||
const captured = await Promise.race([
|
||||
got,
|
||||
new Promise<null>((r) => setTimeout(() => r(null), 3000)),
|
||||
]);
|
||||
worker.stop();
|
||||
await runPromise;
|
||||
return captured;
|
||||
},
|
||||
);
|
||||
} finally {
|
||||
restore();
|
||||
}
|
||||
}
|
||||
|
||||
describe('issue #1801 fix #2 — supervised DB self-defense', () => {
|
||||
it('a SUPERVISED worker with a dead pool emits db_dead (probe runs under supervision)', async () => {
|
||||
const info = await runUntilUnhealthy(true);
|
||||
expect(info).not.toBeNull();
|
||||
expect(info?.reason).toBe('db_dead');
|
||||
}, 10_000);
|
||||
|
||||
it('an UNSUPERVISED worker with a dead pool also emits db_dead (back-compat)', async () => {
|
||||
const info = await runUntilUnhealthy(false);
|
||||
expect(info).not.toBeNull();
|
||||
expect(info?.reason).toBe('db_dead');
|
||||
}, 10_000);
|
||||
|
||||
it('structural: DB probe is NOT gated on !isSupervisedChild; stall detection IS', () => {
|
||||
const src = readFileSync(
|
||||
join(import.meta.dir, '..', 'src', 'core', 'minions', 'worker.ts'),
|
||||
'utf8',
|
||||
);
|
||||
// Outer self-health guard must NOT require non-supervised anymore.
|
||||
expect(src).toContain('if (this.opts.healthCheckInterval > 0) {');
|
||||
expect(src).not.toContain('if (!isSupervisedChild && this.opts.healthCheckInterval > 0)');
|
||||
// Stall detection must still be wrapped in a non-supervised guard.
|
||||
expect(src).toMatch(/Stall detection \(NON-supervised only\)[\s\S]*?if \(!isSupervisedChild\)/);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user