diff --git a/CHANGELOG.md b/CHANGELOG.md index 0bc4160ed..660af18aa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,51 @@ All notable changes to GBrain will be documented in this file. +## [0.35.5.1] - 2026-05-16 + +**`gbrain doctor` stops counting clean supervisor exits as crashes — the "120x/24h" alarm finally reflects real crashes, with per-cause breakdown for operator triage.** + +`doctor.ts:1013` and `gbrain jobs supervisor status` at `jobs.ts:805` both counted every `worker_exited` audit event as a crash, regardless of cause. After v0.34.3.0's RSS-watchdog work added more code=0 worker drains, the count inflated to 120+/day on a healthy brain — the exact alarm pattern users started seeing ("Supervisor crashes: 120x/24h, was 62x — nearly doubled"). The classifier upstream in `child-worker-supervisor.ts:309-321` was already stamping a five-value `likely_cause` field on every exit (clean_exit, graceful_shutdown, runtime_error, oom_or_external_kill, unknown); neither read site looked at it. v0.35.5.1 ships a shared `summarizeCrashes(events)` helper colocated with `readSupervisorEvents`, denylist semantics so future failure modes surface by default, threshold dropped from `>3` to `>=1`, and per-cause breakdown in the messages so an operator triages in one glance. + +**Relationship to v0.35.4.0:** that release shipped a binary `classifyWorkerExit({code})` helper (clean if `code === 0`, crash otherwise). It correctly stopped counting RSS-watchdog drains as crashes, but treated SIGTERM-driven graceful shutdowns as crashes (because `null !== 0`) and lost the cause distinction operators need to triage. v0.35.5.1 layers on top: reads `likely_cause` so `graceful_shutdown` is correctly clean, and bucketizes real crashes into `runtime_error` (code bugs), `oom_or_external_kill` (memory pressure), `unknown` (other), `legacy` (pre-v0.34 or future unrecognized). The `classifyWorkerExit` helper from v0.35.4.0 remains in use by the supervisor's internal restart policy where the binary check is the right shape; the doctor + jobs surfaces use the richer classifier. + +### What you can now do + +**Trust the doctor's supervisor count.** `gbrain doctor` reports actual crashes, not RSS-watchdog drains or SIGTERM stops. The "120x/24h" alarm class is dead. + +**See per-cause breakdown at a glance.** The warn message reads `Worker crashed Nx in last 24h (runtime=A oom=B unknown=C legacy=D)` — distinguishes memory pressure (oom) from code bugs (runtime) without grep'ing the JSONL audit. + +**Hook `crashes_by_cause` into monitoring.** `gbrain jobs supervisor status --json` now exposes `crashes_by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}` and `clean_exits_24h` fields. Dashboards bind to named buckets. + +**Future failure modes surface by default.** Denylist semantics: only `clean_exit` and `graceful_shutdown` are non-crashes; everything else (including new `likely_cause` values added upstream in future releases) counts as a crash. The bug we just fixed cannot silently recur on the next schema addition. + +### Itemized changes + +- `src/core/minions/handlers/supervisor-audit.ts` exports new `isCrashExit(event)`, `summarizeCrashes(events)`, `CrashSummary` type, and `CLEAN_EXIT_CAUSES` constant. Single regression point — both CLI surfaces import from here so they cannot drift. +- `src/commands/doctor.ts:1011-1043` replaces the ad-hoc `events.filter(e => e.event === 'worker_exited').length` with `summarizeCrashes(events)`. Drops the warn threshold from `>3` to `>=1` (any real crash is signal now that the counter is calibrated). Widens the ok message with `clean_exits_24h=N` and the warn message with `runtime=N oom=M unknown=K legacy=L`. +- `src/commands/jobs.ts:803-826` same wiring. JSON output adds `crashes_by_cause` and `clean_exits_24h` fields. Human output gains a per-cause line under `Crashes (24h)` plus a `Clean exits (24h)` line. +- `test/supervisor-audit.test.ts` (new) — 14 unit cases: 9-case `isCrashExit` branch matrix (every `likely_cause` value, denylist regression guard for unrecognized future causes, legacy fallback paths for pre-v0.34 entries, non-exit-event defensive case), 5-case `summarizeCrashes` aggregator (mixed-stream bucket assertions, empty input, non-exit-only, unrecognized-cause routing to legacy, null-code edge case). +- `test/doctor.test.ts` — 4 source-grep wiring assertions guarding both surfaces against drift: doctor + jobs.ts use `summarizeCrashes`, the ad-hoc filter pattern is gone, threshold is `>=1`, per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) appear in both files. +- Why denylist semantics (codex outside-voice catch during `/plan-eng-review`): the set of clean-exit causes is small and explicit (the worker exited because we asked it to); the set of crash causes is open-ended. Allowlist would silently underreport future failure modes; denylist surfaces them by default. The bug being fixed today is itself an allowlist-of-event-names — the denylist shape forecloses the recurrence. +- Plan + review trail: `/plan-eng-review` cleared the plan with 3 substantive findings resolved (shared helper extraction, denylist over allowlist, threshold rebaseline pulled into scope). Codex outside-voice surfaced the duplicate bug at `jobs.ts:805` (would have shipped doctor-only otherwise and let the two surfaces drift). Coverage audit reports 100% on the 22 logical branches the diff introduces. Adversarial review surfaced 13 follow-up observations, all either pre-existing surfaces or accepted plan trade-offs — none in-scope for this PR. + +## To take advantage of v0.35.5.1 + +`gbrain upgrade` is all you need. No schema migration, no config change, no manual action. + +1. Run `gbrain upgrade`. +2. Verify the count: + ```bash + gbrain doctor 2>&1 | grep -i supervisor + gbrain jobs supervisor status --json | jq '{crashes_24h, clean_exits_24h, crashes_by_cause}' + ``` + Both commands MUST report identical `crashes_24h` (cross-surface parity is the regression guard). If the new count is still high, the watchdog OOMs / SIGKILLs are real — investigate via `~/.gbrain/audit/supervisor-*.jsonl` or check the per-cause breakdown in the warn message. +3. Cross-check with raw JSONL if you want ground truth: + ```bash + jq -r 'select(.event=="worker_exited") | .likely_cause' ~/.gbrain/audit/supervisor-*.jsonl | sort | uniq -c + ``` +4. If anything looks wrong, file an issue: https://github.com/garrytan/gbrain/issues with the doctor output and the relevant lines from the audit JSONL. + ## [0.35.5.0] - 2026-05-16 **Upgrade goes through on stuck Supabase brains. Orphan counts get honest. `gbrain think` over MCP actually answers. Worktrees stop being misclassified. node_modules stops leaking.** diff --git a/CLAUDE.md b/CLAUDE.md index 35be02d32..f7daedfc9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -144,6 +144,7 @@ strict behavior when unset. - `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules. - `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`). - `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values. +- `src/core/minions/handlers/supervisor-audit.ts` — supervisor lifecycle JSONL audit at `~/.gbrain/audit/supervisor-YYYY-Www.jsonl` (ISO-week rotation; shares `computeIsoWeekName()` helper with `shell-audit.ts`). `writeSupervisorEvent(emission, supervisorPid)` appends one line per supervisor event (`started`, `worker_spawned`, `worker_exited`, `backoff`, `health_warn`, `health_error`, `max_crashes_exceeded`, `shutting_down`, `stopped`, `worker_spawn_failed`). `readSupervisorEvents({sinceMs})` is the readback path for `gbrain doctor`. **v0.35.5.0:** new exports `isCrashExit(event)`, `summarizeCrashes(events)`, `CrashSummary` type, and `CLEAN_EXIT_CAUSES` denylist (`'clean_exit' | 'graceful_shutdown'`). Single regression point — both `gbrain doctor` (Lane D supervisor check at `doctor.ts:1011-1043`) and `gbrain jobs supervisor status` (`jobs.ts:803-826`) import from here so the two CLI surfaces cannot drift. `isCrashExit` classifies a single `worker_exited` event against the denylist: `clean_exit` / `graceful_shutdown` are NON-crashes; everything else (`runtime_error`, `oom_or_external_kill`, `unknown`, AND any future `likely_cause` value added upstream in `child-worker-supervisor.ts`) is a crash. Pre-v0.34 audit lines lacking `likely_cause` fall back to `code !== 0`. `summarizeCrashes` returns `{total, by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}, clean_exits}` so dashboards bind to named buckets — the `legacy` bucket catches BOTH pre-v0.34 fallback entries AND future unrecognized `likely_cause` values, fail-loud instead of silent underreport. Denylist-over-allowlist was a codex outside-voice catch during `/plan-eng-review` — the bug being fixed (read sites counting every `worker_exited` as a crash, inflating to 120+/day on healthy brains after v0.34.3.0 watchdog drains) was itself an allowlist-of-event-names. Pinned by `test/supervisor-audit.test.ts` (14 cases: 9-case `isCrashExit` branch matrix including denylist regression guard for unrecognized future causes + non-exit-event defensive case, 5-case `summarizeCrashes` aggregator including unrecognized-cause routing to legacy + null-code edge case) and 4 source-grep wiring assertions in `test/doctor.test.ts` guarding both surfaces against drift. - `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced. - `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. **v0.30.2:** Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times before dead-lettering. Catches both initial-prompt overflow and turn-N tool-loop accumulation that the chunker in `synthesize.ts` can't bound ahead of time. - `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15. @@ -157,7 +158,7 @@ strict behavior when unset. - `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection) - `src/commands/agent.ts` (v0.16) — `gbrain agent run [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel. - `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs. -- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. **v0.28.1:** `case 'work'` now wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging on failure. Replaces the prior call inside `MinionWorker.start()` (which violated engine ownership: the worker disconnected an engine it didn't own, and clobbered the module-level singleton on PostgresEngine via the now-fixed idempotency bug). Pool slots now free immediately on shutdown instead of waiting for TCP keepalive (~minutes). v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase). +- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. **v0.28.1:** `case 'work'` now wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging on failure. Replaces the prior call inside `MinionWorker.start()` (which violated engine ownership: the worker disconnected an engine it didn't own, and clobbered the module-level singleton on PostgresEngine via the now-fixed idempotency bug). Pool slots now free immediately on shutdown instead of waiting for TCP keepalive (~minutes). v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase). **v0.35.5.0:** `gbrain jobs supervisor status` at `jobs.ts:803-826` now consumes `summarizeCrashes()` from `src/core/minions/handlers/supervisor-audit.ts` for cross-surface parity with `gbrain doctor`. JSON output adds `crashes_by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}` + `clean_exits_24h` fields so dashboards bind to named buckets; human output gains a per-cause line under `Crashes (24h)` plus a `Clean exits (24h)` line. Pre-fix the read site at `jobs.ts:805` counted every `worker_exited` event as a crash regardless of `likely_cause` — the same bug class the v0.35.5.0 doctor fix closes. Pinned by the 4 source-grep wiring assertions in `test/doctor.test.ts` that require the per-cause breakdown substrings (`crashes_by_cause`, `clean_exits_24h=`) to appear in BOTH `doctor.ts` and `jobs.ts`. - `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman - `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart). **v0.34.3.0:** inline spawn-and-respawn loop replaced with a `ChildWorkerSupervisor` instance. Drops `crashCount`, `lastWorkerStartTime`, `STABLE_RUN_RESET_MS`, `startWorker`, and the inline `child.on('exit')` block — all consolidated into the shared core. `--max-rss 2048` and `maxCrashes: 5` preserved from the legacy loop. `onMaxCrashesExceeded` now routes through autopilot's own `shutdown('max_crashes')` so the autopilot lockfile gets cleaned up (pre-refactor the inline loop called `process.exit(1)` directly and bypassed cleanup). `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of `workerProc.kill()`. Pinned by `test/autopilot-supervisor-wiring.test.ts` (6 static-shape regression guards: composes ChildWorkerSupervisor not the legacy inline names, `--max-rss 2048` in argv, `maxCrashes: 5` literal, shutdown-via-callback wiring, no workerProc reference). Closes the parallel-supervisor bug class Codex flagged during plan-eng-review. - `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. **v0.34.1.0 (#870):** stdin `'end'` / `'close'` shutdown hooks are skipped when `process.env.MCP_STDIO === '1'`. Gateway-piped stdio MCP wrappers (OpenClaw's `bundle-mcp`, similar) pipe the JSON-RPC handshake then close their stdin half; pre-fix this killed the server before the first tool call landed. Signal handlers (SIGTERM / SIGINT / SIGHUP) and the parent-process watchdog still cover legitimate disconnects. `src/commands/serve.ts` exposes `ServeOptions.mcpStdio?: boolean` as a test seam so the runtime guard is exercisable without process.env mutation. Pinned by `test/serve-stdio-lifecycle.test.ts`. @@ -180,7 +181,7 @@ strict behavior when unset. - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. diff --git a/VERSION b/VERSION index 7513a6391..d4fe04f21 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.35.5.0 +0.35.5.1 \ No newline at end of file diff --git a/llms-full.txt b/llms-full.txt index 7e8637271..9c59c4848 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -252,6 +252,7 @@ strict behavior when unset. - `src/core/minions/protected-names.ts` — side-effect-free constant module exporting `PROTECTED_JOB_NAMES` + `isProtectedJobName()`. Kept pure so queue core can import without loading handler modules. - `src/core/minions/handlers/shell.ts` — `shell` job handler. Spawns `/bin/sh -c cmd` (absolute path, PATH-override-safe) or `argv[0] argv[1..]` (no shell). Env allowlist: `PATH, HOME, USER, LANG, TZ, NODE_ENV` + caller `env:` overrides. UTF-8-safe stdout/stderr tail via `string_decoder.StringDecoder`. Abort (either `ctx.signal` or `ctx.shutdownSignal`) fires SIGTERM → 5s grace → SIGKILL on child. Requires `GBRAIN_ALLOW_SHELL_JOBS=1` on worker (gated by `registerBuiltinHandlers`). - `src/core/minions/handlers/shell-audit.ts` — per-submission JSONL audit trail at `~/.gbrain/audit/shell-jobs-YYYY-Www.jsonl` (ISO-week rotation; override via `GBRAIN_AUDIT_DIR`). Best-effort: `mkdirSync(recursive)` + `appendFileSync`; failures logged to stderr, submission not blocked. Logs cmd (first 80 chars) or argv (JSON array). Never logs env values. +- `src/core/minions/handlers/supervisor-audit.ts` — supervisor lifecycle JSONL audit at `~/.gbrain/audit/supervisor-YYYY-Www.jsonl` (ISO-week rotation; shares `computeIsoWeekName()` helper with `shell-audit.ts`). `writeSupervisorEvent(emission, supervisorPid)` appends one line per supervisor event (`started`, `worker_spawned`, `worker_exited`, `backoff`, `health_warn`, `health_error`, `max_crashes_exceeded`, `shutting_down`, `stopped`, `worker_spawn_failed`). `readSupervisorEvents({sinceMs})` is the readback path for `gbrain doctor`. **v0.35.5.0:** new exports `isCrashExit(event)`, `summarizeCrashes(events)`, `CrashSummary` type, and `CLEAN_EXIT_CAUSES` denylist (`'clean_exit' | 'graceful_shutdown'`). Single regression point — both `gbrain doctor` (Lane D supervisor check at `doctor.ts:1011-1043`) and `gbrain jobs supervisor status` (`jobs.ts:803-826`) import from here so the two CLI surfaces cannot drift. `isCrashExit` classifies a single `worker_exited` event against the denylist: `clean_exit` / `graceful_shutdown` are NON-crashes; everything else (`runtime_error`, `oom_or_external_kill`, `unknown`, AND any future `likely_cause` value added upstream in `child-worker-supervisor.ts`) is a crash. Pre-v0.34 audit lines lacking `likely_cause` fall back to `code !== 0`. `summarizeCrashes` returns `{total, by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}, clean_exits}` so dashboards bind to named buckets — the `legacy` bucket catches BOTH pre-v0.34 fallback entries AND future unrecognized `likely_cause` values, fail-loud instead of silent underreport. Denylist-over-allowlist was a codex outside-voice catch during `/plan-eng-review` — the bug being fixed (read sites counting every `worker_exited` as a crash, inflating to 120+/day on healthy brains after v0.34.3.0 watchdog drains) was itself an allowlist-of-event-names. Pinned by `test/supervisor-audit.test.ts` (14 cases: 9-case `isCrashExit` branch matrix including denylist regression guard for unrecognized future causes + non-exit-event defensive case, 5-case `summarizeCrashes` aggregator including unrecognized-cause routing to legacy + null-code edge case) and 4 source-grep wiring assertions in `test/doctor.test.ts` guarding both surfaces against drift. - `src/core/minions/backpressure-audit.ts` (v0.19.1) — sibling of shell-audit.ts for `maxWaiting` coalesce events. JSONL at `~/.gbrain/audit/backpressure-YYYY-Www.jsonl`. Fires one line per coalesce with `(queue, name, waiting_count, max_waiting, returned_job_id, ts)`. Closes the silent-drop vector the v0.19.0 maxWaiting guard introduced. - `src/core/minions/handlers/subagent.ts` (v0.15) — LLM-loop handler. Two-phase tool persistence (pending → complete/failed), replay reconciliation for mid-dispatch crashes, dual-signal abort (`ctx.signal` + `ctx.shutdownSignal`), Anthropic prompt caching on system + tool defs. `makeSubagentHandler({engine, client?, ...})` factory; `MessagesClient` is an injectable interface the real SDK implements structurally. Throws `RateLeaseUnavailableError` (renewable) when rate-lease capacity is full. **v0.30.2:** Anthropic 400 `prompt is too long` responses (status 400 + body matches `/prompt is too long|prompt_too_long|context.*length/i`) classify as `UnrecoverableError` so the job goes straight to `dead` on first attempt instead of stalling three times before dead-lettering. Catches both initial-prompt overflow and turn-N tool-loop accumulation that the chunker in `synthesize.ts` can't bound ahead of time. - `src/core/minions/handlers/subagent-aggregator.ts` (v0.15) — `subagent_aggregator` handler. Claims AFTER all children resolve (queue changes guarantee every terminal child posts a `child_done` inbox message with outcome). Reads inbox via `ctx.readInbox()`, builds deterministic mixed-outcome markdown summary. No LLM call in v0.15. @@ -265,7 +266,7 @@ strict behavior when unset. - `src/core/minions/attachments.ts` — Attachment validation (path traversal, null byte, oversize, base64, duplicate detection) - `src/commands/agent.ts` (v0.16) — `gbrain agent run [flags]` CLI. Submits `subagent` (or N children + 1 aggregator) under `{allowProtectedSubmit: true}`. Single-entry `--fanout-manifest` short-circuits. Children get `on_child_fail: 'continue'` + `max_stalled: 3`. `--follow` is the default on TTY; streams logs + polls `waitForCompletion` in parallel. Ctrl-C detaches, does not cancel. - `src/commands/agent-logs.ts` (v0.16) — `gbrain agent logs [--follow] [--since]`. Merges JSONL heartbeat audit + `subagent_messages` into a chronological timeline. `parseSince` accepts ISO-8601 or relative (`5m`, `1h`, `2d`). Transcript tail renders only for terminal jobs. -- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. **v0.28.1:** `case 'work'` now wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging on failure. Replaces the prior call inside `MinionWorker.start()` (which violated engine ownership: the worker disconnected an engine it didn't own, and clobbered the module-level singleton on PostgresEngine via the now-fixed idempotency bug). Pool slots now free immediately on shutdown instead of waiting for TCP keepalive (~minutes). v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase). +- `src/commands/jobs.ts` — `gbrain jobs` CLI subcommands + `gbrain jobs work` daemon. **v0.28.1:** `case 'work'` now wraps `worker.start()` in try/finally and owns engine lifecycle — calls `engine.disconnect()` on shutdown with loud error logging on failure. Replaces the prior call inside `MinionWorker.start()` (which violated engine ownership: the worker disconnected an engine it didn't own, and clobbered the module-level singleton on PostgresEngine via the now-fixed idempotency bug). Pool slots now free immediately on shutdown instead of waiting for TCP keepalive (~minutes). v0.13.1 surfaces the full `MinionJobInput` retry/backoff/timeout/idempotency surface as first-class CLI flags on `jobs submit`: `--max-stalled`, `--backoff-type fixed|exponential`, `--backoff-delay`, `--backoff-jitter`, `--timeout-ms`, `--idempotency-key`. `jobs smoke --sigkill-rescue` is the opt-in regression guard for #219. v0.16 wires `registerBuiltinHandlers` to always register `subagent` + `subagent_aggregator` (no env flag — `ANTHROPIC_API_KEY` is the natural cost gate, trust is via `PROTECTED_JOB_NAMES`) and loads `GBRAIN_PLUGIN_PATH` plugins at worker startup with a loud startup-line per plugin. `shell` handler still gated by `GBRAIN_ALLOW_SHELL_JOBS=1` (RCE surface, separate concern). v0.22.10 (#521): the `autopilot-cycle` handler now forwards `job.data.phases` to `runCycle` (was previously discarded — caller-supplied phase selection silently became a full cycle). Phases are validated against `ALL_PHASES` from `src/core/cycle.ts`; invalid names are filtered out and an empty/missing array falls back to the default 6-phase cycle. v0.22.13 (PR #490 CODEX-1+CODEX-4): `sync` handler now resolves `sourceId` at entry by looking up `sources.local_path` (mirrors `cycle.ts:480`'s autopilot fix from PR #475) so multi-source brains read the per-source `last_commit` anchor instead of the global config key. Concurrency routed through the shared `autoConcurrency()` policy in `src/core/sync-concurrency.ts` instead of the prior hardcoded `4`; PGLite stays serial. `noEmbed` default is `true` (embed is a separate job — submit `gbrain embed --stale` after sync, or rely on the autopilot cycle's embed phase). **v0.35.5.0:** `gbrain jobs supervisor status` at `jobs.ts:803-826` now consumes `summarizeCrashes()` from `src/core/minions/handlers/supervisor-audit.ts` for cross-surface parity with `gbrain doctor`. JSON output adds `crashes_by_cause: {runtime_error, oom_or_external_kill, unknown, legacy}` + `clean_exits_24h` fields so dashboards bind to named buckets; human output gains a per-cause line under `Crashes (24h)` plus a `Clean exits (24h)` line. Pre-fix the read site at `jobs.ts:805` counted every `worker_exited` event as a crash regardless of `likely_cause` — the same bug class the v0.35.5.0 doctor fix closes. Pinned by the 4 source-grep wiring assertions in `test/doctor.test.ts` that require the per-cause breakdown substrings (`crashes_by_cause`, `clean_exits_24h=`) to appear in BOTH `doctor.ts` and `jobs.ts`. - `src/commands/features.ts` — `gbrain features --json --auto-fix`: usage scan + feature adoption salesman - `src/commands/autopilot.ts` — `gbrain autopilot --install`: self-maintaining brain daemon (sync+extract+embed). **v0.28.1:** consumes `detectTini()` from `src/core/minions/spawn-helpers.ts` and resolves it once at startup instead of per worker respawn (was paying an `execFileSync` cost on every restart). **v0.34.3.0:** inline spawn-and-respawn loop replaced with a `ChildWorkerSupervisor` instance. Drops `crashCount`, `lastWorkerStartTime`, `STABLE_RUN_RESET_MS`, `startWorker`, and the inline `child.on('exit')` block — all consolidated into the shared core. `--max-rss 2048` and `maxCrashes: 5` preserved from the legacy loop. `onMaxCrashesExceeded` now routes through autopilot's own `shutdown('max_crashes')` so the autopilot lockfile gets cleaned up (pre-refactor the inline loop called `process.exit(1)` directly and bypassed cleanup). `shutdown()` drains via `childSupervisor.killChild('SIGTERM')` + `awaitChildExit(35_000)` instead of `workerProc.kill()`. Pinned by `test/autopilot-supervisor-wiring.test.ts` (6 static-shape regression guards: composes ChildWorkerSupervisor not the legacy inline names, `--max-rss 2048` in argv, `maxCrashes: 5` literal, shutdown-via-callback wiring, no workerProc reference). Closes the parallel-supervisor bug class Codex flagged during plan-eng-review. - `src/mcp/server.ts` — MCP stdio server (generated from operations). v0.22.7: tool-call handler delegates to `dispatchToolCall` from `src/mcp/dispatch.ts` so stdio + HTTP transports share one validation, context-build, and error-format path. **v0.34.1.0 (#870):** stdin `'end'` / `'close'` shutdown hooks are skipped when `process.env.MCP_STDIO === '1'`. Gateway-piped stdio MCP wrappers (OpenClaw's `bundle-mcp`, similar) pipe the JSON-RPC handshake then close their stdin half; pre-fix this killed the server before the first tool call landed. Signal handlers (SIGTERM / SIGINT / SIGHUP) and the parent-process watchdog still cover legitimate disconnects. `src/commands/serve.ts` exposes `ServeOptions.mcpStdio?: boolean` as a test seam so the runtime guard is exercisable without process.env mutation. Pinned by `test/serve-stdio-lifecycle.test.ts`. @@ -288,7 +289,7 @@ strict behavior when unset. - `test/fixtures/whoknows-eval.jsonl` (v0.33) — 10-row synthetic placeholder demonstrating the eval-fixture schema (`{query, expected_top_3_slugs, notes?}` JSONL). End users replace with their own real queries before shipping; the placeholder uses obviously-example slugs (`wiki/people/example-alice`) so production data isn't conflated with the test fixture. Drives `test/e2e/whoknows.test.ts` (which seeds a matching synthetic brain and asserts the >=80% gate) and the `whoknows_health` doctor check. - `src/commands/transcripts.ts` (v0.29) — `gbrain transcripts recent [--days N] [--full] [--json]`: recent raw `.txt` transcripts from the dream-cycle corpus dirs. Imports `listRecentTranscripts` from `src/core/transcripts.ts` (the same library the gated `get_recent_transcripts` MCP op uses). Local-only by construction — the CLI always runs with `ctx.remote=false`. - `src/commands/integrity.ts` — `gbrain integrity check|auto|review|extract`: bare-tweet detection, dead-link detection, three-bucket repair (auto-repair / review-queue / skip). `scanIntegrity()` is the shared library function called from `gbrain doctor` (sampled at limit=500) and `cmdCheck` (full scan). v0.22.8: batch-load fast path on Postgres uses a single SQL query to fix the PgBouncer round-trip timeout (60s → ~6s). Gated by `engine.kind === 'postgres'` at the call site so PGLite never enters batch; fallback `catch` logs at `GBRAIN_DEBUG=1` so real Postgres errors are diagnosable. **v0.32.8 (PR #860):** batch projection switched from `SELECT DISTINCT ON (slug)` to `SELECT ... ORDER BY source_id, slug` so multi-source brains scan each `(source, slug)` row independently (pre-fix the DISTINCT collapsed same-slug-different-source pages into one scan, the same bug class this PR fixes). Sequential and auto-repair loops use `listAllPageRefs()` to enumerate `(slug, source_id)` pairs and thread `sourceId` to `getPage`. Batch + sequential paths now report the same page count on multi-source brains. -- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. +- `src/commands/doctor.ts` — `gbrain doctor [--json] [--fast] [--fix] [--dry-run] [--index-audit]`: health checks. v0.12.3 added `jsonb_integrity` + `markdown_body_completeness` reliability checks. v0.14.1: `--fix` delegates inlined cross-cutting rules to `> **Convention:** see [path](path).` callouts (pipes DRY violations into `src/core/dry-fix.ts`); `--fix --dry-run` previews without writing. v0.14.2: `schema_version` check fails loudly when `version=0` (migrations never ran — the #218 `bun install -g` signature) and routes users to `gbrain apply-migrations --yes`; new opt-in `--index-audit` flag (Postgres-only) reports zero-scan indexes from `pg_stat_user_indexes` (informational only, no auto-drop). v0.15.2: every DB check is wrapped in a progress phase; `markdown_body_completeness` runs under a 1s heartbeat timer so 10+ min scans are observable on 50K-page brains. v0.19.1 added `queue_health` (Postgres-only) with two subchecks: stalled-forever active jobs (started_at > 1h) and waiting-depth-per-name > threshold (default 10, override via `GBRAIN_QUEUE_WAITING_THRESHOLD`). Worker-heartbeat subcheck intentionally deferred to follow-up B7 because it needs a `minion_workers` table to produce ground-truth signal. Fix hints point at `gbrain repair-jsonb`, `gbrain sync --force`, `gbrain apply-migrations`, and `gbrain jobs get/cancel `. v0.22.12 (#500): `sync_failures` check shows `[CODE=N, ...]` breakdown for both unacked entries (warn) and acked-historical entries (ok), surfacing systemic failure modes (`SLUG_MISMATCH=2685`) instead of a bare count. v0.26.7 (#612): `rls_event_trigger` check (post-install drift detector for migration v35's auto-RLS event trigger). Lives outside the `// 5. RLS` slice that the structural doctor.test.ts guards anchor on, so the existing test guards stay intact. Healthy `evtenabled` set is `('O','A')` only — `R` is replica-only and would not fire in normal sessions; `D` is disabled. Fix hint is `gbrain apply-migrations --force-retry 35`. **v0.30.2:** `queue_health` gains a fourth subcheck — surfaces dead-lettered subagent jobs with `last_error` matching the `prompt_too_long` classifier within the last 24h. Fix hint points at `gbrain dream --phase synthesize --dry-run --json` to identify the offending transcript and `gbrain jobs prune --status dead --queue default` to clean up. Postgres-only. **v0.31.7:** `runDoctor` switches to `autoDetectSkillsDirReadOnly` (from `src/core/repo-root.ts`) so `bun install -g github:garrytan/gbrain && cd ~ && gbrain doctor` finds the bundled `skills/` via the install-path fallback instead of warning "Could not find skills directory" + docking the health score. `--fix` carries a D6 safety gate: when `detected.source === 'install_path'`, the command refuses auto-repair with a stderr message pointing at `$GBRAIN_SKILLS_DIR` / `$OPENCLAW_WORKSPACE` / `--skills-dir`, because `autoFixDryViolations` writes to SKILL.md files and would otherwise silently rewrite the install tree. The `graph_coverage` check now short-circuits to `ok: 'No entity pages — graph_coverage not applicable (markdown-only brain)'` when `SELECT COUNT(*) FROM pages WHERE type IN ('entity','person','company','organization')` returns 0 (closes #530); the entity count is woven into the warn message and the WARN hint switches from the long-deprecated `gbrain link-extract && gbrain timeline-extract` (gone since v0.16) to the canonical `gbrain extract all`. Pinned by an IRON-RULE regression assertion in `test/doctor.test.ts` that bans the stale verb names from the source string. **v0.32.4:** new `sync_freshness` check (exported `checkSyncFreshness` at the same file) added to both `runDoctor` (local) and `doctorReportRemote` (thin-client). Pure staleness probe — queries `sources.last_sync_at` only, no filesystem access. Warns at 24h, fails at 72h (or never-synced). Future-`last_sync_at` warns ("clock skew or corrupted timestamp") instead of silently falling through as ok — codex outside-voice caught the negative-ageMs bug pre-merge. Env-var overrides `GBRAIN_SYNC_FRESHNESS_WARN_HOURS` / `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS`; invalid values fall back to defaults with a once-per-process stderr warn (`_resolveSyncFreshnessHours`). Failure messages embed `source.id` (not `source.name`) so the printed fix command `gbrain sync --source ` matches what the user copy-pastes. Filesystem-vs-DB page drift detection was deliberately stripped from the v0.32.4 scope — `doctorReportRemote` runs in the HTTP MCP server (`src/commands/serve-http.ts`), and walking DB-supplied `local_path` from a remote-callable endpoint crosses a trust boundary (OAuth write scope could mutate `sources.local_path`). Drift detection will resurface in a separate PR routed through `multi_source_drift`'s existing guard infrastructure (`GBRAIN_DRIFT_LIMIT` / `GBRAIN_DRIFT_TIMEOUT_MS`) with slug normalization tests and a meta-file allow-list. Pinned by 12 cases in `test/doctor.test.ts` ("v0.32.4 — sync_freshness check" describe block): empty sources, never-synced fail, >72h fail, exact 72h boundary, 24h-72h warn, exact 24h boundary, <24h ok, future-timestamp warn, mixed sources (highest severity wins), `executeRaw` throws → outer-catch warn, `GBRAIN_SYNC_FRESHNESS_FAIL_HOURS=6` override fires at 7h, source.id-in-message regression. **v0.35.5.0:** the Lane D supervisor check at `doctor.ts:1011-1043` now consumes `summarizeCrashes(events)` from `src/core/minions/handlers/supervisor-audit.ts` instead of the pre-fix `events.filter(e => e.event === 'worker_exited').length`. The warn threshold drops from `>3` to `>=1` (any real crash is signal now that the counter is calibrated against clean exits). The ok message gains `clean_exits_24h=N`; the warn message gains `runtime=A oom=B unknown=C legacy=D` per-cause breakdown so an operator triages OOM vs runtime-error vs unknown-future-cause at a glance without grep'ing the JSONL audit. Closes the "Supervisor crashes: 120x/24h, was 62x — nearly doubled" alarm class that bit users on healthy brains after v0.34.3.0's RSS-watchdog work added more code=0 worker drains — both `doctor` and `gbrain jobs supervisor status` were counting every `worker_exited` event as a crash regardless of cause. Cross-surface parity is the regression guard: 4 source-grep wiring assertions in `test/doctor.test.ts` ban the ad-hoc filter pattern, pin the `>=1` threshold, and require the per-cause breakdown substrings (`runtime=`, `oom=`, `unknown=`, `legacy=`, `clean_exits_24h=`, `crashes_by_cause`) to appear in BOTH `doctor.ts` and `jobs.ts`. - `src/core/migrate.ts` — schema-migration runner. Owns the `MIGRATIONS` array (source of truth for schema DDL). **v40 (v0.29):** `pages_emotional_weight` adds `pages.emotional_weight REAL NOT NULL DEFAULT 0.0`. Column-only (no index). On Postgres 11+ and PGLite, `ADD COLUMN` with a constant DEFAULT is metadata-only — instant on tables of any size. v0.14.2 extended the `Migration` interface with `sqlFor?: { postgres?, pglite? }` (engine-specific SQL overrides `sql`) and `transaction?: boolean` (set to false for `CREATE INDEX CONCURRENTLY`, which Postgres refuses inside a transaction; ignored on PGLite since it has no concurrent writers). Migration v14 (fix wave) uses a handler branching on `engine.kind` to run CONCURRENTLY on Postgres (with a pre-drop of any invalid remnant via `pg_index.indisvalid`) and plain `CREATE INDEX` on PGLite. v15 bumps `minion_jobs.max_stalled` default 1→5 and backfills existing non-terminal rows. v0.22.6.1: migration v24 (`rls_backfill_missing_tables`) uses `sqlFor: { pglite: '' }` to no-op on PGLite — PGLite has no RLS engine and is single-tenant by definition, and the v24 ALTERs target subagent tables that don't exist in pglite-schema.ts. Closes #395 (contributed by @jdcastro2). **v30 (v0.23):** creates `dream_verdicts (file_path TEXT, content_hash TEXT, worth_processing BOOL, reasons JSONB, judged_at TIMESTAMPTZ, PK(file_path, content_hash))`. RLS-enabled when running as a BYPASSRLS role. The synthesize phase reads/writes this table to avoid re-judging on backfill re-runs. **v35 (v0.26.7):** auto-RLS event trigger + one-time backfill. `auto_rls_on_create_table` fires on `ddl_command_end` for `WHEN TAG IN ('CREATE TABLE','CREATE TABLE AS','SELECT INTO')` and runs `ALTER TABLE … ENABLE ROW LEVEL SECURITY` on every new `public.*` table — no FORCE (matches v24/v29/schema.sql posture so non-BYPASSRLS apps can still read their own tables). The same migration backfills RLS on every existing `public.*` base table whose comment doesn't match the doctor regex (`^GBRAIN:RLS_EXEMPT\s+reason=\S.{3,}`). Per-table failure aborts the offending CREATE TABLE (event triggers fire inside the DDL transaction); no EXCEPTION wrap — that would convert loud rollback into silent permissive default. PGLite no-op via `sqlFor.pglite: ''`. Breaking change: operators with intentionally-RLS-off public tables must add the GBRAIN:RLS_EXEMPT comment BEFORE upgrade or the backfill will flip them on. **v46 (v0.31.3):** `mcp_request_log_params_jsonb_normalize` rewrites pre-v0.31.3 rows where `mcp_request_log.params` was stored as a JSON-encoded string (`jsonb_typeof = 'string'`) up to a real JSONB object via `UPDATE ... SET params = params::text::jsonb WHERE jsonb_typeof(params) = 'string'`. Single statement, idempotent — second-run finds no string-shaped rows and is a no-op. Closes the bug where `/admin/api/requests` returned a quoted string instead of the parsed object. **v0.34.1.0 (#861 + #876, v60-v65):** six-migration chain wires source-scoping into the OAuth client table. v60 (`oauth_clients_source_id_fk`) adds `oauth_clients.source_id TEXT` with NULL→`'default'` backfill and an FK to `sources(id) ON DELETE SET NULL`. v61 (`oauth_clients_federated_read_column`) adds `federated_read TEXT[] NOT NULL DEFAULT '{}'`. v62 (`oauth_clients_federated_read_backfill`) explicit-CASE backfills so `source_id IS NULL` produces `'{}'` not an array-containing-NULL. v63 (`oauth_clients_federated_read_validate`) is the fail-loud check that every row's source_id is in its federated_read array post-backfill. v64 (`oauth_clients_source_id_fk_restrict`) flips the FK to `ON DELETE RESTRICT` now that federated_read provides the alternative scope-loss path — source delete is refused if any client references it. v65 (`oauth_clients_federated_read_gin_index`) is the GIN index for the array-containment queries the read paths run. PGLite parity via `sqlFor.pglite` where needed. - `src/core/progress.ts` — Shared bulk-action progress reporter. Writes to stderr. Modes: `auto` (TTY: `\r`-rewriting; non-TTY: plain lines), `human`, `json` (JSONL), `quiet`. Rate-gated by `minIntervalMs` and `minItems`. `startHeartbeat(reporter, note)` helper for single long queries. `child()` composes phase paths. Singleton SIGINT/SIGTERM coordinator emits `abort` events for every live phase. EPIPE defense on both sync throws and stream `'error'` events. Zero dependencies. Introduced in v0.15.2. - `src/core/cli-options.ts` — Global CLI flag parser. `parseGlobalFlags(argv)` returns `{cliOpts, rest}` with `--quiet` / `--progress-json` / `--progress-interval=` stripped. `getCliOptions()` / `setCliOptions()` expose a module-level singleton so commands reach the resolved flags without parameter threading. `cliOptsToProgressOptions()` maps to reporter options. `childGlobalFlags()` returns the flag suffix to append to `execSync('gbrain ...')` calls in migration orchestrators. `OperationContext.cliOpts` extends shared-op dispatch for MCP callers. diff --git a/package.json b/package.json index 8f104bf65..2f1907aaf 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.35.5.0", + "version": "0.35.5.1", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index c5658fc52..129f32598 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -8,7 +8,6 @@ import { loadCompletedMigrations } from '../core/preferences.ts'; import { compareVersions } from './migrations/index.ts'; import { createProgress, startHeartbeat, type ProgressReporter } from '../core/progress.ts'; import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; -import { classifyWorkerExit } from '../core/minions/exit-classification.ts'; import type { DbUrlSource } from '../core/config.ts'; import { join } from 'path'; import { existsSync, readFileSync, readdirSync } from 'fs'; @@ -994,7 +993,7 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo // Does NOT run the supervisor itself — this is a read-only health check. try { const { DEFAULT_PID_FILE } = await import('../core/minions/supervisor.ts'); - const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts'); + const { readSupervisorEvents, summarizeCrashes } = await import('../core/minions/handlers/supervisor-audit.ts'); let supervisorPid: number | null = null; let running = false; @@ -1011,12 +1010,20 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 }); const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null; - // Only count non-zero exits as crashes; clean restarts (code 0) are normal - // worker lifecycle — the worker finishes its queue and exits cleanly. - const allExits = events.filter(e => e.event === 'worker_exited'); - const realCrashes = allExits.filter(e => classifyWorkerExit(e as { code?: number | null }) === 'crash'); - const cleanRestarts = allExits.length - realCrashes.length; - const crashes24h = realCrashes.length; + // Shared classifier — same code path runs in `gbrain jobs supervisor + // status` (src/commands/jobs.ts). Counts only events whose `likely_cause` + // is NOT in the clean denylist (clean_exit, graceful_shutdown). Pre-v0.34 + // entries lacking `likely_cause` fall back to `code !== 0`. Supersedes + // v0.35.4.0's binary `classifyWorkerExit({code})` on this surface: the + // `likely_cause` read correctly classifies SIGTERM (code=null, + // likely_cause='graceful_shutdown') as clean, and produces per-cause + // buckets so operators triage memory pressure (oom) vs code bugs + // (runtime) without grep'ing JSONL. `classifyWorkerExit` is still + // used by the supervisor's internal restart policy where the binary + // shape is the right contract. + const summary = summarizeCrashes(events); + const crashes24h = summary.total; + const causeStr = `runtime=${summary.by_cause.runtime_error} oom=${summary.by_cause.oom_or_external_kill} unknown=${summary.by_cause.unknown} legacy=${summary.by_cause.legacy}`; const maxCrashesEvent = events.filter(e => e.event === 'max_crashes_exceeded').pop() ?? null; // Only surface a Check if the supervisor was ever observed (stops the @@ -1034,17 +1041,20 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo status: 'warn', message: `Supervisor not running (last_start=${lastStart ?? 'unknown'}). Restart with: gbrain jobs supervisor start --detach`, }); - } else if (crashes24h > 3) { + } else if (crashes24h >= 1) { + // Threshold dropped from `>3` (pre-fix, inflated by clean exits being + // miscounted) to `>=1` (any real crash is signal). Per-cause breakdown + // gives operators triage context without grep'ing the JSONL. checks.push({ name: 'supervisor', status: 'warn', - message: `Supervisor running but worker crashed ${crashes24h}x in last 24h (${cleanRestarts} clean restarts excluded). Check ~/.gbrain/audit/supervisor-*.jsonl for causes.`, + message: `Worker crashed ${crashes24h}x in last 24h (${causeStr}). Check ~/.gbrain/audit/supervisor-*.jsonl for context.`, }); } else { checks.push({ name: 'supervisor', status: 'ok', - message: `running=true pid=${supervisorPid} last_start=${lastStart ?? 'unknown'} crashes=${crashes24h}${cleanRestarts > 0 ? ` clean_restarts=${cleanRestarts}` : ''}`, + message: `running=true pid=${supervisorPid} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h} clean_exits_24h=${summary.clean_exits}`, }); } } diff --git a/src/commands/jobs.ts b/src/commands/jobs.ts index c2c81072b..60efeb9be 100644 --- a/src/commands/jobs.ts +++ b/src/commands/jobs.ts @@ -6,7 +6,6 @@ import type { BrainEngine } from '../core/engine.ts'; import { MinionQueue } from '../core/minions/queue.ts'; import { MinionWorker } from '../core/minions/worker.ts'; -import { classifyWorkerExit } from '../core/minions/exit-classification.ts'; import type { MinionJob, MinionJobStatus } from '../core/minions/types.ts'; import { loadConfig, isThinClient } from '../core/config.ts'; import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts'; @@ -786,7 +785,7 @@ HANDLER TYPES (built in) // ----- status subcommand ----- if (isStatusCmd) { const { existsSync, readFileSync } = await import('fs'); - const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts'); + const { readSupervisorEvents, summarizeCrashes } = await import('../core/minions/handlers/supervisor-audit.ts'); let supervisorPid: number | null = null; let running = false; @@ -803,10 +802,11 @@ HANDLER TYPES (built in) const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 }); const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null; - const allExits = events.filter(e => e.event === 'worker_exited'); - const realCrashes = allExits.filter(e => classifyWorkerExit(e as { code?: number | null }) === 'crash'); - const cleanRestarts = allExits.length - realCrashes.length; - const crashes24h = realCrashes.length; + // Shared classifier — same code path runs in `gbrain doctor` so the + // two surfaces cannot drift on what counts as a crash. Supersedes + // v0.35.4.0's binary `classifyWorkerExit({code})` on this surface; + // see doctor.ts for the layering rationale. + const summary = summarizeCrashes(events); const maxCrashesEvent = events.filter(e => e.event === 'max_crashes_exceeded').pop() ?? null; const status = { @@ -814,8 +814,9 @@ HANDLER TYPES (built in) supervisor_pid: supervisorPid, pid_file: pidFile, last_start: lastStart, - crashes_24h: crashes24h, - clean_restarts_24h: cleanRestarts, + crashes_24h: summary.total, + clean_exits_24h: summary.clean_exits, + crashes_by_cause: summary.by_cause, max_crashes_exceeded: !!maxCrashesEvent, }; @@ -826,7 +827,8 @@ HANDLER TYPES (built in) if (supervisorPid) console.log(` PID: ${supervisorPid}`); console.log(` PID file: ${pidFile}`); if (lastStart) console.log(` Last start: ${lastStart}`); - console.log(` Crashes (24h): ${crashes24h}${cleanRestarts > 0 ? ` (${cleanRestarts} clean restarts excluded)` : ''}`); + console.log(` Crashes (24h): ${summary.total} (runtime=${summary.by_cause.runtime_error} oom=${summary.by_cause.oom_or_external_kill} unknown=${summary.by_cause.unknown} legacy=${summary.by_cause.legacy})`); + console.log(` Clean exits (24h): ${summary.clean_exits}`); if (maxCrashesEvent) console.log(` ⚠ Max crashes exceeded at ${maxCrashesEvent.ts}`); } process.exit(running ? 0 : 1); diff --git a/src/core/minions/handlers/supervisor-audit.ts b/src/core/minions/handlers/supervisor-audit.ts index 49138a328..44c463076 100644 --- a/src/core/minions/handlers/supervisor-audit.ts +++ b/src/core/minions/handlers/supervisor-audit.ts @@ -105,3 +105,90 @@ export function readSupervisorEvents(opts: { sinceMs?: number } = {}): Superviso } return events; } + +/** + * Denylist of clean-exit `likely_cause` values. Anything not in this set — + * including future unrecognized values — counts as a crash. Matches the + * domain asymmetry: clean exits are explicit (the worker exited because we + * asked it to); crashes are an open catch-all. If a future maintainer adds a + * new `likely_cause` upstream in `child-worker-supervisor.ts` (e.g. + * `lock_lost`, `panic`), the doctor surfaces it by default instead of + * silently underreporting — denylist semantics close the bug class this + * helper was added to fix. + */ +const CLEAN_EXIT_CAUSES = new Set(['clean_exit', 'graceful_shutdown']); + +/** + * Per-cause crash bucket shape returned by `summarizeCrashes()`. Bucket names + * mirror the upstream `likely_cause` values: `runtime_error` (code=1), + * `oom_or_external_kill` (SIGKILL), `unknown` (other signals/codes). The + * `legacy` bucket catches pre-v0.34 entries lacking `likely_cause` that fall + * through to the `code !== 0` fallback. + */ +export interface CrashSummary { + total: number; + by_cause: { + runtime_error: number; + oom_or_external_kill: number; + unknown: number; + legacy: number; + }; + clean_exits: number; +} + +/** + * Classify a single audit event. Returns true when the event represents a + * worker crash (not a clean shutdown, watchdog drain, or non-exit lifecycle + * event). Pre-v0.34 audit lines lacking `likely_cause` fall back to + * `code !== 0`. + */ +export function isCrashExit(event: SupervisorEmission): boolean { + if (event.event !== 'worker_exited') return false; + const cause = event.likely_cause as string | undefined; + if (cause === undefined) { + // Legacy fallback for pre-v0.34 entries lacking `likely_cause`. Treat + // any non-zero exit code as a crash; missing/null `code` also counts + // (truly malformed line — fail-loud, the user can investigate the audit + // file directly). + const code = event.code as number | null | undefined; + return code !== 0; + } + return !CLEAN_EXIT_CAUSES.has(cause); +} + +/** + * Summarize crash counts across a window of supervisor audit events. Both + * `gbrain doctor` and `gbrain jobs supervisor status` consume this — single + * regression point, single test target. + * + * Bucketing rule: `worker_exited` events classified as crashes by + * `isCrashExit()` are dispatched to `by_cause` based on `likely_cause`. The + * `legacy` bucket catches BOTH (a) pre-v0.34 entries lacking `likely_cause` + * that fell through to the `code !== 0` fallback, AND (b) future + * unrecognized `likely_cause` values not in the explicit allowlist + * (`runtime_error` / `oom_or_external_kill` / `unknown`). Operators + * watching `legacy=N` rise know the upstream classifier added a value the + * doctor doesn't yet name — that's the intended signal for "extend my + * bucket vocabulary." + */ +export function summarizeCrashes(events: SupervisorEmission[]): CrashSummary { + const summary: CrashSummary = { + total: 0, + by_cause: { runtime_error: 0, oom_or_external_kill: 0, unknown: 0, legacy: 0 }, + clean_exits: 0, + }; + for (const e of events) { + if (e.event !== 'worker_exited') continue; + if (!isCrashExit(e)) { + summary.clean_exits++; + continue; + } + summary.total++; + const cause = e.likely_cause as string | undefined; + if (cause === 'runtime_error') summary.by_cause.runtime_error++; + else if (cause === 'oom_or_external_kill') summary.by_cause.oom_or_external_kill++; + else if (cause === 'unknown') summary.by_cause.unknown++; + else summary.by_cause.legacy++; // pre-v0.34 fallback OR future unrecognized cause + } + return summary; +} diff --git a/test/doctor.test.ts b/test/doctor.test.ts index 389d50cc5..d5b422226 100644 --- a/test/doctor.test.ts +++ b/test/doctor.test.ts @@ -579,6 +579,65 @@ describe('v0.32.4 — sync_freshness check', () => { }); }); +// Supervisor crash classifier wiring. Pre-fix, doctor.ts:1013 counted every +// `worker_exited` event as a crash regardless of `likely_cause`, inflating +// `crashes_24h` to 120+/day from RSS-watchdog drains and SIGTERM stops. +// These tests pin the read-side wiring so doctor and `gbrain jobs supervisor +// status` (jobs.ts:805) cannot drift: both go through `summarizeCrashes`. +describe('supervisor crash classifier wiring (v0.35.x)', () => { + test('doctor.ts uses summarizeCrashes — no ad-hoc worker_exited filter', async () => { + const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text(); + // Wired to the shared helper. + expect(source).toContain('summarizeCrashes'); + // The pre-fix ad-hoc filter pattern must NOT survive. The exact buggy + // expression was `events.filter(e => e.event === 'worker_exited').length`. + // Match the structural fingerprint, not whitespace. + expect(source).not.toMatch( + /events\.filter\([^)]*e\.event\s*===\s*'worker_exited'[^)]*\)\.length/, + ); + }); + + test('doctor.ts warn threshold dropped from >3 to >=1', async () => { + const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text(); + // The pre-fix `crashes24h > 3` threshold made sense only because the + // counter was over-counting clean exits. Under accurate counts, any real + // crash is signal — threshold lands at `>=1`. + expect(source).toMatch(/crashes24h\s*>=\s*1/); + // The old `> 3` predicate must not survive on the supervisor check. + expect(source).not.toMatch(/crashes24h\s*>\s*3/); + }); + + test('doctor.ts ok + warn messages include per-cause breakdown and clean_exits_24h', async () => { + const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text(); + // Per-cause breakdown surfaces qualitative signal (oom vs runtime vs unknown + // vs legacy) so operators can triage without grep'ing JSONL. + expect(source).toContain('runtime='); + expect(source).toContain('oom='); + expect(source).toContain('unknown='); + expect(source).toContain('legacy='); + // Clean-exit count surfaces alongside crash count for transparency. + expect(source).toContain('clean_exits_24h='); + }); + + test('jobs.ts supervisor status uses summarizeCrashes — same wiring as doctor', async () => { + const source = await Bun.file(new URL('../src/commands/jobs.ts', import.meta.url)).text(); + // Both surfaces MUST go through the shared helper. Without this, the two + // CLI commands report drifting crash counts (the bug class codex caught + // during the eng review outside-voice pass). + expect(source).toContain('summarizeCrashes'); + expect(source).not.toMatch( + /events\.filter\([^)]*e\.event\s*===\s*'worker_exited'[^)]*\)\.length/, + ); + // JSON output exposes the per-cause breakdown so dashboards/monitors can + // distinguish memory pressure from code bugs without re-classifying. + expect(source).toContain('crashes_by_cause'); + expect(source).toContain('clean_exits_24h'); + }); +}); + +// v0.34.5 stub-guard observability tests (from v0.35.4.0). Doctor surfaces +// the 24h fire count for the resolver-stub-guard. WARN at >10 hits is the +// signal that prefix-expansion in resolveEntitySlug is missing a case. describe('stub_guard_24h check (v0.34.5)', () => { test('doctor source defines the stub_guard_24h check', async () => { const source = await Bun.file(new URL('../src/commands/doctor.ts', import.meta.url)).text(); diff --git a/test/exit-classification.test.ts b/test/exit-classification.test.ts index ace310b83..b71ccabf4 100644 --- a/test/exit-classification.test.ts +++ b/test/exit-classification.test.ts @@ -34,22 +34,29 @@ describe('consumer wire-up — helper used by all 3 sites (no inline filters lef // accidentally inlines the rule again gets caught at test time, not at // production-divergence time. + // v0.35.5.0: doctor.ts and jobs.ts moved from `classifyWorkerExit` (binary + // code-based) to `summarizeCrashes` (per-cause via `likely_cause`). The + // wire-up contract is "must use a shared helper, not an inline filter" — + // the specific helper differs by site. The supervisor's internal restart + // policy still uses `classifyWorkerExit` (binary is the right shape there). const SITES = [ - { label: 'doctor.ts', path: 'src/commands/doctor.ts' }, - { label: 'jobs.ts', path: 'src/commands/jobs.ts' }, - { label: 'child-worker-supervisor.ts', path: 'src/core/minions/child-worker-supervisor.ts' }, + { label: 'doctor.ts', path: 'src/commands/doctor.ts', helper: 'summarizeCrashes' }, + { label: 'jobs.ts', path: 'src/commands/jobs.ts', helper: 'summarizeCrashes' }, + { label: 'child-worker-supervisor.ts', path: 'src/core/minions/child-worker-supervisor.ts', helper: 'classifyWorkerExit' }, ]; for (const site of SITES) { - it(`${site.label} imports classifyWorkerExit`, () => { + it(`${site.label} uses a shared classifier helper (${site.helper})`, () => { const source = readFileSync(join(import.meta.dir, '..', site.path), 'utf8'); - // Either named import OR import-from of the helper file — both count. - expect(source).toMatch(/(import\s+\{[^}]*classifyWorkerExit[^}]*\}|from\s+['"][^'"]*exit-classification)/); + // Helper is either imported by name (top-level) or via dynamic import. + const helperRe = new RegExp(`\\b${site.helper}\\b`); + expect(source).toMatch(helperRe); }); - it(`${site.label} calls classifyWorkerExit at least once`, () => { + it(`${site.label} calls ${site.helper} at least once`, () => { const source = readFileSync(join(import.meta.dir, '..', site.path), 'utf8'); - expect(source).toMatch(/classifyWorkerExit\s*\(/); + const callRe = new RegExp(`\\b${site.helper}\\s*\\(`); + expect(source).toMatch(callRe); }); } diff --git a/test/supervisor-audit.test.ts b/test/supervisor-audit.test.ts new file mode 100644 index 000000000..7f128c4b4 --- /dev/null +++ b/test/supervisor-audit.test.ts @@ -0,0 +1,177 @@ +/** + * Unit tests for the shared crash classifier used by `gbrain doctor` and + * `gbrain jobs supervisor status`. Both surfaces import `isCrashExit` + + * `summarizeCrashes` from `src/core/minions/handlers/supervisor-audit.ts`; + * pinning them here keeps the two CLI surfaces from drifting. + * + * Why this file exists: pre-fix the doctor counted every `worker_exited` + * event as a crash, regardless of `likely_cause`. Clean SIGTERM shutdowns + * and RSS-watchdog drains (code=0) inflated `crashes_24h` to 120+/day on + * Garry's brain. The classifier upstream in child-worker-supervisor.ts + * already stamped `likely_cause` correctly; the read sites just ignored it. + * These tests pin every branch of the new shared classifier so the bug + * cannot silently recur. + */ + +import { describe, test, expect } from 'bun:test'; +import { + isCrashExit, + summarizeCrashes, + type CrashSummary, +} from '../src/core/minions/handlers/supervisor-audit.ts'; +import type { SupervisorEmission } from '../src/core/minions/supervisor.ts'; + +// Helper: build a SupervisorEmission of the given event with arbitrary extra +// fields. `ts` is required by the type but irrelevant to the classifier; we +// stamp a constant so failures show predictable fixtures. +function evt( + event: SupervisorEmission['event'], + extra: Record = {}, +): SupervisorEmission { + return { event, ts: '2026-05-16T00:00:00Z', ...extra }; +} + +describe('isCrashExit — branch matrix', () => { + // Case 1: explicit clean exit (worker returned code=0 voluntarily, + // e.g. RSS watchdog drain). The most common cause of the original + // bug — every drain was being counted as a crash. + test('clean_exit is not a crash', () => { + expect(isCrashExit(evt('worker_exited', { likely_cause: 'clean_exit', code: 0 }))).toBe(false); + }); + + // Case 2: SIGTERM-driven shutdown (operator stop, OS-initiated graceful + // termination). Also not a crash. + test('graceful_shutdown is not a crash', () => { + expect(isCrashExit(evt('worker_exited', { likely_cause: 'graceful_shutdown', signal: 'SIGTERM' }))).toBe(false); + }); + + // Case 3: code=1 from the worker process — a real runtime error. + test('runtime_error is a crash', () => { + expect(isCrashExit(evt('worker_exited', { likely_cause: 'runtime_error', code: 1 }))).toBe(true); + }); + + // Case 4: SIGKILL (kernel OOM kill, external `kill -9`). Real crash. + test('oom_or_external_kill is a crash', () => { + expect(isCrashExit(evt('worker_exited', { likely_cause: 'oom_or_external_kill', signal: 'SIGKILL' }))).toBe(true); + }); + + // Case 5: catch-all bucket from the upstream classifier (unusual code or + // signal combination). Real crash. + test('unknown is a crash', () => { + expect(isCrashExit(evt('worker_exited', { likely_cause: 'unknown', code: 137 }))).toBe(true); + }); + + // Case 6: denylist regression guard. If a future maintainer adds a NEW + // `likely_cause` value upstream (e.g. `lock_lost`, `panic`, + // `db_connection_lost`), the doctor MUST surface it by default. Allowlist + // semantics would have silently misclassified this as clean — the exact + // bug class this fix exists to close. + test('unrecognized future likely_cause is a crash (denylist regression guard)', () => { + expect(isCrashExit(evt('worker_exited', { likely_cause: 'future_value_not_known' }))).toBe(true); + }); + + // Case 7: legacy fallback path — pre-v0.34 audit lines lacking + // `likely_cause`. Use `code` to classify: code=0 is clean. + test('legacy (no likely_cause) with code=0 is not a crash', () => { + expect(isCrashExit(evt('worker_exited', { code: 0 }))).toBe(false); + }); + + // Case 8: legacy fallback path — pre-v0.34 entry with code=1 (real crash). + test('legacy (no likely_cause) with code=1 is a crash', () => { + expect(isCrashExit(evt('worker_exited', { code: 1 }))).toBe(true); + }); + + // Case 9: defensive — non-exit lifecycle events MUST never be counted as + // crashes regardless of their other fields. Catches a future caller that + // forgets the upstream `event === 'worker_exited'` filter. + test('non-exit event is never a crash', () => { + expect(isCrashExit(evt('started'))).toBe(false); + expect(isCrashExit(evt('worker_spawned', { code: 1 }))).toBe(false); + expect(isCrashExit(evt('max_crashes_exceeded', { likely_cause: 'runtime_error' }))).toBe(false); + }); +}); + +describe('summarizeCrashes — aggregation', () => { + // Feed a representative mixed stream and assert every bucket. The mix + // exercises every classifier branch so the message-format consumers + // (doctor.ts and jobs.ts) get a stable shape. + test('aggregates a mixed stream into per-cause buckets and clean_exits', () => { + const events: SupervisorEmission[] = [ + evt('worker_exited', { likely_cause: 'runtime_error', code: 1 }), + evt('worker_exited', { likely_cause: 'runtime_error', code: 1 }), + evt('worker_exited', { likely_cause: 'oom_or_external_kill', signal: 'SIGKILL' }), + evt('worker_exited', { likely_cause: 'unknown', code: 137 }), + evt('worker_exited', { code: 1 }), // legacy (no likely_cause) + evt('worker_exited', { likely_cause: 'clean_exit', code: 0 }), + evt('worker_exited', { likely_cause: 'clean_exit', code: 0 }), + evt('worker_exited', { likely_cause: 'clean_exit', code: 0 }), + evt('worker_exited', { likely_cause: 'graceful_shutdown', signal: 'SIGTERM' }), + // Non-exit events MUST be ignored (no double-counting against either bucket). + evt('started'), + evt('worker_spawned'), + evt('health_warn'), + ]; + + const summary: CrashSummary = summarizeCrashes(events); + + expect(summary.total).toBe(5); + expect(summary.by_cause.runtime_error).toBe(2); + expect(summary.by_cause.oom_or_external_kill).toBe(1); + expect(summary.by_cause.unknown).toBe(1); + expect(summary.by_cause.legacy).toBe(1); + expect(summary.clean_exits).toBe(4); + // total + clean_exits should equal the count of worker_exited events, + // proving non-exit lifecycle events were excluded from both buckets. + const exitCount = events.filter((e) => e.event === 'worker_exited').length; + expect(summary.total + summary.clean_exits).toBe(exitCount); + }); + + test('empty input returns zero summary', () => { + const summary = summarizeCrashes([]); + expect(summary).toEqual({ + total: 0, + by_cause: { runtime_error: 0, oom_or_external_kill: 0, unknown: 0, legacy: 0 }, + clean_exits: 0, + }); + }); + + test('only non-exit events returns zero summary', () => { + const summary = summarizeCrashes([evt('started'), evt('worker_spawned'), evt('stopped')]); + expect(summary.total).toBe(0); + expect(summary.clean_exits).toBe(0); + }); + + // Denylist regression guard at the AGGREGATOR level. isCrashExit Case 6 + // proves an unrecognized future `likely_cause` is counted as a crash; this + // pins which BUCKET it lands in. The `else` branch in summarizeCrashes + // routes any crash-classified event whose cause doesn't match the three + // explicit buckets into `legacy`. Operators watching `legacy=N` rise know + // the upstream classifier added a value the doctor doesn't yet name — + // that's the intended signal. + test('unrecognized future likely_cause routes to legacy bucket', () => { + const summary = summarizeCrashes([ + evt('worker_exited', { likely_cause: 'lock_lost' }), + evt('worker_exited', { likely_cause: 'panic' }), + ]); + expect(summary.total).toBe(2); + expect(summary.by_cause.legacy).toBe(2); + expect(summary.by_cause.runtime_error).toBe(0); + expect(summary.by_cause.oom_or_external_kill).toBe(0); + expect(summary.by_cause.unknown).toBe(0); + }); + + // Truly malformed legacy line — `likely_cause` missing AND `code` null + // (or undefined). The classifier comment explicitly says "fail-loud, the + // user can investigate the audit file directly", which means count it. + // null !== 0 is true so isCrashExit returns true; summarizeCrashes then + // lands it in legacy. Pinning this prevents a regression where a future + // refactor adds `code != null` and silently drops malformed entries. + test('legacy entry with null code counts as crash in legacy bucket', () => { + const summary = summarizeCrashes([ + evt('worker_exited', { code: null }), + ]); + expect(summary.total).toBe(1); + expect(summary.by_cause.legacy).toBe(1); + expect(summary.clean_exits).toBe(0); + }); +});