Merge remote-tracking branch 'origin/master' into garrytan/cathedral-1

# Conflicts:
#	CHANGELOG.md
#	TODOS.md
#	VERSION
#	package.json
This commit is contained in:
Garry Tan
2026-06-25 09:47:42 -07:00
59 changed files with 2689 additions and 156 deletions
+43
View File
@@ -20,6 +20,49 @@ concurrency:
cancel-in-progress: true
jobs:
jsonb-parity:
# Dedicated required guard for the JSONB double-encode bug-class (#2339).
# PGLite parses a double-encoded jsonb string silently, so this assertion can
# ONLY be made on real Postgres — a normal gated e2e file would skip without
# DATABASE_URL and let the bug ship green (as #2339 did). This job provisions
# Postgres and HARD-FAILS if DATABASE_URL is missing, so the guard can never
# silently skip.
name: JSONB parity (#2339 regression guard)
runs-on: ubuntu-latest
timeout-minutes: 15
services:
postgres:
image: pgvector/pgvector:pg16
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: gbrain_test
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: 1.3.13
- run: bun install
- name: Require DATABASE_URL (no silent skip)
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: |
if [ -z "$DATABASE_URL" ]; then
echo "::error::DATABASE_URL must be set for the jsonb-parity job — the #2339 guard would silently skip (the exact failure PGLite hides). Failing the job." >&2
exit 1
fi
- name: Run JSONB double-encode parity tests on real Postgres
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/gbrain_test
run: bun test test/e2e/op-checkpoint-jsonb-parity.test.ts test/e2e/jsonb-roundtrip.test.ts
tier1:
name: Tier 1 (Mechanical)
runs-on: ubuntu-latest
+36
View File
@@ -49,6 +49,42 @@ complete operation catalog — both speak the verbs). Run `gbrain protocol confo
to self-certify, and `gbrain protocol stats` to watch adoption. Memories your agent
saves are readable by every agent connected to the brain by default; pass
`visibility: "private"` for local-only facts.
## [0.42.53.0] - 2026-06-23
**`gbrain sync` works again on managed Postgres brains: the durable-checkpoint pin write was encoding its value the wrong way, so every multi-source sync aborted at the very first checkpoint. Fixed, plus a repo-wide sweep of the same JSONB footgun and a new CI guard so it can't come back.** A recent release added a structural check on the sync checkpoint table; the pin write that runs before every drain bound its value as a string rather than a real array, so the check rejected it and the run bailed before importing anything. The bug was invisible on the embedded engine (its driver parses the value either way) and only bit managed Postgres.
### Fixed
- **Multi-source sync no longer aborts at the first checkpoint.** The sync-target pin write now binds its value so Postgres stores a genuine JSONB array instead of a double-encoded string scalar. A dedicated Postgres CI job exercises this on a real database, because the embedded test engine masks the failure — which is exactly why it shipped.
- **The same JSONB double-encode footgun is swept across the codebase.** Every raw write that serialized a value into a JSONB column the bug-prone way is corrected to the safe form (search cache, source config, calibration profiles, subagent tool records, eval receipts, code-intel cache, symbol resolver, and others). Readers were already defensive, so existing rows self-heal as each is rewritten.
- **`gbrain eval suspected-contradictions` no longer crashes on an exact-alias query.** An alias-matched result was missing its page id, which aborted the whole probe on Postgres; the id is now carried through, with a finite-id filter as a defensive backstop.
### Added
- **A CI guard for the positional JSONB double-encode pattern.** The existing guard caught only the template-string spelling; a new static check (`scripts/check-jsonb-params.mjs`) catches the positional-parameter form — the one behind this wave — across the codebase, with its own self-test. The embedded engine's native path is intentionally not flagged, since the bug can't occur there.
### To take advantage of v0.42.53.0
`gbrain upgrade`. Multi-source Postgres brains that had stopped syncing resume on the next `gbrain sync` — no migration, no manual step. Rows written in the double-encoded form before the fix self-heal as each is rewritten; re-running the affected write (or a sync) repairs them eagerly if you'd rather not wait.
## [0.42.52.0] - 2026-06-18
**Autopilot stops manufacturing dead jobs and wedging its own queue, plus four operational rough edges get fixed: minion attempt-accounting, `agent run` flag parsing, honest `sources status`, and a budgeted `gbrain status`.** On a multi-source Postgres brain, autopilot could fan out a continuous stream of dead `autopilot-cycle` jobs while the supervisor periodically wedged the very queue it exists to keep alive. The root cause was one disease with several interacting parts; this wave addresses all of them, then cleans up four smaller reliability bugs found alongside.
### Changed
- **Autopilot runs one brain-wide maintenance pass, not one per source.** The cycle is split: per-source jobs run only source-scoped phases, and a single `autopilot-global-maintenance` job runs the brain-wide phases once per window. This removes the per-cycle memory blow-up that was the shared root cause of the dead-job storm and the queue wedge. Per-source filesystem phases bind to the source's own path, so the freshness stamp and the work agree on which source ran.
- **The supervisor self-heals instead of giving up.** A transient database blip no longer trips the crash-budget breaker into a permanent stop; the supervisor degrades to capped-backoff retry and recovers, with a hard ceiling as the backstop. It detects a live sibling supervisor through the queue's database lock (not a `$HOME`-derived pidfile), so two supervisors under a split home directory can't both claim the queue.
- **`gbrain sources status` tells a running sync apart from an idle source.** A source holding a live sync lock now reads as actively syncing instead of "idle," matching the honest-freshness signal `gbrain doctor` already shows.
### Added
- **Per-source failure cooldown + fan-out clamp.** A source that fails backs off (bounded exponential) instead of being re-dispatched every tick; per-tick fan-out is clamped to the worker concurrency, with a `gbrain doctor` check that warns on a mismatch.
- **`gbrain status --deadline-ms` / `--fast`.** A budgeted status snapshot returns whatever sections completed within the budget (marked partial) instead of hanging a poller; the JSON envelope also carries the CLI `version`.
- **A sync stall watchdog.** If the import drain makes no forward progress for `GBRAIN_SYNC_STALL_ABORT_SECONDS` (default 900), the run aborts and releases its per-source lock so the next `gbrain sync` resumes from the checkpoint — no manual `pkill`. It keys off import progress (not the lock heartbeat) and reports a distinct `stall_timeout` reason. (Limit: a hang inside a single file's import is observed between files, not mid-file; the wall-clock deadline remains the backstop there.)
### Fixed
- **A timed-out minion run counts as a spent attempt.** Wall-clock dead-lettering already did; the per-job timeout path didn't, so long-lane jobs (subagent / embed-backfill / autopilot-cycle) could read `attempts: 0/N (started: N)`. Accounting is now honest across all dead-letter paths.
- **`gbrain agent run` no longer swallows flags after the prompt.** A trailing `--detach` / `--follow` is recognized instead of being captured into the prompt string; a `--word` inside the prompt stays verbatim, and an explicit `--` ends flag parsing anywhere.
### To take advantage of v0.42.52.0
`gbrain upgrade`. Existing brains pick up the cycle split, supervisor backoff, and per-source cooldown on the next autopilot tick (one catch-up global-maintenance pass on the first tick) — no migration, all on by default. Tune the sync stall watchdog with `GBRAIN_SYNC_STALL_ABORT_SECONDS` if 900s doesn't fit your largest files; budget a status poller with `gbrain status --fast` or `--deadline-ms=<n>`.
## [0.42.51.0] - 2026-06-17
**`gbrain sync` stops bottlenecking all its workers on a single database row, a malformed checkpoint can no longer wedge a source, and `gbrain doctor` tells an actively-running sync apart from a stuck one.** A slow source that fell behind HEAD could read as permanently stale even while it imported every cycle: sync was single-core-bound at the database layer, so handing it more workers didn't help, and the freshness check couldn't see that a sync was in fact running.
+10 -4
View File
@@ -59,9 +59,14 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
- **Source isolation.** Every read-side op routes through `sourceScopeOpts(ctx)`; precedence
is federated array (`ctx.auth.allowedSources`) > scalar (`ctx.sourceId`) > nothing. Don't
hand-roll source filtering — a missed thread is a cross-source data leak.
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it;
PGLite hides the bug. Pass raw objects to `engine.executeRaw`, or use `executeRawJsonb`.
Guarded by `scripts/check-jsonb-pattern.sh`.
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it (a jsonb
string scalar); PGLite hides the bug. This bites BOTH spellings — the template form
(`${JSON.stringify(x)}::jsonb`) AND the positional form (`executeRaw(\`…$N::jsonb\`, [JSON.stringify(x)])`,
the #2339 class that aborted every sync). Fix: pass a raw object to `engine.executeRaw` / use
`executeRawJsonb` / `sql.json()`; or for the positional path bind through `$N::text::jsonb` (binds as
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
@@ -390,7 +395,7 @@ Progress banks into the append-only `op_checkpoint_paths` table (one row per dra
path, written via the direct session pool so it survives `EMAXCONNSESSION`); a killed
run resumes from the checkpoint and `last_commit` only advances on true completion. The
per-source lock heartbeats through the direct pool and refuses to steal a live,
recently-refreshed holder. Five env knobs tune it (all env-only, incident-time escape
recently-refreshed holder. Six env knobs tune it (all env-only, incident-time escape
hatches — no config-dashboard surface by design):
| Env var | Default | What it does |
@@ -400,6 +405,7 @@ hatches — no config-dashboard surface by design):
| `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` | 3 | Consecutive failed flushes (each already retried ~12s) before the run aborts with `reason: 'checkpoint_unavailable'` instead of importing work it can never bank. |
| `GBRAIN_SYNC_YIELD_EVERY` | 64 | Yield the event loop (`setTimeout(0)`, NOT `setImmediate` — Bun starves the timers phase under a tight setImmediate loop) every N files so the lock-refresh `setInterval` heartbeat fires mid-import. |
| `GBRAIN_LOCK_STEAL_GRACE_SECONDS` | derived (~600 at 30min TTL) | A holder that refreshed within this window is NOT stolen even if its TTL lapsed (starved-but-alive). Dead holders stop refreshing, age past the grace, and become stealable; TTL stays the backstop. |
| `GBRAIN_SYNC_STALL_ABORT_SECONDS` | 900 | Progress-aware stall watchdog (#1950): if the import drain makes no forward progress (keyed on file-import progress, NOT the lock heartbeat) for N seconds, abort the run and release the per-source lock so the next `gbrain sync` resumes from the checkpoint. Reports `reason: 'stall_timeout'`. Observed BETWEEN files; a hang inside one file's import isn't interrupted until it returns (the wall-clock hard deadline is that backstop). 0 disables. |
## Pace Mode (DB-contention-aware backfill pacing)
+21
View File
@@ -23,6 +23,27 @@ and the scope record at `~/.gstack/projects/garrytan-gbrain/ceo-plans/2026-06-12
`src/core/verbs/entity-card.ts` open-threads assembly + a new schema table
(additive — the card field already exists, so this is a quality upgrade, not
a contract change).
## reliability fix-wave follow-ups (filed v0.42.52.0)
Deferred from the autopilot/supervisor + sync/status/minion reliability wave
(plan-eng-review + codex + adversarial diff review CLEARED). Both surfaced by the
ship-stage pre-landing review; neither blocks the wave.
- [ ] **P2 — Thread a cancellation signal through `importFile` (#1950).** The sync
stall watchdog aborts `opts.signal`, but the per-iteration abort checks observe
it BETWEEN files — a hang inside one `importFile` call (e.g. a stuck embed
network request) isn't interrupted until that call returns. Thread an
`AbortSignal` into `importFromContent`/`importFromFile` and check it at the async
phase boundaries (post-parse, pre-embed, pre-DB-write) so an in-flight wedge is
reaped too. Core hot path (engine-parity + downstream-client surface) — scope it
on its own. Where: `src/core/import-file.ts`, `src/commands/sync.ts`.
- [ ] **P3 — Centralize live-sync liveness onto `liveSyncStatus` (#1950).**
`gbrain sources status` now uses the shared `liveSyncStatus(engine, sourceId)`
helper; retrofit `gbrain doctor` (its own inline lock probe) and `gbrain status`
onto the same helper so there's one source of truth for "is this source
syncing." Where: `src/core/db-lock.ts`, `src/commands/doctor.ts`,
`src/commands/status.ts`.
## Pace Mode follow-ups (filed v0.42.49.0)
Deferred from the paced-backfill wave (CEO + eng review CLEARED). Core shipped:
+33
View File
@@ -176,6 +176,39 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
## JSONB writes: never double-encode (the #2339 trap)
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
wrong and the write succeeds on PGLite but stores a **jsonb string scalar** on
real Postgres — `col ->> 'k'` returns NULL, `jsonb_array_elements` throws, and a
`jsonb_typeof = 'array'` CHECK rejects the row (this aborted every sync in #2339).
| Form | Verdict |
|---|---|
| Template tag: `` sql`... ${sql.json(obj)}` `` (postgres-engine only) | ✅ native jsonb serialization |
| Positional raw call, raw object: `executeRawJsonb(engine, sql, scalars, [obj])` | ✅ object reaches the wire as jsonb |
| Positional raw call, stringified: `executeRaw(\`... $N::text::jsonb\`, [JSON.stringify(x)])` | ✅ binds as text, the cast parses it |
| Positional raw call, BARE cast: `executeRaw(\`... $N::jsonb\`, [JSON.stringify(x)])` | ❌ **double-encodes** under postgres.js `.unsafe()` |
| Template literal interpolation: `` `... ${JSON.stringify(x)}::jsonb` `` | ❌ double-encodes |
**Why:** postgres.js `.unsafe(sql, params)` (the path behind `executeRaw` /
`executeRawDirect`) binds a JS **string** as a text param. A bare `$N::jsonb`
cast then wraps that already-JSON string into a jsonb scalar string instead of
parsing it. Casting through `$N::text::jsonb` forces a text→jsonb parse.
**PGLite's `db.query` parses text→jsonb natively, so it hides the bug** — which is
why a regression only shows up on Postgres (and why the parity test must run there).
**Two CI guards enforce this, both wired into `scripts/check-jsonb-pattern.sh`:**
- the template-tag grep (`${JSON.stringify(x)}::jsonb`), and
- `scripts/check-jsonb-params.mjs`, an AST-lite scanner for the positional
`$N::jsonb` + `JSON.stringify` form the grep misses. Sanctioned escapes:
`$N::text::jsonb`, `$N::text[]`, `executeRawJsonb`, `sql.json`, or an inline
`jsonb-guard-ok` comment.
The real backstop is `test/e2e/op-checkpoint-jsonb-parity.test.ts` +
`test/e2e/jsonb-roundtrip.test.ts`, which round-trip writes through real Postgres
and assert `jsonb_typeof` — the assertion PGLite cannot make.
## Adding a new engine
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
File diff suppressed because one or more lines are too long
+43 -4
View File
@@ -208,9 +208,14 @@ Per-file detail is in `docs/architecture/KEY_FILES.md`.
- **Source isolation.** Every read-side op routes through `sourceScopeOpts(ctx)`; precedence
is federated array (`ctx.auth.allowedSources`) > scalar (`ctx.sourceId`) > nothing. Don't
hand-roll source filtering — a missed thread is a cross-source data leak.
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it;
PGLite hides the bug. Pass raw objects to `engine.executeRaw`, or use `executeRawJsonb`.
Guarded by `scripts/check-jsonb-pattern.sh`.
- **JSONB: never `JSON.stringify` into a `::jsonb` cast.** postgres.js double-encodes it (a jsonb
string scalar); PGLite hides the bug. This bites BOTH spellings — the template form
(`${JSON.stringify(x)}::jsonb`) AND the positional form (`executeRaw(\`…$N::jsonb\`, [JSON.stringify(x)])`,
the #2339 class that aborted every sync). Fix: pass a raw object to `engine.executeRaw` / use
`executeRawJsonb` / `sql.json()`; or for the positional path bind through `$N::text::jsonb` (binds as
text, the cast parses it). Guarded by `scripts/check-jsonb-pattern.sh` (template grep) +
`scripts/check-jsonb-params.mjs` (positional AST scanner); the real backstop is the DATABASE_URL-gated
e2e parity tests, since PGLite can't surface the bug. Full rule in `docs/ENGINES.md`.
- **Engine parity.** `src/core/postgres-engine.ts` and `src/core/pglite-engine.ts` move in
lockstep — a new method/SQL shape lands in BOTH, pinned by `test/e2e/engine-parity.test.ts`.
Forward-referenced columns/indexes go in the bootstrap probe set (guarded by
@@ -539,7 +544,7 @@ Progress banks into the append-only `op_checkpoint_paths` table (one row per dra
path, written via the direct session pool so it survives `EMAXCONNSESSION`); a killed
run resumes from the checkpoint and `last_commit` only advances on true completion. The
per-source lock heartbeats through the direct pool and refuses to steal a live,
recently-refreshed holder. Five env knobs tune it (all env-only, incident-time escape
recently-refreshed holder. Six env knobs tune it (all env-only, incident-time escape
hatches — no config-dashboard surface by design):
| Env var | Default | What it does |
@@ -549,6 +554,7 @@ hatches — no config-dashboard surface by design):
| `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` | 3 | Consecutive failed flushes (each already retried ~12s) before the run aborts with `reason: 'checkpoint_unavailable'` instead of importing work it can never bank. |
| `GBRAIN_SYNC_YIELD_EVERY` | 64 | Yield the event loop (`setTimeout(0)`, NOT `setImmediate` — Bun starves the timers phase under a tight setImmediate loop) every N files so the lock-refresh `setInterval` heartbeat fires mid-import. |
| `GBRAIN_LOCK_STEAL_GRACE_SECONDS` | derived (~600 at 30min TTL) | A holder that refreshed within this window is NOT stolen even if its TTL lapsed (starved-but-alive). Dead holders stop refreshing, age past the grace, and become stealable; TTL stays the backstop. |
| `GBRAIN_SYNC_STALL_ABORT_SECONDS` | 900 | Progress-aware stall watchdog (#1950): if the import drain makes no forward progress (keyed on file-import progress, NOT the lock heartbeat) for N seconds, abort the run and release the per-source lock so the next `gbrain sync` resumes from the checkpoint. Reports `reason: 'stall_timeout'`. Observed BETWEEN files; a hang inside one file's import isn't interrupted until it returns (the wall-clock hard deadline is that backstop). 0 disables. |
## Pace Mode (DB-contention-aware backfill pacing)
@@ -2117,6 +2123,39 @@ RRF fusion, multi-query expansion, and 4-layer dedup are engine-agnostic. They o
**Migration:** `gbrain migrate --to supabase` exports everything (pages, chunks, embeddings, links, tags, timeline) and imports into Supabase. `gbrain migrate --to pglite` goes the other direction. Bidirectional, lossless.
## JSONB writes: never double-encode (the #2339 trap)
Writing a JS value into a `jsonb` column has exactly two correct forms. Get this
wrong and the write succeeds on PGLite but stores a **jsonb string scalar** on
real Postgres — `col ->> 'k'` returns NULL, `jsonb_array_elements` throws, and a
`jsonb_typeof = 'array'` CHECK rejects the row (this aborted every sync in #2339).
| Form | Verdict |
|---|---|
| Template tag: `` sql`... ${sql.json(obj)}` `` (postgres-engine only) | ✅ native jsonb serialization |
| Positional raw call, raw object: `executeRawJsonb(engine, sql, scalars, [obj])` | ✅ object reaches the wire as jsonb |
| Positional raw call, stringified: `executeRaw(\`... $N::text::jsonb\`, [JSON.stringify(x)])` | ✅ binds as text, the cast parses it |
| Positional raw call, BARE cast: `executeRaw(\`... $N::jsonb\`, [JSON.stringify(x)])` | ❌ **double-encodes** under postgres.js `.unsafe()` |
| Template literal interpolation: `` `... ${JSON.stringify(x)}::jsonb` `` | ❌ double-encodes |
**Why:** postgres.js `.unsafe(sql, params)` (the path behind `executeRaw` /
`executeRawDirect`) binds a JS **string** as a text param. A bare `$N::jsonb`
cast then wraps that already-JSON string into a jsonb scalar string instead of
parsing it. Casting through `$N::text::jsonb` forces a text→jsonb parse.
**PGLite's `db.query` parses text→jsonb natively, so it hides the bug** — which is
why a regression only shows up on Postgres (and why the parity test must run there).
**Two CI guards enforce this, both wired into `scripts/check-jsonb-pattern.sh`:**
- the template-tag grep (`${JSON.stringify(x)}::jsonb`), and
- `scripts/check-jsonb-params.mjs`, an AST-lite scanner for the positional
`$N::jsonb` + `JSON.stringify` form the grep misses. Sanctioned escapes:
`$N::text::jsonb`, `$N::text[]`, `executeRawJsonb`, `sql.json`, or an inline
`jsonb-guard-ok` comment.
The real backstop is `test/e2e/op-checkpoint-jsonb-parity.test.ts` +
`test/e2e/jsonb-roundtrip.test.ts`, which round-trip writes through real Postgres
and assert `jsonb_typeof` — the assertion PGLite cannot make.
## Adding a new engine
1. Create `src/core/<name>-engine.ts` implementing `BrainEngine`
+128
View File
@@ -0,0 +1,128 @@
#!/usr/bin/env node
/**
* CI guard for the POSITIONAL jsonb double-encode footgun (#2339 / #2324 class).
*
* The legacy scripts/check-jsonb-pattern.sh only catches the template-tag form
* (`${JSON.stringify(x)}::jsonb`). It MISSES the positional-param form:
*
* engine.executeRaw(`... $3::jsonb ...`, [a, b, JSON.stringify(x)])
*
* Under postgres.js `.unsafe(sql, params)` a JS STRING bound to a `$N::jsonb`
* param double-encodes — the text→jsonb cast wraps the already-JSON string into a
* jsonb *string scalar*. PGLite parses it silently, so the bug is invisible in
* unit tests and only bites on real Postgres (it aborted every sync in #2339).
*
* This scanner flags any executeRaw / executeRawDirect / .unsafe(...) call whose
* balanced argument span contains BOTH a positional `$N::jsonb` cast
* (NOT `$N::text::jsonb`, NOT `$N::text[]`) AND a `JSON.stringify(` — the exact
* double-encode shape. It is heuristic by design (whole-span correlation); the
* real backstop is the DATABASE_URL-gated e2e parity test. Keep both.
*
* Allowed forms (NOT flagged):
* - `$N::text::jsonb` + JSON.stringify (the fix: binds as text, cast parses it)
* - `$N::text[]` (the unnest path — arrays bind fine)
* - executeRawJsonb(...) (passes raw objects, not strings)
* - sql.json(x) (postgres.js native jsonb serializer)
* - a `jsonb-guard-ok` comment anywhere in the call span (explicit opt-out)
*
* Exit 0 = clean, 1 = violations found. Runs under node or bun.
*/
import { readdirSync, readFileSync, statSync } from 'node:fs';
import { join } from 'node:path';
// Default scan roots; overridable via argv so the guard's own test can point it
// at a fixture dir (e.g. `node check-jsonb-params.mjs /tmp/fixtures`).
const ROOTS = process.argv.slice(2).length > 0 ? process.argv.slice(2) : ['src', 'scripts'];
// executeRawDirect must precede executeRaw in the alternation so the longer name
// wins; executeRawJsonb is deliberately excluded (it passes objects). The
// optional `<...>` handles generic type args, e.g. `executeRaw<{ id: string }>(`.
//
// Only the postgres.js raw path is scanned (executeRaw/executeRawDirect/.unsafe).
// PGLite's native `this.db.query(...)` is intentionally NOT matched: its driver
// parses a text→jsonb cast natively, so the double-encode that bites postgres.js
// `.unsafe()` does not occur there (the `pglite-masks` invariant). The engine
// parity test pins that the resulting jsonb_typeof agrees across both engines.
const CALL_RE = /\b(executeRawDirect|executeRaw|unsafe)\s*(?:<[^>;]*>)?\s*\(/g;
/** Walk from the '(' at openIdx and return [start,end) of the balanced span,
* respecting strings, template literals, and comments. */
function findSpan(src, openIdx) {
let depth = 0;
let mode = 'code'; // code | line | block | sq | dq | tpl
for (let i = openIdx; i < src.length; i++) {
const c = src[i];
const n = src[i + 1];
if (mode === 'line') { if (c === '\n') mode = 'code'; continue; }
if (mode === 'block') { if (c === '*' && n === '/') { mode = 'code'; i++; } continue; }
if (mode === 'sq') { if (c === '\\') { i++; continue; } if (c === "'") mode = 'code'; continue; }
if (mode === 'dq') { if (c === '\\') { i++; continue; } if (c === '"') mode = 'code'; continue; }
if (mode === 'tpl') { if (c === '\\') { i++; continue; } if (c === '`') mode = 'code'; continue; }
// mode === 'code'
if (c === '/' && n === '/') { mode = 'line'; i++; continue; }
if (c === '/' && n === '*') { mode = 'block'; i++; continue; }
if (c === "'") { mode = 'sq'; continue; }
if (c === '"') { mode = 'dq'; continue; }
if (c === '`') { mode = 'tpl'; continue; }
if (c === '(') depth++;
else if (c === ')') { depth--; if (depth === 0) return [openIdx + 1, i]; }
}
return [openIdx + 1, src.length];
}
/** Blank out comments so a commented-out example doesn't trip the JSON.stringify probe. */
function stripComments(s) {
return s.replace(/\/\/[^\n]*/g, '').replace(/\/\*[\s\S]*?\*\//g, '');
}
const violations = [];
function scanFile(file) {
const src = readFileSync(file, 'utf8');
CALL_RE.lastIndex = 0;
let m;
while ((m = CALL_RE.exec(src))) {
const method = m[1];
const openIdx = m.index + m[0].length - 1; // index of the '('
const [s, e] = findSpan(src, openIdx);
const span = src.slice(s, e);
if (/jsonb-guard-ok/.test(span)) continue;
if (!/JSON\.stringify\s*\(/.test(stripComments(span))) continue;
// A positional `$N::jsonb` that is NOT `$N::text::jsonb`.
const jsonbRe = /\$\d+\s*::\s*jsonb\b/g;
let j;
let badText = '';
while ((j = jsonbRe.exec(span))) {
const pre = span.slice(Math.max(0, j.index - 12), j.index);
if (/::\s*text\s*$/.test(pre)) continue; // $N::text::jsonb is the fix — allowed
badText = j[0].replace(/\s+/g, '');
break;
}
if (!badText) continue;
const line = src.slice(0, s).split('\n').length;
violations.push(
`${file}:${line} ${method}(...) binds JSON.stringify into ${badText} — use $N::text::jsonb or pass a raw object (executeRawJsonb / sql.json)`,
);
}
}
function walk(dir) {
let ents;
try { ents = readdirSync(dir); } catch { return; }
for (const ent of ents) {
if (ent === 'node_modules') continue;
const p = join(dir, ent);
const st = statSync(p);
if (st.isDirectory()) walk(p);
else if (p.endsWith('.ts') && !p.endsWith('.test.ts')) scanFile(p);
}
}
for (const root of ROOTS) walk(root);
if (violations.length) {
console.error('JSONB positional double-encode violations (#2339 class):\n');
for (const v of violations) console.error(' ' + v);
console.error(`\n${violations.length} violation(s). Fix: bind through $N::text::jsonb (keeping JSON.stringify), or pass a raw object via executeRawJsonb / sql.json. See docs/ENGINES.md.`);
process.exit(1);
}
console.log('check-jsonb-params: clean (no positional $N::jsonb + JSON.stringify double-encodes)');
+14
View File
@@ -44,3 +44,17 @@ if grep -rEn "$MAX_STALLED_PATTERN" src/schema.sql src/core/migrate.ts src/core/
fi
echo "OK: max_stalled defaults are 5 in all schema sources"
# v0.42.x (#2339 / #2324): positional `$N::jsonb` + JSON.stringify double-encode.
# The template-string grep above only catches `${JSON.stringify(x)}::jsonb`. It
# MISSES the positional-param form — executeRaw(`... $N::jsonb ...`,
# [JSON.stringify(x)]) — which is the exact shape that double-encoded the
# op_checkpoints pin and aborted every sync in #2339. The AST-lite scanner below
# catches it. `set -e` propagates its non-zero exit.
if command -v node >/dev/null 2>&1; then
node scripts/check-jsonb-params.mjs
elif command -v bun >/dev/null 2>&1; then
bun scripts/check-jsonb-params.mjs
else
echo "WARN: neither node nor bun on PATH; skipping check-jsonb-params.mjs" >&2
fi
+78 -19
View File
@@ -74,8 +74,12 @@ SUBMITTING
--follow Tail status until terminal (default on TTY)
--detach Submit + print job id, exit immediately
Flags after \`run\` up to the first unrecognized token are parsed; the
remainder is the prompt. Use \`--\` to explicitly terminate flag parsing.
Flags before the prompt are parsed normally. The no-value switches
--detach, --follow and --no-follow are ALSO recognized when they trail
the prompt, so \`gbrain agent run "do X" --detach\` detaches. Any other
--word is treated as prompt text (no error). Use \`--\` to end flag
parsing and pass the rest verbatim:
gbrain agent run -- "literally --detach this, with --flags"
VIEWING
gbrain agent logs <job_id>
@@ -102,31 +106,86 @@ interface RunFlags {
detach: boolean;
}
/** No-value switches that may also trail the prompt and get hoisted out (#1738). */
const BOOLEAN_TAIL_FLAGS = new Set(['--follow', '--no-follow', '--detach']);
function applyBooleanFlag(flags: RunFlags, a: string): void {
if (a === '--follow') flags.follow = true;
else if (a === '--no-follow') flags.follow = false;
else { flags.detach = true; flags.follow = false; } // --detach
}
/** Read the value for a value-flag, rejecting a missing or flag-shaped value. */
function requireFlagValue(args: string[], i: number, flag: string): string {
const v = args[i];
if (v === undefined || v.startsWith('--')) {
throw new Error(`gbrain agent run: ${flag} requires a value. Run \`gbrain agent run --help\`.`);
}
return v;
}
function parseIntFlagValue(v: string, flag: string): number {
const n = parseInt(v, 10);
if (Number.isNaN(n)) {
throw new Error(`gbrain agent run: ${flag} expects a number, got "${v}".`);
}
return n;
}
/**
* Parse `agent run` args into flags + prompt (#1738).
*
* args ──► [ leading flag zone ] [ ── ? ] [ prompt … (trailing booleans) ]
*
* Leading zone: known flags (value + boolean) are consumed left-to-right until
* the first positional token, an UNKNOWN --flag, or an explicit `--`. An
* unknown --flag is NOT an error — it begins the freeform prompt, so
* `agent run "--note: do X"` works without `--`. Value-flags missing their
* value throw a usage error instead of silently capturing `undefined`/`NaN`.
*
* Prompt zone: a trailing run of the no-value switches (--detach/--follow/
* --no-follow) is hoisted out so `agent run "do X" --detach` detaches. Only
* trailing switches are hoisted; a `--word` elsewhere in the prompt stays
* verbatim. After an explicit `--`, nothing is hoisted.
*/
function parseRunFlags(args: string[]): { flags: RunFlags; rest: string[] } {
const flags: RunFlags = {
follow: process.stdout.isTTY === true,
detach: false,
};
let i = 0;
while (i < args.length) {
const a = args[i];
if (a === '--') { i++; break; }
if (!isKnownFlag(a!)) break;
let escaped = false;
for (; i < args.length; i++) {
const a = args[i]!;
if (a === '--') { i++; escaped = true; break; }
if (!a.startsWith('--')) break;
let known = true;
switch (a) {
case '--subagent-def': flags.subagentDef = args[++i]; i++; break;
case '--model': flags.model = args[++i]; i++; break;
case '--max-turns': flags.maxTurns = parseInt(args[++i] ?? '', 10); i++; break;
case '--tools': flags.tools = (args[++i] ?? '').split(',').map(s => s.trim()).filter(Boolean); i++; break;
case '--timeout-ms': flags.timeoutMs = parseInt(args[++i] ?? '', 10); i++; break;
case '--fanout-manifest': flags.fanoutManifest = args[++i]; i++; break;
case '--follow': flags.follow = true; i++; break;
case '--no-follow': flags.follow = false; i++; break;
case '--detach': flags.detach = true; flags.follow = false; i++; break;
default:
throw new Error(`unknown flag: ${a}. Run \`gbrain agent run --help\` for usage.`);
case '--subagent-def': flags.subagentDef = requireFlagValue(args, ++i, a); break;
case '--model': flags.model = requireFlagValue(args, ++i, a); break;
case '--max-turns': flags.maxTurns = parseIntFlagValue(requireFlagValue(args, ++i, a), a); break;
case '--tools': flags.tools = requireFlagValue(args, ++i, a).split(',').map(s => s.trim()).filter(Boolean); break;
case '--timeout-ms': flags.timeoutMs = parseIntFlagValue(requireFlagValue(args, ++i, a), a); break;
case '--fanout-manifest': flags.fanoutManifest = requireFlagValue(args, ++i, a); break;
case '--follow': flags.follow = true; break;
case '--no-follow': flags.follow = false; break;
case '--detach': flags.detach = true; flags.follow = false; break;
default: known = false; break;
}
if (!known) break; // unknown --flag → first token of the (freeform) prompt
}
const rest = args.slice(i);
// An explicit `--` terminates flag parsing wherever it appears — leading
// zone (escaped) OR after a positional (the leading loop breaks before it,
// so `escaped` stays false). Honor both: when the prompt carries a literal
// `--`, hoist nothing, so `agent run note -- --detach` keeps `--detach`
// verbatim instead of silently flipping detach mode.
if (!escaped && !rest.includes('--')) {
while (rest.length > 0 && BOOLEAN_TAIL_FLAGS.has(rest[rest.length - 1]!)) {
applyBooleanFlag(flags, rest.pop()!);
}
}
return { flags, rest: args.slice(i) };
return { flags, rest };
}
export async function runAgentRun(engine: BrainEngine, args: string[]): Promise<void> {
@@ -251,7 +310,7 @@ async function runFanout(engine: BrainEngine, queue: MinionQueue, flags: RunFlag
// do this after submission because each add() returns the committed
// row's id; the aggregator's seed started with an empty array.
await engine.executeRaw(
`UPDATE minion_jobs SET data = jsonb_set(data, '{children_ids}', $1::jsonb) WHERE id = $2`,
`UPDATE minion_jobs SET data = jsonb_set(data, '{children_ids}', $1::text::jsonb) WHERE id = $2`,
[JSON.stringify(childIds), aggregator.id],
);
+306 -5
View File
@@ -32,9 +32,26 @@
import type { BrainEngine, SourceRow } from '../core/engine.ts';
import type { MinionQueue } from '../core/minions/queue.ts';
import { NON_GLOBAL_PHASES, GLOBAL_PHASES, LAST_GLOBAL_AT_KEY } from '../core/cycle.ts';
const FULL_CYCLE_FLOOR_MIN = 60;
// #2194 fix #2: failure cooldown. A source whose autopilot-cycle keeps
// failing/timing-out re-dispatches every tick today (only SUCCESS gates
// dispatch), so the same handful of sources fail and re-fan-out forever — the
// self-perpetuating dead-job storm. Back a failed source off with bounded
// exponential cooldown so a chronically-slow source can't re-dispatch every
// tick. Disabled with autopilot.failure_cooldown_min=0.
const FAILURE_COOLDOWN_BASE_MIN = 10;
const FAILURE_COOLDOWN_CAP_MIN = 120;
const FAILURE_COOLDOWN_EXP_CAP = 4; // 2^4 = 16× base before the cap clamps
/** Recent-failure record for one source (from minion_jobs dead/failed rows). */
export interface SourceFailure { count: number; lastFailedAt: Date; }
/** Resolved cooldown knobs. baseMin <= 0 means the cooldown is disabled. */
export interface CooldownOpts { baseMin: number; capMin: number; }
export interface FanoutOpts {
repoPath: string;
slot: string;
@@ -58,6 +75,8 @@ export interface FanoutResult {
skipped_fresh: string[];
/** Source ids beyond the fanoutMax cap (will retry next tick). */
skipped_cap: string[];
/** Source ids skipped because they're in failure cooldown (#2194 fix #2). */
skipped_cooldown: string[];
/** True when this tick fell back to the legacy single-job path
* (no sources rows / engine empty). */
legacy_fallback: boolean;
@@ -83,6 +102,62 @@ export async function resolveFanoutMax(engine: BrainEngine): Promise<number> {
return engine.kind === 'pglite' ? 1 : 4;
}
/**
* Read the worker concurrency the supervisor most recently STARTED with, from
* its `started` audit event (the lowest-coupling source — no extra lock-row
* column). Filesystem read; returns null when no supervisor has ever started
* (or the event lacks concurrency). Filtered by queue so a `shell`-queue
* supervisor's concurrency doesn't leak into the `default`-queue decision.
*
* ADVISORY use only (doctor warning). Behavior-changing callers (the fanout
* clamp) must additionally gate on a LIVE supervisor — see
* resolveEffectiveFanoutMax — because a stale `started` row can otherwise
* shrink fan-out for a supervisor that isn't running that config (codex #9/D5).
*/
export async function readSupervisorConcurrency(queue = 'default'): Promise<number | null> {
try {
const { readSupervisorEvents } = await import('../core/minions/handlers/supervisor-audit.ts');
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const started = events
.filter((e) => e.event === 'started' && (e.queue === undefined || e.queue === queue))
.pop();
const c = started?.concurrency;
return typeof c === 'number' && Number.isFinite(c) ? c : null;
} catch {
return null;
}
}
/**
* Resolve fanoutMax CLAMPED to the worker's effective concurrency (#2194 fix #1).
*
* Fanning out more cycles than the worker can run guarantees waiters that then
* race the stalled-sweeper. Clamp to `max(1, concurrency - 1)` — reserving ≥1
* slot for targeted sync/embed jobs that share the `default` queue.
*
* codex #9 / D5: the clamp is BEHAVIOR-changing, so it trusts only a
* proven-alive supervisor (live DB-lock holder, `ttl_expires_at`-gated). With
* no live holder the concurrency is UNKNOWN and we fall back to the unclamped
* default (4 pg / 1 pglite) — the safe direction (never starve on stale data).
* Operators can disable the clamp via `autopilot.fanout_clamp_to_concurrency`.
*/
export async function resolveEffectiveFanoutMax(engine: BrainEngine, queue = 'default'): Promise<number> {
const base = await resolveFanoutMax(engine);
const clampCfg = await engine.getConfig('autopilot.fanout_clamp_to_concurrency');
if (clampCfg === 'false' || clampCfg === '0') return base; // operator opt-out
try {
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
const snap = await inspectLock(engine, supervisorLockId(queue));
if (!snap || !isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) return base; // no live holder → unknown → no clamp
const concurrency = await readSupervisorConcurrency(queue);
if (concurrency === null) return base;
return Math.max(1, Math.min(base, concurrency - 1));
} catch {
return base;
}
}
/**
* Read `last_full_cycle_at` ISO string from a source's config JSONB.
* Returns null when missing or unparseable. Pure function over the row
@@ -111,6 +186,133 @@ export function isSourceStale(src: SourceRow, now = Date.now(), floorMin = FULL_
return ageMin >= floorMin;
}
/**
* Most recent SUCCESSFUL cycle for a source. Prefers `last_source_cycle_at`
* (per-source phases, written by the split cycle) and falls back to the legacy
* `last_full_cycle_at`, so this works before AND after the cycle split.
*/
export function readLastSuccessAt(src: SourceRow): Date | null {
const c = src.config ?? {};
const raw = (typeof c.last_source_cycle_at === 'string' && c.last_source_cycle_at)
|| (typeof c.last_full_cycle_at === 'string' && c.last_full_cycle_at)
|| null;
if (!raw) return null;
const d = new Date(raw);
return Number.isFinite(d.getTime()) ? d : null;
}
/** Bounded exponential cooldown window (minutes) for a given failure count. */
export function cooldownMinForCount(count: number, opts: CooldownOpts): number {
if (count <= 0 || opts.baseMin <= 0) return 0;
const mult = Math.pow(2, Math.min(count - 1, FAILURE_COOLDOWN_EXP_CAP));
return Math.min(opts.baseMin * mult, opts.capMin);
}
/**
* Is a source currently in failure cooldown? Pure — drives both the dispatch
* gate and the claim-time guard. A SUCCESS at-or-after the most recent failure
* clears the cooldown (codex #7: operator repair / manual cycle re-eligibility),
* so a recovered source is never suppressed by stale failure history.
*/
export function isInFailureCooldown(
failure: SourceFailure | undefined,
lastSuccessAt: Date | null,
now: number,
opts: CooldownOpts,
): boolean {
if (opts.baseMin <= 0) return false; // disabled
if (!failure || failure.count <= 0) return false;
if (lastSuccessAt && lastSuccessAt.getTime() >= failure.lastFailedAt.getTime()) return false;
const cooldownMs = cooldownMinForCount(failure.count, opts) * 60_000;
return (now - failure.lastFailedAt.getTime()) < cooldownMs;
}
/**
* Resolve cooldown knobs from config. `autopilot.failure_cooldown_min` overrides
* the base (0 = disable entirely — exactly today's behavior);
* `autopilot.failure_cooldown_cap_min` overrides the ceiling.
*/
export async function resolveFailureCooldownOpts(engine: BrainEngine): Promise<CooldownOpts> {
let baseMin = FAILURE_COOLDOWN_BASE_MIN;
let capMin = FAILURE_COOLDOWN_CAP_MIN;
const baseCfg = await engine.getConfig('autopilot.failure_cooldown_min');
if (baseCfg !== null && baseCfg !== undefined && baseCfg !== '') {
const n = parseInt(baseCfg, 10);
if (Number.isFinite(n) && n >= 0) baseMin = n;
}
const capCfg = await engine.getConfig('autopilot.failure_cooldown_cap_min');
if (capCfg) {
const n = parseInt(capCfg, 10);
if (Number.isFinite(n) && n >= 1) capMin = n;
}
return { baseMin, capMin };
}
/**
* Read recent dead/failed autopilot-cycle jobs grouped by source. Read-at-
* dispatch (NOT a write hook) because timeouts/RSS-kills/stalls dead-letter via
* SQL in queue.ts and never run handler code — a write-only cooldown would miss
* the exact failures that drive the storm. Engine-parity-safe via executeRaw
* (one query, both engines); cutoff is precomputed in JS to avoid INTERVAL
* portability concerns. codex #6: rows with a null source_id are excluded.
*/
export async function readRecentSourceFailures(
engine: BrainEngine,
opts: { sinceMin?: number; sourceId?: string } = {},
): Promise<Map<string, SourceFailure>> {
const sinceMin = opts.sinceMin ?? FAILURE_COOLDOWN_CAP_MIN;
const cutoff = new Date(Date.now() - sinceMin * 60_000).toISOString();
const map = new Map<string, SourceFailure>();
try {
const params: unknown[] = [cutoff];
let sql =
`SELECT data->>'source_id' AS source_id,
count(*)::int AS fail_count,
max(finished_at) AS last_failed_at
FROM minion_jobs
WHERE name = 'autopilot-cycle'
AND status IN ('dead','failed')
AND data->>'source_id' IS NOT NULL
AND finished_at IS NOT NULL
AND finished_at > $1`;
if (opts.sourceId) { params.push(opts.sourceId); sql += ` AND data->>'source_id' = $${params.length}`; }
sql += ` GROUP BY data->>'source_id'`;
const rows = await engine.executeRaw<{ source_id: string | null; fail_count: number; last_failed_at: string | Date }>(sql, params);
for (const r of rows) {
if (!r.source_id) continue; // codex #6 null-source guard (defensive)
const last = r.last_failed_at instanceof Date ? r.last_failed_at : new Date(r.last_failed_at);
if (!Number.isFinite(last.getTime())) continue;
map.set(r.source_id, { count: Number(r.fail_count) || 0, lastFailedAt: last });
}
} catch {
// Pre-migration / transient DB error → no cooldown data (fail open: dispatch).
}
return map;
}
/**
* Claim-time cooldown guard (codex #5 / D4): a job already queued or retrying
* (max_attempts:2) can reach the worker after the dispatch gate decided. The
* handler calls this immediately before runCycle; an in-cooldown claim becomes
* a no-op skip (NOT a failure — it must not re-arm the cooldown). Shares the
* exact cooldown math with the dispatch gate (DRY).
*/
export async function isSourceInCooldown(engine: BrainEngine, sourceId: string, now = Date.now()): Promise<boolean> {
const opts = await resolveFailureCooldownOpts(engine);
if (opts.baseMin <= 0) return false;
const failures = await readRecentSourceFailures(engine, { sinceMin: opts.capMin, sourceId });
const failure = failures.get(sourceId);
if (!failure) return false;
let lastSuccessAt: Date | null = null;
try {
const rows = await engine.executeRaw<{ config: Record<string, unknown> | null }>(
`SELECT config FROM sources WHERE id = $1`, [sourceId],
);
if (rows[0]) lastSuccessAt = readLastSuccessAt({ config: rows[0].config ?? {} } as SourceRow);
} catch { /* treat as no success */ }
return isInFailureCooldown(failure, lastSuccessAt, now, opts);
}
/**
* Decide which sources to dispatch this tick. Pure function so tests can
* exercise the freshness gate + cap math without an engine.
@@ -126,11 +328,21 @@ export function selectSourcesForDispatch(
fanoutMax: number,
now = Date.now(),
floorMin = FULL_CYCLE_FLOOR_MIN,
): { dispatch: SourceRow[]; skippedFresh: SourceRow[]; skippedCap: SourceRow[] } {
recentFailures: Map<string, SourceFailure> = new Map(),
cooldownOpts: CooldownOpts = { baseMin: FAILURE_COOLDOWN_BASE_MIN, capMin: FAILURE_COOLDOWN_CAP_MIN },
): { dispatch: SourceRow[]; skippedFresh: SourceRow[]; skippedCap: SourceRow[]; skippedCooldown: SourceRow[] } {
const stale: SourceRow[] = [];
const fresh: SourceRow[] = [];
const cooldown: SourceRow[] = [];
for (const s of sources) {
(isSourceStale(s, now, floorMin) ? stale : fresh).push(s);
if (!isSourceStale(s, now, floorMin)) { fresh.push(s); continue; }
// #2194 fix #2: a stale source that recently failed is held in cooldown so
// it can't re-dispatch every tick (the storm). Success clears it.
if (isInFailureCooldown(recentFailures.get(s.id), readLastSuccessAt(s), now, cooldownOpts)) {
cooldown.push(s);
continue;
}
stale.push(s);
}
// Oldest-first ordering: NULL last_full_cycle_at sorts before any timestamp.
stale.sort((a, b) => {
@@ -141,7 +353,7 @@ export function selectSourcesForDispatch(
});
const dispatch = stale.slice(0, fanoutMax);
const skippedCap = stale.slice(fanoutMax);
return { dispatch, skippedFresh: fresh, skippedCap };
return { dispatch, skippedFresh: fresh, skippedCap, skippedCooldown: cooldown };
}
/**
@@ -193,10 +405,27 @@ export async function dispatchPerSource(
} else {
log(`[dispatch] job #${job.id} autopilot-cycle (legacy single-source)`);
}
return { dispatched: [], skipped_fresh: [], skipped_cap: [], legacy_fallback: true };
return { dispatched: [], skipped_fresh: [], skipped_cap: [], skipped_cooldown: [], legacy_fallback: true };
}
const { dispatch, skippedFresh, skippedCap } = selectSourcesForDispatch(sources, opts.fanoutMax);
// #2194 fix #2: load recent per-source failures + cooldown knobs so a
// chronically-failing source is backed off instead of re-dispatched every
// tick. Fail-open: cooldown is an optimization, not a correctness gate — if
// config/job-history reads fail (or the engine lacks them), dispatch proceeds
// with no cooldown rather than blocking.
let cooldownOpts: CooldownOpts = { baseMin: 0, capMin: FAILURE_COOLDOWN_CAP_MIN };
let recentFailures = new Map<string, SourceFailure>();
try {
cooldownOpts = await resolveFailureCooldownOpts(engine);
if (cooldownOpts.baseMin > 0) {
recentFailures = await readRecentSourceFailures(engine, { sinceMin: cooldownOpts.capMin });
}
} catch {
cooldownOpts = { baseMin: 0, capMin: FAILURE_COOLDOWN_CAP_MIN };
}
const { dispatch, skippedFresh, skippedCap, skippedCooldown } =
selectSourcesForDispatch(sources, opts.fanoutMax, Date.now(), FULL_CYCLE_FLOOR_MIN, recentFailures, cooldownOpts);
const dispatched: string[] = [];
for (const src of dispatch) {
@@ -208,6 +437,11 @@ export async function dispatchPerSource(
repoPath: opts.repoPath,
source_id: src.id,
pull: !!remoteUrl,
// #2194 fix #3 (cycle split): per-source cycles run ONLY source-scoped
// (+ mixed) phases. The brain-wide global phases (embed, orphans,
// purge, …) run once in autopilot-global-maintenance, not N times
// concurrently here — the fix for the 4→10GB RSS blowout.
phases: NON_GLOBAL_PHASES,
},
{
queue: 'default',
@@ -261,10 +495,77 @@ export async function dispatchPerSource(
}));
}
if (skippedCooldown.length > 0 && opts.jsonMode) {
emit(JSON.stringify({
event: 'fanout_cooldown_skipped',
sources: skippedCooldown.map(s => s.id),
}));
}
return {
dispatched,
skipped_fresh: skippedFresh.map(s => s.id),
skipped_cap: skippedCap.map(s => s.id),
skipped_cooldown: skippedCooldown.map(s => s.id),
legacy_fallback: false,
};
}
const GLOBAL_FLOOR_MIN = 60;
/** Is the brain-wide maintenance overdue? Null/unparseable → overdue. */
export function isGlobalMaintenanceStale(lastGlobalAtIso: string | null, now = Date.now(), floorMin = GLOBAL_FLOOR_MIN): boolean {
if (!lastGlobalAtIso) return true;
const d = new Date(lastGlobalAtIso);
if (!Number.isFinite(d.getTime())) return true;
return (now - d.getTime()) / 60_000 >= floorMin;
}
/**
* #2194 fix #3 / #2227 bug #3 — dispatch the single brain-wide maintenance job
* that runs the `global` cycle phases (embed, orphans, purge, …) ONCE per
* window, instead of N per-source cycles each running them concurrently (the
* RSS blowout). Single-flight is structural: one `idempotency_key` +
* `maxWaiting:1`, so a slow run never stacks. Gated on `autopilot.last_global_at`
* (stamped by the handler on success). Postgres-only fan-out concern; on PGLite
* the file lock already serializes, but the job is still correct there.
*/
export async function dispatchGlobalMaintenance(
engine: BrainEngine,
queue: MinionQueue,
opts: { repoPath: string; slot: string; timeoutMs: number; jsonMode: boolean; emit?: (l: string) => void; log?: (l: string) => void },
): Promise<{ dispatched: boolean; reason: 'stale' | 'fresh' }> {
const emit = opts.emit ?? ((line) => process.stderr.write(line + '\n'));
const log = opts.log ?? ((line) => console.log(line));
let floorMin = GLOBAL_FLOOR_MIN;
const floorCfg = await engine.getConfig('autopilot.global_floor_min');
if (floorCfg) {
const n = parseInt(floorCfg, 10);
if (Number.isFinite(n) && n >= 1) floorMin = n;
}
const lastGlobalAt = await engine.getConfig(LAST_GLOBAL_AT_KEY);
if (!isGlobalMaintenanceStale(lastGlobalAt, Date.now(), floorMin)) {
return { dispatched: false, reason: 'fresh' };
}
const job = await queue.add(
'autopilot-global-maintenance',
{ repoPath: opts.repoPath, phases: GLOBAL_PHASES },
{
queue: 'default',
// Structural single-flight: one global job per slot; maxWaiting:1 coalesces
// any surplus so a slow brain-wide pass never stacks duplicates.
idempotency_key: `autopilot-global:${opts.slot}`,
max_attempts: 2,
timeout_ms: opts.timeoutMs,
maxWaiting: 1,
},
);
if (opts.jsonMode) {
emit(JSON.stringify({ event: 'dispatched', job_id: job.id, mode: 'global_maintenance', slot: opts.slot }));
} else {
log(`[dispatch] job #${job.id} autopilot-global-maintenance (brain-wide phases)`);
}
return { dispatched: true, reason: 'stale' };
}
+21 -3
View File
@@ -871,8 +871,12 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
// codex P1-3). Fresh-install brains with no sources rows fall
// back to the legacy single autopilot-cycle so existing
// behavior is preserved.
const { dispatchPerSource, resolveFanoutMax } = await import('./autopilot-fanout.ts');
const fanoutMax = await resolveFanoutMax(engine);
const { dispatchPerSource, dispatchGlobalMaintenance, resolveEffectiveFanoutMax } = await import('./autopilot-fanout.ts');
// #2194 fix #1: clamp fan-out to the worker's effective concurrency
// (reserve ≥1 slot), gated on a LIVE supervisor so a stale audit row
// can't shrink throughput (codex #9/D5). autopilot-cycle jobs run on
// the 'default' queue, so that's the concurrency we compare against.
const fanoutMax = await resolveEffectiveFanoutMax(engine, 'default');
const result = await dispatchPerSource(engine, queue, {
repoPath,
slot,
@@ -880,6 +884,18 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
fanoutMax,
jsonMode,
});
// #2194 fix #3 / #2227 bug #3: dispatch the single brain-wide
// maintenance job (embed/orphans/purge/…) once per window — the per-
// source cycles above no longer run global phases, so this is where
// the brain-wide work happens (single-flight, no RSS blowout). Only on
// the per-source path (legacy single-source still runs everything).
if (!result.legacy_fallback) {
try {
await dispatchGlobalMaintenance(engine, queue, { repoPath, slot, timeoutMs, jsonMode });
} catch (e) {
if (jsonMode) process.stderr.write(JSON.stringify({ event: 'global_maintenance_dispatch_failed', error: e instanceof Error ? e.message : String(e) }) + '\n');
}
}
if (result.dispatched.length > 0 || result.legacy_fallback) {
lastFullCycleAt = Date.now();
}
@@ -889,6 +905,7 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
dispatched: result.dispatched,
skipped_fresh: result.skipped_fresh,
skipped_cap: result.skipped_cap,
skipped_cooldown: result.skipped_cooldown,
legacy_fallback: result.legacy_fallback,
fanout_max: fanoutMax,
score,
@@ -896,7 +913,8 @@ export async function runAutopilot(engine: BrainEngine, args: string[]) {
} else if (!result.legacy_fallback) {
console.log(
`[dispatch] fanout: ${result.dispatched.length} dispatched, ` +
`${result.skipped_fresh.length} fresh, ${result.skipped_cap.length} capped ` +
`${result.skipped_fresh.length} fresh, ${result.skipped_cap.length} capped, ` +
`${result.skipped_cooldown.length} cooldown ` +
`(score=${score}, max=${fanoutMax})`,
);
}
+66 -2
View File
@@ -695,6 +695,9 @@ export async function doctorReportRemote(engine: BrainEngine): Promise<DoctorRep
// issue #1801 — wedged_queue (cross-surface parity with buildChecks).
checks.push(await computeWedgedQueueCheck(engine));
// #2194 fix #5 — warn when autopilot fan-out exceeds worker concurrency.
checks.push(await computeAutopilotFanoutConcurrencyCheck(engine));
// v0.41 Bug 2 / Eng D8 — subagent_health surfaces rate-lease pressure to the operator.
checks.push(await checkSubagentHealth(engine));
@@ -1563,6 +1566,49 @@ export async function computeWedgedQueueCheck(engine: BrainEngine): Promise<Chec
}
}
/**
* #2194 fix #5: warn when autopilot's per-tick fan-out exceeds the worker's
* effective concurrency. Fanning out more cycles than there are worker slots
* guarantees waiters that race the stalled-sweeper a silent misconfig today.
* Advisory (started-event concurrency is fine here; the behavior-changing clamp
* in resolveEffectiveFanoutMax is the one that gates on liveness). Surfaces only
* when a supervisor has actually started (no noise on never-supervised brains).
*/
export async function computeAutopilotFanoutConcurrencyCheck(engine: BrainEngine): Promise<Check> {
if (engine.kind !== 'postgres') {
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: 'PGLite — single-writer, fan-out is 1' };
}
try {
const { resolveFanoutMax, readSupervisorConcurrency } = await import('./autopilot-fanout.ts');
const concurrency = await readSupervisorConcurrency('default');
if (concurrency === null) {
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: 'No supervisor observed — skipping fan-out/concurrency check' };
}
const fanoutMax = await resolveFanoutMax(engine);
const effectiveSlots = Math.max(1, concurrency - 1);
if (fanoutMax > effectiveSlots) {
return {
name: 'autopilot_fanout_concurrency',
status: 'warn',
message:
`autopilot fan-out (${fanoutMax}/tick) exceeds worker concurrency (${concurrency}). ` +
`Surplus cycles queue behind the worker and race the stalled-sweeper. ` +
`Lower fan-out: \`gbrain config set autopilot.fanout_max_per_tick ${effectiveSlots}\`, ` +
`or raise the supervisor's \`--concurrency\` to ${fanoutMax + 1}. ` +
`(The clamp in autopilot does this automatically unless disabled.)`,
details: { fanout_max: fanoutMax, concurrency, effective_slots: effectiveSlots },
};
}
return {
name: 'autopilot_fanout_concurrency',
status: 'ok',
message: `fan-out ${fanoutMax}/tick within worker concurrency ${concurrency}`,
};
} catch (e) {
return { name: 'autopilot_fanout_concurrency', status: 'ok', message: `Skipped (${e instanceof Error ? e.message : String(e)})` };
}
}
export async function checkBatchRetryHealth(_engine: BrainEngine): Promise<Check> {
try {
// Codex M-10: surface bad env config at doctor time.
@@ -4395,7 +4441,22 @@ export async function buildChecks(
const pidStatus = readSupervisorPid(DEFAULT_PID_FILE);
const supervisorPid = pidStatus.pid;
const running = pidStatus.running;
const pidfileRunning = pidStatus.running;
// issue #2227 fix #1/#3: DEFAULT_PID_FILE is HOME-derived, so a supervisor
// started under a different $HOME reads as "not running" even when healthy.
// Consult the queue-scoped DB singleton lock (#1849, HOME-independent) before
// warning. PID-reuse-safe (isLockHolderLive keys on lock freshness).
let detectedViaDbLock = false;
if (!pidfileRunning && engine) {
try {
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
const snap = await inspectLock(engine, supervisorLockId('default'));
if (snap && isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) detectedViaDbLock = true;
} catch { /* pre-migration / transient: pidfile-only */ }
}
const running = pidfileRunning || detectedViaDbLock;
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
@@ -4443,7 +4504,7 @@ export async function buildChecks(
checks.push({
name: 'supervisor',
status: 'ok',
message: `running=true pid=${supervisorPid} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h} clean_exits_24h=${summary.clean_exits}`,
message: `running=true${detectedViaDbLock ? ' (detected via DB lock; pidfile not at the HOME-derived path)' : ` pid=${supervisorPid}`} last_start=${lastStart ?? 'unknown'} crashes_24h=${crashes24h} clean_exits_24h=${summary.clean_exits}`,
});
}
}
@@ -7218,6 +7279,9 @@ export async function buildChecks(
// waiting, zero live-lock active, stale completions) as a health error.
progress.heartbeat('wedged_queue');
checks.push(await computeWedgedQueueCheck(engine));
// #2194 fix #5 — autopilot fan-out vs worker concurrency mismatch.
progress.heartbeat('autopilot_fanout_concurrency');
checks.push(await computeAutopilotFanoutConcurrencyCheck(engine));
// v0.40.4 graph_signals_coverage — global inbound-link density when
// graph_signals is enabled in the active mode bundle.
progress.heartbeat('graph_signals_coverage');
+119 -9
View File
@@ -1020,10 +1020,38 @@ HANDLER TYPES (built in)
const pidStatus = readSupervisorPid(pidFile);
const supervisorPid = pidStatus.pid;
const running = pidStatus.running;
const pidfileRunning = pidStatus.running;
const events = readSupervisorEvents({ sinceMs: 24 * 60 * 60 * 1000 });
const lastStart = events.filter(e => e.event === 'started').pop()?.ts ?? null;
// issue #2227 fix #1/#3: the pidfile is HOME-derived, so a supervisor
// started under a different $HOME (keeper=/root vs ops=/data) reads as
// "not running" here even when it is healthy — the false signal that
// makes an operator spawn a duplicate. Fall back to the queue-scoped DB
// singleton lock (#1849), the HOME-independent authority. PID-reuse-safe:
// isLockHolderLive keys on lock freshness, never process.kill.
const supQueue = parseFlag(args, '--queue') ?? 'default';
let detectedViaDbLock = false;
let dbLockHolder: { holder_pid: number; holder_host: string } | null = null;
if (!pidfileRunning) {
try {
const { inspectLock, isLockHolderLive } = await import('../core/db-lock.ts');
const { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } = await import('../core/minions/supervisor.ts');
const snap = await inspectLock(engine, supervisorLockId(supQueue));
if (snap && isLockHolderLive(snap, SUPERVISOR_LOCK_TTL_MIN)) {
detectedViaDbLock = true;
dbLockHolder = { holder_pid: snap.holder_pid, holder_host: snap.holder_host };
}
} catch {
// Pre-migration brains / transient DB errors: fall back to pidfile-only.
}
}
const running = pidfileRunning || detectedViaDbLock;
// Surface the supervisor's recorded config from the latest `started`
// event (concurrency + effective --max-rss) so split-$HOME deployments
// see what the live-but-pidfile-invisible supervisor is running.
const startedEvt = events.filter(e => e.event === 'started').pop() ?? null;
// Shared classifier — same code path runs in `gbrain doctor` so the
// two surfaces cannot drift on what counts as a crash. Supersedes
// v0.35.4.0's binary `classifyWorkerExit({code})` on this surface;
@@ -1038,15 +1066,20 @@ HANDLER TYPES (built in)
nice_requested: w.nice_requested,
nice: w.nice_now,
}));
const supervisorNice = running && supervisorPid !== null
const supervisorNice = pidfileRunning && supervisorPid !== null
? getEffectiveNiceness(supervisorPid)
: null;
const status = {
running,
supervisor_pid: supervisorPid,
detected_via: detectedViaDbLock ? 'db_lock' : (pidfileRunning ? 'pidfile' : null),
supervisor_pid: supervisorPid ?? dbLockHolder?.holder_pid ?? null,
db_lock_holder: dbLockHolder,
pid_file: pidFile,
queue: supQueue,
last_start: lastStart,
concurrency: typeof startedEvt?.concurrency === 'number' ? startedEvt.concurrency : null,
max_rss_mb: typeof startedEvt?.max_rss_mb === 'number' ? startedEvt.max_rss_mb : null,
crashes_24h: summary.total,
clean_exits_24h: summary.clean_exits,
crashes_by_cause: summary.by_cause,
@@ -1058,9 +1091,11 @@ HANDLER TYPES (built in)
if (jsonMode) {
console.log(JSON.stringify(status, null, 2));
} else {
console.log(`Supervisor: ${running ? 'running' : 'not running'}`);
if (supervisorPid) console.log(` PID: ${supervisorPid}`);
const via = detectedViaDbLock ? ' (detected via DB lock; pidfile not found at the configured path)' : '';
console.log(`Supervisor: ${running ? 'running' : 'not running'}${via}`);
if (status.supervisor_pid) console.log(` PID: ${status.supervisor_pid}${detectedViaDbLock ? ` @ ${dbLockHolder?.holder_host}` : ''}`);
console.log(` PID file: ${pidFile}`);
if (detectedViaDbLock && status.concurrency !== null) console.log(` Concurrency: ${status.concurrency}${status.max_rss_mb !== null ? ` (max-rss ${status.max_rss_mb}MB)` : ''}`);
if (lastStart) console.log(` Last start: ${lastStart}`);
console.log(` Crashes (24h): ${summary.total} (runtime=${summary.by_cause.runtime_error} oom=${summary.by_cause.oom_or_external_kill} unknown=${summary.by_cause.unknown} legacy=${summary.by_cause.legacy})`);
console.log(` Clean exits (24h): ${summary.clean_exits}`);
@@ -1607,6 +1642,13 @@ export async function registerBuiltinHandlers(
// archived between fan-out and worker claim, skip cleanly.
const rawSourceId = job.data.source_id;
let sourceId: string | undefined;
// issue #2227/#2194 (TODOS:634, codex #8): a per-source cycle must run its
// FILESYSTEM phases (sync/lint/extract) against the SOURCE's own checkout,
// not the global brain's. Pre-fix it inherited `repoPath` (the default
// checkout) while writing DB freshness for `source_id` — mixed scope that
// made cooldown/freshness attribute to the wrong source. We resolve the
// source's `local_path` here and use it as the cycle's brainDir below.
let sourceLocalPath: string | null = null;
if (rawSourceId !== undefined && rawSourceId !== null) {
if (typeof rawSourceId !== 'string') {
throw new Error(`autopilot-cycle: invalid source_id (not a string): ${JSON.stringify(rawSourceId)}`);
@@ -1620,9 +1662,10 @@ export async function registerBuiltinHandlers(
}
// Archive recheck (codex r1 P1-5): cheap pre-cycle lookup. Returns
// immediately if source is gone or archived; runCycle never even
// acquires a lock.
const rows = await engine.executeRaw<{ archived: boolean | null }>(
`SELECT archived FROM sources WHERE id = $1`,
// acquires a lock. Also fetches local_path so FS phases bind to the
// source's own checkout (the #2227/#2194 mixed-scope fix).
const rows = await engine.executeRaw<{ archived: boolean | null; local_path: string | null }>(
`SELECT archived, local_path FROM sources WHERE id = $1`,
[rawSourceId],
);
if (rows.length === 0) {
@@ -1640,8 +1683,17 @@ export async function registerBuiltinHandlers(
};
}
sourceId = rawSourceId;
sourceLocalPath = typeof rows[0].local_path === 'string' && rows[0].local_path.length > 0
? rows[0].local_path
: null;
}
// Effective checkout for FS phases. For a per-source cycle, bind to the
// SOURCE's local_path (or null → skip FS phases for a pure-DB source);
// NEVER fall through to the global repoPath, which would run sync/lint
// against the wrong tree. Legacy (no source_id) keeps the global repoPath.
const effectiveBrainDir: string | null = sourceId ? sourceLocalPath : repoPath;
// Allow callers to select phases via job data (e.g. skip embed for
// fast cycles). Validates against ALL_PHASES to prevent injection.
const { ALL_PHASES } = await import('../core/cycle.ts');
@@ -1653,8 +1705,23 @@ export async function registerBuiltinHandlers(
// Pull default: legacy `true` for back-compat; explicit boolean wins.
const pull = typeof job.data.pull === 'boolean' ? job.data.pull : true;
// #2194 fix #2 / codex #5 (D4): claim-time cooldown guard. A job already
// queued or retrying (max_attempts:2) can reach the worker after the
// dispatch gate decided to back this source off. Skip it here as a NO-OP
// (status 'skipped', NOT a failure — a failure would re-arm the cooldown).
if (sourceId) {
const { isSourceInCooldown } = await import('./autopilot-fanout.ts');
if (await isSourceInCooldown(engine, sourceId)) {
return {
partial: false,
status: 'skipped',
report: { reason: 'source_in_cooldown', source_id: sourceId },
};
}
}
const report = await runCycle(engine, {
brainDir: repoPath,
brainDir: effectiveBrainDir,
pull,
signal: job.signal, // propagate abort so cycle bails on timeout/cancel
...(sourceId ? { sourceId } : {}),
@@ -1672,6 +1739,49 @@ export async function registerBuiltinHandlers(
};
});
// #2194 fix #3 / #2227 bug #3 — brain-wide maintenance. Runs the `global`
// cycle phases (embed, orphans, purge, resolve_symbol_edges, grade_takes,
// calibration_profile, synthesize_concepts, skillopt) ONCE per window instead
// of N times concurrently across per-source cycles (the 4→10GB RSS blowout).
// No source_id → uses the legacy global cycle lock; stamps autopilot.last_global_at
// on success so the dispatch gate backs off.
worker.register('autopilot-global-maintenance', async (job) => {
const { runCycle, GLOBAL_PHASES, LAST_GLOBAL_AT_KEY, ALL_PHASES } = await import('../core/cycle.ts');
const repoPath: string | null = typeof job.data.repoPath === 'string'
? job.data.repoPath
: (await engine.getConfig('sync.repo_path')) ?? null;
const validPhases = new Set(ALL_PHASES);
const requested = Array.isArray(job.data.phases)
? (job.data.phases as string[]).filter((p) => validPhases.has(p as never))
: GLOBAL_PHASES;
const phases = (requested.length > 0 ? requested : GLOBAL_PHASES) as typeof GLOBAL_PHASES;
const report = await runCycle(engine, {
brainDir: repoPath,
pull: false, // brain-wide DB/maintenance work never git-pulls
signal: job.signal,
phases,
yieldBetweenPhases: async () => { await new Promise<void>((r) => setImmediate(r)); },
});
// Stamp last_global_at only on a non-failed run so a failed pass stays stale
// and re-dispatches next tick (self-healing retry).
if (report.status === 'ok' || report.status === 'clean' || report.status === 'partial') {
try {
await engine.setConfig(LAST_GLOBAL_AT_KEY, new Date().toISOString());
} catch (e) {
console.warn(`[autopilot-global-maintenance] failed to stamp last_global_at: ${e instanceof Error ? e.message : String(e)}`);
}
}
return {
partial: report.status === 'partial' || report.status === 'failed',
status: report.status,
report,
};
});
// Shell handler is always registered. Runtime env guard lives inside the
// handler so claimed jobs emit a clear rejection log on workers missing
// GBRAIN_ALLOW_SHELL_JOBS=1.
+38 -13
View File
@@ -692,7 +692,7 @@ async function runFederate(engine: BrainEngine, args: string[], value: boolean):
const config = parseConfig(src.config);
config.federated = value;
await engine.executeRaw(
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(config), id],
);
console.log(`Source "${id}" is now ${value ? 'federated (appears in cross-source default search)' : 'isolated (only searched when explicitly named)'}.`);
@@ -747,8 +747,26 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
// caught-up source reports lag 0 instead of growing wall-clock (v0.41.32.0).
const metrics = await computeAllSourceMetrics(engine, sources, { probeContent: true });
// #1950: a source holding a live (non-TTL-expired) per-source sync lock is
// actively syncing RIGHT NOW. Without this it printed "idle" while a sync
// proc was live (the bug reported). Read the SAME live-lock signal `gbrain
// doctor` uses, via the shared helper, so the two surfaces never disagree.
const { liveSyncStatus } = await import('../core/db-lock.ts');
const syncRunning = new Map<string, { holder_pid: number; holder_host: string }>();
await Promise.all(
metrics.map(async (m) => {
const live = await liveSyncStatus(engine, m.source_id);
if (live) syncRunning.set(m.source_id, live);
}),
);
if (json) {
console.log(JSON.stringify({ schema_version: 1, sources: metrics }, null, 2));
const enriched = metrics.map((m) => ({
...m,
sync_running: syncRunning.has(m.source_id),
sync_holder: syncRunning.get(m.source_id) ?? null,
}));
console.log(JSON.stringify({ schema_version: 1, sources: enriched }, null, 2));
return;
}
@@ -765,11 +783,15 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
const embed = `${m.embed_coverage_pct.toFixed(0)}%`;
// v0.41.31: embed-backfill state (active beats queued beats idle) so a
// cron operator sees deferred embedding work after `sync --all`.
const backfill = m.backfill_active > 0
? `active(${m.backfill_active})`
: m.backfill_queued > 0
? `queued(${m.backfill_queued})`
: 'idle';
// #1950: a live sync lock wins the column — surfacing "actively syncing"
// matters more than deferred embed-backfill state.
const backfill = syncRunning.has(m.source_id)
? 'running'
: m.backfill_active > 0
? `active(${m.backfill_active})`
: m.backfill_queued > 0
? `queued(${m.backfill_queued})`
: 'idle';
const fails = String(m.failed_jobs_24h);
const queue = String(m.queue_depth);
const pages = m.total_pages.toLocaleString();
@@ -780,7 +802,10 @@ async function runStatus(engine: BrainEngine, args: string[]): Promise<void> {
for (const m of metrics) {
const warns: string[] = [];
if (!m.local_path) warns.push('no local_path');
if (m.lag_seconds === null) warns.push(`never synced — run \`gbrain sync --source ${m.source_id}\``);
// #1950: don't cry "never synced" while a sync lock is live — it's syncing now.
if (m.lag_seconds === null && !syncRunning.has(m.source_id)) {
warns.push(`never synced — run \`gbrain sync --source ${m.source_id}\``);
}
if (m.embed_coverage_pct < 95 && m.total_chunks > 100) {
warns.push(`${(100 - m.embed_coverage_pct).toFixed(1)}% un-embedded — run \`gbrain embed --stale --source ${m.source_id}\``);
}
@@ -854,7 +879,7 @@ async function runWebhookSet(engine: BrainEngine, args: string[]): Promise<void>
cfg.webhook_secret = secret;
cfg.github_repo = githubRepo;
await engine.executeRaw(
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(cfg), id],
);
@@ -910,7 +935,7 @@ async function runWebhookRotate(engine: BrainEngine, args: string[]): Promise<vo
const cfg = parseConfig(src.config);
cfg.webhook_secret = secret;
await engine.executeRaw(
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(cfg), id],
);
console.log(`New webhook secret for source "${id}":`);
@@ -934,7 +959,7 @@ async function runWebhookClear(engine: BrainEngine, args: string[]): Promise<voi
delete cfg.webhook_secret;
delete cfg.github_repo;
await engine.executeRaw(
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(cfg), id],
);
console.log(`Webhook configuration cleared for source "${id}".`);
@@ -959,7 +984,7 @@ async function runTrackedBranch(engine: BrainEngine, args: string[]): Promise<vo
if (setArg) {
cfg.tracked_branch = setArg;
await engine.executeRaw(
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(cfg), id],
);
console.log(`Tracked branch for source "${id}" set to "${setArg}".`);
@@ -975,7 +1000,7 @@ async function runTrackedBranch(engine: BrainEngine, args: string[]): Promise<vo
const branch = execFileSync('git', ['-C', src.local_path, 'rev-parse', '--abbrev-ref', 'HEAD'], { encoding: 'utf8' }).trim();
cfg.tracked_branch = branch;
await engine.executeRaw(
`UPDATE sources SET config = $1::jsonb WHERE id = $2`,
`UPDATE sources SET config = $1::text::jsonb WHERE id = $2`,
[JSON.stringify(cfg), id],
);
console.log(`Detected branch "${branch}" for source "${id}"; persisted to config.tracked_branch.`);
+152 -23
View File
@@ -39,6 +39,7 @@ import type { BrainEngine } from '../core/engine.ts';
import { existsSync, readFileSync } from 'node:fs';
import { gbrainPath, loadConfig, isThinClient } from '../core/config.ts';
import { callRemoteTool, unpackToolResult } from '../core/mcp-client.ts';
import { VERSION } from '../version.ts';
import {
buildSyncStatusReport,
type SyncStatusReport,
@@ -104,6 +105,10 @@ export interface AutopilotStatus {
export interface StatusReport {
schema_version: typeof SCHEMA_VERSION;
/** #1984: the local gbrain CLI version, so a poller can pin behavior to a build. */
version: string;
/** #1984: the remote brain server's version (thin-client only; present when reported). */
remote_version?: string;
generated_at: string;
mode: 'local' | 'thin-client';
sync?: SyncStatusReport;
@@ -113,6 +118,33 @@ export interface StatusReport {
queue?: QueueCounts | { local_only_remote: true };
autopilot?: AutopilotStatus | { local_only_remote: true };
warnings?: string[];
/** #1984: true when a --deadline-ms budget elided one or more sections. */
partial?: boolean;
/** #1984: names of the sections skipped/timed-out under the deadline. */
stale_sections?: string[];
}
/**
* #1984: race a section's work against the remaining deadline budget. Returns
* the value, or undefined if the budget elapses first (the underlying query is
* abandoned — the process is a one-shot, so a stranded read is fine). `ms`
* undefined / <=0 means "no deadline" and the promise is awaited as-is.
*/
export async function withSectionDeadline<T>(
p: Promise<T>,
ms: number | undefined,
onTimeout: () => void,
): Promise<T | undefined> {
if (ms === undefined || ms <= 0) return p;
let timer: ReturnType<typeof setTimeout> | undefined;
const timeout = new Promise<undefined>((resolve) => {
timer = setTimeout(() => { onTimeout(); resolve(undefined); }, ms);
});
try {
return await Promise.race([p, timeout]);
} finally {
if (timer) clearTimeout(timer);
}
}
// ---------------------------------------------------------------------------
@@ -305,6 +337,8 @@ function buildAutopilotStatus(): AutopilotStatus {
interface BuildOpts {
sections?: Set<Section>;
/** #1984: total wall-clock budget; sections share it, later ones get the remainder. */
deadlineMs?: number;
}
async function buildLocalReport(
@@ -313,23 +347,44 @@ async function buildLocalReport(
): Promise<StatusReport> {
const want = (s: Section) => !opts.sections || opts.sections.has(s);
const warnings: string[] = [];
const staleSections: string[] = [];
const report: StatusReport = {
schema_version: SCHEMA_VERSION,
version: VERSION,
generated_at: new Date().toISOString(),
mode: 'local',
};
// #1984: a shared budget so `gbrain status --deadline-ms=N` never blocks a
// poller past N. Each async section is raced against the REMAINING budget, so
// one slow/hung section (cross-region DB, lock contention) can't strand the
// whole snapshot — it's marked stale and the rest still return.
const deadlineAt = opts.deadlineMs && opts.deadlineMs > 0 ? Date.now() + opts.deadlineMs : null;
const remaining = (): number | undefined =>
deadlineAt === null ? undefined : Math.max(1, deadlineAt - Date.now());
const markStale = (name: Section) => {
staleSections.push(name);
report.partial = true;
warnings.push(`${name} section exceeded the --deadline-ms budget (returned stale)`);
};
if (want('sync')) {
try {
const sources = await engine.executeRaw<{
id: string;
name: string;
local_path: string | null;
config: Record<string, unknown> | null;
}>(`SELECT id, name, local_path, config FROM sources ORDER BY id`);
report.sync = await buildSyncStatusReport(
engine,
sources.map((s) => ({ id: s.id, name: s.name, local_path: s.local_path, config: s.config ?? {} })),
report.sync = await withSectionDeadline(
(async () => {
const sources = await engine.executeRaw<{
id: string;
name: string;
local_path: string | null;
config: Record<string, unknown> | null;
}>(`SELECT id, name, local_path, config FROM sources ORDER BY id`);
return buildSyncStatusReport(
engine,
sources.map((s) => ({ id: s.id, name: s.name, local_path: s.local_path, config: s.config ?? {} })),
);
})(),
remaining(),
() => markStale('sync'),
);
} catch (err) {
warnings.push(`sync section failed: ${(err as Error).message}`);
@@ -337,23 +392,24 @@ async function buildLocalReport(
}
if (want('cycle')) {
try {
report.cycle = await buildCycleSnapshot(engine);
report.cycle = await withSectionDeadline(buildCycleSnapshot(engine), remaining(), () => markStale('cycle'));
} catch (err) {
warnings.push(`cycle section failed: ${(err as Error).message}`);
}
}
if (want('locks')) {
report.locks = await buildLocks(engine);
report.locks = await withSectionDeadline(buildLocks(engine), remaining(), () => markStale('locks'));
}
if (want('workers')) {
report.workers = buildWorkerSummary();
}
if (want('queue')) {
report.queue = await buildQueueCounts(engine);
report.queue = await withSectionDeadline(buildQueueCounts(engine), remaining(), () => markStale('queue'));
}
if (want('autopilot')) {
report.autopilot = buildAutopilotStatus();
}
if (staleSections.length > 0) report.stale_sections = staleSections;
if (warnings.length > 0) report.warnings = warnings;
return report;
}
@@ -366,20 +422,51 @@ async function buildThinClientReport(
const warnings: string[] = [];
const report: StatusReport = {
schema_version: SCHEMA_VERSION,
version: VERSION,
generated_at: new Date().toISOString(),
mode: 'thin-client',
};
if (want('sync') || want('cycle')) {
try {
const raw = await callRemoteTool(cfg!, 'get_status_snapshot', {});
const payload = unpackToolResult<{
schema_version: number;
sync: SyncStatusReport;
cycle: CycleSnapshot;
}>(raw);
if (want('sync')) report.sync = payload.sync;
if (want('cycle')) report.cycle = payload.cycle;
const payload = await withSectionDeadline(
(async () => {
// #1984: pass the budget as the request timeout so the LOSING side of
// the race actually cancels the in-flight MCP call instead of leaking
// it (the section deadline only abandons the promise locally).
const raw = await callRemoteTool(
cfg!,
'get_status_snapshot',
{},
opts.deadlineMs && opts.deadlineMs > 0 ? { timeoutMs: opts.deadlineMs } : {},
);
return unpackToolResult<{
schema_version: number;
version?: string;
sync: SyncStatusReport;
cycle: CycleSnapshot;
}>(raw);
})(),
opts.deadlineMs && opts.deadlineMs > 0 ? opts.deadlineMs : undefined,
() => {
report.partial = true;
// Only name the sections the caller actually requested; the remote
// fetch backs both sync+cycle, but `--section sync` must not report
// `cycle` (a section it excluded) as stale. Matches the local path.
const elided: Section[] = [
...(want('sync') ? (['sync'] as Section[]) : []),
...(want('cycle') ? (['cycle'] as Section[]) : []),
];
report.stale_sections = [...(report.stale_sections ?? []), ...elided];
warnings.push('remote snapshot exceeded the --deadline-ms budget (returned stale)');
},
);
if (payload) {
// #1984: surface the brain server's version for thin-client parity.
if (payload.version) report.remote_version = payload.version;
if (want('sync')) report.sync = payload.sync;
if (want('cycle')) report.cycle = payload.cycle;
}
} catch (err) {
warnings.push(`remote snapshot failed: ${(err as Error).message}`);
}
@@ -401,7 +488,13 @@ function renderHuman(report: StatusReport): string {
lines.push('');
lines.push('GBrain Status');
lines.push('=============');
lines.push(`Mode: ${report.mode} · ${report.generated_at}`);
const ver = report.remote_version && report.remote_version !== report.version
? `v${report.version} (remote v${report.remote_version})`
: `v${report.version}`;
lines.push(`Mode: ${report.mode} · ${ver} · ${report.generated_at}`);
if (report.partial) {
lines.push(`⚠ partial snapshot — stale sections: ${(report.stale_sections ?? []).join(', ')} (--deadline-ms budget hit)`);
}
lines.push('');
// Sync
@@ -528,6 +621,36 @@ function renderHuman(report: StatusReport): string {
* - Set<Section> → only these sections
* - 'usage_error' → bad section name (caller exits 2)
*/
/** #1984: `--fast` preset budget (ms) when no explicit `--deadline-ms` is given. */
export const FAST_DEADLINE_MS = 2000;
/**
* #1984: parse the status deadline budget. `--deadline-ms=N` (or `--deadline-ms N`)
* wins; bare `--fast` applies FAST_DEADLINE_MS. Returns undefined (no budget),
* a positive ms value, or 'usage_error' on a non-positive / non-numeric value.
*/
export function parseDeadlineFlag(args: string[]): number | undefined | 'usage_error' {
let raw: string | undefined;
for (let i = 0; i < args.length; i++) {
const a = args[i];
if (a === '--deadline-ms') {
// #1984: a bare `--deadline-ms` with no following value is a usage error,
// not a silent fall-through to no-budget / --fast (which would mask a
// typo'd budget and let a poller hang it never meant to).
if (i + 1 >= args.length) return 'usage_error';
raw = args[i + 1];
break;
}
if (a.startsWith('--deadline-ms=')) { raw = a.slice('--deadline-ms='.length); break; }
}
if (raw == null) {
return args.includes('--fast') ? FAST_DEADLINE_MS : undefined;
}
const n = Number(raw);
if (!Number.isFinite(n) || n <= 0) return 'usage_error';
return n;
}
export function parseSectionFlag(args: string[]): Set<Section> | undefined | 'usage_error' {
let raw: string | undefined;
for (let i = 0; i < args.length; i++) {
@@ -575,19 +698,25 @@ export async function runStatus(
const sections = sectionFlag;
const json = args.includes('--json');
const deadlineMs = parseDeadlineFlag(args);
if (deadlineMs === 'usage_error') {
stderr('gbrain status: --deadline-ms must be a positive number of milliseconds\n');
return { exitCode: 2 };
}
const cfg = loadConfig();
const useThinClient = cfg ? isThinClient(cfg) : false;
let report: StatusReport;
try {
if (useThinClient) {
report = await buildThinClientReport(cfg, { sections });
report = await buildThinClientReport(cfg, { sections, deadlineMs });
} else {
if (!engine) {
stderr('gbrain status: no engine connected (DB unreachable?). Run `gbrain doctor` to diagnose.\n');
return { exitCode: 1 };
}
report = await buildLocalReport(engine, { sections });
report = await buildLocalReport(engine, { sections, deadlineMs });
}
} catch (err) {
stderr(`gbrain status: snapshot failed: ${(err as Error).message}\n`);
+74 -6
View File
@@ -212,7 +212,7 @@ export interface SyncResult {
* cron operators can disambiguate timeout vs pull-timeout in monitoring.
*/
filesImported?: number;
reason?: 'timeout' | 'pull_timeout' | 'checkpoint_unavailable';
reason?: 'timeout' | 'pull_timeout' | 'stall_timeout' | 'checkpoint_unavailable';
/**
* v0.42.x (#1794): cumulative file paths durably banked to the checkpoint
* across THIS run + prior resumed runs. Surfaced on every partial/blocked
@@ -1440,7 +1440,7 @@ function buildPartialResult(opts: {
modified: number;
deleted: number;
renamed: number;
reason: 'timeout' | 'pull_timeout' | 'checkpoint_unavailable';
reason: 'timeout' | 'pull_timeout' | 'stall_timeout' | 'checkpoint_unavailable';
bankedFiles?: number;
}): SyncResult {
return {
@@ -2050,7 +2050,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// `bankedFiles` is surfaced so a killed run shows banked progress instead of
// looking like total loss. toCommit reports the PINNED target; last_commit is
// never advanced on a partial (the next run resumes from the checkpoint).
const partial = async (reason: 'timeout' | 'pull_timeout'): Promise<SyncResult> => {
const partial = async (reason: 'timeout' | 'pull_timeout' | 'stall_timeout'): Promise<SyncResult> => {
deregisterCheckpointCleanup();
if (!checkpointDead) {
try { await flushCheckpoint(); } catch { /* best effort — we're aborting */ }
@@ -2409,6 +2409,48 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
pacer = createNoopPacer();
}
// #1950: progress-aware stall watchdog for the import drain. The incident
// was a sync wedged ~29min while ALIVE — so the lock heartbeat kept
// refreshing (it fires on its own timer) and the wall-clock deadline hadn't
// hit yet, leaving only a manual `pkill`. This keys off FORWARD IMPORT
// PROGRESS (progress.tick below bumps `progressAt`), not the heartbeat: if no
// file completes for `resolveStallAbortSeconds()`, abort. The abort signal is
// composed into `opts.signal`, so the existing per-iteration abort checks,
// pacer.acquire/pace, and parallel-worker break-loops all observe it; the
// drain returns partial() (last_commit unchanged, next run resumes from the
// checkpoint) and withRefreshingLock's finally releases the lock. Limits
// (TODOS: #1950 follow-up — thread a cancellation signal through importFile):
// the abort is observed BETWEEN files (the per-iteration checks + the next
// importOnePath's pre-acquire check), so a hang INSIDE a single importFile
// call is not interrupted until that call returns — the watchdog fires and
// logs, but the in-flight file finishes (or the wall-clock hard deadline is
// the eventual backstop). This catches the documented #1950 incident shape
// (a slow-but-progressing drain, many files) and a stalled between-file
// drain; a single wedged file or a fully starved event loop is out of scope
// here. stallAborted distinguishes this from a user --timeout/SIGINT so the
// partial result reports `stall_timeout`, not `timeout`.
const stallSeconds = resolveStallAbortSeconds();
const progressAt = { last: Date.now() };
let stallAborted = false;
let stallTimer: ReturnType<typeof setInterval> | undefined;
if (stallSeconds > 0) {
const stallMs = stallSeconds * 1000;
const stallController = new AbortController();
stallTimer = setInterval(() => {
if (Date.now() - progressAt.last >= stallMs) {
serr(
`[sync] no import progress for ${stallSeconds}s — aborting (stall watchdog). ` +
`The per-source lock will release; the next 'gbrain sync' resumes from the checkpoint.`,
);
stallAborted = true;
stallController.abort();
}
}, Math.min(5000, stallMs));
// Don't keep the process alive on the watchdog alone.
(stallTimer as unknown as { unref?: () => void }).unref?.();
opts = { ...opts, signal: composeAbortSignals(opts.signal, stallController.signal) };
}
async function importOnePath(eng: BrainEngine, path: string): Promise<void> {
const filePath = join(syncRepoPath, path);
if (!existsSync(filePath)) {
@@ -2429,6 +2471,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// its row so it can't age doctor to a permanent FAIL. (This covers the
// net-zero add-then-delete range where the path isn't in filtered.deleted.)
succeededPaths.push(path);
progressAt.last = Date.now(); // #1950: forward progress → reset stall watchdog
progress.tick(1, `skip:${path}`);
return;
}
@@ -2484,6 +2527,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
} finally {
permit.release();
}
progressAt.last = Date.now(); // #1950: forward progress → reset stall watchdog
progress.tick(1, path);
// v0.42.x (#1794): keep the lock-refresh heartbeat alive on big imports.
await maybeYield();
@@ -2510,7 +2554,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// serial fallback inside the parallel branch (database_url unset).
if (opts.signal?.aborted) {
progress.finish();
return await partial('timeout');
return await partial(stallAborted ? 'stall_timeout' : 'timeout');
}
await importOnePath(engine, path);
}
@@ -2574,7 +2618,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// primary serial site.
if (opts.signal?.aborted) {
progress.finish();
return await partial('timeout');
return await partial(stallAborted ? 'stall_timeout' : 'timeout');
}
await importOnePath(engine, path);
}
@@ -2583,6 +2627,10 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// paced-backfill: release any blocked acquirers + clear pacer state on
// every exit path (including the early partial('timeout') returns above).
pacer.dispose();
// #1950: tear down the stall watchdog on every import-phase exit (normal,
// partial('timeout'), or throw). The try wrapping the import loop guarantees
// this runs before any post-import bookmark/anchor work.
if (stallTimer) clearInterval(stallTimer);
}
progress.finish();
@@ -2594,7 +2642,7 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// the bookmark write below. By returning partial here, we preserve
// the D-V3-1 invariant that abort means "never advance last_commit."
if (opts.signal?.aborted) {
return await partial('timeout');
return await partial(stallAborted ? 'stall_timeout' : 'timeout');
}
}
@@ -3144,6 +3192,26 @@ export interface HardDeadlineResolution {
reason: string;
}
/**
* #1950: default no-import-progress window before the in-band stall watchdog
* aborts the drain. Generous on purpose — it must clear one legitimately large
* file (cross-region, big page) without false-tripping; one file taking longer
* than this trips it (documented limit). Distinct from the wall-clock hard
* deadline (whole-run cap) and the lock heartbeat (refreshes regardless of
* import progress). Env-tunable; <=0 disables.
*/
export const DEFAULT_SYNC_STALL_ABORT_SEC = 900;
export function resolveStallAbortSeconds(
env: Record<string, string | undefined> = process.env,
): number {
const raw = env.GBRAIN_SYNC_STALL_ABORT_SECONDS;
if (raw === undefined || raw === '') return DEFAULT_SYNC_STALL_ABORT_SEC;
const n = Number(raw);
if (!Number.isFinite(n)) return DEFAULT_SYNC_STALL_ABORT_SEC;
return n; // n <= 0 disables the watchdog
}
/**
* Resolve the out-of-band hard-deadline for a `gbrain sync` invocation (#1633).
* Pure + argv/env-only so it runs BEFORE `connectEngine` (so a connect-phase hang
Binary file not shown.
+1 -1
View File
@@ -123,7 +123,7 @@ export async function putCachedTraversal<T>(
`INSERT INTO code_traversal_cache
(symbol_qualified, depth, source_id, response_json,
max_chunk_updated_at, xmin_max, cluster_generation)
VALUES ($1, $2, $3, $4::jsonb, $5::timestamptz, $6, $7)
VALUES ($1, $2, $3, $4::text::jsonb, $5::timestamptz, $6, $7)
ON CONFLICT (symbol_qualified, depth, source_id)
DO UPDATE SET
response_json = EXCLUDED.response_json,
+4 -4
View File
@@ -266,13 +266,13 @@ async function writeDbCache<T>(
): Promise<void> {
const [, , contentSha] = splitCacheKey(key);
if (!contentSha) return;
// executeRaw with positional binding for JSONB. Per the sql-query.ts
// contract: object values passed via positional params reach the
// wire as proper jsonb when cast.
// #2339 class: this binds JSON.stringify(value) (a STRING) positionally, so it
// must cast through $4::text::jsonb — a bare $4::jsonb double-encodes under
// postgres.js .unsafe() (PGLite hides it). Pass a raw object to use $N::jsonb.
await engine.executeRaw(
`INSERT INTO conversation_parser_llm_cache
(content_sha256, model_id, call_shape, value_json)
VALUES ($1, $2, $3, $4::jsonb)
VALUES ($1, $2, $3, $4::text::jsonb)
ON CONFLICT (content_sha256, model_id, call_shape) DO NOTHING`,
[contentSha, modelStr, shape, JSON.stringify(value)],
);
+30 -2
View File
@@ -242,6 +242,25 @@ export const PHASE_SCOPE: Record<CyclePhase, PhaseScope> = {
skillopt: 'global',
};
/**
* #2194 fix #3 / #2227 bug #3 — the cycle split.
*
* Per-source autopilot cycles run ONLY the source-scoped (and mixed) phases;
* the brain-wide `global` phases (embed, orphans, purge, resolve_symbol_edges,
* grade_takes, calibration_profile, synthesize_concepts, skillopt) run ONCE in
* a separate `autopilot-global-maintenance` job instead of N times concurrently
* across per-source cycles (the 4→10GB RSS blowout). Single-flight is
* structural: one global job, not a skip-and-pretend-fresh hack (codex #1/#2).
*
* GLOBAL_PHASES NON_GLOBAL_PHASES == ALL_PHASES, with no overlap — pinned by
* test/autopilot-global-maintenance.test.ts.
*/
export const GLOBAL_PHASES: CyclePhase[] = ALL_PHASES.filter((p) => PHASE_SCOPE[p] === 'global');
export const NON_GLOBAL_PHASES: CyclePhase[] = ALL_PHASES.filter((p) => PHASE_SCOPE[p] !== 'global');
/** Config key holding the ISO timestamp of the last successful global-maintenance run. */
export const LAST_GLOBAL_AT_KEY = 'autopilot.last_global_at';
/**
* Phases that mutate state (filesystem or DB) and therefore should
* coordinate via the cycle lock. Only orphans is truly read-only
@@ -2305,12 +2324,21 @@ export async function runCycle(
// the cost of missing a successful write (next cycle will redo work).
if (opts.sourceId && engine && !dryRun && !aborted && (status === 'ok' || status === 'clean' || status === 'partial')) {
try {
const nowIso = new Date().toISOString();
// #2194 fix #3 (the cycle split): `last_source_cycle_at` is the NEW gate
// for per-source dispatch (source-scoped phases done). We ALSO keep
// `last_full_cycle_at` current so doctor's cycle-freshness check and any
// legacy reader stay valid — it's no longer a *gate* for the brain-wide
// phases (those gate on autopilot.last_global_at), so writing it on a
// source-only cycle does not re-introduce the freshness poisoning codex
// flagged in the rejected skip-based design.
await engine.updateSourceConfig(opts.sourceId, {
last_full_cycle_at: new Date().toISOString(),
last_source_cycle_at: nowIso,
last_full_cycle_at: nowIso,
});
} catch (e) {
// Best-effort; cycle already succeeded by the time we get here.
console.warn(`[cycle] failed to write last_full_cycle_at for source ${opts.sourceId}: ${e instanceof Error ? e.message : String(e)}`);
console.warn(`[cycle] failed to write last_source_cycle_at for source ${opts.sourceId}: ${e instanceof Error ? e.message : String(e)}`);
}
}
+1 -1
View File
@@ -356,7 +356,7 @@ class CalibrationProfilePhase extends BaseCyclePhase {
active_bias_tags, model_id, cost_usd, judge_model_agreement
) VALUES ($1, $2, now(), false,
$3, $4, $5, $6, $7,
$8::jsonb, $9::text[],
$8::text::jsonb, $9::text[],
$10, $11,
$12::text[], $13, NULL, NULL)`,
[
+54
View File
@@ -132,6 +132,28 @@ export function isHolderDeadLocally(
return classifyHolderLiveness(holderPid, holderHost, ageMs, opts) === 'dead_eligible';
}
/**
* issue #2227: is a lock-row holder live enough to count as "running" for an
* observability surface (`gbrain jobs supervisor status`, `gbrain doctor`)?
*
* PID-reuse-safe by design (`pid-liveness-alone-pid-reuse`): keys on the lock's
* own freshness, NEVER `process.kill`. A live holder refreshes its TTL on a
* timer; a dead one stops, so its `ttl_expires_at` lapses and (after the steal
* grace) `last_refreshed_at` ages out. The primary signal is `!ttl_expired`;
* the heartbeat grace covers a starved-but-alive holder whose TTL briefly
* lapsed between refresh ticks (the #1794 thrash class), mirroring the
* steal-grace semantics `tryAcquireDbLock` already uses. Cross-host holders are
* still "running" for visibility (a supervisor exists, just elsewhere) — we
* report freshness, not takeover-eligibility, so host is not consulted here.
*/
export function isLockHolderLive(snap: LockSnapshot, ttlMinutes: number = DEFAULT_TTL_MINUTES): boolean {
if (!snap.ttl_expired) return true;
if (snap.ms_since_last_refresh !== null) {
return snap.ms_since_last_refresh < resolveStealGraceSeconds(ttlMinutes) * 1000;
}
return false;
}
/**
* Try to acquire a named DB lock.
*
@@ -707,6 +729,38 @@ export function syncLockId(sourceId: string): string {
*/
export const SYNC_LOCK_ID = syncLockId('default');
/**
* #1950: is `sourceId` actively holding a live (non-TTL-expired) sync lock?
*
* Centralizes the live-sync signal that `gbrain doctor` already computes inline
* so `gbrain sources status` (and future surfaces) read the SAME truth instead
* of each re-deriving it. Returns the holder when a live lock is held, else
* null (idle, or a stale/expired lock that's structurally available for the
* next acquire). Inspect failures swallow to null — a status surface should
* degrade to "no indicator", never crash.
*
* Honest scope: a live lock proves the holder process is heartbeating, NOT that
* the import is making forward progress — `withRefreshingLock` refreshes
* `last_refreshed_at` on its own timer regardless of import progress. So callers
* report "running", NOT "healthy"; a wedged-but-alive holder still reads as
* running here. Forward-progress stall detection lives in the sync drain loop
* (#1950 stall-abort); stale-lock triage lives in `gbrain doctor`.
*/
export async function liveSyncStatus(
engine: BrainEngine,
sourceId: string,
): Promise<{ holder_pid: number; holder_host: string } | null> {
try {
const snap = await inspectLock(engine, syncLockId(sourceId));
if (snap && !snap.ttl_expired) {
return { holder_pid: snap.holder_pid, holder_host: snap.holder_host };
}
return null;
} catch {
return null;
}
}
/**
* v0.30.1 (T4 + A4): wrap long-running work in a refreshing TTL lock.
*
+1
View File
@@ -128,6 +128,7 @@ export const SKILL_CHECK_NAMES: ReadonlySet<string> = new Set([
*/
export const OPS_CHECK_NAMES: ReadonlySet<string> = new Set([
'alternative_providers',
'autopilot_fanout_concurrency',
'autopilot_lock_scope',
'batch_retry_health',
'brainstorm_health',
+10 -2
View File
@@ -171,11 +171,19 @@ async function generateIntraPagePairs(
results: SearchResult[],
): Promise<ContradictionPair[]> {
if (results.length === 0) return [];
// Unique page_ids only.
const pageIds = Array.from(new Set(results.map((r) => r.page_id)));
// Unique, FINITE page_ids only. Defensive backstop for the alias-hop bug
// (#2339 sibling): an alias-injected synthetic result with an undefined/NaN
// page_id must never reach `ANY($1::int[])` — postgres.js rejects it with
// UNDEFINED_VALUE and aborts the whole probe. Mirrors the hybrid.ts:63 filter.
const pageIds = Array.from(
new Set(
results.map((r) => r.page_id).filter((n): n is number => typeof n === 'number' && Number.isFinite(n)),
),
);
const takesByPage = await engine.listActiveTakesForPages(pageIds);
const out: ContradictionPair[] = [];
for (const r of results) {
if (typeof r.page_id !== 'number' || !Number.isFinite(r.page_id)) continue;
const takes = takesByPage.get(r.page_id) ?? [];
if (takes.length === 0) continue;
const chunkMember = searchResultToMember(r);
+63 -7
View File
@@ -61,9 +61,12 @@ export type ChildSupervisorEvent =
}
| {
kind: 'health_warn';
reason: 'clean_restart_budget_exceeded' | 'rss_watchdog_loop';
reason: 'clean_restart_budget_exceeded' | 'rss_watchdog_loop' | 'crash_budget_degraded';
count: number;
windowMs: number;
/** Present for window-scoped warnings (budget/watchdog loops). */
windowMs?: number;
/** Present for crash_budget_degraded: the soft budget that was crossed. */
max?: number;
};
export interface ChildWorkerSupervisorOpts {
@@ -73,8 +76,24 @@ export interface ChildWorkerSupervisorOpts {
args: string[];
/** Child env. Defaults to a clone of process.env. */
env?: NodeJS.ProcessEnv;
/** Give up after this many consecutive code != 0 exits. */
/**
* Soft crash budget. issue #1994 (#2227 tail): crossing this NO LONGER
* permanently gives up. Instead the supervisor enters DEGRADED mode — it
* keeps respawning with capped exponential backoff (60s cap) and emits a
* loud `crash_budget_degraded` health_warn — so a transient DB-pooler outage
* that trips the counter self-heals when the DB returns (the stable-run reset
* clears crashCount once a respawn runs > stableRunResetMs) instead of
* wedging the queue until a human restart.
*/
maxCrashes: number;
/**
* Hard ceiling: permanently give up (fire onMaxCrashesExceeded) ONLY after
* this many consecutive crashes — the runaway backstop for a genuinely
* unrecoverable hot crash-loop that the stable-run reset never escapes.
* Default: maxCrashes * HARD_STOP_CRASH_MULTIPLIER. Set to 0 to disable
* permanent give-up entirely (retry-forever-with-backoff).
*/
hardStopMaxCrashes?: number;
/** Stable-run reset window: code != 0 after this duration resets crashCount to 1. Default 5 min. */
stableRunResetMs?: number;
@@ -130,6 +149,12 @@ export interface ChildWorkerSupervisorOpts {
_now?: () => number;
}
/** issue #1994: how many multiples of the soft crash budget before the hard
* permanent-give-up backstop fires. 10× the default soft budget of 10 = 100
* consecutive crashes (each within the stable-run window) before we conclude
* it's a genuine code bug and stop. A transient outage recovers long before. */
export const HARD_STOP_CRASH_MULTIPLIER = 10;
const DEFAULTS = {
stableRunResetMs: 5 * 60 * 1000,
cleanRestartBudget: 10,
@@ -287,19 +312,50 @@ export class ChildWorkerSupervisor {
/**
* Run the spawn-and-respawn loop. Resolves when:
* 1. composer.isStopping() returns true, OR
* 2. crashCount reaches maxCrashes (after firing onMaxCrashesExceeded).
* 2. crashCount reaches the HARD ceiling (after firing onMaxCrashesExceeded).
*
* issue #1994 (#2227 tail): crossing the SOFT budget (`maxCrashes`) no longer
* stops the supervisor. It enters degraded mode — keep respawning with capped
* exponential backoff (60s cap, so it's a paced retry, not a hot loop) and
* announce loudly — so a transient DB-pooler outage that trips the counter
* recovers on its own (a respawn that runs > stableRunResetMs resets
* crashCount to 1) instead of permanently wedging the queue. Permanent
* give-up fires only at the much-higher hard ceiling: the runaway backstop
* for a genuinely unrecoverable hot crash-loop.
*/
async run(): Promise<void> {
while (!this.opts.isStopping() && this._crashCount < this.opts.maxCrashes) {
const hardStop = this.opts.hardStopMaxCrashes ??
this.opts.maxCrashes * HARD_STOP_CRASH_MULTIPLIER;
let degradedAnnounced = false;
while (!this.opts.isStopping()) {
await this.spawnOnce();
if (this.opts.isStopping()) return;
if (this._crashCount >= this.opts.maxCrashes) {
this.opts.onMaxCrashesExceeded(this._crashCount, this.opts.maxCrashes);
// Hard ceiling: permanent give-up (the runaway backstop). hardStop <= 0
// disables it entirely (retry-forever-with-backoff for deployments that
// would rather never auto-stop a recoverable supervisor).
if (hardStop > 0 && this._crashCount >= hardStop) {
this.opts.onMaxCrashesExceeded(this._crashCount, hardStop);
return;
}
// Soft budget crossed → degraded mode. Announce once per degradation
// episode; re-arm after a stable-run reset drops us back under budget.
if (this._crashCount >= this.opts.maxCrashes) {
if (!degradedAnnounced) {
degradedAnnounced = true;
this.opts.onEvent({
kind: 'health_warn',
reason: 'crash_budget_degraded',
count: this._crashCount,
max: this.opts.maxCrashes,
});
}
} else {
degradedAnnounced = false;
}
await this.applyBackoff();
}
}
+3
View File
@@ -34,6 +34,9 @@ export const HANDLER_DEFAULT_TIMEOUT_MS: Readonly<Record<string, number>> = {
subagent_aggregator: THIRTY_MIN_MS,
'embed-backfill': THIRTY_MIN_MS,
'autopilot-cycle': THIRTY_MIN_MS,
// #2194 fix #3: brain-wide maintenance (embed-all/orphans/purge/…) can run
// longer than a single source cycle; give it the same 30-min budget.
'autopilot-global-maintenance': THIRTY_MIN_MS,
};
/**
+11 -9
View File
@@ -878,7 +878,7 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
const rows = await engine.executeRaw<{ gbrain_tool_use_id: string }>(
`INSERT INTO subagent_tool_executions
(job_id, message_idx, tool_use_id, tool_name, input, status, schema_version, ordinal, gbrain_tool_use_id, provider_id)
VALUES ($1, $2, $3, $4, $5::jsonb, 'pending', 2, $6, $7, $8)
VALUES ($1, $2, $3, $4, $5::text::jsonb, 'pending', 2, $6, $7, $8)
ON CONFLICT (job_id, message_idx, ordinal) DO UPDATE
SET status = subagent_tool_executions.status
RETURNING gbrain_tool_use_id::text AS gbrain_tool_use_id`,
@@ -891,7 +891,7 @@ async function runSubagentViaGateway(args: GatewayRunArgs): Promise<SubagentResu
onToolCallComplete: async (gbrainToolUseId, output) => {
await engine.executeRaw(
`UPDATE subagent_tool_executions
SET status = 'complete', output = $1::jsonb, ended_at = now()
SET status = 'complete', output = $1::text::jsonb, ended_at = now()
WHERE gbrain_tool_use_id::text = $2`,
[JSON.stringify(output ?? null), gbrainToolUseId],
);
@@ -1107,7 +1107,7 @@ async function persistMessage(engine: BrainEngine, jobId: number, msg: Persisted
await engine.executeRaw(
`INSERT INTO subagent_messages (job_id, message_idx, role, content_blocks,
tokens_in, tokens_out, tokens_cache_read, tokens_cache_create, model)
VALUES ($1, $2, $3, $4::jsonb, $5, $6, $7, $8, $9)
VALUES ($1, $2, $3, $4::text::jsonb, $5, $6, $7, $8, $9)
ON CONFLICT (job_id, message_idx) DO NOTHING`,
[
jobId,
@@ -1131,13 +1131,15 @@ async function persistToolExecPending(
toolName: string,
input: unknown,
): Promise<void> {
// Serialize to JSON string for the ::jsonb cast. When `input` is already a
// string (e.g. pre-serialized), avoid double-encoding which produces a jsonb
// scalar string instead of a jsonb object — breaking `input->>'key'` lookups.
// Serialize to a JSON string, then bind through $5::text::jsonb. The value is
// ALWAYS a string here (pre-serialized input, or JSON.stringify) — binding a
// string to a bare $5::jsonb double-encodes it into a jsonb scalar string under
// postgres.js .unsafe() (#2339 class; PGLite hides it). The ::text cast makes
// the text→jsonb parse produce a real jsonb object.
const jsonStr = typeof input === 'string' ? input : JSON.stringify(input);
await engine.executeRaw(
`INSERT INTO subagent_tool_executions (job_id, message_idx, tool_use_id, tool_name, input, status)
VALUES ($1, $2, $3, $4, $5::jsonb, 'pending')
VALUES ($1, $2, $3, $4, $5::text::jsonb, 'pending')
ON CONFLICT (job_id, tool_use_id) DO NOTHING`,
[jobId, messageIdx, toolUseId, toolName, jsonStr],
);
@@ -1151,7 +1153,7 @@ async function persistToolExecComplete(
): Promise<void> {
await engine.executeRaw(
`UPDATE subagent_tool_executions
SET status = 'complete', output = $3::jsonb, ended_at = now()
SET status = 'complete', output = $3::text::jsonb, ended_at = now()
WHERE job_id = $1 AND tool_use_id = $2`,
[jobId, toolUseId, typeof output === 'string' ? output : JSON.stringify(output)],
);
@@ -1170,7 +1172,7 @@ async function persistToolExecFailed(
// rejected upfront) and "pending row exists" (tool threw mid-execute).
await engine.executeRaw(
`INSERT INTO subagent_tool_executions (job_id, message_idx, tool_use_id, tool_name, input, status, error, ended_at)
VALUES ($1, $2, $3, $4, $5::jsonb, 'failed', $6, now())
VALUES ($1, $2, $3, $4, $5::text::jsonb, 'failed', $6, now())
ON CONFLICT (job_id, tool_use_id) DO UPDATE
SET status = 'failed', error = EXCLUDED.error, ended_at = now()`,
[jobId, messageIdx, toolUseId, toolName, typeof input === 'string' ? input : JSON.stringify(input), error],
+9
View File
@@ -647,9 +647,18 @@ export class MinionQueue {
async handleTimeouts(): Promise<MinionJob[]> {
return this.engine.transaction(async (tx) => {
const rows = await tx.executeRaw<Record<string, unknown>>(
// #1737: count the timed-out run as a spent attempt (terminal, no retry),
// mirroring handleWallClockTimeouts + handleStalled. handleTimeouts is the
// FIRST killer to fire for the long-lane handlers (timeout_ms stamped at
// submit), so without this the job reads `attempts: 0/N (started: N)`.
// Safe against double-count: the worker sweep runs handleStalled ->
// handleTimeouts -> handleWallClockTimeouts sequentially and awaited, and
// each guards on `status = 'active'`, so the first to set status='dead'
// excludes the row from the later sweeps.
`UPDATE minion_jobs SET
status = 'dead',
error_text = 'timeout exceeded',
attempts_made = attempts_made + 1,
lock_token = NULL,
lock_until = NULL,
finished_at = now(),
+28 -1
View File
@@ -30,6 +30,7 @@ import { detectTini } from './spawn-helpers.ts';
import { resolveDefaultMaxRssMb } from './rss-default.ts';
import {
ChildWorkerSupervisor,
HARD_STOP_CRASH_MULTIPLIER,
type ChildSupervisorEvent,
} from './child-worker-supervisor.ts';
import {
@@ -191,6 +192,21 @@ export function buildWorkerArgs(
* shutdown() drain window (issue #1801, D3). */
const WEDGE_RESTART_GRACE_MS = 35_000;
/**
* issue #1994: resolve the hard permanent-give-up ceiling. Default
* maxCrashes × HARD_STOP_CRASH_MULTIPLIER; operators override (or disable with
* 0 = never auto-stop) via GBRAIN_SUPERVISOR_HARD_STOP_CRASHES. A negative or
* non-integer override is ignored (falls back to the default).
*/
export function resolveHardStopMaxCrashes(maxCrashes: number): number {
const raw = process.env.GBRAIN_SUPERVISOR_HARD_STOP_CRASHES;
if (raw !== undefined && raw !== '') {
const n = Number(raw);
if (Number.isInteger(n) && n >= 0) return n;
}
return maxCrashes * HARD_STOP_CRASH_MULTIPLIER;
}
/** Calculate backoff: 1s, 2s, 4s, 8s, 16s, 32s, 60s cap. */
export function calculateBackoffMs(crashCount: number): number {
const base = Math.min(1000 * Math.pow(2, Math.max(crashCount, 0)), 60_000);
@@ -293,7 +309,11 @@ export const ExitCodes = {
* pidfile path. TTL > refresh-interval × max-failures so we always exit
* before our lock could lapse and let a second supervisor take over.
*/
const SUPERVISOR_LOCK_TTL_MIN = 5;
// Exported (issue #2227) so observability surfaces (`gbrain jobs supervisor
// status`, `gbrain doctor`) compute the lock-freshness steal grace with the
// SAME TTL the supervisor refreshes against, when detecting a live supervisor
// via the DB lock instead of the (possibly split-$HOME) pidfile.
export const SUPERVISOR_LOCK_TTL_MIN = 5;
const SUPERVISOR_LOCK_REFRESH_MS = 60_000;
const SUPERVISOR_LOCK_REFRESH_MAX_FAILURES = 3; // 3 × 60s = 180s < 5min TTL
@@ -825,6 +845,10 @@ export class MinionSupervisor {
args: workerArgs,
env,
maxCrashes: this.opts.maxCrashes,
// issue #1994: hard permanent-give-up ceiling (the runaway backstop).
// Operators can raise/lower or disable (0 = never auto-stop) via
// GBRAIN_SUPERVISOR_HARD_STOP_CRASHES; default is maxCrashes × 10.
hardStopMaxCrashes: resolveHardStopMaxCrashes(this.opts.maxCrashes),
_backoffFloorMs: this.opts._backoffFloorMs,
isStopping: () => this.stopping,
onMaxCrashesExceeded: (count, max) => {
@@ -907,6 +931,9 @@ export class MinionSupervisor {
// ("raise --max-rss") is one glance away. Peak RSS stays in the
// worker's own stderr line (the supervisor never sees it).
...(event.reason === 'rss_watchdog_loop' ? { max_rss_mb: this.opts.maxRssMb } : {}),
// issue #1994: degraded mode crossed the soft crash budget — surface
// it so doctor/status show "retrying with backoff" instead of silence.
...(event.max !== undefined ? { max_crashes: event.max } : {}),
});
return;
}
+1 -1
View File
@@ -120,7 +120,7 @@ export async function writeImpactLogRow(
remediation_id, metric_name, metric_before, metric_after,
job_id, source_id, brain_id, started_at, idempotency_key,
applied_by, details
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::jsonb)`,
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11::text::jsonb)`,
[
attribution.remediation_id,
metricName,
+12 -5
View File
@@ -179,15 +179,22 @@ export async function recordCompleted(
// REPLACE semantics (kept deliberately — #1794 V3). Callers like
// extract-conversation-facts serialize a MUTABLE map through here and rely on
// stale keys being REMOVED; an append would make them unremovable. The full
// set lands in the parent `completed_keys` JSONB column via a single UPSERT
// exactly as before. JSON.stringify into `$3::jsonb` is correct (the text→jsonb
// cast yields a proper array; NOT the double-encode trap, which is the template
// form). Sync uses `appendCompleted` (below) instead, never this.
// set lands in the parent `completed_keys` JSONB column via a single UPSERT.
// #2339: bind through `$3::text::jsonb`, NOT `$3::jsonb`. Under postgres.js
// `.unsafe(sql, params)` (executeRawDirect's path) a JS string bound to a
// `$N::jsonb` param double-encodes — the text→jsonb cast wraps the already-JSON
// string into a jsonb *string scalar*, which fails the v119
// `op_checkpoints_completed_keys_array CHECK (jsonb_typeof = 'array')` and aborts
// every sync on real Postgres (PGLite parses it silently, which hid the bug).
// Casting through `text` first binds it as a plain text param so the text→jsonb
// cast parses it into a genuine jsonb array. This is the positional-param form of
// the CLAUDE.md double-encode trap (the grep guard only caught the template form).
// Sync uses `appendCompleted` (below, `unnest($3::text[])`) instead, never this.
const sorted = [...keys].sort();
return durableWrite(engine, key, 'write', () =>
engine.executeRawDirect(
`INSERT INTO op_checkpoints (op, fingerprint, completed_keys, updated_at)
VALUES ($1, $2, $3::jsonb, now())
VALUES ($1, $2, $3::text::jsonb, now())
ON CONFLICT (op, fingerprint) DO UPDATE
SET completed_keys = EXCLUDED.completed_keys,
updated_at = now()`,
+4 -1
View File
@@ -2473,7 +2473,10 @@ const get_status_snapshot: Operation = {
}
const sync = await buildSyncStatusReport(ctx.engine, sources);
const cycle = await buildCycleSnapshot(ctx.engine);
return { schema_version: 1 as const, sync, cycle };
// #1984: report the brain server's version so a thin-client `gbrain status`
// can surface remote_version alongside its own local CLI version.
const { VERSION } = await import('../version.ts');
return { schema_version: 1 as const, version: VERSION, sync, cycle };
},
scope: 'admin',
localOnly: false,
+5
View File
@@ -647,6 +647,11 @@ export async function applyAliasHop(
if (!page) continue;
injectScore += 1e-6;
out.push({
// #2339-sibling: include page_id. The `as SearchResult` cast hid its
// absence, so any consumer reading page_id off an alias-injected result got
// undefined — e.g. listActiveTakesForPages bound undefined/NaN into
// ANY($1::int[]) and crashed the contradiction probe on real Postgres.
page_id: page.id,
slug: page.slug,
title: page.title,
type: page.type,
+1 -1
View File
@@ -236,7 +236,7 @@ export class SemanticQueryCache {
// the v0.40.3.0 IRON-RULE).
await this.engine.executeRaw(
`INSERT INTO query_cache (id, query_text, source_id, knobs_hash, embedding, results, meta, ttl_seconds, page_generations, max_generation_at_store, created_at)
VALUES ($1, $2, $3, $4, $5::vector, $6::jsonb, $7::jsonb, $8, $9::jsonb, $10, now())
VALUES ($1, $2, $3, $4, $5::vector, $6::text::jsonb, $7::text::jsonb, $8, $9::text::jsonb, $10, now())
ON CONFLICT (id) DO UPDATE SET
query_text = EXCLUDED.query_text,
knobs_hash = EXCLUDED.knobs_hash,
+2 -2
View File
@@ -408,7 +408,7 @@ export async function addSource(
try {
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config)
VALUES ($1, $2, $3, $4::jsonb)`,
VALUES ($1, $2, $3, $4::text::jsonb)`,
[opts.id, displayName, finalPath, JSON.stringify(config)],
);
} catch (e) {
@@ -454,7 +454,7 @@ export async function addSource(
const displayName = opts.name ?? opts.id;
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config)
VALUES ($1, $2, $3, $4::jsonb)`,
VALUES ($1, $2, $3, $4::text::jsonb)`,
[opts.id, displayName, finalPath, JSON.stringify(config)],
);
}
+16 -9
View File
@@ -79,15 +79,22 @@ function assertSqlValue(value: unknown): asserts value is SqlValue {
* auth/admin surface that a focused helper preserves the contract without
* forcing every call site to remember which positions hold JSONB.
*
* Why this is safe vs the v0.12.0 double-encode bug: the bug was specific
* to postgres.js's template-tag auto-stringify path interacting with
* sql.json() not to positional binding through `unsafe()`. JS objects
* passed as positional params reach the wire protocol with the correct
* type oid (jsonb when cast in the SQL string), so there is no double-
* encode. The CI guard (scripts/check-jsonb-pattern.sh) doesn't fire
* because the source pattern is a method call (`executeRawJsonb(...)`),
* not the banned literal-template-tag interpolation pattern with
* JSON.stringify cast to jsonb.
* Why this is safe vs the double-encode bug: this helper binds a JS **object**
* (not a pre-stringified string) to each `$N::jsonb` position. postgres.js
* `unsafe()` and PGLite both serialize a JS object to the jsonb wire type
* correctly, so there is no double-encode.
*
* IMPORTANT (the #2339 distinction): positional binding is NOT universally safe.
* Binding `JSON.stringify(x)` (a **string**) to a `$N::jsonb` position via
* `unsafe()`/`executeRawDirect` DOES double-encode the textjsonb cast wraps
* the already-JSON string into a jsonb *string scalar* (PGLite hides it; real
* Postgres exposes it, and it broke every sync in #2339). The fixes are: pass a
* raw object (this helper), or cast through `$N::text::jsonb` so the string is
* parsed, never `$N::jsonb` + JSON.stringify. The legacy grep guard
* (scripts/check-jsonb-pattern.sh) only caught the template-tag form; the
* positional `$N::jsonb` + JSON.stringify form is caught by the AST guard
* scripts/check-jsonb-params.mjs. This helper's `executeRawJsonb(...)` method-call
* shape trips neither guard because it passes objects, which is correct.
*
* Usage:
* await executeRawJsonb(
+2 -2
View File
@@ -32,8 +32,8 @@ export async function writeReceiptToDb(engine: BrainEngine, receipt: TakesQualit
receipt_json, receipt_disk_path, created_at
) VALUES (
$1, $2, $3, $4,
$5, $6, $7, $8::jsonb, $9,
$10::jsonb, $11, $12::timestamptz
$5, $6, $7, $8::text::jsonb, $9,
$10::text::jsonb, $11, $12::timestamptz
)
ON CONFLICT (receipt_sha8_corpus, receipt_sha8_prompt, receipt_sha8_models, receipt_sha8_rubric)
DO NOTHING`,
+55 -3
View File
@@ -43,7 +43,7 @@ describe('parseRunFlags', () => {
expect(rest).toEqual(['hello', 'world']);
});
test('flags before prompt are parsed, unknown token ends flag parsing', () => {
test('leading flags parsed; first positional begins the prompt', () => {
const { flags, rest } = agentTesting.parseRunFlags([
'--model', 'claude-opus-4-7', '--max-turns', '30', 'summarize', 'everything',
]);
@@ -69,8 +69,60 @@ describe('parseRunFlags', () => {
expect(rest).toEqual(['--not-a-flag']);
});
test('unknown flag throws', () => {
expect(() => agentTesting.parseRunFlags(['--what', 'x'])).toThrow(/unknown flag/);
test('#1738: unknown --flag is prompt text, not an error', () => {
const { rest } = agentTesting.parseRunFlags(['--what', 'x']);
expect(rest).toEqual(['--what', 'x']);
});
test('#1738: trailing --detach is hoisted out of the prompt', () => {
const { flags, rest } = agentTesting.parseRunFlags(['do', 'the', 'thing', '--detach']);
expect(flags.detach).toBe(true);
expect(flags.follow).toBe(false);
expect(rest).toEqual(['do', 'the', 'thing']);
});
test('#1738: leading flags + trailing switch both apply', () => {
const { flags, rest } = agentTesting.parseRunFlags(['--model', 'm', 'summarize', '--detach', '--no-follow']);
expect(flags.model).toBe('m');
expect(flags.detach).toBe(true);
expect(flags.follow).toBe(false);
expect(rest).toEqual(['summarize']);
});
test('#1738: a --switch mid-prompt (not trailing) stays verbatim', () => {
const { flags, rest } = agentTesting.parseRunFlags(['summarize', '--detach', 'the', 'doc']);
expect(flags.detach).toBe(false);
expect(rest).toEqual(['summarize', '--detach', 'the', 'doc']);
});
test('#1738: a freeform prompt starting with --word is preserved', () => {
const { rest } = agentTesting.parseRunFlags(['--note:', 'do', 'the', 'thing']);
expect(rest).toEqual(['--note:', 'do', 'the', 'thing']);
});
test('#1738: -- suppresses trailing-switch hoisting', () => {
const { flags, rest } = agentTesting.parseRunFlags(['--', 'do', 'x', '--detach']);
expect(flags.detach).toBe(false);
expect(rest).toEqual(['do', 'x', '--detach']);
});
test('#1738: -- AFTER a positional also suppresses hoisting (no silent detach flip)', () => {
// The leading-flag loop breaks at the first positional, so the `escaped`
// flag never fires for a `--` placed later. A literal `--` ANYWHERE must
// still mean "hoist nothing" — otherwise `agent run note -- body --detach`
// silently detaches and drops the `--` as junk.
const { flags, rest } = agentTesting.parseRunFlags(['note', '--', 'body', '--detach']);
expect(flags.detach).toBe(false);
expect(rest).toEqual(['note', '--', 'body', '--detach']);
});
test('#1738: value-flag missing its value throws a usage error', () => {
expect(() => agentTesting.parseRunFlags(['--model'])).toThrow(/requires a value/);
expect(() => agentTesting.parseRunFlags(['--model', '--detach', 'x'])).toThrow(/requires a value/);
});
test('#1738: numeric value-flag rejects a non-number', () => {
expect(() => agentTesting.parseRunFlags(['--max-turns', 'abc', 'x'])).toThrow(/expects a number/);
});
test('--subagent-def + --timeout-ms parsed', () => {
@@ -0,0 +1,108 @@
/**
* issue #2227/#2194 (TODOS:634, codex #8) a per-source `autopilot-cycle`
* binds its filesystem phases to the SOURCE's own checkout (`local_path`),
* never the global brain's `sync.repo_path`.
*
* Pre-fix the handler fed `repoPath` (the global checkout) into runCycle even
* when `source_id` was set, so FS phases (sync/lint/extract) ran against the
* wrong tree while DB freshness was stamped for `source_id` mixed scope.
* That made the failure-cooldown and freshness gates attribute work to the
* wrong source, the prerequisite codex flagged before the storm-breaker could
* be trusted.
*
* Drives the REAL handler captured from registerBuiltinHandlers (not a
* source-grep) so a reintroduced repoPath fallthrough fails here. PGLite
* in-memory. The report's `brain_dir` mirrors the cycle's effective brainDir
* (cycle.ts:2324), so it's the observable proxy for "which checkout did FS
* phases bind to".
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { mkdtempSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { registerBuiltinHandlers } from '../src/commands/jobs.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
async function captureHandlers(): Promise<Map<string, (job: any) => Promise<any>>> {
const handlers = new Map<string, (job: any) => Promise<any>>();
const fakeWorker = { register(name: string, fn: (job: any) => Promise<any>) { handlers.set(name, fn); } };
await registerBuiltinHandlers(fakeWorker as never, engine);
return handlers;
}
describe('autopilot-cycle handler — per-source checkout binding (#2227/#2194)', () => {
test('source_id with local_path → brainDir is the SOURCE checkout, not the global repo', async () => {
const sourceDir = mkdtempSync(join(tmpdir(), 'gbrain-src-'));
// A DIFFERENT global checkout must NOT win for a per-source job.
await engine.setConfig('sync.repo_path', '/some/global/brain/checkout');
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, archived, created_at)
VALUES ('repo-a', 'Repo A', $1, '{}'::jsonb, false, now())`,
[sourceDir],
);
const handlers = await captureHandlers();
const handler = handlers.get('autopilot-cycle')!;
// DB-only phase keeps the test cheap; brain_dir is stamped from opts.brainDir
// regardless of which phases run, so it still proves the binding.
const result = await handler({
data: { source_id: 'repo-a', phases: ['resolve_symbol_edges'] },
signal: undefined,
});
expect(result.report.brain_dir).toBe(sourceDir);
expect(result.report.brain_dir).not.toBe('/some/global/brain/checkout');
});
test('source_id with NULL local_path → brainDir is null (FS phases skip), never the global repo', async () => {
// The mixed-scope bug: a pure-DB source must NOT fall through to repoPath.
await engine.setConfig('sync.repo_path', '/some/global/brain/checkout');
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path, config, archived, created_at)
VALUES ('db-only', 'DB Only', NULL, '{}'::jsonb, false, now())`,
[],
);
const handlers = await captureHandlers();
const handler = handlers.get('autopilot-cycle')!;
const result = await handler({
data: { source_id: 'db-only', phases: ['resolve_symbol_edges'] },
signal: undefined,
});
expect(result.report.brain_dir).toBeNull();
expect(result.report.brain_dir).not.toBe('/some/global/brain/checkout');
});
test('legacy (no source_id) keeps the global repoPath — back-compat', async () => {
const globalDir = mkdtempSync(join(tmpdir(), 'gbrain-global-'));
await engine.setConfig('sync.repo_path', globalDir);
const handlers = await captureHandlers();
const handler = handlers.get('autopilot-cycle')!;
const result = await handler({
data: { phases: ['resolve_symbol_edges'] },
signal: undefined,
});
expect(result.report.brain_dir).toBe(globalDir);
});
});
+151
View File
@@ -0,0 +1,151 @@
/**
* #2194 fix #2 failure cooldown (the storm-breaker).
*
* A source whose autopilot-cycle keeps failing re-dispatched every 5-min tick
* (only SUCCESS gated dispatch), so the same handful of sources failed and
* re-fanned-out forever 200+ dead jobs/24h. The cooldown backs a failed
* source off with bounded exponential delay, read at dispatch from minion_jobs
* (dead/failed rows) AND re-checked at claim time (codex #5). A success clears
* it (codex #7). These tests pin the pure math, the engine query, the
* null-source guard (codex #6), and the dispatch/claim-time gates.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import {
cooldownMinForCount,
isInFailureCooldown,
readRecentSourceFailures,
isSourceInCooldown,
selectSourcesForDispatch,
resolveFailureCooldownOpts,
type SourceFailure,
type CooldownOpts,
} from '../src/commands/autopilot-fanout.ts';
import type { SourceRow } from '../src/core/engine.ts';
const OPTS: CooldownOpts = { baseMin: 10, capMin: 120 };
describe('cooldownMinForCount — bounded exponential', () => {
test('grows 10, 20, 40, 80 then caps at 120', () => {
expect(cooldownMinForCount(1, OPTS)).toBe(10);
expect(cooldownMinForCount(2, OPTS)).toBe(20);
expect(cooldownMinForCount(3, OPTS)).toBe(40);
expect(cooldownMinForCount(4, OPTS)).toBe(80);
expect(cooldownMinForCount(5, OPTS)).toBe(120); // 160 capped to 120
expect(cooldownMinForCount(99, OPTS)).toBe(120);
});
test('zero/negative count or disabled base → 0', () => {
expect(cooldownMinForCount(0, OPTS)).toBe(0);
expect(cooldownMinForCount(3, { baseMin: 0, capMin: 120 })).toBe(0);
});
});
describe('isInFailureCooldown — pure decision', () => {
const now = Date.UTC(2026, 5, 16, 12, 0, 0);
const ago = (min: number) => new Date(now - min * 60_000);
test('no failure record → not in cooldown', () => {
expect(isInFailureCooldown(undefined, null, now, OPTS)).toBe(false);
});
test('disabled (baseMin 0) → never in cooldown', () => {
expect(isInFailureCooldown({ count: 5, lastFailedAt: ago(1) }, null, now, { baseMin: 0, capMin: 120 })).toBe(false);
});
test('failed 5min ago, count 1 (cooldown 10) → in cooldown', () => {
expect(isInFailureCooldown({ count: 1, lastFailedAt: ago(5) }, null, now, OPTS)).toBe(true);
});
test('failed 15min ago, count 1 (cooldown 10) → recovered by time', () => {
expect(isInFailureCooldown({ count: 1, lastFailedAt: ago(15) }, null, now, OPTS)).toBe(false);
});
test('success at/after the latest failure → cleared (codex #7)', () => {
const failure: SourceFailure = { count: 3, lastFailedAt: ago(5) };
expect(isInFailureCooldown(failure, ago(4), now, OPTS)).toBe(false); // success 4min ago > fail 5min ago
});
test('success BEFORE the latest failure, still in window → suppressed', () => {
const failure: SourceFailure = { count: 3, lastFailedAt: ago(5) };
expect(isInFailureCooldown(failure, ago(30), now, OPTS)).toBe(true);
});
});
describe('selectSourcesForDispatch — cooldown bucket', () => {
const src = (id: string): SourceRow => ({ id, name: id, config: {} } as SourceRow);
test('a stale source in cooldown is held in skippedCooldown, not dispatched', () => {
const sources = [src('a'), src('b')];
const failures = new Map<string, SourceFailure>([
['a', { count: 1, lastFailedAt: new Date(Date.now() - 60_000) }], // 1min ago, cooldown 10min
]);
const r = selectSourcesForDispatch(sources, 4, Date.now(), 60, failures, OPTS);
expect(r.dispatch.map(s => s.id)).toEqual(['b']);
expect(r.skippedCooldown.map(s => s.id)).toEqual(['a']);
});
});
describe('readRecentSourceFailures + isSourceInCooldown (PGLite)', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 30000);
afterAll(async () => { await engine.disconnect(); });
beforeEach(async () => { await resetPgliteState(engine); });
async function addJob(status: string, sourceId: string | null, finishedMinAgo: number): Promise<void> {
const finished = new Date(Date.now() - finishedMinAgo * 60_000).toISOString();
const data = sourceId === null ? {} : { source_id: sourceId };
await engine.executeRaw(
`INSERT INTO minion_jobs (name, status, data, finished_at) VALUES ('autopilot-cycle', $1, $2, $3)`,
[status, data, finished],
);
}
test('groups dead/failed jobs by source with count + max(finished_at)', async () => {
await addJob('dead', 'repo-a', 5);
await addJob('failed', 'repo-a', 2);
await addJob('dead', 'repo-b', 10);
await addJob('completed', 'repo-a', 1); // not counted
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
expect(map.get('repo-a')?.count).toBe(2);
expect(map.get('repo-b')?.count).toBe(1);
expect(map.has('repo-a')).toBe(true);
// last failed is the most recent of the two (2 min ago).
const lastA = map.get('repo-a')!.lastFailedAt.getTime();
expect(Date.now() - lastA).toBeLessThan(4 * 60_000);
});
test('null source_id rows are excluded (codex #6)', async () => {
await addJob('dead', null, 3);
await addJob('dead', 'repo-c', 3);
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
expect(map.has('repo-c')).toBe(true);
expect([...map.keys()].some(k => !k)).toBe(false);
expect(map.size).toBe(1);
});
test('failures older than the window are not counted', async () => {
await addJob('dead', 'repo-old', 500); // way outside a 120min window
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
expect(map.has('repo-old')).toBe(false);
});
test('isSourceInCooldown: recent failure → true; cleared after a success stamp', async () => {
await engine.executeRaw(
`INSERT INTO sources (id, name, config, created_at) VALUES ('repo-cd', 'r', '{}'::jsonb, now())`, [],
);
await addJob('dead', 'repo-cd', 1); // 1 min ago, count 1 → 10min cooldown
expect(await isSourceInCooldown(engine, 'repo-cd')).toBe(true);
// Operator repairs + a successful cycle stamps last_source_cycle_at NOW.
await engine.updateSourceConfig('repo-cd', { last_source_cycle_at: new Date().toISOString() });
expect(await isSourceInCooldown(engine, 'repo-cd')).toBe(false);
});
test('isSourceInCooldown returns false when cooldown disabled (failure_cooldown_min=0)', async () => {
await engine.setConfig('autopilot.failure_cooldown_min', '0');
await addJob('dead', 'repo-dis', 1);
expect(await isSourceInCooldown(engine, 'repo-dis')).toBe(false);
const opts = await resolveFailureCooldownOpts(engine);
expect(opts.baseMin).toBe(0);
});
});
+110
View File
@@ -0,0 +1,110 @@
/**
* #2194 fix #1 (codex #9 / D5): resolveEffectiveFanoutMax clamps the per-tick
* fan-out to the worker's effective concurrency (max(1, concurrency-1),
* reserving 1 slot) but ONLY when a LIVE supervisor holds the queue lock.
* A stale `started` audit row must not shrink throughput for a supervisor that
* isn't running that config, so with no live holder the clamp is skipped and
* the unclamped base is used.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { tryAcquireDbLock } from '../src/core/db-lock.ts';
import { supervisorLockId, SUPERVISOR_LOCK_TTL_MIN } from '../src/core/minions/supervisor.ts';
import { computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts';
import { resolveEffectiveFanoutMax } from '../src/commands/autopilot-fanout.ts';
let engine: PGLiteEngine;
let auditDir: string;
const prevAuditDir = process.env.GBRAIN_AUDIT_DIR;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 30000);
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
auditDir = mkdtempSync(join(tmpdir(), 'gbrain-clamp-'));
process.env.GBRAIN_AUDIT_DIR = auditDir;
await engine.executeRaw(`DELETE FROM gbrain_cycle_locks WHERE id LIKE 'gbrain-supervisor:%'`);
// base fan-out: override to 8 so the clamp's effect is visible on PGLite
// (whose natural default is 1). The clamp logic is engine-agnostic.
await engine.setConfig('autopilot.fanout_max_per_tick', '8');
await engine.setConfig('autopilot.fanout_clamp_to_concurrency', 'true');
});
afterEach(() => {
if (prevAuditDir === undefined) delete process.env.GBRAIN_AUDIT_DIR;
else process.env.GBRAIN_AUDIT_DIR = prevAuditDir;
try { rmSync(auditDir, { recursive: true, force: true }); } catch { /* noop */ }
});
function writeStarted(concurrency: number): void {
const file = join(auditDir, computeSupervisorAuditFilename());
writeFileSync(file, JSON.stringify({
event: 'started', ts: new Date().toISOString(), supervisor_pid: 4242,
queue: 'default', concurrency,
}) + '\n', 'utf8');
}
describe('resolveEffectiveFanoutMax — clamp gated on live supervisor (#2194/codex #9)', () => {
test('NO live holder → no clamp (stale audit row cannot shrink throughput)', async () => {
writeStarted(3); // audit says concurrency 3, but no live lock holder
const n = await resolveEffectiveFanoutMax(engine, 'default');
expect(n).toBe(8); // unclamped base
});
test('live holder + concurrency 3 → clamp to max(1, 3-1) = 2', async () => {
writeStarted(3);
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
expect(holder).not.toBeNull();
try {
const n = await resolveEffectiveFanoutMax(engine, 'default');
expect(n).toBe(2);
} finally {
await holder!.release();
}
});
test('live holder but clamp disabled → unclamped base', async () => {
await engine.setConfig('autopilot.fanout_clamp_to_concurrency', 'false');
writeStarted(3);
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
try {
const n = await resolveEffectiveFanoutMax(engine, 'default');
expect(n).toBe(8);
} finally {
await holder!.release();
}
});
test('live holder + concurrency 1 → floor at 1 (never below 1)', async () => {
writeStarted(1);
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
try {
const n = await resolveEffectiveFanoutMax(engine, 'default');
expect(n).toBe(1);
} finally {
await holder!.release();
}
});
test('live holder but no started event (concurrency unknown) → no clamp', async () => {
// lock row exists but audit has no concurrency → fall back to base.
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
try {
const n = await resolveEffectiveFanoutMax(engine, 'default');
expect(n).toBe(8);
} finally {
await holder!.release();
}
});
});
+5 -2
View File
@@ -28,8 +28,11 @@ describe('autopilot.ts ↔ dispatchPerSource wiring', () => {
);
});
test('imports resolveFanoutMax (so PGLite gets fanoutMax=1 per codex P1-3)', () => {
expect(AUTOPILOT_SRC).toMatch(/resolveFanoutMax/);
test('imports resolveEffectiveFanoutMax (clamps to worker concurrency; PGLite base still 1)', () => {
// #2194 fix #1: autopilot now resolves the CLAMPED fan-out (gated on a live
// supervisor) instead of the raw resolveFanoutMax. The clamp wraps
// resolveFanoutMax, so PGLite's base-1 still holds (codex P1-3).
expect(AUTOPILOT_SRC).toMatch(/resolveEffectiveFanoutMax/);
});
test('calls dispatchPerSource within the shouldFullCycle branch', () => {
+146
View File
@@ -0,0 +1,146 @@
/**
* #2194 fix #3 / #2227 bug #3 the cycle split.
*
* Per-source autopilot cycles run ONLY source-scoped (+ mixed) phases; the
* brain-wide `global` phases run ONCE in a separate autopilot-global-maintenance
* job. This replaces the rejected skip-and-stamp-fresh design (codex #1/#2): the
* split makes single-flight structural (one global job, not N concurrent embeds)
* and never marks a source "fresh" for global work it didn't do. These tests pin
* the phase partition, the dispatch gate, the per-source phase set, and the
* global handler stamping autopilot.last_global_at.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { registerBuiltinHandlers } from '../src/commands/jobs.ts';
import {
ALL_PHASES,
GLOBAL_PHASES,
NON_GLOBAL_PHASES,
PHASE_SCOPE,
LAST_GLOBAL_AT_KEY,
} from '../src/core/cycle.ts';
import {
dispatchGlobalMaintenance,
isGlobalMaintenanceStale,
dispatchPerSource,
} from '../src/commands/autopilot-fanout.ts';
import type { BrainEngine } from '../src/core/engine.ts';
describe('cycle phase partition (#2194 fix #3)', () => {
test('GLOBAL NON_GLOBAL == ALL_PHASES, no overlap', () => {
const union = new Set([...GLOBAL_PHASES, ...NON_GLOBAL_PHASES]);
expect(union.size).toBe(ALL_PHASES.length);
for (const p of ALL_PHASES) expect(union.has(p)).toBe(true);
// No phase in both.
const overlap = GLOBAL_PHASES.filter((p) => NON_GLOBAL_PHASES.includes(p));
expect(overlap).toEqual([]);
});
test('every GLOBAL phase is PHASE_SCOPE==="global"; embed is global, lint is not', () => {
for (const p of GLOBAL_PHASES) expect(PHASE_SCOPE[p]).toBe('global');
expect(GLOBAL_PHASES).toContain('embed');
expect(GLOBAL_PHASES).toContain('orphans');
expect(GLOBAL_PHASES).toContain('purge');
expect(NON_GLOBAL_PHASES).toContain('lint');
expect(NON_GLOBAL_PHASES).toContain('sync');
expect(NON_GLOBAL_PHASES).not.toContain('embed');
});
});
describe('isGlobalMaintenanceStale', () => {
const now = Date.UTC(2026, 5, 16, 12, 0, 0);
test('null/unparseable → stale (must run)', () => {
expect(isGlobalMaintenanceStale(null, now)).toBe(true);
expect(isGlobalMaintenanceStale('not-a-date', now)).toBe(true);
});
test('older than floor → stale; within floor → fresh', () => {
expect(isGlobalMaintenanceStale(new Date(now - 61 * 60_000).toISOString(), now, 60)).toBe(true);
expect(isGlobalMaintenanceStale(new Date(now - 10 * 60_000).toISOString(), now, 60)).toBe(false);
});
});
describe('dispatchGlobalMaintenance — single-flight gate', () => {
function stubs(lastGlobalAt: string | null) {
const added: Array<{ name: string; data: any; opts: any }> = [];
const engine = {
kind: 'postgres' as const,
getConfig: async (k: string) => (k === LAST_GLOBAL_AT_KEY ? lastGlobalAt : null),
} as unknown as BrainEngine;
const queue = {
add: async (name: string, data: unknown, opts: Record<string, unknown>) => {
added.push({ name, data, opts }); return { id: 1 };
},
} as any;
return { engine, queue, added };
}
test('stale (never run) → dispatches one global job with single-flight opts', async () => {
const { engine, queue, added } = stubs(null);
const r = await dispatchGlobalMaintenance(engine, queue, { repoPath: '/tmp', slot: 's1', timeoutMs: 1, jsonMode: true, emit: () => {} });
expect(r.dispatched).toBe(true);
expect(added.length).toBe(1);
expect(added[0].name).toBe('autopilot-global-maintenance');
expect(added[0].opts.idempotency_key).toBe('autopilot-global:s1');
expect(added[0].opts.maxWaiting).toBe(1); // structural single-flight
expect(added[0].data.phases).toEqual(GLOBAL_PHASES);
});
test('fresh → does NOT dispatch', async () => {
const { engine, queue, added } = stubs(new Date().toISOString());
const r = await dispatchGlobalMaintenance(engine, queue, { repoPath: '/tmp', slot: 's1', timeoutMs: 1, jsonMode: true, emit: () => {} });
expect(r.dispatched).toBe(false);
expect(added.length).toBe(0);
});
});
describe('dispatchPerSource — per-source jobs carry NON_GLOBAL phases (no embed)', () => {
test('each per-source job sets phases = NON_GLOBAL_PHASES', async () => {
const sources = [{ id: 'repo-a', name: 'a', config: {} }, { id: 'repo-b', name: 'b', config: {} }];
const added: any[] = [];
const engine = {
kind: 'postgres' as const,
listAllSources: async () => sources,
getConfig: async () => null,
executeRaw: async () => [],
} as unknown as BrainEngine;
const queue = { add: async (name: string, data: unknown, opts: unknown) => { added.push({ name, data, opts }); return { id: added.length }; } } as any;
await dispatchPerSource(engine, queue, { repoPath: '/tmp', slot: 's', timeoutMs: 1, fanoutMax: 4, jsonMode: true, emit: () => {}, log: () => {} });
expect(added.length).toBe(2);
for (const j of added) {
expect(j.data.phases).toEqual(NON_GLOBAL_PHASES);
expect(j.data.phases).not.toContain('embed');
}
});
});
describe('autopilot-global-maintenance handler stamps last_global_at (PGLite)', () => {
let engine: PGLiteEngine;
beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); }, 30000);
afterAll(async () => { await engine.disconnect(); });
beforeEach(async () => { await resetPgliteState(engine); });
async function captureHandlers() {
const handlers = new Map<string, (job: any) => Promise<any>>();
const fakeWorker = { register(name: string, fn: (job: any) => Promise<any>) { handlers.set(name, fn); } };
await registerBuiltinHandlers(fakeWorker as never, engine);
return handlers;
}
test('runs global phases (no source_id) and stamps autopilot.last_global_at on success', async () => {
expect(await engine.getConfig(LAST_GLOBAL_AT_KEY)).toBeNull();
const handlers = await captureHandlers();
const handler = handlers.get('autopilot-global-maintenance');
expect(handler).toBeTruthy();
const result = await handler!({ data: { phases: ['orphans', 'embed'] }, signal: undefined });
// The cycle ran the requested global phases (DB-only on an empty brain).
expect(result.report.phases.some((p: any) => p.phase === 'orphans')).toBe(true);
expect(['ok', 'clean', 'partial']).toContain(result.report.status);
// Freshness stamped so the dispatch gate backs off.
const stamped = await engine.getConfig(LAST_GLOBAL_AT_KEY);
expect(stamped).not.toBeNull();
expect(Number.isFinite(new Date(stamped!).getTime())).toBe(true);
});
});
+83
View File
@@ -0,0 +1,83 @@
/**
* Self-test for scripts/check-jsonb-params.mjs the positional jsonb
* double-encode guard (#2339 / #2324 class). Verifies it catches the bug shape
* (including generic-typed calls and the `jsonStr` variable case is acknowledged
* as out of scope) and does NOT false-positive on the sanctioned forms.
*
* Fixtures are written to a temp dir and the scanner is pointed at it via argv,
* so this never touches the real src/ tree.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
const SCRIPT = join(import.meta.dir, '..', 'scripts', 'check-jsonb-params.mjs');
let root: string;
let badDir: string;
let goodDir: string;
function runGuard(dir: string): { code: number; err: string } {
const res = Bun.spawnSync([process.execPath, SCRIPT, dir]);
return { code: res.exitCode, err: res.stderr.toString() + res.stdout.toString() };
}
beforeAll(() => {
root = mkdtempSync(join(tmpdir(), 'jsonb-guard-'));
badDir = join(root, 'bad');
goodDir = join(root, 'good');
mkdirSync(badDir, { recursive: true });
mkdirSync(goodDir, { recursive: true });
// BAD: positional $1::jsonb bound to a JSON.stringify'd value.
writeFileSync(
join(badDir, 'bad.ts'),
"await engine.executeRaw(`INSERT INTO t (a) VALUES ($1::jsonb)`, [JSON.stringify(x)]);\n",
);
// BAD: generic-typed executeRaw<T>(...) must still be caught.
writeFileSync(
join(badDir, 'bad_generic.ts'),
"await engine.executeRaw<{ id: string }>(`UPDATE t SET a = $2::jsonb WHERE id = $1`, [id, JSON.stringify(x)]);\n",
);
// GOOD: the fix — $1::text::jsonb.
writeFileSync(
join(goodDir, 'good_text.ts'),
"await engine.executeRaw(`INSERT INTO t (a) VALUES ($1::text::jsonb)`, [JSON.stringify(x)]);\n",
);
// GOOD: text[] array path (the appendCompleted unnest shape).
writeFileSync(
join(goodDir, 'good_array.ts'),
"await engine.executeRaw(`INSERT INTO t (a) SELECT unnest($1::text[])`, [JSON.stringify(arr)]);\n",
);
// GOOD: executeRawJsonb passes a raw object, not a string — excluded.
writeFileSync(
join(goodDir, 'good_helper.ts'),
"await executeRawJsonb(engine, `INSERT INTO t (a) VALUES ($1::jsonb)`, [], [JSON.stringify(x)]);\n",
);
// GOOD: explicit opt-out for a rare legitimate object-binding case.
writeFileSync(
join(goodDir, 'good_optout.ts'),
"await engine.executeRaw(`INSERT INTO t (a) VALUES ($1::jsonb)` /* jsonb-guard-ok */, [JSON.stringify(x)]);\n",
);
});
afterAll(() => {
rmSync(root, { recursive: true, force: true });
});
describe('check-jsonb-params guard', () => {
test('flags positional $N::jsonb + JSON.stringify (incl. generic-typed calls)', () => {
const { code, err } = runGuard(badDir);
expect(code).toBe(1);
expect(err).toContain('bad.ts');
expect(err).toContain('bad_generic.ts');
});
test('passes the sanctioned forms (::text::jsonb, ::text[], executeRawJsonb, opt-out)', () => {
const { code, err } = runGuard(goodDir);
expect(code).toBe(0);
expect(err).toContain('clean');
});
});
+67 -1
View File
@@ -50,6 +50,7 @@ async function runUntilTerminal(
h: Harness,
overrides: Partial<{
maxCrashes: number;
hardStopMaxCrashes: number;
_backoffFloorMs: number;
cleanRestartBudget: number;
cleanRestartWindowMs: number;
@@ -71,6 +72,7 @@ async function runUntilTerminal(
cliPath: h.workerScript,
args: [],
maxCrashes: overrides.maxCrashes ?? 3,
hardStopMaxCrashes: overrides.hardStopMaxCrashes,
_backoffFloorMs: overrides._backoffFloorMs ?? 5,
cleanRestartBudget: overrides.cleanRestartBudget,
cleanRestartWindowMs: overrides.cleanRestartWindowMs,
@@ -159,12 +161,15 @@ if [ $((NEXT % 2)) -eq 1 ]; then exit 1; else exit 0; fi
try {
const res = await runUntilTerminal(h, {
maxCrashes: 3,
// issue #1994: the soft budget no longer gives up; pin the hard
// ceiling to 3 so this counting test still fires give-up at 3.
hardStopMaxCrashes: 3,
_backoffFloorMs: 5,
stopAfterEvents: 200,
});
expect(res.maxCrashesFired).not.toBeNull();
// 3 code!=0 exits → max_crashes=3
// 3 code!=0 exits → hard ceiling=3
expect(res.maxCrashesFired!.count).toBe(3);
const exits = res.events.filter((e) => e.kind === 'worker_exited');
@@ -235,6 +240,7 @@ esac
const res = await runUntilTerminal(h, {
maxCrashes: 3,
hardStopMaxCrashes: 3, // issue #1994: pin give-up to 3 for this counting test
_backoffFloorMs: 5,
_now: fakeNow,
stopAfterEvents: 200,
@@ -471,6 +477,66 @@ esac
});
});
// issue #1994 (#2227 tail): crossing the SOFT crash budget no longer
// permanently gives up. The supervisor enters degraded mode (capped backoff
// + loud warn) so a transient outage self-heals; permanent give-up fires only
// at the much-higher hard ceiling.
describe('degraded-mode crash backoff (issue #1994)', () => {
it('crossing the soft budget does NOT give up; it warns and keeps retrying to the hard ceiling', async () => {
const h = makeHarness('degraded-softbudget', 'exit 1');
try {
const { events, maxCrashesFired } = await runUntilTerminal(h, {
maxCrashes: 3, // soft budget
hardStopMaxCrashes: 6, // hard ceiling
_backoffFloorMs: 1,
stopAfterEvents: 200,
});
// Permanent give-up fired at the HARD ceiling (6), not the soft budget (3).
expect(maxCrashesFired).not.toBeNull();
expect(maxCrashesFired!.count).toBe(6);
expect(maxCrashesFired!.max).toBe(6);
// The soft-budget crossing announced degraded mode (at least once).
const degraded = events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'health_warn' }> =>
e.kind === 'health_warn' && e.reason === 'crash_budget_degraded',
);
expect(degraded.length).toBeGreaterThanOrEqual(1);
expect(degraded[0].max).toBe(3);
expect(degraded[0].count).toBeGreaterThanOrEqual(3);
// It kept respawning past the soft budget (more than 3 crash exits).
const crashes = events.filter(
(e): e is Extract<ChildSupervisorEvent, { kind: 'worker_exited' }> =>
e.kind === 'worker_exited' && e.code === 1,
);
expect(crashes.length).toBe(6);
} finally {
h.cleanup();
}
});
it('hardStopMaxCrashes=0 disables permanent give-up (retry-forever-with-backoff)', async () => {
const h = makeHarness('degraded-noforever', 'exit 1');
try {
const { events, maxCrashesFired } = await runUntilTerminal(h, {
maxCrashes: 3,
hardStopMaxCrashes: 0, // never permanently stop
_backoffFloorMs: 1,
stopAfterEvents: 40, // the safety net stops the test, not a give-up
});
// Never gave up despite many crashes past the soft budget.
expect(maxCrashesFired).toBeNull();
const crashes = events.filter(
(e) => e.kind === 'worker_exited' && (e as any).code === 1,
);
expect(crashes.length).toBeGreaterThan(3);
} finally {
h.cleanup();
}
});
});
describe('issue #1801 — restartCurrentChild + killChild liveness fix', () => {
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
// ESRCH = no such process (dead). EPERM = process exists but we can't
+38
View File
@@ -21,6 +21,8 @@ import {
inspectLock,
listStaleLocks,
deleteLockRow,
liveSyncStatus,
syncLockId,
} from '../src/core/db-lock.ts';
let engine: PGLiteEngine;
@@ -183,3 +185,39 @@ describe('deleteLockRow', () => {
await handle!.release();
});
});
describe('liveSyncStatus (#1950)', () => {
test('returns null when the source holds no lock (idle)', async () => {
const live = await liveSyncStatus(engine, 'never-synced');
expect(live).toBeNull();
});
test('returns the holder when a live (non-expired) sync lock is held', async () => {
const handle = await tryAcquireDbLock(engine, syncLockId('busy-source'));
expect(handle).not.toBeNull();
const live = await liveSyncStatus(engine, 'busy-source');
expect(live).not.toBeNull();
expect(live!.holder_pid).toBe(process.pid);
expect(typeof live!.holder_host).toBe('string');
expect(live!.holder_host.length).toBeGreaterThan(0);
await handle!.release();
});
test('returns null for a stale (TTL-expired) lock — structurally available, not running', async () => {
await (engine as any).db.query(
`INSERT INTO gbrain_cycle_locks (id, holder_pid, holder_host, acquired_at, ttl_expires_at)
VALUES ($1, $2, $3, NOW() - INTERVAL '2 hours', NOW() - INTERVAL '1 hour')`,
[syncLockId('wedged-dead'), 31337, 'old-host'],
);
const live = await liveSyncStatus(engine, 'wedged-dead');
expect(live).toBeNull();
});
test('is scoped per source — one source live does not mark another running', async () => {
const handle = await tryAcquireDbLock(engine, syncLockId('source-a'));
expect(handle).not.toBeNull();
expect(await liveSyncStatus(engine, 'source-a')).not.toBeNull();
expect(await liveSyncStatus(engine, 'source-b')).toBeNull();
await handle!.release();
});
});
@@ -0,0 +1,84 @@
/**
* #2194 fix #5: `gbrain doctor` warns when autopilot's per-tick fan-out exceeds
* the worker's effective concurrency. Fanning out more cycles than there are
* worker slots guarantees waiters that race the stalled-sweeper a silent
* misconfig the operator never saw before this check.
*
* Drives computeAutopilotFanoutConcurrencyCheck directly with a fake engine so
* the fan-out (config) and concurrency (audit) inputs are controllable without
* spawning a supervisor. The audit read is stubbed via GBRAIN_AUDIT_DIR + a
* hand-written started event.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { computeAutopilotFanoutConcurrencyCheck } from '../src/commands/doctor.ts';
import { computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts';
// Minimal fake engine: postgres-kind + a config map for fanout override.
function fakeEngine(config: Record<string, string> = {}) {
return {
kind: 'postgres' as const,
getConfig: async (k: string) => config[k] ?? null,
} as any;
}
let auditDir: string;
const prevAuditDir = process.env.GBRAIN_AUDIT_DIR;
beforeEach(() => {
auditDir = mkdtempSync(join(tmpdir(), 'gbrain-fanout-doctor-'));
process.env.GBRAIN_AUDIT_DIR = auditDir;
});
afterEach(() => {
if (prevAuditDir === undefined) delete process.env.GBRAIN_AUDIT_DIR;
else process.env.GBRAIN_AUDIT_DIR = prevAuditDir;
try { rmSync(auditDir, { recursive: true, force: true }); } catch { /* noop */ }
});
/** Write a `started` audit event with the given concurrency for queue 'default'. */
function writeStarted(concurrency: number): void {
const file = join(auditDir, computeSupervisorAuditFilename());
const line = JSON.stringify({
event: 'started',
ts: new Date().toISOString(),
supervisor_pid: 4242,
queue: 'default',
concurrency,
});
writeFileSync(file, line + '\n', 'utf8');
}
describe('computeAutopilotFanoutConcurrencyCheck (#2194 fix #5)', () => {
test('warns when fan-out (4) exceeds effective slots (concurrency 2 → 1)', async () => {
writeStarted(2);
const check = await computeAutopilotFanoutConcurrencyCheck(fakeEngine());
expect(check.status).toBe('warn');
expect(check.message).toContain('exceeds worker concurrency');
expect(check.details).toMatchObject({ fanout_max: 4, concurrency: 2, effective_slots: 1 });
});
test('ok when fan-out fits (override 1, concurrency 4)', async () => {
writeStarted(4);
const check = await computeAutopilotFanoutConcurrencyCheck(
fakeEngine({ 'autopilot.fanout_max_per_tick': '1' }),
);
expect(check.status).toBe('ok');
});
test('ok/skip when no supervisor has ever started (no noise on unsupervised brains)', async () => {
// No started event written.
const check = await computeAutopilotFanoutConcurrencyCheck(fakeEngine());
expect(check.status).toBe('ok');
expect(check.message).toContain('No supervisor observed');
});
test('PGLite short-circuits (single-writer, fan-out is 1)', async () => {
const check = await computeAutopilotFanoutConcurrencyCheck({ kind: 'pglite', getConfig: async () => null } as any);
expect(check.status).toBe('ok');
expect(check.message).toContain('PGLite');
});
});
@@ -0,0 +1,62 @@
/**
* #2194 fix #2 engine-parity for the failure-cooldown query.
*
* readRecentSourceFailures runs ONE SQL through engine.executeRaw (the
* engine-agnostic path), so PGLite and Postgres must return identical
* groupings. This pins that: it seeds the SAME dead/failed autopilot-cycle rows
* into both engines and asserts the per-source counts + the null-source
* exclusion (codex #6) match. PGLite always runs; Postgres runs only when
* DATABASE_URL is set (mirrors engine-parity.test.ts's gating).
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { PGLiteEngine } from '../../src/core/pglite-engine.ts';
import type { BrainEngine } from '../../src/core/engine.ts';
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
import { readRecentSourceFailures } from '../../src/commands/autopilot-fanout.ts';
const SKIP_PG = !hasDatabase();
async function seed(engine: BrainEngine): Promise<void> {
const rows: Array<[string, Record<string, unknown>, number]> = [
['dead', { source_id: 'repo-a' }, 5],
['failed', { source_id: 'repo-a' }, 2],
['dead', { source_id: 'repo-b' }, 10],
['completed', { source_id: 'repo-a' }, 1], // excluded (not a failure)
['dead', {}, 3], // excluded (null source_id, codex #6)
];
for (const [status, data, minAgo] of rows) {
await engine.executeRaw(
`INSERT INTO minion_jobs (name, status, data, finished_at) VALUES ('autopilot-cycle', $1, $2, $3)`,
[status, data, new Date(Date.now() - minAgo * 60_000).toISOString()],
);
}
}
function summarize(map: Map<string, { count: number }>): Record<string, number> {
const out: Record<string, number> = {};
for (const [k, v] of map) out[k] = v.count;
return out;
}
describe('failure-cooldown query — PGLite', () => {
let engine: PGLiteEngine;
beforeAll(async () => { engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); await seed(engine); }, 30000);
afterAll(async () => { await engine.disconnect(); });
test('groups failures by source, excludes completed + null-source', async () => {
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
expect(summarize(map)).toEqual({ 'repo-a': 2, 'repo-b': 1 });
});
});
(SKIP_PG ? describe.skip : describe)('failure-cooldown query — Postgres parity', () => {
let engine: BrainEngine;
beforeAll(async () => { await setupDB(); engine = await getEngine(); await seed(engine); }, 60000);
afterAll(async () => { await teardownDB(); });
test('Postgres returns the SAME groupings as PGLite', async () => {
const map = await readRecentSourceFailures(engine, { sinceMin: 120 });
expect(summarize(map)).toEqual({ 'repo-a': 2, 'repo-b': 1 });
});
});
+4
View File
@@ -138,6 +138,10 @@ describeE2E('E2E: Minions resilience (OpenClaw real-world patterns)', () => {
const final = await queue.getJob(job.id);
expect(final?.error_text).toMatch(/timeout exceeded/i);
// #1737 regression: the runaway run is counted as a spent attempt by
// handleTimeouts (it's the first killer to fire), so accounting reads
// honestly instead of `attempts: 0/1 (started: 1)`.
expect(final?.attempts_made).toBe(1);
} finally {
await a.disconnect();
await b.disconnect();
@@ -0,0 +1,80 @@
/**
* E2E: op_checkpoints.completed_keys JSONB parity #2339 regression guard.
*
* #2339: `recordCompleted` bound `JSON.stringify(array)` to a `$3::jsonb` param
* via postgres.js `.unsafe()` (executeRawDirect). That double-encodes the value
* into a jsonb *string scalar*, which violates the v119
* `op_checkpoints_completed_keys_array CHECK (jsonb_typeof = 'array')` and aborts
* EVERY sync on real Postgres at the first checkpoint write. PGLite's driver
* parses the string silently, which is exactly why the unit suite stayed green
* and the bug shipped so this assertion can ONLY be made on real Postgres.
*
* This file uses the standard `hasDatabase()` skip gate (consistent with the
* other e2e tests). The X2-A guarantee that it actually RUNS lives in a dedicated
* CI job (.github/workflows) that provisions a Postgres service so DATABASE_URL
* is always present there rather than a fail-on-skip hack inside this file,
* which would red-fail legitimate DB-less local runs.
*
* Fix under test: `$3::text::jsonb` binds the value as text, so the textjsonb
* cast parses it into a genuine jsonb array.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
import { recordCompleted, loadOpCheckpoint } from '../../src/core/op-checkpoint.ts';
const describeE2E = hasDatabase() ? describe : describe.skip;
describeE2E('E2E: op_checkpoints completed_keys jsonb parity (#2339)', () => {
beforeAll(async () => {
await setupDB();
});
afterAll(async () => {
await teardownDB();
});
const key = { op: 'sync-target', fingerprint: 'jsonb-parity-2339' };
test('recordCompleted stores completed_keys as a jsonb ARRAY, not a double-encoded scalar', async () => {
const engine = getEngine();
// Pre-fix this throws SQLSTATE 23514 (the CHECK), durableWrite exhausts its
// retries, and recordCompleted returns false. Post-fix it stores a real array.
const ok = await recordCompleted(engine, key, ['b/2.md', 'a/1.md', 'c/3.md']);
expect(ok).toBe(true);
const sql = getConn();
const [row] = await sql`
SELECT jsonb_typeof(completed_keys) AS t,
jsonb_array_length(completed_keys) AS len,
completed_keys ->> 0 AS first
FROM op_checkpoints
WHERE op = ${key.op} AND fingerprint = ${key.fingerprint}
`;
expect(row.t).toBe('array'); // pre-fix: 'string' (scalar) — the bug
expect(Number(row.len)).toBe(3);
expect(row.first).toBe('a/1.md'); // recordCompleted sorts the set
}, 30_000);
test('loadOpCheckpoint round-trips the recorded set', async () => {
const engine = getEngine();
const got = await loadOpCheckpoint(engine, key);
expect(new Set(got)).toEqual(new Set(['a/1.md', 'b/2.md', 'c/3.md']));
}, 30_000);
test('REPLACE semantics: re-recording a smaller set drops stale keys (stays an array)', async () => {
const engine = getEngine();
const ok = await recordCompleted(engine, key, ['only/1.md']);
expect(ok).toBe(true);
const got = await loadOpCheckpoint(engine, key);
expect(new Set(got)).toEqual(new Set(['only/1.md']));
const sql = getConn();
const [row] = await sql`
SELECT jsonb_typeof(completed_keys) AS t
FROM op_checkpoints
WHERE op = ${key.op} AND fingerprint = ${key.fingerprint}
`;
expect(row.t).toBe('array');
}, 30_000);
});
+5
View File
@@ -82,6 +82,11 @@ describe('gbrain status E2E (PGLite)', () => {
expect(parsed).toHaveProperty('workers');
expect(parsed).toHaveProperty('queue');
expect(parsed).toHaveProperty('autopilot');
// #1984: version field present (local CLI version) for poller pinning;
// a normal (un-budgeted) run is never partial.
expect(typeof parsed.version).toBe('string');
expect(parsed.version.length).toBeGreaterThan(0);
expect(parsed.partial).toBeUndefined();
});
test('dual cycle rows: last_full + last_targeted surface independently', async () => {
+2
View File
@@ -64,6 +64,8 @@ describe('get_status_snapshot handler shape', () => {
expect(result.schema_version).toBe(1);
expect(result).toHaveProperty('sync');
expect(result).toHaveProperty('cycle');
// #1984: the op now also reports the brain server's version (thin-client parity).
expect(typeof result.version).toBe('string');
expect(result).not.toHaveProperty('locks');
expect(result).not.toHaveProperty('workers');
expect(result).not.toHaveProperty('queue');
+6
View File
@@ -1300,6 +1300,12 @@ describe('MinionQueue: handleTimeouts', () => {
const dead = await queue.getJob(job.id);
expect(dead!.status).toBe('dead');
expect(dead!.error_text).toBe('timeout exceeded');
// #1737 regression: the timed-out run counts as a spent attempt, mirroring
// the wall-clock + stall dead-letter paths. Without this the job reads
// `attempts: 0/N (started: N)`. Asserted on both the RETURNING row and the
// persisted row so a future refactor can't silently drop the increment.
expect(timedOut[0].attempts_made).toBe(1);
expect(dead!.attempts_made).toBe(1);
});
test('handleTimeouts ignores stalled jobs (lock_until > now guard)', async () => {
+69 -1
View File
@@ -14,7 +14,65 @@
*/
import { describe, test, expect } from 'bun:test';
import { parseSectionFlag, runStatus } from '../src/commands/status.ts';
import {
parseSectionFlag,
parseDeadlineFlag,
withSectionDeadline,
runStatus,
FAST_DEADLINE_MS,
} from '../src/commands/status.ts';
describe('parseDeadlineFlag (#1984)', () => {
test('no flag → undefined (no budget)', () => {
expect(parseDeadlineFlag([])).toBeUndefined();
expect(parseDeadlineFlag(['--json'])).toBeUndefined();
});
test('--fast applies the preset budget', () => {
expect(parseDeadlineFlag(['--fast'])).toBe(FAST_DEADLINE_MS);
});
test('--deadline-ms in both forms', () => {
expect(parseDeadlineFlag(['--deadline-ms', '500'])).toBe(500);
expect(parseDeadlineFlag(['--deadline-ms=750'])).toBe(750);
});
test('explicit --deadline-ms wins over --fast', () => {
expect(parseDeadlineFlag(['--fast', '--deadline-ms=100'])).toBe(100);
});
test('non-positive / non-numeric → usage_error', () => {
expect(parseDeadlineFlag(['--deadline-ms', '0'])).toBe('usage_error');
expect(parseDeadlineFlag(['--deadline-ms', '-5'])).toBe('usage_error');
expect(parseDeadlineFlag(['--deadline-ms', 'soon'])).toBe('usage_error');
});
test('bare --deadline-ms with no value → usage_error (not a silent no-budget fallthrough)', () => {
expect(parseDeadlineFlag(['--deadline-ms'])).toBe('usage_error');
expect(parseDeadlineFlag(['--fast', '--deadline-ms'])).toBe('usage_error');
});
});
describe('withSectionDeadline (#1984)', () => {
test('resolves the value when it beats the budget', async () => {
let timedOut = false;
const v = await withSectionDeadline(Promise.resolve(42), 1000, () => { timedOut = true; });
expect(v).toBe(42);
expect(timedOut).toBe(false);
});
test('returns undefined + fires onTimeout when the budget elapses', async () => {
let timedOut = false;
const v = await withSectionDeadline(new Promise<number>(() => {}), 10, () => { timedOut = true; });
expect(v).toBeUndefined();
expect(timedOut).toBe(true);
});
test('no budget (undefined/<=0) awaits the promise as-is', async () => {
expect(await withSectionDeadline(Promise.resolve('x'), undefined, () => {})).toBe('x');
expect(await withSectionDeadline(Promise.resolve('y'), 0, () => {})).toBe('y');
});
});
describe('parseSectionFlag', () => {
test('no --section flag → undefined (all sections)', () => {
@@ -73,4 +131,14 @@ describe('runStatus exit codes', () => {
expect(r.exitCode).toBe(1);
expect(captured).toMatch(/snapshot failed|no engine connected/);
});
test('#1984: invalid --deadline-ms → exit 2 (usage error)', async () => {
let captured = '';
const r = await runStatus(null, ['--deadline-ms', '0'], {
stdout: () => {},
stderr: (s: string) => { captured += s; },
});
expect(r.exitCode).toBe(2);
expect(captured).toContain('--deadline-ms');
});
});
+42
View File
@@ -199,3 +199,45 @@ describe('summarizeCrashes — aggregation', () => {
expect(summary.clean_exits).toBe(0);
});
});
// issue #2227 bug #1/#2: a duplicate supervisor (different $HOME / --pid-file)
// passes the pidfile guard but loses the queue-scoped DB singleton lock
// (#1849) and `process.exit(ExitCodes.LOCK_HELD)`s. That fence-exit happens in
// supervisor.ts:start() BEFORE the worker is ever spawned and BEFORE the
// `started` event is emitted (the DB-lock acquire at :528 precedes the
// `started` emit at :555), so the loser contributes NO `worker_exited` events
// to the audit trail. The crash ledger (summarizeCrashes) moves ONLY on
// `worker_exited`, so a fence-collision can never burn the crash budget the
// guard exists to protect. The field report claimed 20 "unknown" crashes were
// fence exits; this pins that the fence path is structurally uncountable, so a
// future refactor that (wrongly) logs a `worker_exited` on the LOCK_HELD path
// would fail here instead of silently re-introducing the breaker-trip loop.
describe('summarizeCrashes — LOCK_HELD fence-exit is never a crash (issue #2227)', () => {
test('a losing duplicate supervisor (no worker_exited) contributes zero crashes', () => {
// The loser exits at the DB-lock fence before emitting `started`, so its
// audit contribution is empty. Even paired with the winner's healthy
// stream, total crashes stay 0.
const winnerHealthy: SupervisorEmission[] = [
evt('started', { supervisor_pid: 4242, queue: 'default', concurrency: 3 }),
evt('worker_spawned', { worker_pid: 4243 }),
// ... no worker_exited yet (winner still running)
];
expect(summarizeCrashes(winnerHealthy).total).toBe(0);
});
test('started + stopped with no worker_exited (clean fence path) is zero crashes', () => {
// Defensive: even if a future change made the loser emit lifecycle events
// (started/shutting_down/stopped) around the LOCK_HELD exit, none of those
// are `worker_exited`, so the ledger must stay at 0. The only way this test
// fails is if someone logs a `worker_exited` on the fence path — which is
// exactly the regression #2227 fix #2 guards against.
const fenceStream: SupervisorEmission[] = [
evt('started', { supervisor_pid: 5252 }),
evt('shutting_down', { reason: 'LOCK_HELD' }),
evt('stopped', { exit_code: 2 }),
];
const summary = summarizeCrashes(fenceStream);
expect(summary.total).toBe(0);
expect(summary.clean_exits).toBe(0);
});
});
+62 -3
View File
@@ -14,9 +14,26 @@ import { existsSync, unlinkSync } from 'fs';
import { join } from 'path';
import { tmpdir } from 'os';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { tryAcquireDbLock } from '../src/core/db-lock.ts';
import { MinionSupervisor, ExitCodes, supervisorLockId, classifySupervisorSingleton } from '../src/core/minions/supervisor.ts';
import type { DbLockHandle } from '../src/core/db-lock.ts';
import { tryAcquireDbLock, inspectLock, isLockHolderLive } from '../src/core/db-lock.ts';
import { MinionSupervisor, ExitCodes, supervisorLockId, classifySupervisorSingleton, SUPERVISOR_LOCK_TTL_MIN } from '../src/core/minions/supervisor.ts';
import type { DbLockHandle, LockSnapshot } from '../src/core/db-lock.ts';
// Build a LockSnapshot fixture for the isLockHolderLive matrix. Only ttl_expired
// and ms_since_last_refresh are consulted; the rest are filled for shape.
function snap(over: Partial<LockSnapshot>): LockSnapshot {
return {
id: 'gbrain-supervisor:default',
holder_pid: 4242,
holder_host: 'box',
acquired_at: new Date(),
ttl_expires_at: new Date(),
age_ms: 1000,
ttl_expired: false,
last_refreshed_at: new Date(),
ms_since_last_refresh: 0,
...over,
};
}
let engine: PGLiteEngine;
@@ -147,6 +164,48 @@ describe('#1849 LOCK_HELD path does not strand the pidfile', () => {
});
describe('#2227 isLockHolderLive — PID-reuse-safe supervisor liveness', () => {
test('fresh TTL → live (the normal running case)', () => {
expect(isLockHolderLive(snap({ ttl_expired: false }), SUPERVISOR_LOCK_TTL_MIN)).toBe(true);
});
test('expired TTL but refreshed within the steal grace → live (starved-but-alive #1794)', () => {
// ttl lapsed but the holder heartbeat is recent → it is alive, just starved.
expect(isLockHolderLive(snap({ ttl_expired: true, ms_since_last_refresh: 5_000 }), SUPERVISOR_LOCK_TTL_MIN)).toBe(true);
});
test('expired TTL and stale heartbeat → dead (a gone supervisor stops refreshing)', () => {
expect(isLockHolderLive(snap({ ttl_expired: true, ms_since_last_refresh: 36_000_000 }), SUPERVISOR_LOCK_TTL_MIN)).toBe(false);
});
test('expired TTL and no heartbeat column → dead', () => {
expect(isLockHolderLive(snap({ ttl_expired: true, ms_since_last_refresh: null }), SUPERVISOR_LOCK_TTL_MIN)).toBe(false);
});
test('liveness NEVER consults process.kill (PID reuse cannot false-positive)', () => {
// A row whose holder_pid happens to be a live, unrelated process (PID reuse)
// but whose lock is stale must read as NOT live — proving freshness, not the
// PID probe, is the signal. holder_pid=1 (init, always alive) + expired/stale.
expect(isLockHolderLive(snap({ holder_pid: 1, ttl_expired: true, ms_since_last_refresh: 36_000_000 }), SUPERVISOR_LOCK_TTL_MIN)).toBe(false);
});
});
describe('#2227 status detects a live supervisor via the DB lock (split-$HOME)', () => {
test('a live queue lock with no local pidfile reads as running via inspectLock', async () => {
// Simulate the keeper holding the queue lock under a different $HOME: there
// is a live lock row but the local pidfile path is empty.
const holder = await tryAcquireDbLock(engine, supervisorLockId('default'), SUPERVISOR_LOCK_TTL_MIN);
expect(holder).not.toBeNull();
const live = await inspectLock(engine, supervisorLockId('default'));
expect(live).not.toBeNull();
expect(isLockHolderLive(live!, SUPERVISOR_LOCK_TTL_MIN)).toBe(true);
await holder!.release();
// After release the row is gone → not running.
const gone = await inspectLock(engine, supervisorLockId('default'));
expect(gone).toBeNull();
});
});
describe('#1849 refresh-failure fails safe (F1A)', () => {
test('exits LOCK_LOST after the failure threshold; tolerates a single blip', async () => {
const sup = new MinionSupervisor(engine, { cliPath: '/bin/sh', healthInterval: 0, json: true });
+38 -1
View File
@@ -4,7 +4,7 @@ import { spawn } from 'child_process';
import { join } from 'path';
import { tmpdir } from 'os';
import { readSupervisorEvents, computeSupervisorAuditFilename } from '../src/core/minions/handlers/supervisor-audit.ts';
import { calculateBackoffMs } from '../src/core/minions/supervisor.ts';
import { calculateBackoffMs, resolveHardStopMaxCrashes } from '../src/core/minions/supervisor.ts';
const TEST_PID_FILE = '/tmp/gbrain-supervisor-test.pid';
@@ -58,6 +58,16 @@ function spawnSupervisor(h: IntegrationHarness, overrides: Record<string, string
SUP_HEALTH_INTERVAL_MS: '999999', // effectively off
...overrides,
};
// issue #1994: the soft crash budget now DEGRADES (retry-with-backoff) rather
// than permanently giving up; permanent give-up fires at a much-higher hard
// ceiling (maxCrashes × 10). These integration tests assert the give-up
// LIFECYCLE (audit events, exit code, pidfile cleanup), so pin the hard
// ceiling to the soft budget by default — the degraded path is unit-tested in
// child-worker-supervisor.test.ts. Tests that want true degraded behavior
// pass GBRAIN_SUPERVISOR_HARD_STOP_CRASHES explicitly.
if (env.GBRAIN_SUPERVISOR_HARD_STOP_CRASHES === undefined) {
env.GBRAIN_SUPERVISOR_HARD_STOP_CRASHES = env.SUP_MAX_CRASHES;
}
const child = spawn('bun', [join(import.meta.dir, 'fixtures/supervisor-runner.ts')], {
env,
@@ -104,6 +114,31 @@ async function waitFor(pred: () => boolean, timeoutMs: number, tickMs = 20): Pro
}
describe('MinionSupervisor', () => {
describe('resolveHardStopMaxCrashes (issue #1994)', () => {
const KEY = 'GBRAIN_SUPERVISOR_HARD_STOP_CRASHES';
afterEach(() => { delete process.env[KEY]; });
it('defaults to maxCrashes × 10 when no override', () => {
delete process.env[KEY];
expect(resolveHardStopMaxCrashes(10)).toBe(100);
expect(resolveHardStopMaxCrashes(3)).toBe(30);
});
it('honors a valid non-negative integer override', () => {
process.env[KEY] = '0'; // 0 = disable permanent give-up
expect(resolveHardStopMaxCrashes(10)).toBe(0);
process.env[KEY] = '5';
expect(resolveHardStopMaxCrashes(10)).toBe(5);
});
it('ignores a negative or non-integer override (falls back to default)', () => {
process.env[KEY] = '-1';
expect(resolveHardStopMaxCrashes(10)).toBe(100);
process.env[KEY] = 'abc';
expect(resolveHardStopMaxCrashes(10)).toBe(100);
});
});
describe('calculateBackoffMs', () => {
it('returns ~1s for first crash', () => {
const backoff = calculateBackoffMs(0);
@@ -197,6 +232,8 @@ describe('MinionSupervisor', () => {
// hit max-crashes, then exit via shutdown() with code 1.
const h = makeHarness('max-crashes', 'exit 1');
try {
// hard ceiling defaults to SUP_MAX_CRASHES in the harness (see
// spawnSupervisor) so this give-up lifecycle still fires at 3 (#1994).
const sup = spawnSupervisor(h, { SUP_MAX_CRASHES: '3' });
const { code } = await sup.exited;
+22
View File
@@ -6,11 +6,33 @@ import { describe, test, expect } from 'bun:test';
import {
resolveSyncHardDeadline,
composeAbortSignals,
resolveStallAbortSeconds,
DEFAULT_SYNC_STALL_ABORT_SEC,
HARD_DEADLINE_GRACE_SEC,
} from '../src/commands/sync.ts';
const GRACE_MS = HARD_DEADLINE_GRACE_SEC * 1000;
describe('resolveStallAbortSeconds (#1950)', () => {
test('defaults to 900s when the env var is unset or empty', () => {
expect(resolveStallAbortSeconds({})).toBe(DEFAULT_SYNC_STALL_ABORT_SEC);
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: '' })).toBe(900);
});
test('honors a positive override', () => {
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: '120' })).toBe(120);
});
test('<=0 disables the watchdog (returned verbatim)', () => {
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: '0' })).toBe(0);
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: '-1' })).toBe(-1);
});
test('falls back to the default on a non-numeric value', () => {
expect(resolveStallAbortSeconds({ GBRAIN_SYNC_STALL_ABORT_SECONDS: 'nope' })).toBe(900);
});
});
describe('resolveSyncHardDeadline', () => {
test('--no-hard-deadline disables everything (even with --timeout)', () => {
const r = resolveSyncHardDeadline(['--source', 'x', '--timeout', '60', '--no-hard-deadline'], { isTty: false });