mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-31 04:07:52 +00:00
2d9f4ec243c19fbd4e2a43e95f13e18e0cb8d1e4
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
6a10bad8e5 |
v0.41.0.0 feat(minions): fleet you supervise (4 field bugs + cathedral) (#1367)
* v0.41: migration v93 — minions audit tables + budget columns Three new audit tables for the v0.41 minions cathedral (each with SET NULL FK so audit rows survive `gbrain jobs prune`, denormalized context columns so post-NULL rows still carry forensic value): - minion_lease_pressure_log — Bug 2 audit (one row per lease-full bounce) - minion_budget_log — D5 audit (reserve/refund/spent/halted) - minion_self_fix_log — E6 audit (classifier-gated auto-resubmit chain) Three new columns on minion_jobs: - budget_remaining_cents — D5 parent spendable balance - budget_owner_job_id — Eng D7 immutable budget owner (FK SET NULL) - budget_root_owner_id — Eng D10 denormalized historical owner (no FK) Eng D10 closes the codex-pass-3 #4 ambiguity bug: when the budget owner is pruned mid-batch, `budget_owner_job_id` becomes NULL via SET NULL, which is indistinguishable from "never had a budget." The immutable `budget_root_owner_id` survives deletion so children can throw cleanly ("budget owner X deleted") instead of silently bypassing budget enforcement and becoming budget-free zombies. Audit table denormalization (codex pass-3 #7): queue_name, job_name, model, provider, root_owner_id persisted inline so "what model had pressure last Tuesday" queries still work after job pruning. Both Postgres + PGLite parity. Indexed for the read patterns the doctor check + jobs stats consume. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.41: subagent hardening — Bug 1 + Bug 3 + Approach C composable prompt Three independent fixes to src/core/minions/handlers/subagent.ts. Each is covered by its own test set; bundled in one commit because they touch overlapping lines of subagent.ts (cleaner than 3 hunk-split commits). Bug 1 — rate-lease default 8 → 32 + `unlimited` sentinel src/core/minions/handlers/subagent.ts:61 Pre-v0.41 the default cap of 8 starved 10-concurrency batches on upstreams with no provider-side rate limit (Azure/Bedrock/self-hosted). New resolveLeaseCap() bumps default to 32, accepts `unlimited`/`none` as POSITIVE_INFINITY sentinel, throws on NaN/negative/zero with a paste-ready hint. Codex pass-1 #7 caught the original `=0`/`NaN`-uncapped semantics as dangerous (universal convention is "0 means disabled"). Pinned by test/rate-leases-uncapped.test.ts (15 cases). Bug 3 — strip `provider:` prefix at Anthropic SDK call site src/core/minions/handlers/subagent.ts:439, ~:895 `gbrain agent run --model anthropic:claude-sonnet-4-6` pre-fix sent the qualified string straight to client.messages.create which Anthropic rejects with "model not found." New stripProviderPrefix() applies at the one SDK call site; `model` stays qualified everywhere else (persistence, recipe lookup, capability gate). Pinned by 4 new test/subagent-handler.test.ts cases. Approach C — composable system prompt renderer w/ per-tool usage_hint src/core/minions/system-prompt.ts (NEW) src/core/minions/types.ts (ToolDef.usage_hint + SubagentHandlerData.system_no_tool_preamble) src/core/minions/tools/brain-allowlist.ts (BRAIN_TOOL_USAGE_HINTS) src/core/minions/handlers/subagent.ts (wiring) Bug 4 absorbed: pre-v0.41 DEFAULT_SYSTEM was one generic line that gave the model no guidance on WHICH tool to reach for. The field-report case was a `shell` tool sitting unused because nothing told the model to reach for it. New deterministic renderer splices a tool-usage preamble listing each tool's name + usage_hint; closing paragraph names shell/bash explicitly + tells the model brain tools write to the DB (not local files). Determinism preserved for Anthropic prompt-cache marker stability. Pinned by 13 cases in test/system-prompt.test.ts (determinism, opt-out, plugin tools, cache safety). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.41: Bug 2 — lease-full bypass that doesn't burn attempts The field-report dead-letter loop closed at the root. Pre-v0.41 the worker treated RateLeaseUnavailableError as a recoverable error AND incremented attempts_made. After 3 lease-full bounces the job hit max_attempts (default 3) and dead-lettered with message `rate lease "anthropic:messages" full (8/8)`. The operator who reported the bug submitted 100 jobs at --concurrency 10 with a default cap of 8; all 100 dead-lettered before the upstream had a chance to drain. Fix: MinionQueue.releaseLeaseFullJob(jobId, lockToken, errorText, backoffMs) Mirrors failJob() but skips the attempts_made increment. Same lock_token + status='active' idempotency guard as failJob; returns null on lock-token mismatch so racing stall sweeps / cancels still win. Worker catch block (src/core/minions/worker.ts:741-792) Detects `err instanceof RateLeaseUnavailableError` BEFORE the existing `isUnrecoverable || attemptsExhausted` gate. Routes through releaseLeaseFullJob with 1-3s jittered backoff. The handler comment at subagent.ts:425 ("treat as renewable error so the worker re-claims") is now actually true. src/core/minions/lease-pressure-audit.ts (NEW) Best-effort logLeasePressure() writes one row to migration v93's minion_lease_pressure_log per bounce. Denormalized context columns (queue_name, job_name, model, provider, root_owner_id) populated inline so post-prune forensic queries still see context (Eng D8 / codex pass-3 #7). Stderr-warn on write failure; never blocks the bypass path. Pinned by test/minions-lease-full-retry.test.ts (7 cases): - flips status to delayed without incrementing attempts_made - returns null on lock_token mismatch - 5 bounces leaves attempts_made=0; failJob comparison shows the asymmetry (failJob DOES bump) - logLeasePressure writes denormalized columns - countRecentLeasePressure for doctor + jobs stats consumers - audit row survives hard-delete via SET NULL FK - best-effort no-throw contract on write failure Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.41: doctor subagent_health + jobs stats lease_pressure line Operator visibility for the v0.41 Bug 2 audit data. src/commands/doctor.ts checkSubagentHealth(engine) — new exported check function. Reads the last 24h of minion_lease_pressure_log and classifies by bounce volume + forward progress: 0 bounces → ok 1-99 bounces → ok ("transient") 100+ bounces + subagent jobs completing → ok ("healthy backpressure") 100+ bounces + NO completed subagent jobs → warn (paste-ready hint) 1000+ bounces → fail (blocking) Warn/fail messages embed `export GBRAIN_ANTHROPIC_MAX_INFLIGHT=64` for copy-paste. Pre-v93 brains (no table) silently skip with OK. Works on both Postgres + PGLite. src/commands/jobs.ts (case 'stats') Adds `Lease pressure (1h)` line to the stats output. When >0 bounces, cross-checks completed subagent count and surfaces the same binding-but-healthy vs cap-too-tight distinction inline so operators don't have to run `gbrain doctor` to see it. Pre-v93 silent skip. test/doctor-subagent-health.test.ts (NEW) 4 cases pinning all threshold bands. Uses `allowProtectedSubmit: true` on the queue.add for `subagent`-named owner jobs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.41: Wave B — visibility cathedral (error clustering + jobs watch + cost cathedral) Five new modules + one SPA tab + one CLI command, all wired into the v0.41 audit substrate from migration v93. Each module is unit-tested in isolation; integration smoke tests live in the e2e suite. NEW MODULES: src/core/minions/error-classify.ts (D3 + E6 shared classifier) Conservative regex set classifying minion_jobs.last_error into stable buckets. Narrowed tool-error sub-types per codex pass-2 #4: only tool_schema_mismatch self-fixes; tool_crash + tool_unavailable + tool_permission stay visible. RECOVERABLE_CLUSTERS export gates E6 self-fix qualification. clusterErrors() groups + sorts for D3 surfaces. Pinned by 21 cases against real production error strings. src/core/minions/batch-projection.ts (D4 submit-time projection) Pure-function projectBatch() computes total cost + duration with ±30% band (or sample-stddev when historical). Cold-start fallback uses model-default per-token pricing + 5s mean latency guess; annotates "(no history; estimate is a wide guess)" so operators don't trust approximations. Unknown-model returns tagged variant so --budget-usd refuses to gate. Raise-cap hint fires when lease is binding AND a 4x raise meaningfully helps. Pinned by 16 cases. src/core/minions/budget-tracker.ts (D5 + Eng D7 + Eng D10) Reservation pattern that bounds overspend even under N parallel children of one owner. SQL UPDATE CAS WHERE budget_remaining_cents >= cost RETURNING balance; CAS miss → BudgetExhausted; on return → refundBudget unspent cents. Eng D10 NULL-bypass: jobs without an owner skip reservation cleanly. Eng D10 owner-deleted disambiguation: when budget_owner_job_id is NULL but budget_root_owner_id is set, the owner was pruned mid-batch; child throws BudgetOwnerDeleted instead of silently bypassing. haltBudgetSubtree() recursive halt walks budget_owner_job_id = X to flip the entire subtree to dead with reason. Pinned by 10 cases covering: reservation+refund, CAS miss, NULL bypass, owner-deleted throw, halt sweep, grandchild inheritance, active-job preservation. NEW SURFACES: src/commands/jobs-watch.ts + GET /admin/api/jobs/watch + JobsWatchPage Live TTY dashboard via readSnapshot() + renderSnapshot(). 1s refresh, ANSI-colored lease pressure by severity, top-5 clustered errors, budget owners panel. Non-TTY mode emits JSON snapshots per tick. Admin SPA tab consumes the same /admin/api/jobs/watch endpoint so TTY + browser dashboards stay 1:1. src/commands/jobs.ts — --cluster-errors flag on `gbrain jobs stats` Groups dead/failed jobs from last 24h by classifier bucket; surfaces top 5 with paste-ready `gbrain jobs get <id>` example. src/core/minions/types.ts — SubagentHandlerData additions no_self_fix (E6 per-job opt-out), is_self_fix_child (chain-depth marker), self_fix_cluster (audit metadata). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.41: Wave C — self-tuning fleet (E5 controller + E6 self-fix + shared election) The "magic layer" the wave promises: workers tune their own lease cap based on real upstream signals; failed jobs auto-heal one layer deep for known-recoverable failure modes. Both default ON for fresh installs + upgrades; off-switches per CLAUDE.md. src/core/db-lock.ts — tryWithDbElection convenience (Eng D9) Thin wrapper over the existing tryAcquireDbLock: acquires, runs fn, releases. For per-tick election use cases (controller tick chooses one writer per cluster). Codex pass-3 #8/#9 audit picked this shape over building a parallel new primitive — the existing gbrain_cycle_locks table works for both engines. src/core/minions/lease-cap-controller.ts (E5 reframed + Eng D6 correction) Auto-adapts the rate-lease cap based on bounce rate + upstream 429s + latency stability. CORRECTED control law per codex pass-2 #9: * Ramp DOWN only when upstream pushes back (429s OR latency unstable) * Ramp UP fast when workers starve (bounces > 1/min + no 429s) * Ramp UP slow on healthy headroom (util > 50% + 0 bounces + 0 429s) * Deadband otherwise My first draft had the bounce sign inverted; would have cratered cap during a healthy 100-job burst — exactly the field-report case. IRON- RULE regression test (test/lease-cap-controller.test.ts) pins the correct sign so future "let's simplify" PRs can't silently regress it. Per-tick election via tryWithDbElection — only ONE worker per cluster runs the WRITE side; all workers READ lease_cap_current fresh on every acquire. Asymmetric AIMD steps (rampDown=8, rampUp=4) — TCP congestion control wisdom. Latency signal sourced from subagent job durations in window; full upstream-SDK-latency tracking is v0.42. Pinned by 14 cases including the field-report scenario simulation ("starving workers get MORE capacity, not less"). src/core/minions/self-fix.ts (E6 with narrowed classifier per codex pass-2 #4) Classifier-gated auto-resubmit on terminal failures. ONLY three buckets qualify: prompt_too_long, tool_schema_mismatch, malformed_json. Explicitly NOT recoverable: tool_crash (real bug), tool_unavailable (config issue), tool_permission (needs human). Chain depth cap = 2 (D15 default); per-job opt-out via data.no_self_fix; global off-switch via config. buildSelfFixPrompt cluster-specific prep: prompt_too_long → truncate-with-leaf-preservation (v0.41 ships simple; semantic reduction in v0.42) tool_schema_mismatch → surface error verbatim + "check input_schema" malformed_json → "respond with JSON only — no prose, no fences" Children inherit budget owner from parent (Eng D7 + D10) but DO NOT copy remaining cents (codex pass-3 #5 caught the original plan's contradiction; only owner row holds spendable balance). Pinned by 16 cases. scripts/e5-lease-cap-ab.ts (D11 + codex pass-2 #7 spec) Manually-runnable A/B harness with committed receipt-fixture baseline. Spec: 500 jobs, log-normal prompt distribution, $8 budget per arm, synthetic 429 burst at minute 15, PR-gate verdict (controller must beat fixed-cap by ≥5% on throughput AND match within ±2% on cost efficiency). v0.41 ships the spec + dry-run + fixture shape; real-run dispatcher deferred to v0.41.1 (filed in TODOS). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.41.0.0 release — VERSION + CHANGELOG + TODOS + llms.txt regen Trio audit passes: VERSION: 0.41.0.0 package.json: 0.41.0.0 CHANGELOG: ## [0.41.0.0] - 2026-05-24 CHANGELOG entry written in ELI10-lead-first voice per CLAUDE.md voice rules. Lead with what the user gets (100-job batch now completes); itemized changes after; "To take advantage of v0.41.0.0" block at the end with paste-ready upgrade verification. TODOS.md updates filed via CEO D13 + D16 + Eng D9 + codex pass-1 #11: - v0.41+: per-key rate-lease caps (P2; deferred until gateway-default flip) - v0.41+: audit retention sweep in autopilot purge phase (P3) - v0.41.1: full E5 A/B dispatcher (currently dry-run only) - v0.41.1: tryWithDbElection retrofit of existing rate-leases + queue paths - v0.42: semantic-aware prompt_too_long reduction llms.txt + llms-full.txt regenerated to absorb the CHANGELOG entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.41 test gap-fills — 6 E2E suites covering every user flow Six new test/e2e/ files, 12 tests total, all passing inline against PGLite (no DATABASE_URL needed). Each pairs with a load-bearing claim in the v0.41 CHANGELOG so a future regression has somewhere to scream. minions-field-report-repro.test.ts THE BUG THIS WAVE FIXES. Submits 12 subagent jobs; stubbed handler bounces each twice then succeeds. Pre-v0.41 all 12 would dead-letter at attempt 3. Post-v0.41 all 12 complete with attempts_made=0 + 24 audit rows visible. minions-prefix-strip-smoke.test.ts Bug 3 end-to-end: stubbed MessagesClient records params.model; asserts the SDK call site receives 'claude-sonnet-4-6' (bare) when the job was submitted with 'anthropic:claude-sonnet-4-6' (qualified). minions-budget-cathedral.test.ts D5 enforcement under fan-out. Two scenarios: 1. Mid-batch budget exhaustion: 10 children of one budget-bearing parent; first 5 reserve, last 5 hit CAS miss, haltBudgetSubtree flips remaining 10 to dead (owner row preserved). 2. Parallel reservation cannot exceed budget: 8 concurrent reserves at 10c each on a 30c budget → exactly 3 succeed, 5 hit exhausted, owner balance stays 0 (NOT negative). minions-self-fix-flow.test.ts E6 classifier-gated retry. 4 scenarios pinning codex pass-2 #4: 1. prompt_too_long → child submitted with self-fix prompt + audit 2. tool_crash → NOT recoverable; no child submitted 3. no_self_fix opt-out bypasses recoverable cluster 4. Chain depth cap (default=2) refuses grandchild self-fix minions-controller-bounce-only.test.ts IRON-RULE REGRESSION for Eng D6 sign correction. 100 bounce events in audit, no 429s → controller MUST ramp cap UP (not down). 50 bounces + 10 dead jobs with 429-shaped errors → controller MUST ramp cap DOWN. If a future "simplify the rule" PR ever inverts the sign, this test screams. jobs-watch-readsnapshot.test.ts Engine-aggregation half of D2 (the renderer half lives in the unit suite). Verifies snapshot includes lease pressure, clustered errors, budget owners with cents. Total: 12 new E2E tests, all passing in 42s on PGLite. Plus the new unit tests already shipped in Waves A-C: ~120 unit tests total across 9 new test files. All pass; verify gate green; typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.41 follow-up: regen src/admin-embedded.ts + TS strict fixes + withEnv Three fixes the verify + admin-embed-serial-test gauntlet found: src/admin-embedded.ts AUTO-GENERATED file. v0.41 admin SPA build (T13) changed the hashed asset filename from index-DFgMZhBE.js to index-DqP-zmqH.js but the build-admin-embedded.ts generator wasn't re-run after `bun run build` in admin/. Result: src/admin-embedded.ts kept the old hash and `gbrain serve --http` failed to load the admin SPA with `Cannot find module '../admin/dist/assets/index-DFgMZhBE.js'`. Caught by test/admin-embed-spawn.serial.test.ts. Regenerated via `bun run scripts/build-admin-embedded.ts`. src/core/minions/self-fix.ts TS strict-mode fixes caught by `bun run typecheck`: - `rows` implicit-any → explicit Array<{...}> annotation. - childData typed as SubagentHandlerData & {...} → not assignable to Record<string, unknown> for queue.add's signature. Added narrow cast at the call site. test/batch-projection.test.ts check-test-isolation R1 violation: raw `process.env` mutation caught by the lint. Switched to `withEnv()` from test/helpers/with-env.ts (the canonical pattern per CLAUDE.md test-isolation rules). After: `bun run verify` green, `bun test test/admin-embed-spawn.serial.test.ts` 4/4 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(e2e): 4 root-cause fixes for pre-existing E2E flakes (master polish) After merging origin/master (which landed v0.40.8.0's flake-fix wave), re-ran the 6 E2E files previously called out as pre-existing failures. v0.40.8.0 had already fixed 3; the remaining 3 had real root causes: 1. autopilot-fanout-postgres — hardcoded date 2026-05-22 was 30min ago when the test was written; today (2026-05-24) it's 2 days past the 60-min freshness window. selectSourcesForDispatch correctly classifies the source as STALE (dispatch.length=1) instead of FRESH (length=0). Fix: replace literal date with Date.now() - 30 * 60 * 1000 so the timestamp stays relative-fresh forever. 2. ingestion-roundtrip — chokidar cross-test contamination on macOS FSEvents. Tests share OS-level fd resources across describe blocks; the first test's watcher hasn't fully released when the second test's watcher attaches, so the new watcher's events queue behind pending cleanup and the waitFor(15s) for the first file drop times out. Fixes: - Move fs.mkdirSync(inboxDir) BEFORE createInboxFolderSource + daemon.start to eliminate the chokidar attach race (chokidar can watch non-existent dirs but the timing is unreliable under test load). - Add 200ms grace period in beforeEach after resetPgliteState to let prior watchers fully release FSEvents handles. - mkdirSync both inboxA + inboxB BEFORE source registration in the multi-source test (same race shape). - Bump waitFor timeouts 6s → 15s for fs.watch flake tolerance. 3. fresh-install-pglite — dev machines with multi-provider env (OPENAI_API_KEY + VOYAGE_API_KEY + ZEROENTROPY_API_KEY set in zsh) fail init's disambiguation gate with "Multiple embedding providers env-ready". The test sets ZE_API_KEY but doesn't NEGATE the others. Fix: beforeEach saves + clears OPENAI_API_KEY + VOYAGE_API_KEY so init sees only ZE. afterEach restores. Hermetic per dev machine. 4. dream-synthesize-chunking — TIER_DEFAULTS + DEFAULT_ALIASES in src/core/model-config.ts had BARE Anthropic model ids (e.g. 'claude-sonnet-4-6' instead of 'anthropic:claude-sonnet-4-6'). The v0.40.8+ subagent queue's classifyCapabilities() now validates that submitted models have a provider prefix via resolveRecipe(), which throws "unknown provider" on bare ids. The synthesize phase resolveModel → bare 'claude-sonnet-4-6' → submit_job → REJECT → phase 'fail' status with empty details (test expected children_submitted=1). Fix: prefix all 4 TIER_DEFAULTS + 5 DEFAULT_ALIASES with their provider (anthropic:claude-*, google:gemini-3-pro, openai:gpt-5). Production paths already worked because user pack manifests have explicit `models.tier.subagent = anthropic:...`; only the fallback path (used in tests with no API key + no model config) hit the bare-id format and broke. Verification (all run against DATABASE_URL=...:5434/gbrain_test): test/e2e/autopilot-fanout-postgres.test.ts → 6/6 pass test/e2e/dream-cycle-phase-order-pglite.test.ts → 5/5 pass test/e2e/dream-synthesize-chunking.test.ts → 4/4 pass test/e2e/fresh-install-pglite.test.ts → 2/2 pass test/e2e/http-transport.test.ts → 8/8 pass test/e2e/ingestion-roundtrip.test.ts → 3/3 pass test/e2e/mechanical.test.ts → 78/78 pass Total: 106/106 pass, 0 fail. Adjacent unit tests verified green: test/anthropic-model-ids.test.ts → 6/6 pass test/model-config.serial.test.ts → 19/19 pass typecheck clean. Plan: v0.41 wave (~/.claude/plans/system-instruction-you-are-working-toasty-milner.md). Post-merge polish — every E2E failure surfaced in the v0.41 ship reports is now green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(test): isolate HOME in run-e2e.sh to stop config corruption Replaces #517 (re-ported fresh against current scripts/run-e2e.sh after v0.23.1 rewrote the script — original cherry-pick would not apply). E2E tests call setupDB which writes $HOME/.gbrain/config.json pointing at the docker test container. When the container tears down, the user's real autopilot daemon wedges trying to connect to a vanished postgres. Three operators hit this within 16 days before the original PR filed. Fix: wrapper exports HOME + GBRAIN_HOME to a mktemp tmpdir BEFORE bun starts so config writes land in the tmpdir, with a post-run breach detector that compares md5 of the user's real config against pre-run. Both env vars required: loadConfig/saveConfig resolve via HOME while configPath honors GBRAIN_HOME. HOME set before bun starts because os.homedir() caches at first call. Test seam: test/gbrain-home-isolation.test.ts updated to assert against homedir() === configDir() when GBRAIN_HOME unset (correct under the safety wrapper itself) instead of the prior "not /tmp/" sentinel. Revert path: git revert <this-sha> if test:e2e regresses on master. Co-Authored-By: orendi84 <orendi84@users.noreply.github.com> * fix(engines): silence pg NOTICEs + redirect migration progress to stderr Two changes that share a single root cause — stdout pollution breaking JSON-parsing callers like `gbrain jobs submit --json | jq` and the `zombie-reaping.test.ts` execSync flow. 1. **postgres NOTICE silencing.** postgres.js's default `onnotice` calls `console.log(notice)`, which flooded stdout with `{severity:"NOTICE", message:"relation already exists, skipping"}` objects under idempotent `CREATE INDEX IF NOT EXISTS` migrations + `initSchema`. Silenced by default in both `src/core/db.ts` (singleton) and `src/core/postgres-engine.ts` (instance pools). Opt back in with `GBRAIN_PG_NOTICES=1`. 2. **Migration progress to stderr.** `console.log` calls in `src/core/migrate.ts` (`Schema version N → M`, `[N] name...`, `[N] ✓ name`) and the wrappers in both engines (`N migration(s) applied`, `Schema verify: ...`, `HNSW sweep: ...`, `Pre-v0.21 brain detected`) now route to `process.stderr.write`. Progress messages were never the program's data output; they belong on stderr. Closes the cross-test flake class where any test invoking `bun run src/cli.ts jobs submit --json` mid-suite would JSON.parse a mix of migration progress + the actual job row. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(e2e): close 3 remaining flake classes after cebu-v4 + halifax merge 1. **dream-cycle-phase-order-pglite**: EXPECTED_PHASES was missing `schema-suggest` (v0.39.0.0 added it between `orphans` and `purge`). Hand-port of cebu-v4's |
||
|
|
1666ec427e |
v0.39.2.0 feat(autopilot): per-source fan-out + cycle lock primitive + phase taxonomy (#1295)
* feat(source-id): canonical dependency-free source_id validator module
New src/core/source-id.ts consolidates the three regex sites that drifted
across the codebase (utils.ts permissive, sources-ops.ts strict,
source-resolver.ts strict). Exports:
- SOURCE_ID_RE: ^[a-z0-9](?:[a-z0-9-]{0,30}[a-z0-9])?$ (strict kebab,
1-32 chars, no underscores, alphanumeric boundaries)
- isValidSourceId(s): boolean — for silent-fallback tiers (dotfile,
brain_default config)
- assertValidSourceId(s): void, throws — for explicit-validation tiers
(explicit --source flag, GBRAIN_SOURCE env, cycleLockIdFor primitive)
Dependency-free by design (no engine imports), so both PGLite and
Postgres engines can pull it without circular-import risk. Replaces
the soon-to-be-removed local validators in utils.ts and sources-ops.ts;
preserves both call shapes (boolean + throwing) per the codex outside-voice
finding that resolver tiers need both.
19 unit tests covering valid ids, length boundary (32-char max), underscore
rejection, path-traversal shapes, edge hyphens, whitespace, non-ASCII,
non-string inputs, and TypeScript narrowing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(source-id): migrate three regex sites to canonical source-id.ts
Consolidates source_id validation through src/core/source-id.ts:
- src/core/utils.ts: validateSourceId is now a back-compat re-export of
assertValidSourceId. Regex TIGHTENS from permissive ^[a-z0-9_-]+$ to
the strict kebab. The path-safety boundary now matches what sources-ops
enforces at source creation time; no production source IDs break because
sources-ops always rejected underscored IDs at creation. Picks up the
blast-radius callers in cycle/patterns.ts and cycle/synthesize.ts
reverse-write paths.
- src/core/sources-ops.ts: deletes local SOURCE_ID_RE + validateSourceId;
imports isValidSourceId from source-id.ts. Keeps the thin SourceOpError-
wrapping validator so `gbrain sources add` keeps its user-facing error
envelope.
- src/core/source-resolver.ts: imports SOURCE_ID_RE + isValidSourceId from
source-id.ts. Per codex outside-voice P1-F, silent-fallback tiers
(dotfile read at tier 3, brain_default config at tier 5) use
isValidSourceId so an invalid dotfile/config value falls through to the
next resolver tier instead of throwing. Explicit + env tiers keep their
inline regex-test-and-throw shape because they need tailored error
messages.
Behaviour: validateSourceId('snake_id') NOW THROWS where pre-PR it accepted.
Documented as the intentional tightening; no existing IDs in production
contain underscores.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): per-source lock primitive + db-lock consolidation + PGLite ordering
Three intertwined cycle.ts changes that landed as one logical unit:
1) DELETE acquirePostgresLock + acquirePGLiteLock (~75 LOC of duplicated
UPSERT-with-TTL SQL). Replace with tryAcquireDbLock from
src/core/db-lock.ts, which was extracted in v0.22.13 and should have
been adopted here at that time. New acquireDbCycleLock(engine, sourceId)
is a 6-line adapter that keeps cycle.ts's LockHandle shape.
Deliberately uses tryAcquireDbLock NOT withRefreshingLock (codex r2 P0-A):
- tryAcquireDbLock returns null on busy → cycle returns
{status:'skipped', reason:'cycle_already_running'} (existing contract)
- withRefreshingLock throws → would convert busy cycles into failures
- withRefreshingLock's background timer would skip Minion job-lock
renewal (codex r2 P0-B) and add in-phase DB traffic on PGLite's
single connection (codex r2 P1-A)
2) Add cycleLockIdFor(sourceId?: string) primitive:
- undefined → 'gbrain-cycle' (legacy default, back-compat for autopilot
and every existing caller)
- valid kebab → 'gbrain-cycle:<source_id>' (per-source DB lock row)
- invalid → throws via assertValidSourceId (codex r2 P1-B defense-
in-depth at the primitive layer, since CycleOpts.sourceId is a new
direct API surface that becomes part of a DB lock ID AND a PGLite
file path component)
Add CycleOpts.sourceId; thread through to acquireDbCycleLock. Documents
that this only scopes the LOCK — embed/orphans/purge/etc remain
brain-global per PHASE_SCOPE.
3) PGLite file+DB ordering invariant (codex r2 P0-C + P0-D):
- PGLite engines acquire the GLOBAL file lock (cycle.lock, no source
suffix) BEFORE the per-source DB lock. PGLite's process-level
write-lock is the single-writer guard; per-source DB lock IDs
alone would let two PGLite cycles run concurrently.
- File lock release on DB acquisition failure (cleanup guarantee)
- Compose both handles into one LockHandle whose release() is
reverse-of-acquire (DB first, file last) so file lock isn't released
while DB lock is still live.
- Postgres engines skip the file lock entirely — per-source DB IDs
are the full granularity.
13 unit tests in test/cycle-lock-per-source.test.ts pin the back-compat
default, per-source ID shape, distinct-ID property, and the internal-
validation throws.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): PHASE_SCOPE taxonomy + doctor cycle_phase_scope check
Static documentation of each cycle phase's scope: 'source' (safe to
parallelize per source), 'global' (must serialize brain-wide), or
'mixed' (per-phase decomposition needed before parallelizing).
The PHASE_SCOPE record is the load-bearing input for any future
autopilot fan-out wave. It surfaces what codex round-1 P0-1 was
warning about: not all 14 cycle phases are source-scoped today.
embed/orphans/purge/resolve_symbol_edges/grade_takes/calibration_profile
walk brain-wide regardless of sourceId. Per-source cycle LOCKS (this
PR) let two cycles RUN concurrently, but global-scoped phases inside
each will still touch the same rows.
The taxonomy is documentation, not runtime enforcement (runtime
enforcement deferred per plan; filed as TODO).
New doctor check cycle_phase_scope renders the taxonomy as an
operator-facing message AND surfaces phase_scope_map under
Check.details for JSON consumers. Added optional Check.details field
to the doctor types — mirrors PhaseResult.details. Additive; no
schema_version bump.
11 unit tests across phase-scope-coverage + doctor-cycle-phase-scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(source-id): blast-radius regression for strict-regex callers
Pins the codex round-2 P1-D finding: utils.validateSourceId is also
used in cycle reverse-write paths at patterns.ts:263 and
synthesize.ts:909. Pre-PR they used the permissive regex; post-PR
they share the strict kebab regex with sources-ops creation-time
validation. Existing underscore IDs would fail at THOSE cycle sites,
not just at source add/remove.
Structural assertions guard against future drift:
- utils.ts validateSourceId === assertValidSourceId from source-id.ts
- patterns.ts + synthesize.ts both import validateSourceId from utils
- validation call precedes the join() at both reverse-write sites
- utils.ts no longer contains the inline ^[a-z0-9_-]+$ permissive regex
- utils.ts re-exports assertValidSourceId-as-validateSourceId from source-id.ts
9 cases pin the contract. IRON-RULE: source-text grep regressions land
on the offending refactor first.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(engine): listAllSources + updateSourceConfig for per-source autopilot
Two lean engine-layer methods that the v0.38 per-source autopilot wave
consumes. Both have parity implementations on Postgres + PGLite.
listAllSources(opts?):
- Returns the bare SourceRow shape (id, name, local_path, last_sync_at,
config) without sources-ops.listSources's per-source page_count
enrichment (N+1 expensive; out of scope for hot-loop callers).
- `includeArchived` defaults false (matches sources-ops semantics).
- `localPathOnly` filters local_path IS NOT NULL so autopilot fan-out
doesn't dispatch jobs for pure-DB sources whose handler would fall
back to global sync.repo_path (codex r1 P1-4).
- Ordering: (id = 'default') DESC, id — same as sources-ops for
operator-output stability.
updateSourceConfig(sourceId, patch):
- Atomic JSONB merge via Postgres `config || $patch::jsonb` operator.
No read-modify-write race; same-key overwrites (no deep merge —
flat patches only, matches the v0.38 use case of last_full_cycle_at).
- Returns true when a row was updated, false when sourceId doesn't
exist (best-effort no-op; caller decides how to handle).
- Postgres: sql.json(patch) per the canonical pattern; PGLite:
JSON.stringify + ::jsonb cast on positional param.
New SourceRow type exported from engine.ts. Imported by both engine
impls. 11 integration tests in test/list-all-sources.test.ts cover
defaults, filters, JSONB round-trip, archived flag, and merge semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(cycle): write last_full_cycle_at to sources.config on per-source cycle exit
Closes codex round-1 P0-5 (write site for last_full_cycle_at was
unspecified pre-PR). runCycle's exit hook persists
{ last_full_cycle_at: '<ISO>' } to sources.config JSONB when a
successful per-source cycle completes. Autopilot's v0.38 per-source
fan-out gate reads this field next tick to decide whether to skip a
source (60-min freshness floor).
Conditions for write (all required):
- opts.sourceId is set — legacy callers without sourceId skip the
write (autopilot will keep working today via fallback path)
- engine is non-null — no-DB path skips
- status is 'ok' / 'clean' / 'partial' — failed/skipped cycles do NOT
mark a source as fresh (next cycle will redo work)
- dryRun is false — writes are out of scope
Best-effort: write failure logs a warning but does NOT change the
CycleReport status. The cycle already succeeded by the time we get
here; the cost of missing a stale write is one redundant cycle next
tick, not data loss.
5 PGLite integration tests cover all four gate conditions plus the
"timestamp advances on each successful run" property.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(jobs): autopilot-cycle handler honors source_id + pull + archive recheck
Threads the v0.38 per-source dispatch payload through the autopilot-cycle
handler at src/commands/jobs.ts:1146. Closes three codex round-1 findings:
- P0-2 / P1-B: validates job.data.source_id at handler entry via the
canonical source-id.ts isValidSourceId boolean check. Malformed
source_id from a queue replay dead-letters with a clear error
instead of reaching cycle code.
- P1-2: job.data.pull explicit boolean overrides the legacy hardcoded
`true`, so per-source dispatch for local-only sources can pass
pull: false (no git network round-trip for sources without remote_url).
Missing/undefined preserves the legacy true for back-compat with cron/
launchd callers that don't know about the new field.
- P1-5: archived-source recheck happens BEFORE runCycle is invoked
(cheap SELECT archived FROM sources WHERE id = $1). If the source was
archived between fan-out and worker claim, handler returns
{ status: 'skipped', reason: 'source_archived' } cleanly — no lock
acquired, no phases run, no last_full_cycle_at touched. Same skip
shape for source_not_found (deleted between dispatch and claim).
7 PGLite integration tests cover all five paths (legacy / valid /
not-found / archived / malformed-source_id / non-string source_id /
pull: false override).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(autopilot): per-source fan-out dispatch (the headline parallelism win)
The headline change of the v0.38 federated-sync wave. Replaces autopilot's
single-job-per-tick dispatch with per-source fan-out so a 5-source
federated brain refreshes in ~5min wall-clock instead of ~25min sequential.
src/commands/autopilot-fanout.ts (new) — pure-function dispatch helper:
- resolveFanoutMax(engine): PGLite=1 (codex P1-3 — preserves single-writer
invariant), Postgres=4, operator override via autopilot.fanout_max_per_tick
- readLastFullCycleAt(src): JSONB→Date with NULL/unparseable safety
- isSourceStale(src, now?, floorMin?): 60-min default freshness floor
- selectSourcesForDispatch(sources, fanoutMax): stale-only + oldest-first
+ alphabetical tiebreaker (deterministic for tests)
- dispatchPerSource(engine, queue, opts): the orchestrator
src/commands/autopilot.ts (modified): the existing shouldFullCycle branch
calls dispatchPerSource. Behavior preserved:
- Healthy + recent (60min floor) → sleep (unchanged)
- Targeted-plan path → unchanged (uses computeRecommendations)
- Full-cycle path → NOW fans out per-source rather than ONE job for default
Per-source dispatch shape:
- Idempotency key: `autopilot-cycle:<source_id>:<slot>` — two ticks for
the same source within one slot coalesce; different sources never collide
- pull: !!source.config.remote_url — remotes pull, local-only don't
- maxWaiting: 1 per submit — backpressure when worker can't drain
- Per-submit try/catch (codex E1 F1) — one source's failure doesn't
abort the tick; surfaces as fanout_submit_failed event
Fallback path: empty `sources` table (pre-v0.18 brain or fresh install
before `gbrain sources add`) falls back to the legacy single autopilot-
cycle job with no source_id, preserving today's single-source behavior.
JSON event stream extended:
- `dispatched` event gains source_id + mode='per_source' fields
- new `fanout_summary` event per tick with dispatched/skipped_fresh/
skipped_cap arrays so operators can see what the tick did
- new `fanout_cap_reached` event when sources overflow the cap
Caveat (intentional, codex r1 P0-1 scope): per-source LOCKS let two
cycles RUN concurrently, but several phases (embed, orphans, purge,
resolve_symbol_edges, grade_takes, calibration_profile) still walk the
brain globally inside each cycle. PHASE_SCOPE taxonomy from the prior
commit documents this. Genuine per-phase per-source isolation is the
deferred Phase 2 follow-up.
27 unit tests in test/autopilot-fanout.test.ts pin every branch (stale
gate, cap behavior, idempotency keys, legacy fallback, per-submit error
isolation, oldest-first sort, alphabetical tiebreaker).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(doctor): cycle_freshness check — sibling to sync_freshness
New check_cycle_freshness sibling to checkSyncFreshness. Where
sync_freshness reads sources.last_sync_at (one phase), this check
reads sources.config->>'last_full_cycle_at' — the canonical
"this whole cycle completed" timestamp the v0.38 runCycle exit hook
writes and the v0.38 autopilot fan-out gate reads.
Operator sees exactly what autopilot sees when deciding to skip a
source. Default thresholds tighter than sync_freshness (6h warn /
24h fail vs 24h/72h) because full-cycle staleness compounds: sync
stale → extract stale → embed stale → search returns stale results.
Env overrides:
- GBRAIN_CYCLE_FRESHNESS_WARN_HOURS (default 6)
- GBRAIN_CYCLE_FRESHNESS_FAIL_HOURS (default 24)
Edge cases covered (9 PGLite integration tests):
- empty (no federated sources) → ok
- last_full_cycle_at present + fresh → ok
- last_full_cycle_at present + warn window → warn
- last_full_cycle_at present + fail window → fail
- last_full_cycle_at NULL (never cycled) → fail
- mixed severity → highest wins
- future timestamp (clock skew) → warn
- unparseable timestamp → warn
- local_path NULL sources filtered (codex P1-4 parity)
Failure messages embed source.id so the printed fix command
`gbrain dream --source <id>` matches what the user copy-pastes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: fill gaps surfaced by post-Phase-2 audit (resolver silent-fallback + PGLite ordering)
Two test gaps identified by the test-coverage audit after Phase 1 + 2 shipped:
1. src/core/source-resolver.ts — the migration to isValidSourceId for
silent-fallback tiers (dotfile read, brain_default config) had no
direct test. Pre-PR these used inline regex; post-PR they use the
canonical isValidSourceId. Codex P1-F intent (silent fallback for
invalid input on tiers 3+5, throw on tiers 1+2) deserved an explicit
test.
test/source-resolver-silent-fallback.test.ts (12 cases):
- tier 3: valid dotfile honored; underscore/whitespace/uppercase
silently falls through to next tier
- tier 5: valid brain_default honored; underscore + 33+ char silently
falls through
- tier 1: valid explicit --source returns; underscore/whitespace
THROWS (contract distinction)
- tier 2: valid env GBRAIN_SOURCE returns; underscore THROWS
2. src/core/cycle.ts — the PGLite file+DB ordering invariant (codex r2
P0-C + P0-D) was implemented in Phase 1 (T5) but had no test pinning
the ordering / cleanup / per-source DB lock ID semantics.
test/cycle-pglite-lock-ordering.test.ts (6 cases):
- global file lock acquired during PGLite cycle
- cycle for source A then B serializes (file lock held in turn)
- DB-lock acquire failure releases file lock cleanly (no stranded state)
- engine=null path still uses file lock
- DB lock row uses per-source ID (gbrain-cycle:<source>) not legacy
- consecutive cycles can re-acquire both locks (release-on-exit works)
Plus .context/PHASE_3_ASSESSMENT.md (gitignored) documenting why each
Phase 3 DRY refactor item from the original plan is deferred: each item
on closer inspection is either a premature abstraction or was explicitly
rejected by the original author (BaseCyclePhase header comment).
18 new tests; 0 fails. typecheck clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: autopilot wiring static guard + Postgres parity e2e for fan-out
Closes the remaining test gaps identified by the post-Phase-2 audit:
test/autopilot-fanout-wiring.test.ts (5 cases) — static-shape regression
for autopilot.ts ↔ dispatchPerSource. The fan-out helper itself has 27
unit tests; this file pins the WIRING in autopilot.ts:
- imports dispatchPerSource + resolveFanoutMax
- calls dispatchPerSource inside the shouldFullCycle branch (not the
targeted-plan path)
- updates lastFullCycleAt after dispatch
- does NOT regress to the pre-PR single-job dispatch (regex-grep guard
against the legacy `autopilot-cycle:${slot}` idempotency-key shape
reappearing in autopilot.ts)
Same canonical static-shape pattern as test/autopilot-supervisor-wiring.test.ts.
test/e2e/list-all-sources-postgres.test.ts (10 cases) — Postgres parity
for engine.listAllSources + updateSourceConfig. The PGLite path has
unit-level coverage; the Postgres path has separate impls (sql.json
serialization, sql.count semantics) that could drift. Specifically pins:
- returns rows, filters archived/localPath correctly
- JSONB config parses to object (autopilot reads last_full_cycle_at)
- default source sorts first
- updateSourceConfig: not-found returns false, patch merges, same-key
overwrites, idempotent on repeat
- jsonb_typeof regression: round-trip stores real JSONB object, NOT
a JSON-encoded string (feedback_postgres_jsonb_double_encode class)
test/e2e/autopilot-fanout-postgres.test.ts (6 cases) — end-to-end
integration on Postgres:
- 3 sources fan out as 3 distinct jobs with per-source idempotency keys
- re-dispatch within same slot dedupes (idempotency-key coalesce)
- last_full_cycle_at < 60min ago sources are skipped by gate
- end-to-end: updateSourceConfig → listAllSources → selectSourcesForDispatch
correctly classifies fresh sources
- fan-out cap honored (5 sources, fanoutMax=2 → 2 dispatched)
- empty federated brain falls back to legacy single-job dispatch
21 new test cases. Brings the v0.38 wave coverage to 132 unit + 16 e2e.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(autopilot): drop maxWaiting from per-source submit (E2E found silent coalesce)
The Postgres E2E for fan-out surfaced two real bugs that the unit-stub
tests + PGLite tests couldn't catch:
1) **CRITICAL** — dispatchPerSource passed `maxWaiting: 1` to every
per-source queue.add. maxWaiting is per-(name, queue) — since all
per-source jobs share `name='autopilot-cycle'`, the second + third +
Nth source's submit silently coalesced into the FIRST source's
waiting job. Net result on a real worker: 1 job processed per tick,
not N. The entire fan-out feature was a silent no-op past the first
source.
Per-source idempotency_key (`autopilot-cycle:<source_id>:<slot>`)
already handles "two ticks for same source within slot" dedup, which
is the only thing maxWaiting was buying us. Dropping it fixes fan-out
without losing dedup.
New unit-stub regression test asserts maxWaiting is NOT in the per-
source submit opts so a future refactor that re-adds it gets caught
in 100x faster CI (test/autopilot-fanout.test.ts).
2) **postgres-engine.ts:updateSourceConfig** — initial impl used
sql.json() correctly but my mid-debug rewrite to executeRaw +
positional `$1::jsonb` produced JSONB STRING shape (not OBJECT)
because postgres-js double-encodes JS string params in unsafe mode.
`||` between JSONB object + JSONB string yields a JSONB ARRAY,
wiping every existing config key on update.
Same latent bug class exists at src/commands/sources.ts:482 (gbrain
sources federate/unfederate path); flagged for follow-up but
out-of-scope here.
Reverted to sql.json() inside the template tag (verified via direct
psql round-trip: jsonb_typeof = 'object'). Updated the e2e seed
helper to use sql.json() too — the executeRaw + JSON.stringify
pattern was producing string-shape JSONB at SEED time which made
the failure cascade harder to debug.
Coverage adds:
- test/e2e/list-all-sources-postgres.test.ts: 11 cases pin Postgres
parity for listAllSources + updateSourceConfig including jsonb_typeof
round-trip
- test/e2e/autopilot-fanout-postgres.test.ts: 6 cases end-to-end
including 3-source fan-out producing 3 distinct rows, idempotency
coalesce within slot, cap honored, legacy fallback path
- test/autopilot-fanout.test.ts: +1 regression guard on maxWaiting
This is the kind of bug that justifies the user's "fill test gaps then
run E2E" mandate. The unit tests + PGLite parity tests + typecheck all
passed cleanly; only the real-Postgres E2E found the coalesce.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(e2e): update multi-source-bug-class validateSourceId expectations for strict regex
The v0.32.8 test pinned the OLD permissive ^[a-z0-9_-]+$ behavior including
the underscored case 'jarvis_memory'. The v0.38 wave (this PR's E2 + codex
P1-D) tightens validateSourceId to the strict kebab regex shared with
sources-ops. 'jarvis_memory' now lives in the rejected set, not the
allowed set.
Updated the test to assert the new contract:
- Replaced 'jarvis_memory' allowed case with 'jarvis-memory' (kebab)
- Added 'a' (single-char) to allowed cases
- Added 'jarvis_memory', 'snake_case', '-leading', 'trailing-', and a
33-char string to the rejected cases — the v0.38 strict-regex additions
Comment explains the contract shift so future readers don't see the test
as flapping intent.
Found by running the main E2E suite — the test file is in the canonical
e2e set and would have failed CI otherwise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* v0.39.3.0: per-source autopilot fan-out + cycle lock primitive + phase taxonomy
VERSION + CHANGELOG bump for the parallel-federated-sync wave (14 commits).
Headline change: federated brains refresh all sources in parallel via
per-source autopilot dispatch instead of one source per 5-min tick.
Five-source brain wall-clock: ~25min → ~5min.
Test infra adjustments for the v0.38 test-isolation lint that landed via
upstream master merge:
- test/source-resolver-silent-fallback.test.ts now uses withEnv() for
GBRAIN_SOURCE mutations (was direct process.env mutation)
- test/cycle-pglite-lock-ordering.test.ts → .serial.test.ts (the
file-wide GBRAIN_HOME setup needs quarantine from the parallel pool)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(test): bump cycle-abort handler-source-grep window 2000→6000
CI on
|