diff --git a/AGENTS.md b/AGENTS.md index 9efaca672..846a04ce4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -104,8 +104,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont ## Before shipping Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks, -unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a -fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the +guards + typecheck, then 4-shard parallel unit + E2E against four pgvector +containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL` +unset) and tears down. Use `bun run ci:local:diff` for the diff-aware subset during fast iteration on a focused branch. Requires Docker (Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`). diff --git a/CHANGELOG.md b/CHANGELOG.md index a6972e9a6..662838cf5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,55 @@ All notable changes to GBrain will be documented in this file. +## [0.42.45.0] - 2026-06-13 + +**The daily sync cron stops wedging on cost, and the embedding-spend estimate finally matches what a sync actually does (gbrain#2139).** On an active brain the inline-embed cost gate priced the *entire* corpus every time the working tree was dirty — which is always, since agents and crons write to it constantly — so a routine daily sync estimated ~158M tokens / ~$8 when the real delta was a few hundred files / ~$0.04, then blocked the cron with a confirmation it could never answer. Embeds silently stalled until someone noticed. The estimate now mirrors execution: it fetches first and prices only the files this run will pull and import, through the same diff machinery the sync itself uses. A brain whose commits are caught up but whose tree is dirty estimates $0, because an attached-HEAD sync imports only the committed diff. + +When the gate does fire in a non-interactive session, it no longer exits with an error — it imports now and defers embedding to capped background jobs (which drain via the jobs worker or `gbrain embed --stale`), so a cron is never wedged again. Operators who have decided cost isn't the constraint get one switch — `gbrain config set spend.posture tokenmax` — that makes every cost gate informational across sync, reindex, enrich, and onboard (spend is still recorded; the switch removes the ceiling, not the accounting). The USD knobs accept `off` / `unlimited`, and every gate message now carries paste-ready commands so the controls are discoverable at the moment they fire. + +This release also lifts the rule that blocked `--skip-failed` / `--retry-failed` under parallel sync — failure recovery no longer has to drop to `--serial` (which is what armed the inline gate in the first place). + +### Added +- **`spend.posture` config** — `tokenmax` makes every embedding-cost gate informational (print the estimate, proceed, keep the ledger); `gated` (default) enforces as before. Documented end-to-end in `docs/operations/spend-controls.md`. +- **First-class off switches** — `sync.cost_gate_min_usd`, `embed.backfill_max_usd_per_source_24h`, `embed.backfill_max_usd`, and `reindex --max-cost` / `enrich --max-usd` accept `off` / `unlimited` / `none`. No more sentinel values like `100000`. +- **Single-source `gbrain sync` cost preview** — plain `gbrain sync` previously embedded inline with no preview; it now carries the same gate as `sync --all` (auto-defers in non-TTY sessions, never blocks). +- **Self-describing gate messages** — every cost-gate / FYI line ends with the exact `gbrain config set` commands to widen, disable, or switch posture, plus a docs pointer. + +### Changed +- **`gbrain sync --all` is no longer blocked by the cost gate in cron/agent contexts.** Above the floor in a non-interactive session it auto-defers embeds (exit 0) instead of emitting a `cost_preview_requires_yes` envelope and exiting 2. Cron wrappers that branched on exit 2 now see exit 0 with `status: "auto_deferred"`. A TTY still prompts `[y/N]`; `--yes` still embeds inline. +- **`--skip-failed` / `--retry-failed` now work under parallel sync.** The failure ledger is per-source and lock-serialized, so the previous "not supported under parallel — re-run with --serial" refusal is retired. +- **The six spend-control config keys are now first-class** (`gbrain config set` accepts them without `--force`). + +### To take advantage of v0.42.45.0 +`gbrain upgrade`. Nothing to configure for the headline fix — the daily sync cron stops wedging and the estimate is accurate out of the box. If you run a high-volume brain where cost genuinely isn't the constraint, `gbrain config set spend.posture tokenmax` makes every gate informational. To widen or disable a specific gate instead, see the table in `docs/operations/spend-controls.md` (e.g. `gbrain config set sync.cost_gate_min_usd off`). Failure-recovery syncs can now stay parallel: `gbrain sync --all --skip-failed` no longer forces `--serial`. + +## [0.42.44.0] - 2026-06-13 + +### Fixed + +- **Personal-brain tutorial points at the correct AlphaClaw site.** Step 4 of `docs/tutorials/personal-brain.md` ("Deploy via AlphaClaw on Render") linked to the wrong top-level domain, sending readers to a site that isn't the official AlphaClaw. The link now resolves to the right destination, so the deploy step works as written (gbrain#2165). + +## [0.42.43.0] - 2026-06-12 + +**The brain now volunteers relevant pages instead of waiting to be asked (gbrain#2095).** Retrieval used to be pull-only: a deep session could run for hours with zero brain contributions — not because the brain had nothing, but because nothing prompted the agent to ask, and pages stored under coined names were missed by literal-string queries. Push-based context inverts that, on three channels sharing one zero-LLM, confidence-gated core: the ambient retrieval reflex now reads the last few conversation turns (an entity your assistant introduced two turns ago resolves on the "what did she invest in?" follow-up), a new `volunteer_context` operation gives any agent a per-turn volunteer surface over CLI stdin or MCP, and `gbrain watch` streams volunteered pages as a transcript flows through it. + +Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got retrieved afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the retrieval signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves. + +This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in the local CI gate (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure. + +### Added +- **`volunteer_context` operation** (CLI: `gbrain volunteer-context`, MCP tool) — pipe recent turns in (`user:` / `assistant:` prefixed lines, or plain text), get confidence-gated page pointers with rationales and synopses out. `--stats` returns the volunteered-vs-used precision summary. Per-call knobs: `max_pages`, `min_confidence`, `session_id`/`turn` attribution. +- **`gbrain watch`** — the streaming push transport: feed a transcript on stdin, volunteered pages stream out (`--json` for JSONL), each slug at most once per session. Piped input exits cleanly at end-of-input; interactive sessions run until Ctrl-C. +- **Rolling-window retrieval reflex** — the ambient channel extracts entities from the last 4 turns (configurable via `retrieval_reflex_window_turns` / `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`; 1 restores the previous single-turn behavior). Assistant-introduced entities and named-antecedent follow-ups now surface pointers with zero agent-initiated queries. +- **Volunteered-context feedback log** — volunteered pages are recorded best-effort (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses always strip the takes/facts privacy fences before reaching a prompt. +- **Transaction-mode pooler in the local CI gate** — `bun run ci:local` now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after. + +### Fixed +- **`gbrain doctor` exits 1 on FAIL again on every engine.** Its verdict write predated the v0.42.42.0 exit-verdict channel and was being silently zeroed; swept, plus a structural test that fails CI on the next raw exit-code write anywhere in the CLI. +- **Reflex pointer suppression under multi-turn windows** distinguishes "page already surfaced" from "entity merely mentioned earlier" — without this, window extraction would have suppressed every prior-turn entity by construction. + +### To take advantage of v0.42.43.0 +`gbrain upgrade` (applies the new feedback-log migration automatically). The wider reflex window is on by default for context-engine hosts — set `retrieval_reflex_window_turns: 1` in `~/.gbrain/config.json` to restore single-turn behavior. Agents without the context engine: call `volunteer_context` per turn (window in, pointers out), or pipe a transcript through `gbrain watch --json`. After a few days, `gbrain volunteer-context --stats` shows which resolution arms are earning their keep; raise or lower `min_confidence` accordingly. ## [0.42.42.0] - 2026-06-12 **`gbrain query` no longer pays a flat 10-second exit tax on managed Postgres behind a transaction-mode pooler — and CLI exit codes finally tell the truth on PGLite.** On deployments where the pooler holds sockets open past the bounded pool drain (gbrain#2084, a residual of gbrain#1972), every query printed its results and then sat for 10 seconds until the force-exit banner fired. The cause was two-layered: the hard-deadline timer was armed *before* the operation handler, so a multi-second search on a large brain burned the teardown budget (and any operation slower than 10 seconds was silently killed mid-run with exit 0 and truncated output); and the CLI never exited explicitly on success — it waited for Bun's event loop to drain, which a stuck pooler socket can hold open forever. diff --git a/CLAUDE.md b/CLAUDE.md index b05671eae..7b6859f00 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP +Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -97,6 +97,8 @@ detail on demand.) | any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry | | search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` | | search modes / cost knobs | `docs/guides/search-modes.md` | +| embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` | +| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` | | schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` | | thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` | | the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry | diff --git a/TODOS.md b/TODOS.md index 92c22f146..ed6f5507c 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,88 @@ # TODOS +## Spend-controls wave follow-ups (filed v0.42.45.0, #2139) + +Deferred from the #2139 delta-estimator wave. See plan + GSTACK REVIEW REPORT at +`~/.claude/plans/system-instruction-you-are-working-lovely-balloon.md`. + +- [ ] **P3 — Measured post-import chunk-count gating (#2139 proposal 2b).** + **What:** Gate the inline cost decision on the actual chunk count sync produced + (known after import, before embedding) instead of the pre-sync token estimate. + **Why:** A fully execution-accurate gate with zero estimate error. **Context:** + After v0.42.42.0 the estimator already mirrors execution (fetch-first delta via the + shared `computeSyncDelta`, `--full`=delta+stale, dirty-tree→$0). This is the + belt-and-suspenders fallback if a future case still drifts. **Trigger:** only if the + delta estimator proves insufficient in practice. **Start:** the gate call site in + `src/commands/sync.ts` (`runInlineCostGate`), gate on post-import `chunksCreated`. +- [ ] **P3 — Per-source defer granularity (#2139, D8A road-not-taken).** + **What:** When the aggregate inline gate trips in a non-TTY session, defer embeds + only for sources above a per-source floor; let cheap sources keep embedding inline. + **Why:** Cheap sources would get embeddings minutes sooner instead of waiting for a + backfill-worker drain. **Context:** v0.42.42.0 chose GLOBAL defer (one flag, strictly + dominates the exit-2 it replaced). This is the granularity upgrade. **Trigger:** a + filed embedding-latency-by-minutes complaint. **Start:** thread per-source estimates + through `runOne` (`src/commands/sync.ts`); design worked out at D8A in the plan. + +## gbrain#2095 push-based context follow-ups (v0.43+) + +Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`). +Deliberately scoped OUT of v1 per the eng-review scope decision (success criteria +are the bar). Plan + GSTACK REVIEW REPORT at +`~/.claude/plans/system-instruction-you-are-working-cheerful-elephant.md`. + +- [ ] **P3 — SSE/HTTP push channel via serve-http.** The op + `gbrain watch` cover + pull-per-turn and stdin streaming; a serve-http SSE feed would push volunteered + pages to remote agents without a local CLI. **Why:** thin-client/remote-MCP + deployments get push too. **Cons:** async plumbing + auth scoping; no consumer + wired today. **Where:** `src/commands/serve-http.ts` + `src/core/context/volunteer.ts`. + **Blocked by:** a real consumer (revisit when one exists). +- [ ] **P3 — policy skill + doctor check for push-context.** The ambient reflex + needed doctor visibility because silent failure was invisible; volunteer is + invoked-on-demand so v1 skipped it. If `volunteer-context --stats` adoption shows + agents not discovering the surface, ship a `push-context` recipe (mirror + `recipes/retrieval-reflex/`) + a doctor check reading the events table. + **Where:** `recipes/`, `src/commands/doctor.ts`. +- [ ] **P3 — structured `messages[]` param for volunteer_context.** v1 takes a + string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract. + If MCP callers accumulate parsing bugs, add a structured array param beside it. + **Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`. +- [ ] **P3 — index shapes for the per-turn resolver query.** The arm-2 resolver + (`retrieval-reflex.ts`: `lower(title) = ANY() OR slug = ANY() OR slug LIKE + ANY('%/...')`) predates #2095 but now runs per turn on three channels + (reflex window, volunteer_context, watch) federated across sources. Neither + the leading-wildcard suffix arm nor `lower(title)` is index-served. If + per-turn latency telemetry on large brains comes back hot: add + `(source_id, lower(title))` btree + a reverse(slug) text_pattern_ops (or + gin_trgm) index, or split the OR into three index-friendly queries. + **Where:** `src/core/context/retrieval-reflex.ts`, migration. +- [ ] **P3 — batch the volunteer-events pruner's first run after a long gap.** + `purgeStaleVolunteerEvents` is one unbatched DELETE with a bare + `volunteered_at` predicate (full scan; fine for a TTL-bounded table). Edge: + a brain whose dream cycle was off for months could hit the pooler's ~2min + statement_timeout on the first prune, get swallowed by the catch, and never + make progress. If observed: id-batched chunks (`DELETE ... WHERE id IN + (SELECT ... LIMIT 10000)` looped). **Where:** + `src/core/context/volunteer-events.ts:purgeStaleVolunteerEvents`. +- [ ] **P3 — route `gbrain watch` through the serve resolve-IPC on PGLite.** + `watch` connects directly, so on a PGLite brain it monopolizes the single + connection for its whole (potentially hours-long) session — a concurrent + `gbrain serve` or any write path blocks on the lock until watch exits. + WATCH_HELP documents the monopoly; the fix is an IPC rung in watch's + resolver (reuse `resolveViaIpc` like the ambient reflex's ladder) so a + running serve answers and watch never takes the lock. **Why:** watch + + serve concurrently is the natural agent topology. **Where:** + `src/commands/watch.ts`, `src/core/context/resolve-ipc.ts` (red-team RT2). +- [ ] **P3 — capability/version gate for host-injected reflex resolvers.** + Windowing switched the orchestrator's suppression request to 'slug-only'; + a host resolver built against the pre-window contract that still applies + title-whole-word suppression silently self-suppresses every windowed + entity. The contract is documented at `ResolveEntitiesFn` (reflex.ts), but + nothing detects a stale host. Add a capability handshake (e.g. resolver + advertises `supportsSuppressionModes`) and fall back to + `window_turns: 1` semantics when absent. **Where:** + `src/core/context/reflex.ts:ResolveEntitiesFn` + the OpenClaw plugin + contract (red-team RT4). + ## gbrain triage wave follow-ups (filed v0.42.41.0) Deferred from the v0.42.41.0 fix wave (eng-reviewed as separate scope, not hotfixes). @@ -32,11 +115,13 @@ Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extra is deterministic + precision-biased. See plan + GSTACK REVIEW REPORT at `~/.claude/plans/system-instruction-you-are-working-wild-yeti.md`. -- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The v1 extractor - (`src/core/context/entity-salience.ts`) misses lowercase names, many non-Latin - scripts, pronoun follow-ups ("what about her?"), and assistant-introduced entities. - These need conversation state or an LLM pass. **Why:** higher recall on the read - side. **Where:** `entity-salience.ts` + the orchestrator's `priorContextText`. +- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The extractor + (`src/core/context/entity-salience.ts`) misses lowercase names and many non-Latin + scripts; these need an LLM pass or script-aware heuristics. **Why:** higher recall + on the read side. **Where:** `entity-salience.ts`. *(Partially done by the #2095 + wave: `extractCandidatesFromWindow` now covers assistant-introduced entities and + pronoun follow-ups whose antecedent was NAMED in the rolling window; true pronoun + coreference for never-named antecedents remains with the LLM-pass idea.)* - [ ] **P3 — recall knob: optional fuzzy/prefix-expansion resolution.** The resolver (`src/core/context/retrieval-reflex.ts`) is exact-only (alias + title + slug-suffix) for precision. Revisit adding `resolveEntitySlug`'s trgm-fuzzy / prefix-expansion diff --git a/VERSION b/VERSION index b7fef61ef..360bb9599 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.42.0 \ No newline at end of file +0.42.45.0 diff --git a/docker-compose.ci.yml b/docker-compose.ci.yml index fc42df13a..787899d3a 100644 --- a/docker-compose.ci.yml +++ b/docker-compose.ci.yml @@ -85,6 +85,40 @@ services: volumes: - gbrain-ci-pg-data-4:/var/lib/postgresql/data + # v0.43 (#2084 / eng-review TD1): PgBouncer in TRANSACTION pooling mode + # fronting postgres-1 — the production topology (Supabase direct :5432 + + # pooled :6543) behind three consecutive pooler-teardown waves + # (#1972 → #2015 → #2084) that CI could never reproduce. + # test/e2e/pgbouncer-teardown.test.ts uses a DEDICATED database + # (gbrain_pgbouncer) on postgres-1 so it never races shard 1's + # TRUNCATE-based fixtures; pgbouncer's wildcard [databases] section + # forwards any dbname to DB_HOST. + pgbouncer: + image: edoburu/pgbouncer:latest + environment: + DB_HOST: postgres-1 + DB_PORT: "5432" + DB_USER: postgres + DB_PASSWORD: postgres + POOL_MODE: transaction + # plain (CI-only): pg16 stores SCRAM verifiers, and pgbouncer can only + # answer the server's SCRAM challenge when its userlist holds the + # PLAINTEXT password — an md5-hashed userlist fails with + # "server login failed: wrong password type". + AUTH_TYPE: plain + MAX_CLIENT_CONN: "200" + DEFAULT_POOL_SIZE: "10" + # gbrain's client sets statement_timeout + idle_in_transaction_session_timeout + # as startup parameters (db.ts buildConnectionParams); the Supabase pooler + # whitelists them, so this pooler must too or every connection is refused + # before the teardown path is even reached. + IGNORE_STARTUP_PARAMETERS: extra_float_digits,statement_timeout,idle_in_transaction_session_timeout,search_path + ports: + - "${GBRAIN_CI_PGBOUNCER_PORT:-6543}:5432" + depends_on: + postgres-1: + condition: service_healthy + runner: image: oven/bun:1 working_dir: /app @@ -97,6 +131,8 @@ services: condition: service_healthy postgres-4: condition: service_healthy + pgbouncer: + condition: service_started # No global DATABASE_URL — scripts/ci-local.sh sets per-shard URL via -e. # Unit phase explicitly unsets DATABASE_URL so test/e2e/* gracefully skip. volumes: diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 14ba6765f..3cff19ddc 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -12,15 +12,17 @@ Before shipping (/ship) or reviewing (/review), always run the full test suite. Two equivalent paths: **Path A — local CI gate (recommended, v0.23.1+):** -- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit - tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a - fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to - what nightly Tier 1 catches. Spins up + tears down postgres automatically via - `docker-compose.ci.yml`. Override the host port with - `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides. +- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), + guards + typecheck, then 4-shard parallel unit + E2E against four pgvector + containers plus a transaction-mode PgBouncer service (unit phase keeps + `DATABASE_URL` unset; `--no-shard` for the legacy sequential flow). Stronger + than PR CI's 2-file Tier 1 set; closer to what nightly Tier 1 catches. Spins + up + tears down postgres automatically via `docker-compose.ci.yml`. Override + the host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides. - `bun run ci:local:diff` runs only the E2E files matched by the diff selector - (`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or - schema/skills/package.json changes. Fast iteration during a focused branch. + (`scripts/select-e2e.ts`), falling back to ALL E2E files on unmapped src/ + paths or schema/skills/package.json changes. Fast iteration during a focused + branch. **Path B — manual lifecycle (still supported):** - `bun test` — unit tests (no database required) diff --git a/docs/TESTING.md b/docs/TESTING.md index d39384b98..93b33252c 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -15,7 +15,7 @@ Seven test command tiers, each with a clear scope: | `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. | | `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. | | `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. | -| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. | +| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. | ### CI vs local: intentionally divergent file sets @@ -126,6 +126,12 @@ Unit tests and what they cover: - `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override). - `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling). - `test/cli-should-force-exit.test.ts` — `shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command). +- `test/cli-exit-verdict-pin.test.ts` — #2084 structural class pin: greps `src/` so the NEXT raw `process.exitCode =` write fails CI (a raw write bypasses the gbrain-owned verdict channel and gets silently zeroed by the deliberate flush-exit — the bug that made doctor's FAIL path exit 0). Runtime variants live in `test/cli-finish-teardown.test.ts`; this is the review-time guard. +- `test/cli-pipe-truncation.test.ts` — real-CLI pipe completeness (the #1959 incident class), implementation-agnostic: the actual CLI run the way agents run it (piped stdout) produces complete, parseable, byte-stable `--tools-json` output and exits deliberately, well under the teardown backstop. Synthetic flush-mechanism coverage stays in `test/flush-then-exit-harness.test.ts`. +- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join. +- `test/watch-command.test.ts` — `gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin). +- `test/watch-sigint.serial.test.ts` — `gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`). +- `test/cli-format-volunteer.test.ts` — `formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary. - `test/config.test.ts` — config redaction. - `test/files.test.ts` — MIME/hash. - `test/import-file.test.ts` — import pipeline. @@ -133,7 +139,7 @@ Unit tests and what they cover: - `test/file-migration.test.ts` — file migration. - `test/file-resolver.test.ts` — file resolution. - `test/import-resume.test.ts` — import checkpoints. -- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion. +- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion; v117 `context_volunteer_events` (named + idempotent entry, documented columns + both source-scoped indexes after `initSchema`, insert + 90-day `purgeStaleVolunteerEvents` round-trip). - `test/bootstrap.test.ts` — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on a simulated legacy brain, fresh-install regression guard, legacy `links` shape coverage. - `test/schema-bootstrap-coverage.test.ts` — CI guard. `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in `PGLITE_SCHEMA_SQL`; the test fails loudly if `applyForwardReferenceBootstrap` skips one (extend both arrays when adding a column-with-index to the embedded schema blob). Also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`) and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies — catching the column-only forward-reference class (e.g. `sources.archived`, `oauth_clients.source_id`) that a CREATE INDEX parser alone can't see. `parseBaseTableColumns` strips SQL line + block comments before identifying column names so commented-out lines don't hide adjacent columns. - `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each, snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via `PGLITE_SCHEMA_SQL` or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skips without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`. @@ -223,6 +229,8 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D - `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required). - `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use. - `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner. +- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972→#2015→#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures. +- `test/e2e/volunteer-context-postgres.test.ts` — `volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated. - `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape. - `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory. - `test/e2e/search-exclude.test.ts` — `test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths. diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 667217659..ced7b8326 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -11,16 +11,17 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Exports upload validators `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` is a REQUIRED field flagging untrusted callers; `OperationContext.allowedSlugPrefixes` is the trusted-workspace allow-list set by the dream cycle; `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link skipped only when `remote=true && !trustedWorkspace`. Every `Operation` carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`; `sync_brain`, `file_upload`, `file_list`, `file_url` are `admin + localOnly` (rejected over HTTP). Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) use FAIL-CLOSED semantics: `ctx.remote === false` for trusted-only sites, `ctx.remote !== false` for "untrust unless explicit-false" — anything not strictly `false` is treated as remote (closes the HTTP MCP shell-job RCE where a read+write OAuth token could submit `shell` jobs). `sourceScopeOpts(ctx)` encodes the source-scoped read precedence ladder — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId`/`ctx.auth.sourceId`) over nothing; every read-side op handler routes through it so a source-bound OAuth client can't see neighboring sources via `search`/`query`/`list_pages`/`get_page`/`find_experts`/query's image path. `put_page`'s inline disk write-through is the shared `writePageThrough` helper (`src/core/write-through.ts`), ATOMIC via temp-sibling + rename so a crash or concurrent `gbrain sync` can't read a half-written `.md`; same helper backs `gbrain brainstorm/lsd --save`. Link provenance surface (#1941): `add_link` (`gbrain link`/`link-add`) + `remove_link` (`gbrain unlink`/`link-rm`) expose `link_source`/`link_type`; `add_link` rejects the reconciliation-managed built-ins via `MANAGED_LINK_SOURCES` (`markdown`/`frontmatter`/`mentions`/`wikilink-resolved`) and defaults omitted provenance to `'manual'` (the engine's own default stays `'markdown'` for internal callers); `list_link_sources` (`gbrain link-sources`, read) lists provenances via `sourceScopeOpts`. CLI aliases register through `cliHints.aliases` (collision-guarded in `src/cli.ts`). - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput`/`TimelineBatchInput` for the bulk-insert API (`addLinksBatch`/`addTimelineEntriesBatch`). `readonly kind: 'postgres' | 'pglite'` discriminator lets `src/core/migrate.ts` and others branch without `instanceof` + dynamic imports. Methods: `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[],$2::text[],$3::real[])` composite-keyed on `(slug, source_id)`), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` has `sort?: 'updated_desc'|'updated_asc'|'created_desc'|'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines. `listAllPageRefs(): Promise>` ordered by `(source_id, slug)` — cheap cross-source enumeration replacing the `getAllSlugs()→getPage(slug)` N+1 (which silently defaulted to `source_id='default'`); parity across postgres-engine.ts + pglite-engine.ts; Pinned by `test/e2e/multi-source-bug-class.test.ts`. `SearchOpts`+`PageFilters` add `sourceIds?: string[]` (federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when set, preserve scalar `sourceId` fast path when unset); `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId`/`opts.sourceIds`. `traverseGraph` opts has `frontierCap?: number` (per-iteration recursive-CTE cap, approx per-BFS-layer); return type `Promise` for MCP wire stability; export `TraverseGraphOpts`; Postgres uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, PGLite mirrors with positional params; Pinned by `test/regressions/v0_36_frontier_cap.test.ts`. Phantom-redirect methods: `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (content_hash refresh required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug`+`source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence; parity at `test/phantom-redirect-engine-parity.test.ts`. `getAdjacencyBoosts(pageIds): Promise>` powers the per-query graph-signals stage — one SQL query returning inbound-link counts among top-K plus a cross-source count (links from differing `source_id`); `COALESCE(p.source_id,'default')` null safety, `HAVING >= 1`, cross-source CASE-WHEN excludes the target's own source; parity SQL across both engines; `SearchResult` gains optional `base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta` + internal staging fields; Pinned by `test/e2e/graph-signals-engine.test.ts`. Two REQUIRED methods: `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup); `sourceId` REQUIRED on both at the type level (asymmetric with single-row `deletePage` which keeps optional/'default'); both short-circuit on empty input and throw when `> DELETE_BATCH_SIZE`. Embedding-signature stale-detection quartet: `countStaleChunks(opts?)` gains optional `signature?: string` widening the stale predicate from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (NULL signature is GRANDFATHERED, never counted; omit `signature` for the legacy NULL-only count); `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` = `SUM(LENGTH(chunk_text))` over stale chunks (same predicate + embed_skip filter + optional sourceId scope), used by `gbrain sync --all` cost preview via `estimateCostFromChars`; `setPageEmbeddingSignature(slug, {sourceId?, signature})` stamps `pages.embedding_signature` after a page's chunks (re)embed, idempotent no-op when page absent; `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` NULLs `embedding`+`embedded_at` on every chunk whose page signature is set AND differs, returning the count, called BEFORE `listStaleChunks` so signature-drift pages flow through the NULL-embedding keyset cursor unchanged (NULL never invalidated). Widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. Free-text alias layer: `resolveAliases(aliasNorms, opts?): Promise>>` (READ; maps each normalized alias to declaring `(slug, source_id)` pairs, source-scoped) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE; replaces the full alias set, delete-then-insert, empty clears, idempotent on the unique triple), called by the `importFromContent` ingest projection and the `reindex --aliases` backfill; parity across both engines, Pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines injects the shared `buildBestPerPagePoolCte` per-page max-pool so a page surfaces on its strongest chunk. `executeRawDirect(sql, params?, opts?)` is the lock-hot-path sibling of `executeRaw`: same single-statement contract, but routes to the direct session-mode pool when dual-pool is active (Postgres/Supabase port 5432) so a long-held lock heartbeat survives the transaction pooler's per-transaction connection recycling; PGLite delegates straight to `executeRaw` (no pooler). Both engines implement it; the Minion lock path (`claim`/`renewLock`) is the consumer. `reconnect(ctx?: {error?})` is a REQUIRED lifecycle method on BOTH engines: it recovers a dropped connection using the config captured at the last `connect()`, so callers (autopilot health probe, `batchRetry`) never `disconnect()` + bare `connect()` (which loses the config and throws `database_url undefined` forever, and opens a null-connection window). PostgresEngine rebuilds its pool with a `_reconnecting` reentrancy guard and emits pool-recovery audit; PGLite is single-writer in-process so it just restores the saved data dir for parity. - `src/core/engine-constants.ts` — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry — bounded per-statement work for predictable lock hold time + write amplification. -- `src/core/background-work.ts` (#1762/#1745/#1775) — process background-work registry: the single owner of "drain every fire-and-forget DB-write sink before the CLI disconnects." `registerBackgroundWorkDrainer({name, order, drain(timeoutMs), abort?})` + `drainAllBackgroundWorkForCliExit({timeoutMs})` over a `Map` (idempotent registration by name; `__registerDrainerForTest` returns an unregister handle). Drains in explicit `(order, name)` order — facts FIRST (order 0) so its abort-path DB `logIngest` runs against the freshest live engine — and AWAITS `abort()` only when `drain()` reports `unfinished>0`. Best-effort per drainer: one sink's failure never blocks the others or the disconnect. FOUR sinks register at module import (rule-of-four): `facts/queue.ts` (order 0; `abort`=`shutdown()` cancels a hung facts:absorb Haiku via internalAbort), `last-retrieved.ts` (order 1), `search/hybrid.ts` (order 2; `awaitPendingSearchCacheWrites` bounded via `Promise.race`), `eval-capture.ts` (order 3; `captureEvalCandidate` self-tracks its promise via `awaitPendingEvalCaptures`). Every cli.ts teardown site reaches it through `finishCliTeardown` (`src/core/cli-force-exit.ts`), which drains the registry before `engine.disconnect()` — closing the PGLite busy-loop where `db.close()` raced an in-flight job and pinned the single-writer lock (#1762). Exports `backgroundWorkSinkCount()` so the teardown helper computes its backstop deadline from the registered sink count. CLI-EXIT-ONLY: the facts `shutdown()` abort is permanent process state, never call in a long-lived `gbrain serve`. Companion changes: `src/core/ai/gateway.ts` `withDefaultTimeout(caller, ms)` bounds every outbound AI call (chat 300s, embed+multimodal 60s; env `GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS`; composed with caller signals via `AbortSignal.any`) and the teardown backstop honors an errored op's exit code so a hung disconnect can't mask failure as success (see `cli-force-exit.ts`); `src/core/postgres-engine.ts` `reconnect()` module-mode branch re-establishes via idempotent `db.connect()` + `connectionManager.setReadPool` refresh instead of `db.disconnect()` (no null window for concurrent ops; fail-loud on real connect failure — #1745); `src/core/search/hybrid.ts` `embedQueryBounded` + a shared `QueryEmbedDeadline` (6s, floored 2s per embed via `MIN_QUERY_EMBED_BUDGET_MS`; env `GBRAIN_QUERY_EMBED_TIMEOUT_MS`) bounds the cache-lookup AND inner query embeds so a stalled provider falls back to keyword instead of stalling the whole op (#1775). Incorporates + hardens PR #1763 (@ElliotDrel). Pinned by `test/core/background-work.test.ts`, `test/search/query-embed-deadline.test.ts`, `test/eval-capture-drain.test.ts`, `test/e2e/postgres-reconnect-singleton.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts`, `test/fix-wave-structural.test.ts`. +- `src/core/background-work.ts` (#1762/#1745/#1775) — process background-work registry: the single owner of "drain every fire-and-forget DB-write sink before the CLI disconnects." `registerBackgroundWorkDrainer({name, order, drain(timeoutMs), abort?})` + `drainAllBackgroundWorkForCliExit({timeoutMs})` over a `Map` (idempotent registration by name; `__registerDrainerForTest` returns an unregister handle). Drains in explicit `(order, name)` order — facts FIRST (order 0) so its abort-path DB `logIngest` runs against the freshest live engine — and AWAITS `abort()` only when `drain()` reports `unfinished>0`. Best-effort per drainer: one sink's failure never blocks the others or the disconnect. FIVE sinks register at module import: `facts/queue.ts` (order 0; `abort`=`shutdown()` cancels a hung facts:absorb Haiku via internalAbort), `last-retrieved.ts` (order 1), `search/hybrid.ts` (order 2; `awaitPendingSearchCacheWrites` bounded via `Promise.race`), `eval-capture.ts` (order 3; `captureEvalCandidate` self-tracks its promise via `awaitPendingEvalCaptures`), `context/volunteer-events.ts` (order 4, #2095 — batched volunteer-event INSERTs, drained like the rest). Every cli.ts teardown site reaches it through `finishCliTeardown` (`src/core/cli-force-exit.ts`), which drains the registry before `engine.disconnect()` — closing the PGLite busy-loop where `db.close()` raced an in-flight job and pinned the single-writer lock (#1762). Exports `backgroundWorkSinkCount()` so the teardown helper computes its backstop deadline from the registered sink count. CLI-EXIT-ONLY: the facts `shutdown()` abort is permanent process state, never call in a long-lived `gbrain serve`. Companion changes: `src/core/ai/gateway.ts` `withDefaultTimeout(caller, ms)` bounds every outbound AI call (chat 300s, embed+multimodal 60s; env `GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS`; composed with caller signals via `AbortSignal.any`) and the teardown backstop honors an errored op's exit code so a hung disconnect can't mask failure as success (see `cli-force-exit.ts`); `src/core/postgres-engine.ts` `reconnect()` module-mode branch re-establishes via idempotent `db.connect()` + `connectionManager.setReadPool` refresh instead of `db.disconnect()` (no null window for concurrent ops; fail-loud on real connect failure — #1745); `src/core/search/hybrid.ts` `embedQueryBounded` + a shared `QueryEmbedDeadline` (6s, floored 2s per embed via `MIN_QUERY_EMBED_BUDGET_MS`; env `GBRAIN_QUERY_EMBED_TIMEOUT_MS`) bounds the cache-lookup AND inner query embeds so a stalled provider falls back to keyword instead of stalling the whole op (#1775). Incorporates + hardens PR #1763 (@ElliotDrel). Pinned by `test/core/background-work.test.ts`, `test/search/query-embed-deadline.test.ts`, `test/eval-capture-drain.test.ts`, `test/e2e/postgres-reconnect-singleton.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts`, `test/fix-wave-structural.test.ts`. - `src/core/search/graph-signals.ts` — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring at full score, demote the rest). All three inherit the floor-ratio gate preventing weak pages from being boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width`. `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (incl. the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension — `runPostFusionStages` has a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Each post-fusion stage stamps its multiplier: `applyBacklinkBoost`→`backlink_boost`, `applySalienceBoost`→`salience_boost`, `applyRecencyBoost`→`recency_boost`. `applyReranker` (earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution powers `gbrain search --explain` — every boost surface carries its own field so `formatResultsExplain` reads them all without coupling to internal stage ordering. - `src/core/search/explain-formatter.ts` — renders `SearchResult[]` as a multi-line per-result breakdown for `gbrain search --explain`. Reads every boost-stamping field. Handles the "no boosts applied" empty path. 4-decimal precision with trailing-zero strip. Pinned by `test/search/explain-formatter.test.ts`. - `src/core/search/mode.ts` extension — `graph_signals: boolean` knob in `ModeBundle` (defaults: `conservative=false`, `balanced=true`, `tokenmax=true`). `KNOBS_HASH_VERSION` appends a `gs=` parts entry per the cache-key contamination convention so a graph-on cache write can't be served to a graph-off lookup. `SearchKeyOverrides` + `SearchPerCallOpts` + `loadOverridesFromConfig` + `SEARCH_MODE_CONFIG_KEYS` + `resolveSearchMode` + `attributeKnob` all carry the field. Opt-out: `gbrain config set search.graph_signals false`. Mid-deploy `query_cache` rows from before the upgrade hash differently — natural row segregation, clears within `cache.ttl_seconds` (3600s default). -- `src/core/context-engine.ts` + `src/openclaw-context-engine.ts` — the deterministic context engine OpenClaw loads on every turn (`assemble()` injects the Live Context block, zero-LLM). `createGBrainContextEngine({workspaceDir, resolveEntities?})` accepts an OPTIONAL host-injected resolver (`ENGINE_API_VERSION` 0.2.0, additive — older hosts work unchanged; the plugin entry maps `ctx.resolveEntities`/`ctx.brainQuery` onto it). `assemble()` runs the Retrieval Reflex after the Live Context block: extracts the current turn's user text, builds prior-context text (every message EXCEPT the current turn — suppression must not see the triggering mention), and appends the pointer block. `warmReflex()` fires at construction. -- `src/core/context/` — Retrieval Reflex (Layer 1, issue #1981). `entity-salience.ts`: pure, zero-LLM, precision-biased `extractCandidates(text)` (capitalized runs + `@handles`, STOPWORDS + soft COMMON_WORDS + sentence-start guard, deterministic, capped). `retrieval-reflex.ts`: `resolveEntitiesToPointers(engine, sourceId, candidates, opts)` — alias arm (`resolveAliases`, caught per-arm for pre-v110 brains) + exact title/slug-suffix arm (the recall fix: real slugs are namespaced `people/x` but `slugify` drops the prefix); synopsis runs through `stripTakesFence`/`stripFactsFence` (the same privacy boundary `get_page` applies) so private facts never reach the prompt; suppression scans `priorContextText` only; capped at `MAX_POINTERS`. `reflex.ts`: the orchestrator + engine-aware resolver ladder (host `resolveEntities` → PGLite serve IPC → Postgres cached process-singleton → disabled), zero-candidate fast path, fail-open + timeout, heartbeat write for the doctor check, `reflexEnabled(cfg)` (file/env gate, default ON; DB-plane does NOT gate — `assemble()` is sync). `resolve-ipc.ts`: local unix-socket resolve protocol (client + server) so PGLite resolves through the single connection `gbrain serve` holds (a second opener would hit the exclusive lock; a subprocess would force-steal it past the 5-min staleness window and crash). Wired into `src/mcp/server.ts` (serve binds `/.gbrain-resolve.sock` on PGLite, cleaned up on shutdown). Doctor surface: `retrieval_reflex_health` in `src/commands/doctor.ts` (reads the heartbeat for truthful runtime status; categorized in `doctor-categories.ts`). Config: `retrieval_reflex` + `retrieval_reflex_max_pointers` in `src/core/config.ts`. Policy layer ships as the `retrieval-reflex` recipe (`recipes/retrieval-reflex/`). Pinned by `test/context/entity-salience.test.ts`, `test/retrieval-reflex.test.ts`, `test/context/resolve-ipc.test.ts`, `test/doctor-retrieval-reflex.test.ts`. +- `src/core/context-engine.ts` + `src/openclaw-context-engine.ts` — the deterministic context engine OpenClaw loads on every turn (`assemble()` injects the Live Context block, zero-LLM). `createGBrainContextEngine({workspaceDir, resolveEntities?})` accepts an OPTIONAL host-injected resolver (`ENGINE_API_VERSION` 0.2.0, additive — older hosts work unchanged; the plugin entry maps `ctx.resolveEntities`/`ctx.brainQuery` onto it). `assemble()` runs the Retrieval Reflex after the Live Context block: extracts the current turn's user text, builds prior-context text (every message EXCEPT the current turn — suppression must not see the triggering mention), passes the rolling window (`getWindowTurns`, last 12 user/assistant turns; the reflex slices to its configured `retrieval_reflex_window_turns`), and appends the pointer block. `warmReflex()` fires at construction. +- `src/core/context/` — Retrieval Reflex (Layer 1, issue #1981). `entity-salience.ts`: pure, zero-LLM, precision-biased `extractCandidates(text)` (capitalized runs + `@handles`, STOPWORDS + soft COMMON_WORDS + sentence-start guard, deterministic, capped) + `extractCandidatesFromWindow(turns)` (#2095: merges per-turn extraction across the last N turns by normalizeAlias form with occurrence/newest-turn/user-mention metadata; salience-ordered — recency > frequency > user-role — so the cap drops stale assistant chatter first). `retrieval-reflex.ts`: `resolveEntitiesToPointers(engine, sourceId, candidates, opts)` — alias arm (`resolveAliases`, caught per-arm for pre-v110 brains) + exact title/slug-suffix arm (the recall fix: real slugs are namespaced `people/x` but `slugify` drops the prefix); pointers carry `source_id`/`arm`/`confidence`/`matchedNorm` (#2095 — `ARM_CONFIDENCE` alias 0.9 / title 0.8 / slug-suffix 0.6 lives next to the arm definitions; arm-2 provenance classified in JS since the combined OR can't report which predicate matched); opts: `sourceIds?` federated scope (alias arm loops per source, arm 2 uses `source_id = ANY`), `suppression?` ('slug-and-title' legacy default; 'slug-only' REQUIRED under windowing — the title rule would suppress every entity merely mentioned in a prior window turn), ambient-channel event logging is DELIVERY-side, not in-resolver — `logDeliveredReflexPointers(engine, pointers)` fires only once a block is actually handed to the consumer (serve's resolve-IPC `onDelivered` hook post-write; `buildReflexAddition` post-timeout on the direct rung), so abandoned/timed-out blocks never pollute the volunteered-vs-used stats; synopsis runs through `stripTakesFence`/`stripFactsFence` (the same privacy boundary `get_page` applies) so private facts never reach the prompt; capped at `MAX_POINTERS`. `reflex.ts`: the orchestrator + engine-aware resolver ladder (host `resolveEntities` → PGLite serve IPC → Postgres cached process-singleton → disabled), zero-candidate fast path, fail-open + timeout, heartbeat write for the doctor check, `reflexEnabled(cfg)` (file/env gate, default ON; DB-plane does NOT gate — `assemble()` is sync); windowed extraction when `windowTurns` present and `retrieval_reflex_window_turns` (default 4; 1 = exact legacy behavior) > 1 — switches suppression to slug-only; accept-side reflex-channel logging fires after the per-turn timeout admits the block (direct-Postgres rung only — IPC logs server-side at delivery; host-injected resolvers are a documented gap). `resolve-ipc.ts`: local unix-socket resolve protocol (client + server) so PGLite resolves through the single connection `gbrain serve` holds (a second opener would hit the exclusive lock; a subprocess would force-steal it past the 5-min staleness window and crash). Wired into `src/mcp/server.ts` (serve binds `/.gbrain-resolve.sock` on PGLite, cleaned up on shutdown). Doctor surface: `retrieval_reflex_health` in `src/commands/doctor.ts` (reads the heartbeat for truthful runtime status; categorized in `doctor-categories.ts`). Config: `retrieval_reflex` + `retrieval_reflex_max_pointers` + `retrieval_reflex_window_turns` in `src/core/config.ts` (env `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`). `volunteer.ts` (#2095): `parseWindow` (lenient `user:`/`assistant:` prefixes, unprefixed → one user turn), `volunteerContext` (extract → resolve → +0.05 multi-turn/newest-turn boost → `min_confidence` 0.7 gate → cap 3/5; deterministic rationale strings, never raw conversation text; slug-only suppression), `volunteerUsageStats` (per-arm/channel precision from the `pages.last_retrieved_at > volunteered_at` join — APPROXIMATE: the 5-min last-retrieved throttle causes false negatives, unrelated reads false positives). `volunteer-events.ts` (#2095): `insertVolunteerEvents` (ONE multi-row parameterized INSERT), `logVolunteerEventsFireAndForget` + bounded drain registered as the `volunteer-events` background-work sink (order 4), `purgeStaleVolunteerEvents` (90-day GC, called from the dream cycle's purge phase). Policy layer ships as the `retrieval-reflex` recipe (`recipes/retrieval-reflex/`). Pinned by `test/context/entity-salience.test.ts`, `test/retrieval-reflex.test.ts`, `test/context/resolve-ipc.test.ts`, `test/doctor-retrieval-reflex.test.ts`, `test/volunteer-context.test.ts`, `test/e2e/volunteer-context-postgres.test.ts`. +- `src/commands/watch.ts` — `gbrain watch` (#2095): the push transport. Reads turns from stdin as they arrive (`user:`/`assistant:` prefixes; unprefixed = user turn), keeps a rolling in-process window (`--window-turns`, default 4), calls `volunteerContext` per turn, streams pointers to stdout (`--json` for JSONL with turn attribution), logs `channel: 'watch'` events with a per-session id. Session dedupe feeds already-pushed slugs back as priorContext so the core's slug-only suppression dedupes. Blocks in the stdin iteration (interactive alive until Ctrl-C/Ctrl-D; piped ends at EOF) — deliberately NOT in DAEMON_COMMANDS; SIGINT closes the stream so teardown flows through finishCliTeardown. Per-turn resolution failures are fail-open. Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + THIN_CLIENT_REFUSED_COMMANDS (thin clients use the `volunteer_context` MCP op). Pinned by `test/watch-command.test.ts`. - `src/commands/integrations.ts` — recipe install. The resolver-row install fence is keyed by `manifest.recipe` (`gbrain::resolver-rows`), so a second `copy-into-host-repo` recipe no longer writes a block mislabeled with the first recipe's name. Pinned by `test/integrations-install.test.ts`. - `src/core/audit/audit-writer.ts` — shared JSONL audit primitive consolidating the hand-rolled audit modules. Exports `createAuditWriter({kind, recordSchema})` returning `{log, readRecent}` plus shared helpers `computeIsoWeekFilename(kind, now?)` and `resolveAuditDir()` (honors `GBRAIN_AUDIT_DIR`). ISO-week file rotation; best-effort writes (stderr warn on failure, never throws); read-path scans current-week + previous-week files for boundary spans. Refactored onto it for parity: `src/core/rerank-audit.ts`, `src/core/audit-slug-fallback.ts`, `src/core/minions/handlers/shell-audit.ts`, `src/core/minions/handlers/supervisor-audit.ts`, `src/core/facts/phantom-audit.ts` (each module's public API preserved bit-for-bit). The `graph-signals-failures` audit (`logGraphSignalsFailure`) uses the same primitive. One hand-rolled audit remains at `src/core/skillpack/audit.ts`. Pinned by `test/audit/audit-writer.test.ts`. -- `src/core/cli-force-exit.ts` (#2084) — single owner of one-shot CLI exit + teardown, designed as a PAIR with the `import.meta.main` seam at the bottom of `src/cli.ts`. `finishCliTeardown({engine, drainTimeoutMs?})` is teardown-ONLY (never exits on the clean path): arms a REF'D backstop (unref'd would let a hung teardown exit naturally, skipping the flush and exiting with whatever PGLite scribbled into `process.exitCode`) whose deadline is COMPUTED from the bounds it guards (`computeTeardownDeadlineMs` = sinks × drainTimeoutMs + facts-abort grace + 2 × pool-end bound + slack, floor 10s; `GBRAIN_TEARDOWN_DEADLINE_MS` env override is the incident escape hatch), drains every background-work sink, disconnects the engine (a throw is warned + swallowed — the exit code reports the OPERATION, not the cleanup), then returns. The exit VERDICT lives in a gbrain-owned channel (`setCliExitVerdict`/`currentExitCode`; mirror-writes `process.exitCode` but NEVER reads it back) because PGLite's Emscripten runtime scribbles its own status into `process.exitCode` at arbitrary points mid-run — every writer that means to set the CLI exit code (op-dispatch catch, reindex, frontmatter, transcripts, brainstorm, autopilot) calls `setCliExitVerdict`. The deadline arms at TEARDOWN start, never before the op handler (the pre-#2084 placement measured handler + teardown combined, so PgBouncer deployments paid a flat 10s force-exit tax on every query and any >10s op was killed mid-run with exit 0). All nine cli.ts disconnect sites route through it; the ONE process exit happens in cli.ts's `main().then/catch` via `flushThenExit(currentExitCode())`, gated by `shouldForceExitAfterMain()` (daemon list: `serve`) — the CLI never waits for Bun's event loop to drain, because `endPoolBounded` deliberately races past stuck PgBouncer sockets that would keep it alive. `flushThenExit(code)` fences stdout+stderr (`write('', cb)` raced with an unref'd guard, EPIPE-safe both sync and async) then holds a REF'D aliveness grace for non-TTY stdio before `process.exit` — Bun delivers queued pipe writes only while the process is alive (no flush API reaches `process.stdout`'s native queue; write callbacks fire on accept, not delivery), so the grace IS the flush (#1959 truncation class). Scope claim is deliberately cli.ts-only: command modules' mid-run engine lifecycles stay local (process-exit semantics inside them would be wrong) and are absorbed by the final explicit exit. Pinned by `test/cli-finish-teardown.test.ts`, `test/flush-then-exit-harness.test.ts` (real spawned-Bun pipe semantics), `test/cli-should-force-exit.test.ts`, the `#2084` describes in `test/fix-wave-structural.test.ts` + `test/e2e/pglite-cli-exit.serial.test.ts`. +- `src/core/cli-force-exit.ts` (#2084) — single owner of one-shot CLI exit + teardown, designed as a PAIR with the `import.meta.main` seam at the bottom of `src/cli.ts`. `finishCliTeardown({engine, drainTimeoutMs?})` is teardown-ONLY (never exits on the clean path): arms a REF'D backstop (unref'd would let a hung teardown exit naturally, skipping the flush and exiting with whatever PGLite scribbled into `process.exitCode`) whose deadline is COMPUTED from the bounds it guards (`computeTeardownDeadlineMs` = sinks × drainTimeoutMs + facts-abort grace + 2 × pool-end bound + slack, floor 10s; `GBRAIN_TEARDOWN_DEADLINE_MS` env override is the incident escape hatch), drains every background-work sink, disconnects the engine (a throw is warned + swallowed — the exit code reports the OPERATION, not the cleanup), then returns. The exit VERDICT lives in a gbrain-owned channel (`setCliExitVerdict`/`currentExitCode`; mirror-writes `process.exitCode` but NEVER reads it back) because PGLite's Emscripten runtime scribbles its own status into `process.exitCode` at arbitrary points mid-run — every writer that means to set the CLI exit code (op-dispatch catch, reindex, frontmatter, transcripts, brainstorm, autopilot, doctor's FAIL verdict, extract, and cli.ts's swept inner exits — friction, claw-test, smoke-test, the no-DB eval runners, status/status-thin, whoknows-thin) calls `setCliExitVerdict`; `test/cli-exit-verdict-pin.test.ts` greps src/ so the next raw `process.exitCode =` write fails CI instead of silently reporting success on failure. The deadline arms at TEARDOWN start, never before the op handler (the pre-#2084 placement measured handler + teardown combined, so PgBouncer deployments paid a flat 10s force-exit tax on every query and any >10s op was killed mid-run with exit 0). All nine cli.ts disconnect sites route through it; the ONE process exit happens in cli.ts's `main().then/catch` via `flushThenExit(currentExitCode())`, gated by `shouldForceExitAfterMain()` (daemon list: `serve`) — the CLI never waits for Bun's event loop to drain, because `endPoolBounded` deliberately races past stuck PgBouncer sockets that would keep it alive. `flushThenExit(code)` fences stdout+stderr (`write('', cb)` raced with an unref'd guard, EPIPE-safe both sync and async) then holds a REF'D aliveness grace for non-TTY stdio before `process.exit` — Bun delivers queued pipe writes only while the process is alive (no flush API reaches `process.stdout`'s native queue; write callbacks fire on accept, not delivery), so the grace IS the flush (#1959 truncation class). Scope claim is deliberately cli.ts-only: command modules' mid-run engine lifecycles stay local (process-exit semantics inside them would be wrong) and are absorbed by the final explicit exit. Pinned by `test/cli-finish-teardown.test.ts`, `test/flush-then-exit-harness.test.ts` (real spawned-Bun pipe semantics), `test/cli-should-force-exit.test.ts`, `test/cli-pipe-truncation.test.ts` (real-CLI piped --tools-json byte-stable), `test/cli-exit-verdict-pin.test.ts`, the `#2084` describes in `test/fix-wave-structural.test.ts` + `test/e2e/pglite-cli-exit.serial.test.ts`, and `test/e2e/pgbouncer-teardown.test.ts` (CI transaction-mode pooler — the #1972/#2015/#2084 class, finally reproducible in CI). - `src/core/cli-options.ts` extension — `CliOptions` gains `explain: boolean`. `parseGlobalFlags` recognizes `--explain` anywhere in argv (stripped before command dispatch). `src/cli.ts` `formatResult` for `search` + `query` cases routes to `formatResultsExplain` from `src/core/search/explain-formatter.ts` when `CliOptions.explain` is set; falls through to the existing JSON / human formatters otherwise. - `src/commands/search.ts:gbrain search stats` extension — `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property; `_meta.metric_glossary` adds `graph_signals.enabled` + `graph_signals.failures_by_reason`. Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts`. @@ -34,7 +35,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/embedding-pricing.ts` — `EMBEDDING_PRICING` map keyed `provider:model` for the post-upgrade reindex cost estimate. Sibling to `anthropic-pricing.ts`. Entries: OpenAI text-embedding-3-large ($0.13/1M), 3-small ($0.02/1M), ada-002 ($0.10/1M), Voyage 3-large ($0.18/1M), 3 ($0.06/1M). `lookupEmbeddingPrice(modelString)` returns a tagged union (`known` with price + `unknown` with provider name); `estimateCostFromChars(charCount, pricePerMTok)` uses 3.5 chars/token. Unknown providers degrade to "estimate unavailable" instead of fabricating numbers. - `src/core/post-upgrade-reembed.ts` — Pure functions backing the `gbrain upgrade` chunker-bump cost prompt. `computeReembedEstimate(engine, model)` queries real SQL (`COUNT(*)` + `COALESCE(SUM(LENGTH(compiled_truth)) + SUM(LENGTH(timeline)), 0)`) on `pages WHERE chunker_version < MARKDOWN_CHUNKER_VERSION`. `formatReembedPrompt(est, graceSeconds)` is the stderr-line formatter. `runPostUpgradeReembedPrompt(engine, model, opts)` orchestrates the 10-second Ctrl-C window; TTY-only wait (non-TTY auto-proceeds for CI / cron); `GBRAIN_NO_REEMBED=1` bails with a doctor-warning marker; `GBRAIN_REEMBED_GRACE_SECONDS=0` skips the wait. - `src/commands/reindex.ts` — `gbrain reindex --markdown [--limit N] [--dry-run] [--json] [--no-embed] [--repo PATH]`. Walks `pages WHERE page_kind = 'markdown' AND chunker_version < MARKDOWN_CHUNKER_VERSION` in 100-row batches ordered by id. Rows with non-null `source_path` re-import via `importFromFile`; rows without fall back to `importFromContent`. Both paths pass `forceRechunk: true` to bypass `importFromContent`'s `content_hash` short-circuit — without it the chunker version bump never reaches pages whose source content hasn't changed, AND the stripFactsFence privacy strip never applies to pre-strip chunks. Idempotent — partial-completion re-runs pick up via id-ordered batches. Wired into `src/commands/upgrade.ts:runPostUpgrade` after `apply-migrations`. The DB-only fallback (no source file on disk) does NOT pass body-only `compiled_truth` to `importFromContent` (that path re-parses with EMPTY frontmatter and OVERWRITES the page's real frontmatter/title/timeline); it `getPage`+`getTags`, reconstructs FULL markdown via `serializeMarkdown(frontmatter, compiled_truth, timeline, {type, title, tags})`, and re-imports THAT so re-chunking a DB-only page preserves everything while bumping `chunker_version`. Pinned by `test/reindex-preserve-tags.test.ts`. -- `src/commands/reindex-code.ts` — `gbrain reindex --code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks `pages WHERE type = 'code'` in 100-row batches, replays through `importCodeFile` for chunk + embed + content_hash folding. Idempotent unless `--force` bypasses the content_hash early-return. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). When `--yes` is absent and the caller is non-TTY or passed `--json`, the cost gate refuses (exit 2, no spend) via the pure exported `buildCostRefusal({json, previewMsg, preview, costUsd, model}): {stdout?, stderr?}` — JSON envelope only when `--json` is explicit, otherwise a human refusal on stderr (the spend guardrail is independent of the output format). Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE regression for the cost-preview fix), `test/reindex-cost-refusal.test.ts`. +- `src/commands/reindex-code.ts` — `gbrain reindex --code [--source ID] [--dry-run] [--yes] [--json] [--force] [--no-embed]`. Walks `pages WHERE type = 'code'` in 100-row batches, replays through `importCodeFile` for chunk + embed + content_hash folding. Idempotent unless `--force` bypasses the content_hash early-return. Cost-preview model field reads `getEmbeddingModelName()` from the gateway so preview reflects what the gateway will actually embed with. An informational stderr nudge inside `runReindexCode` (so dry-run + execute both surface it): when the configured embedding model isn't code-tuned (allowlist `{'voyage-code-3'}`, case-insensitive bare match), prints a recommendation to switch to `voyage:voyage-code-3`; suppress with `GBRAIN_NO_CODE_MODEL_NUDGE=1`, `--no-embed`, or `--json`. Pure `shouldNudgeCodeModel(bareName)` returns a tagged `NudgeDecision` union (takes the bare model name, emits qualified `voyage:voyage-code-3` for the paste-ready `gbrain config set` line). When `--yes` is absent and the caller is non-TTY or passed `--json`, the cost gate refuses (exit 2, no spend) via the pure exported `buildCostRefusal({json, previewMsg, preview, costUsd, model}): {stdout?, stderr?}` — JSON envelope only when `--json` is explicit, otherwise a human refusal on stderr (the spend guardrail is independent of the output format). `spend.posture=tokenmax` OR an explicit `--max-cost off`/`unlimited` makes the gate informational and proceeds (#2139); `--max-cost off` also disables the runtime BudgetTracker cap. Pinned by `test/ai/voyage-code-3-recipe.test.ts`, `test/reindex-code-nudge.serial.test.ts`, `test/reindex-code-model-source.serial.test.ts` (IRON-RULE regression for the cost-preview fix), `test/reindex-cost-refusal.test.ts`. - `src/commands/sync.ts:resolveSlugByPathOrSourcePath` — Resolves a slug by `pages.source_path` first (returns the stored slug for frontmatter-fallback pages whose path doesn't derive a slug), then falls back to `resolveSlugForPath(path)`. Threaded into all 4 delete/rename call sites (`performSync`'s un-syncable cleanup at ~:531, deletes at ~:603, rename oldSlug at ~:622). Without this, emoji-only / Thai / Arabic filenames whose slug came from frontmatter would orphan on delete/rename (the delete path would compute the wrong path-derived slug). Best-effort query — pre-migration brains fall through to the legacy path. - `src/core/sources-ops.ts` — Multi-source registration + clone-lifecycle ops (`addSource`, `recloneIfMissing`, `defaultCloneDir`, `isOwnedClone`, `unownedHint`). **Reclone-ownership invariant (must-never-violate): gbrain may only delete/re-clone a clone it created, NEVER a user working tree.** `recloneIfMissing` deletes `local_path`, so it gates on `isOwnedClone(src)` and throws a `SourceOpError('unmanaged_path', ...)` BEFORE any filesystem op when ownership is unprovable — fail-closed. Ownership is proven by `config.managed_clone === true` (written by `addSource`'s `--url` path, covering default-location and `--clone-dir` clones) OR `local_path === defaultCloneDir(id)` (back-compat for pre-marker clones, via exact normalized-path equality, symlink-free). A row with `remote_url` + an unowned `local_path` (a user-registered working tree, e.g. `sources add --path`) is refused untouched; re-add with `--url` to regain auto-reclone. The reclone is EXDEV-safe: clone into a SIBLING temp of `local_path` (not the shared `clones/.tmp`, which may sit on a different mount than a `--clone-dir` target), then swap (move old aside → move new in → drop old) so `local_path` is never left missing-and-unrecoverable; on swap failure the original is restored, and if restore fails the error names the `aside` path so it's never reflexively deleted. A TOCTOU re-check re-confirms ownership immediately before the destructive move and rejects a symlink leaf swapped in after the entry check (`symlink_escape`). `unownedHint(src, state)` is the shared recovery message used by both the core error and the `gbrain sync --source` CLI error; `gbrain sources restore` special-cases `unmanaged_path` to print "DB row restored; gbrain syncs this path read-only" instead of the misleading "try sync to recover" guidance. `SourceOpErrorCode` includes `unmanaged_path`. Pinned by `test/sources-ops.test.ts`, `test/sources-resync-recovery.test.ts`. - `src/core/utils.ts` — Shared SQL utilities extracted from postgres-engine.ts. Exports `parseEmbedding(value)` (throws on unknown input, used by migration + ingest paths where data integrity matters) and `tryParseEmbedding(value)` (returns `null` + warns once per process, used by search/rescore paths where availability matters more than strictness). `isUndefinedColumnError(err)` predicate — pattern-matches Postgres SQLSTATE 42703 / "column ... does not exist" with engine-driver shape variation tolerated; replaces bare `catch {}` blocks in `oauth-provider.ts` so genuine errors (lock timeout, network blip, permission denied) propagate while column-missing falls through to the legacy fallback. `validateSourceId(id)` throws on anything outside `^[a-z0-9_-]+$`, used by the per-source disk-layout fix in patterns.ts/synthesize.ts before any `join(brainDir, '.sources', source_id, slug+'.md')` so source_id can't traverse out of brainDir. `rowToPage` populates the required `Page.source_id` from the SELECT projection (`scripts/check-source-id-projection.sh` enforces every projection feeding `rowToPage` includes the column). @@ -48,7 +49,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/disk-walk.ts` — `walkBrainRepo(repoPath)` returns `Map` from one recursive `readdirSync`. Skips dot-dirs, `node_modules`, non-`.md` files. Used by `gbrain storage status` to replace per-page `existsSync + statSync` (~400K syscalls on 200K-page brains → tens). - `src/core/git-head.ts` — local git HEAD freshness probe for `gbrain doctor`. `isSourceUnchangedSinceSync(localPath, lastCommit, opts?)` returns true iff `localPath` is a git repo whose current HEAD matches `lastCommit`; when `opts.requireCleanWorkingTree` is true also requires a clean working tree (mirrors `gbrain sync`'s force-walk gate at `sync.ts:1075` so doctor and sync agree on "is there work to do?"). `requireCleanWorkingTree` is `boolean | 'ignore-untracked'` — in `'ignore-untracked'` mode the clean probe runs `git status --porcelain --untracked-files=no` so a quiet repo with stray untracked dirs (`?? companies/`, `?? media/`) is still "unchanged" (sync's incremental path keys off the commit diff and never imports untracked files); `GitCleanProbe` gains an `ignoreUntracked?` second arg. Two probe seams (`_setGitHeadProbeForTests`, `_setGitCleanProbeForTests`) keep unit tests R2-compliant (no `mock.module`). Uses `execFileSync` with array args so shell metachars in `local_path` cannot escape to a shell (the regression test runs real `execFileSync` against `'/nonexistent/$(touch )/repo'` and asserts the sentinel is never created). Fail-open on every error (missing path, not a git repo, git not installed, timeout, NULL inputs, dirty-probe errored → false) preserving the caller's prior time-based behavior. The chunker-version-match check lives in the caller (doctor.ts) because it depends on engine state (`sources.chunker_version` vs `CHUNKER_VERSION` from `src/core/chunkers/code.ts`). Pinned by `test/core/git-head.test.ts` (incl. the shell-injection regression guard). - `src/core/source-health.ts` — per-source health metrics for `gbrain sources status` + doctor's `federation_health`. Commit-relative staleness: `newestCommitMs(localPath)` = HEAD committer time via `git log -1 --format=%ct` (fail-open null; NO working-tree mtime parsing — committed content only, robust against the porcelain-mtime bug farm); pure `lagFromContentMs(contentMs|null, lastSyncMs|null, nowMs)` = remote/column comparator (null lastSync → null; negative wall-clock → skew passthrough; `contentMs <= lastSync` → 0; else/null-content → wall-clock). `computeAllSourceMetrics(engine, sources, {probeContent?})`: LOCAL (`probeContent:true`, `gbrain sources status`) → `isSourceUnchangedSinceSync(..., {requireCleanWorkingTree:'ignore-untracked'}) ? 0 : wall-clock` (live commit-hash catches HEAD moving to an old-dated commit a timestamp compare would miss); REMOTE (default, `federation_health` on the HTTP MCP path) → `lagFromContentMs(row.newest_content_at, ...)`, NO git subprocess (trust boundary). `commitTimeMs(localPath, sha)` is the `newestCommitMs` sibling pinned to an arbitrary commit (committer time via `git show -s --format=%ct `, fail-open null, execFileSync array args) — the resumable sync stamps `newest_content_at` against its pinned target commit, not whatever HEAD raced to. Pinned by `test/source-health.test.ts`. -- `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo` and `pullRepo`. Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git clone --depth=1 [--branch X] -- `. `pullRepo` argv: `git -C pull --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). +- `src/core/git-remote.ts` — SSRF-hardened git invocations for remote-source `cloneRepo`, `pullRepo`, and `fetchRemote(repoPath, branch)` (the last added for the sync cost-estimator's fetch-first path, #2139, so a cost preview / dry-run fetches through the same hardened flags + `GIT_TERMINAL_PROMPT=0` as real sync rather than a less-protected route). Exports two distinct flag constants because `git`'s argv grammar treats them differently: `GIT_SSRF_FLAGS` (3 `-c` config flags — `protocol.allow=user`, `protocol.file.allow=never`, `http.allowRedirects=false`) is global config, spread BEFORE the subcommand verb; `GIT_SSRF_SUBCOMMAND_FLAGS = ['--no-recurse-submodules']` is subcommand-scoped, spread AFTER the verb (a combined array would spread `--no-recurse-submodules` before the verb where real git rejects it exit 129). `cloneRepo` argv: `git clone --depth=1 [--branch X] -- `. `pullRepo` argv: `git -C pull --ff-only`. Pinned by `test/git-remote.test.ts` position-anchored regression guard (`argv.indexOf('--no-recurse-submodules') > argv.indexOf(verb)`). - `src/commands/storage.ts` — `gbrain storage status [--repo P] [--json]`. Split into pure data (`getStorageStatus`) + JSON formatter + human formatter (ASCII-only) matching the `orphans.ts` pattern. `PageCountsByTier` and `DiskUsageByTier` are distinct nominal types so swaps fail at compile time. - `gbrain.yml` (brain repo root) — Optional storage tiering config. Top-level `storage:` section with `db_tracked:` and `db_only:` array-valued keys. `gbrain sync` auto-manages `.gitignore` for `db_only` paths on successful sync (skips on dry-run, blocked-by-failures, submodule context, or `GBRAIN_NO_GITIGNORE=1`). `gbrain export --restore-only [--repo P] [--type T] [--slug-prefix S]` repopulates missing `db_only` files from the database. - `src/core/supabase-admin.ts` — Supabase admin API (project discovery, pgvector check). @@ -101,7 +102,9 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/search/hybrid.ts` — Cathedral II `Promise` return shape. `onMeta?: (m: HybridSearchMeta) => void` callback so op-layer capture records what hybridSearch actually did; existing callers leave it undefined. `HybridSearchOpts.types?: PageType[]` (on `SearchOpts`) threads a multi-type filter into per-engine `searchKeyword` + `searchVector` + `searchKeywordChunks` as `AND p.type = ANY($N::text[])` (primary consumer `gbrain whoknows`, filters to `['person','company']`); AND-applies alongside the single-value `type` filter. `hybridSearch` resolves the embedding column at the boundary via `resolveColumn(loadRegistry(cfg), opts.embedding_column, cfg)` from `src/core/search/embedding-column.ts`, threads the `ResolvedColumn` descriptor (not a raw string) into per-engine `searchVector`, and uses `isCacheSafe(resolved, cfg)` for the cache-skip decision so a repointed `embedding` builtin doesn't leak across vector spaces. `cosineReScore` calls `engine.getEmbeddingsByChunkIds(ids, resolved.name)` so rerank uses vectors from the active column, not the hardcoded OpenAI `embedding`. The `query` MCP op accepts `embedding_column` for per-call A/B; `search` (keyword-only) rejects it. Two post-fusion stages + evidence stamp: `applyTitleBoost(results, query, titleBoost, floorThreshold)` multiplies a result's score by the resolved `title_boost` when `isTitlePhraseMatch` fires, stamps `title_match_boost`, inherits the floor-ratio gate so a title match can't shove a much-stronger page below it; `applyAliasHop(engine, results, query, opts)` normalizes the query, calls `engine.resolveAliases`, and on exact normalized-alias match surfaces that page at top-of-organic + epsilon with `alias_hit=true`; `stampEvidence(...)` runs LAST (after the alias hop, before slice) on every path — keyword-only, no-embed, and full hybrid — so MCP callers and `--explain` read the same `evidence` + `create_safety` contract. `title_boost` resolved from the mode bundle and threaded in. - `docs/eval-capture.md` — stable NDJSON schema reference for gbrain-evals consumers. - `test/public-exports.test.ts` — runtime contract test (R2). Imports each of the 17 public subpaths via package name and pins a canary symbol per module. Paired with `scripts/check-exports-count.sh`. -- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. `BATCH_SIZE=100` (per-recipe pre-split + recursive halving + adaptive shrink-on-miss live in the gateway; the outer paginator is for progress-callback granularity, not batch protection). `estimateEmbeddingCostUsd(tokens)` prices against the currently-configured model's rate via `currentEmbeddingPricePerMTok()` (resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate 0.13 only when the gateway is unconfigured or the model is unknown to the pricing table). `EMBEDDING_COST_PER_1K_TOKENS` retained for back-compat with direct importers/tests. `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from current and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so cost gate and embed decision can't drift. `shouldBlockSync(costUsd, floorUsd, mode): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. +- `src/core/embedding.ts` — OpenAI text-embedding-3-large, batch, retry, backoff. `BATCH_SIZE=100` (per-recipe pre-split + recursive halving + adaptive shrink-on-miss live in the gateway; the outer paginator is for progress-callback granularity, not batch protection). `estimateEmbeddingCostUsd(tokens)` prices against the currently-configured model's rate via `currentEmbeddingPricePerMTok()` (resolves the per-1M-token rate via `lookupEmbeddingPrice(gatewayGetModel())` from `embedding-pricing.ts`, falling back to the OpenAI 3-large rate 0.13 only when the gateway is unconfigured or the model is unknown to the pricing table). `EMBEDDING_COST_PER_1K_TOKENS` retained for back-compat with direct importers/tests. `currentEmbeddingSignature(): string` returns the embedding-provenance signature `:` (e.g. `openai:text-embedding-3-large:1536`) stamped onto `pages.embedding_signature` at every embed-write site; DELIBERATELY excludes the chunker version (tracked separately via `pages.chunker_version`) — this signature is strictly the EMBEDDING space, so a model OR dimension swap makes the stored signature differ from current and a page becomes stale. Same unconfigured-gateway fallback as the cost helpers. (See `src/core/sync-delta.ts` + `src/core/spend-posture.ts` for the #2139 cost-gate supporting modules.) `willEmbedSynchronously({v2Enabled, serialFlag, noEmbed}): SyncEmbedMode` is the single source of truth for whether `gbrain sync --all` embeds at sync time (`'inline'`) or defers to per-source `embed-backfill` minion jobs (`'deferred'`) — mirrors `sync.ts`'s `effectiveNoEmbed` resolution exactly (`v2Enabled && !serialFlag && !noEmbed → deferred`) so cost gate and embed decision can't drift. `shouldBlockSync(costUsd, floorUsd, mode, posture='gated'): boolean` is the pure cost-gate decision: blocks ONLY when `mode === 'inline' && costUsd > floorUsd` — deferred mode never blocks (the backfill's $X/source/24h cap is the real money gate), and `posture === 'tokenmax'` never blocks (the operator declared cost isn't the constraint; an `off`/`unlimited` floor is `Infinity` and so is never exceeded). Pinned by `test/sync-cost-preview.test.ts` + `test/embedding-signature-stale.test.ts`. +- `src/core/sync-delta.ts` — the single "what changed since last_commit" helper (#2139), consumed by BOTH `performSyncInner` (sync executor) and `estimateInlineNewTokens` (cost estimator) so the gate's dollar figure can't drift from what the sync imports. `computeSyncDelta(repoPath, fromCommit, toCommit, {detachedManifest?, detached?})` → `{status:'ok', manifest}` | `{status:'unavailable', reason:'anchor_missing'|'diff_failed'}`. Anchor reachability via `git cat-file -t` (#1970 discipline — a gc'd bookmark is `anchor_missing`, but a present-but-non-ancestor bookmark is still diffed tree-to-tree), then `git diff --name-status -M from..to` parsed by `buildSyncManifest`; merges the detached working-tree manifest when detached (`buildDetachedWorkingTreeManifest`, relocated here from sync.ts). NO dirty/untracked probe — attached-HEAD incremental sync imports only the commit diff, so pricing dirty files would re-introduce phantom costs on a busy brain. `execFileSync` array-args (shell-injection safe), 30s / 100 MiB budget. Test seam `_setGitRunnerForTests`. Pinned by `test/sync-delta.test.ts`. +- `src/core/spend-posture.ts` — spend-control surface (#2139). `resolveSpendPosture(engine): 'gated'|'tokenmax'` (DB-plane `spend.posture`, fail-open `gated`); `tokenmax` makes every cost gate informational across sync/reindex/enrich/onboard (spend still ledgered — removes the ceiling, not the accounting). `parseUsdLimit(raw, def, {allowZero?})` accepts `off`/`unlimited`/`none` → `Infinity`; `formatUsdLimit(n)` renders `Infinity` as the string `'unlimited'` (never raw — `JSON.stringify(Infinity)` is `null`); `usdLimitToCap(n)` maps `Infinity` → `undefined` at the BudgetTracker boundary so ledger rows never serialize null. `normalizeSpendPosture`/`isValidSpendPosture` back the `config set` validation. Doc: `docs/operations/spend-controls.md`. Pinned by `test/sync-cost-preview.test.ts` + `test/spend-off-switch.test.ts`. - `src/core/ai/dims.ts` — per-provider `providerOptions` resolver for embed-time dimension passthrough; the single source of truth for "which provider needs which knob to produce `vector(N)`". Exports `dimsProviderOptions(implementation, modelId, dims)` (called by `embed()` in `gateway.ts`), `VOYAGE_OUTPUT_DIMENSION_MODELS` (private const — the 7 hosted Voyage models that accept `output_dimension`: `voyage-4-large`, `voyage-4`, `voyage-4-lite`, `voyage-3-large`, `voyage-3.5`, `voyage-3.5-lite`, `voyage-code-3` — nano deliberately excluded), `VOYAGE_VALID_OUTPUT_DIMS = [256, 512, 1024, 2048] as const`, `supportsVoyageOutputDimension(modelId)`, `isValidVoyageOutputDim(dims)`. Voyage path uses the SDK-supported `dimensions` field (`{ openaiCompatible: { dimensions: N } }`), NOT Voyage's `output_dimension` wire-key — the `voyageCompatFetch` shim in `gateway.ts:541` translates `dimensions → output_dimension` before the HTTP body is built (the AI SDK's openai-compatible adapter doesn't recognize the wire-key, so sending it from here would be silently dropped and Voyage would return its default 1024-dim). Runtime guard: when a Voyage flexible-dim model is configured with `dims` outside `VOYAGE_VALID_OUTPUT_DIMS`, throws `AIConfigError` with a paste-ready `gbrain config set embedding_dimensions <256|512|1024|2048>` hint at the embed boundary (most common trigger: `embedding_model: voyage:voyage-4-large` without `embedding_dimensions`, falling back to `DEFAULT_EMBEDDING_DIMENSIONS=1536`, an OpenAI default not a Voyage one). - `src/core/ai/types.ts` — provider/recipe types. `EmbeddingTouchpoint` has optional `chars_per_token` (default 4, matching OpenAI tiktoken on English) and `safety_factor` (default 0.8, budget-utilization ceiling), both consulted only when `max_batch_tokens` is also set; Voyage declares `chars_per_token=1` + `safety_factor=0.5` to handle dense payloads (CJK/JSON/base64). Pre-split budget = `max_batch_tokens × safety_factor / chars_per_token`. `EmbeddingTouchpoint.multimodal_models?: string[]` model-level allow-list for recipes mixing text-only + multimodal models under one touchpoint (Voyage's 12 models share `supports_multimodal: true` but only `voyage-multimodal-3` accepts `/multimodalembeddings`); when omitted, recipe-level `supports_multimodal` is sufficient. `AIGatewayConfig.embedding_multimodal_model?: string` lets `embedMultimodal()` route to a different model than `embedding_model` (OpenAI text + Voyage images without flipping the primary pipeline). `Recipe.default_headers?: Record` (static) and `Recipe.resolveDefaultHeaders?(env)` (env-templated) seam for per-recipe headers riding alongside auth on every openai-compat touchpoint; mutually exclusive (declaring both throws `AIConfigError` at gateway-configure time); keys conflicting with the resolved auth header (`Authorization`, the resolver's custom header) rejected at `applyResolveAuth` call time so defaults can't shadow auth. Used by OpenRouter for the `HTTP-Referer` + `X-OpenRouter-Title` + `X-Title` attribution triple. - `src/core/ai/gateway.ts` — unified seam for every AI call. `embedQuery(text, opts?)` and `isAvailable(touchpoint, modelOverride?)` accept a model override so the resolved-column path embeds via the column's provider (Voyage / ZeroEntropy / OpenAI) instead of the global default; the hybrid path passes `{embeddingModel: resolved.provider, dimensions: resolved.dimensions}` and the gateway resolves the matching recipe via `instantiateEmbedding()`. `isAvailable('embedding', 'voyage:voyage-3-large')` checks the override's recipe (not the default) so hybrid skips vector search only when the active column's provider is actually down. `zeroEntropyCompatFetch` shim (sibling to `voyageCompatFetch`) handles ZE's non-OpenAI-compatible wire shape — rewrites the request URL `/embeddings → /models/embed`, injects `input_type` (default `'document'`; the threaded `'query'|'document'` crosses the SDK boundary via the module-level `__embedInputTypeStore` AsyncLocalStorage populated in `embedSubBatch()`, because the AI SDK's openai-compatible adapter strips `input_type` from `providerOptions` before building the wire body — #1400; `voyageCompatFetch` injects it opt-in the same way, and `openAICompatAsymmetricFetch` is the fallthrough shim for every other openai-compat recipe — llama-server/litellm/ollama — a strict pass-through when nothing was threaded) and explicit `encoding_format: 'float'`, and rewrites the response `{results: [{embedding}], usage: {total_bytes, total_tokens}}` → `{data: [{embedding, index}], usage: {prompt_tokens, total_tokens}}` so the SDK's openai-compatible Zod schema validates. Layer 1 (Content-Length) + Layer 2 (per-embedding) OOM caps via tagged `ZeroEntropyResponseTooLargeError` (kept separate from `VoyageResponseTooLargeError` because `test/voyage-response-cap.test.ts` does structural source-text greps pinning the Voyage name). Wired in `instantiateEmbedding()` via the `recipe.id === 'zeroentropyai'` branch. `gateway.rerank()` native HTTP path (no AI-SDK reranking abstraction): resolves the configured reranker via `getRerankerModel()`, posts to `${recipe.base_url}/models/rerank` with bearer auth, returns `RerankResult[]` sorted by relevance. `RerankError.reason` classifier: `auth | rate_limit | network | timeout | payload_too_large | unknown`. 5s default timeout (search hot path). Pre-flight payload guard rejects bodies over `recipe.touchpoints.reranker.max_payload_bytes` with `reason: 'payload_too_large'`. `_rerankTransport` test seam mirrors `_embedTransport`. `embedQuery(text)` threads `inputType: 'query'` through `dimsProviderOptions()` (4-arg). `getRerankerModel()` accessor + `isAvailable('reranker')` branch; `configureGateway` + `reconfigureGatewayWithEngine` thread `reranker_model`; `applyResolveAuth` + `defaultResolveAuth` widen touchpoint param to include `'reranker'`. `embedMultimodalOpenAICompat()` routes recipes with `implementation: 'openai-compatible'` (LiteLLM, Anyscale, vLLM, Gemini multimodal via proxy) through the standard `/embeddings` endpoint with content arrays carrying `image_url` entries; the Voyage `/multimodalembeddings` path is unchanged (gateway selects by recipe `implementation` tag). Runtime dimension validation throws `AIConfigError` (with model id + observed + expected) before the vector reaches storage when the provider returns a width that doesn't match the recipe's `default_dims` or the brain's `embedding_dimensions`. Pinned by `test/openai-compat-multimodal.test.ts`. Module-scoped `_embedTransport` defaults to AI SDK `embedMany`, with `__setEmbedTransportForTests(fn)` test seam so tests drive `embed()` with a stubbed transport. `splitByTokenBudget` and `isTokenLimitError` exported `@internal` (pure functions reused by the test file). Module-level `_shrinkState: Map` halves the recipe's effective `safety_factor` on token-limit miss (floor 0.05) and heals back ×1.5 after `SHRINK_HEAL_AFTER=10` consecutive successes. `configureGateway()` walks every registered recipe at construction and emits a once-per-process stderr warning for any embedding touchpoint missing `max_batch_tokens` (excluding the canonical OpenAI fast-path). `resetGateway()` clears `_shrinkState`, the warned-set, and restores the real transport. `embedMultimodal()` reads `cfg.embedding_multimodal_model` first (falls back to `cfg.embedding_model`); after the recipe-level `supports_multimodal` fast-fail, validates the resolved model against `touchpoint.multimodal_models` when declared (closes the Voyage-text-only-into-multimodal-endpoint footgun before any HTTP call). `getMultimodalModel()` accessor mirrors `getEmbeddingModel` / `getChatModel`. Exported `VoyageResponseTooLargeError` tagged class: `voyageCompatFetch`'s two OOM-defense caps (Layer 1 Content-Length at `:595`, Layer 2 per-embedding base64 at `:619`) throw it; the inbound response-rewriter's try/catch (which swallows parse failures so misshaped responses fall through to the SDK parser) checks `instanceof VoyageResponseTooLargeError` and rethrows so the cap is actually effective (regression assertion in `test/voyage-response-cap.test.ts` pins the `instanceof ⇒ throw err` line). AI SDK v6 toolLoop compat (`gbrain skillopt` rollouts AND production background `subagent` jobs both route through `chat()` / `toolLoop`): in `chat()`, tool defs wrap the raw JSON Schema with the SDK's `jsonSchema()` helper (`inputSchema: jsonSchema(t.inputSchema)`) — v6's `asSchema()` treats a bare `{jsonSchema: ...}` object as a thunk and throws "schema is not a function"; new exported pure `toModelMessages(messages: ChatMessage[]): unknown[]` converts gbrain's provider-neutral `ChatMessage[]` into v6 `ModelMessage[]` — tool results (pushed by `toolLoop` as `role:'user'` with bare-value tool-result blocks) become a dedicated `role:'tool'` message with structured `output:{type:'json'|'text'|'error-text', value}` parts; `null` output preserved as `{type:'json', value:null}` (not dropped); text/tool-call blocks pass through with v6 field names (`toolCallId`/`toolName`/`input`); applied at the `generateText` call (`messages: toModelMessages(opts.messages)`). The converter is the load-bearing fix for the production subagent path, not just skillopt. Pinned by `test/gateway-model-messages.test.ts`. Companion fix in `src/core/skillopt/rollout.ts`: the inline `paramsToSchema` dropped `items` on array params; it now uses the shared `paramDefToSchema` from `src/mcp/tool-defs.ts` (single source of truth, recursive on items/enum/default). @@ -295,7 +298,7 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/embedding-dim-check.ts` extension — facts.embedding dim drift surface. `readFactsEmbeddingDim(engine): Promise` covers both `vector(N)` and `halfvec(N)` shapes (migration v40 falls back to `vector` on pgvector < 0.7); regex ordering is halfvec-before-vector (substring "vec" appears in "halfvec"; naive `/vector/i` would shadow). `buildFactsAlterRecipe(dims, configured, type)` emits the paste-ready `DROP INDEX IF EXISTS idx_facts_embedding_hnsw; ALTER TABLE facts ALTER COLUMN embedding TYPE halfvec(N) USING embedding::halfvec(N); CREATE INDEX idx_facts_embedding_hnsw ON facts USING hnsw (embedding halfvec_cosine_ops) WHERE ...` flow (NOT bare REINDEX, which doesn't rewrite the index after a column-type change). `assertFactsEmbeddingDimMatchesConfig(engine)` is the preflight — throws `FactsEmbeddingDimMismatchError` (tagged `tag: 'FACTS_EMBEDDING_DIM_MISMATCH'` for parity with the worker-pool MUST_ABORT semantics) when configured dim ≠ column width; cached per-engine via `WeakMap`; PGLite engines silently skip. Doctor check `facts_embedding_width_consistency` (registered after `embedding_width_consistency`) reuses the same helpers with an identical ALTER recipe. Pinned by `test/embedding-dim-check-facts.test.ts`. - `src/core/postgres-engine.ts` extension — `insertFact` + `insertFacts` no longer hardcode `tx.unsafe(\`'${embedLit}'::vector\`)` for the embedding column. `resolveFactsEmbeddingCast()` (private) probes `pg_attribute` once per engine instance (cached in `_factsEmbeddingCastSuffix`) and returns `'::halfvec'` when migration v40 created the column as halfvec, else `'::vector'`; both insert paths use the cached suffix so the cast matches the actual column type (works on older pgvector that lacks implicit auto-cast). Test seam `__resetFactsEmbeddingCastCacheForTest()` clears the per-engine cache. - `src/core/cycle.ts` + `src/core/cycle/extract-atoms.ts` + `src/core/cycle/synthesize-concepts.ts` + `src/commands/extract.ts` + `src/commands/doctor.ts` + `src/core/op-checkpoint.ts` extensions — six daily-driver ops fixes. (1) Batch idempotency: `atomsExistingForHashes(engine, sourceId, hashes[])` (exported from `src/core/cycle/extract-atoms.ts`) replaces the per-hash loop (7K individual queries) with one batched SQL roundtrip returning already-extracted `content_hash16` values; fail-open (SQL error → empty set, extraction proceeds); powered by migration v104 `pages_atom_source_hash_idx` (partial expression index on `frontmatter->>'source_hash'` for atom rows where `deleted_at IS NULL`; Postgres `CREATE INDEX CONCURRENTLY` with invalid-remnant pre-drop, PGLite plain). (2) Shorter cycle lock TTL + active in-phase refresh: `LOCK_TTL_MINUTES = 5` (was 30); `buildYieldDuringPhase(lock, outer)` (exported, with `LockHandle`) calls `lock.refresh()` + any external hook on every fire, throttled to 30s via `maybeYield`, firing both in the main loop AND immediately after every `await chat(...)`; `synthesize_concepts` uses the same throttled hook. A crashed cycle releases its lock 6x faster while a healthy long-running cycle keeps it alive (residual: a single `await chat()` past 5 min can expire the lock mid-await — TODO-OPS-2). (3) Progress wiring: `progress?: ProgressReporter` opt on `ExtractAtomsOpts` and `SynthesizeConceptsOpts`; cycle.ts passes its phase-level reporter down (NOT a child reporter, which would collide on `cycle.extract_atoms.extract_atoms.work`); phases only call `tick()`/`heartbeat()`, cycle.ts owns `start()`/`finish()`. (4) `by-mention` resume: `mentionsFingerprint({source, type, since, gazetteerHash})` in `src/core/op-checkpoint.ts` — the gazetteer hash is load-bearing (adding entity pages mid-pause shifts the hash → new fingerprint → fresh scan against the new gazetteer, never silent skip); `gbrain extract links --by-mention` resumes via `op_checkpoints` with `flushAndCheckpoint` ordering (links flush to DB FIRST, page keys commit to checkpoint SECOND, persist THIRD, so a crash mid-batch leaves the page un-checkpointed and resume re-scans it); persist every 1000 items OR 30s; clean exit clears the checkpoint; `--dry-run` skips both load and write. (5) `sync_consolidation` doctor check (multi-source brains see a paste-ready `gbrain sync --all --parallel 4 --workers 4 --skip-failed`; single-source "not applicable"; SQL errors return `warn` via the check's own try/catch). (6) Test-isolation: `test/cycle-last-full-cycle-at.test.ts` + `test/schema-cli.test.ts` use per-test `GBRAIN_HOME=tempdir`. Pinned by `test/cycle/extract-atoms-batch.test.ts`, `test/cycle/cycle-lock-ttl.test.ts` (regression pin on `LOCK_TTL_MINUTES === 5`), `test/op-checkpoint-mentions-fingerprint.test.ts`, `test/cycle/extract-atoms-progress.test.ts`, `test/cycle/synthesize-concepts-progress.test.ts`, `test/cycle/yield-during-phase-refresh.test.ts`, `test/cycle/yield-during-phase-throttle.test.ts`, `test/extract-by-mention-resume.test.ts`, `test/doctor-sync-consolidation.test.ts`. Companion `sync --all` recipe block in `skills/cron-scheduler/SKILL.md`. -- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` reject when combined with `--parallel > 1` (the brain-global `sync-failures.jsonl` has no per-source scope); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error, 2 cost-prompt-not-confirmed. The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The `sync --all` cost gate is mode-aware via `willEmbedSynchronously` + `shouldBlockSync` from `src/core/embedding.ts` (federated_v2 resolved once up front): the DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog `sumStaleChunkChars({signature})` priced via `estimateCostFromChars` at `currentEmbeddingPricePerMTok()` + queued-job count, NEVER exits 2); the INLINE path BLOCKS only when the new-content estimate exceeds `sync.cost_gate_min_usd` (default $0.50). The new-content estimate (`estimateInlineNewTokens`) contributes ZERO for sources provably unchanged since last sync (`isSourceUnchangedSinceSync` HEAD==last_commit + clean tree AND `chunker_version === CURRENT`) and the full-tree ceiling for changed sources; the pre-existing stale backlog is shown informationally but NEVER gated on (sync doesn't sweep it inline). Helpers `resolveCostGateFloorUsd(engine)` (fail-open $0.50, accepts 0) + `resolveBackfillCapUsd(engine)`. JSON envelopes gain `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | confirmation_required`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). +- `src/commands/sync.ts` — `gbrain sync` CLI + the `performSync` / `performFullSync` library entrypoints (consumed by the autopilot cycle and the Minion sync handler). `performSync` runs under a writer lock: per-source `gbrain-sync:` whenever `opts.sourceId` is set, wrapped in `withRefreshingLock` from `src/core/db-lock.ts` so long-running sources (250K+ chunks) don't lose the lock at the 30-min TTL mid-run; the bare no-source path uses the SAME refreshing lock (#1794 — it was previously a non-refreshing `tryAcquireDbLock`, stealable mid-run during an incident); `SyncOpts.lockId?: string` is the explicit override. The lock refresh AND its health probe route through the DIRECT session pool so Supavisor transaction-pool exhaustion (`EMAXCONNSESSION`) can't kill renewal; takeover is heartbeat-aware (it will NOT steal a holder whose `last_refreshed_at` is within `GBRAIN_LOCK_STEAL_GRACE_SECONDS`, defending an alive-but-starved holder); the import loop yields the event loop every `GBRAIN_SYNC_YIELD_EVERY` files (`setTimeout(0)`, not `setImmediate` — Bun starves the timers phase) so the refresh `setInterval` heartbeat fires mid-import. This lock-identity invariant prevents a `sync --all` per-source worker racing `sync --source foo` on the global lock from corrupting the same source. `performSync` throws a typed `SyncLockBusyError` when the writer lock is held; the Minion `sync` handler (`src/commands/jobs.ts`) catches it and marks the job SKIPPED (not failed) so a cron/autopilot tick defers to the holder without polluting crash metrics. `performSyncInner` is RESUMABLE (incremental path): it drains a PINNED target commit (`lastCommit..pin`), banking drained file paths via `appendCompleted` (append-only delta into the `op_checkpoint_paths` child table, migration v115 — one row per path, O(delta) not the old O(N²) full-array rewrite), keyed by `syncFingerprint({sourceId, lastCommit})` from `src/core/op-checkpoint.ts` (paths under `op:'sync'`; the pinned target under `op:'sync-target'`), and advances `last_commit`/`last_sync_at` ONLY at full import completion. Checkpoint writes route through the DIRECT session pool + bounded retry so they survive `EMAXCONNSESSION`; the flush cadence is first-file then every `GBRAIN_SYNC_CHECKPOINT_EVERY` (default 1000) files OR `GBRAIN_SYNC_CHECKPOINT_SECONDS` (default 10s), with a race-safe `pendingCheckpointPaths` delta (single-flight swap, re-merge on failure) under parallel workers; a SIGTERM banks the in-flight delta via a no-retry one-shot (`appendCompletedOnce`, ordered before lock release through `registerCleanup`); and sustained flush failure aborts the run with `reason:'checkpoint_unavailable'` after `GBRAIN_SYNC_MAX_CHECKPOINT_FAILURES` consecutive fails rather than importing work it can never bank (every partial/blocked exit logs the banked-file count). A sync killed mid-import banks its progress, leaves the anchor unmoved (the source stays correctly stale to the autopilot scheduler — `last_sync_at` is never bumped on a partial), and the next run `resumeFilter`s the same fixed diff to skip done files. The pin is the checkpoint's stored target when still reachable from HEAD, else live HEAD (a history rewrite / reset re-pins); completion advances to the pin, NOT live HEAD, so commits landing past the pin are a clean next-sync diff — this closes the cross-run staleness window. After import a pin-reachability gate (`git merge-base --is-ancestor pin HEAD`) replaces the old strict head-drift gate: forward commits on top of the pin (e.g. a background enrich process committing to the same repo every ~2 min) are SAFE and no longer block the run; only a real rewrite (pin not an ancestor of HEAD) blocks and discards the checkpoint. A file added in `lastCommit..pin` but gone from disk (deleted by a commit after the pin) is SKIPPED and marked completed, not recorded as a failure. Downstream extract/facts/embed are size-gated: inline only for `totalChanges <= 100`; large syncs defer to the resumable `extract --stale` watermark + `embed --stale`/backfill + the facts cycle phases, so a 44K-page facts/embed pass never re-blocks import convergence (sync convergence == import convergence). Worker engines wrap in try/finally so disconnect always fires; both PGLite-detection sites use `engine.kind === 'pglite'`. CLI accepts `--workers N` (alias `--concurrency N`) validated via `parseWorkers` (explicit bypasses the file-count floor; auto path defers to `autoConcurrency()`). The newest-first descending-lex order uses `sortNewestFirst(addsAndMods)` from `src/core/sort-newest-first.ts` (shared with `gbrain import`). `gbrain sync --all` runs a continuous worker pool: `parseWorkers`-validated `--parallel N` (default `min(sourceCount, --workers, DEFAULT_PARALLEL_SOURCES=4)`), long-lived async workers pulling from a shared FIFO queue (no head-of-line blocking), per-source `withSourcePrefix(src.id, ...)` so every `slog`/`serr` line carries `[]`; `--skip-failed`/`--retry-failed` are scoped per source (`acknowledgeFailures(sourceId)`; `--all` acks every source, single-source acks only its own) and run UNDER parallel — the #1939 failure ledger is per-`(source_id, path)` and serialized through `withLedgerLock`, so the old "not supported under parallel, re-run with --serial" refusal is lifted (#2139, which also removed the forcing-function that pushed recovery syncs to `--serial` and thus armed the inline cost gate); a connection-budget stderr warning fires when `parallel × workers × 2 > 16` (the `× 2 per-file pool` factor: each per-file worker opens its own `PostgresEngine` with `poolSize=2`). Exports `resolveParallelism`, `syncOneSource`, `buildSyncStatusReport`, `printSyncStatusReport`, `SyncStatusReport` back the `gbrain sources status` dashboard. `--json` envelope `{schema_version: 1, sources, parallel, ok_count, error_count}` on stdout; human banners route to stderr via `humanSink` so `jq` parses cleanly. Exit matrix: 0 all ok, 1 any error. (The non-TTY cost gate no longer exits 2 — it auto-defers; #2139.) The dashboard SQL is `content_chunks ch JOIN pages pg ON pg.id = ch.page_id WHERE pg.deleted_at IS NULL` with `archived = false` at the caller; embedding column resolved via `resolveEmbeddingColumn(undefined, cfg)` from `src/core/search/embedding-column.ts` so Voyage/multimodal/non-default-column brains count against the column they use; errors propagate (no swallow-catch). The sync delete loop is interleaved per-batch resolve+delete using `engine.resolveSlugsByPaths` + `engine.deletePages` from `src/core/engine.ts` (73K-delete commit: ~146K SQL round-trips → ~292, closing the cascade-staleness class where one big-delete commit jammed every other source's sync); per-batch try-catch decomposes batch DELETE failures to per-slug `deletePage` fallback, unrecoverable per-slug failures land in `failedFiles`; `pagesAffected` filters to confirmed-deleted slugs. The rename loop gets the same batched slug-resolve; a rename whose destination is un-syncable folds the source path into the delete set so the old page can't orphan (#1970). An entry-time bookmark-reachability guard distinguishes a gc'd anchor (`cat-file` fails → `performFullSync`) from a history-rewrite anchor that is merely no-longer-an-ancestor but still on disk: the latter is diffed tree-to-tree directly (`git diff lastCommit..pin` is an endpoint-tree compare, ancestry not required) so a force-push / `master`→`main` consolidation imports only the real delta instead of re-walking the whole tree forever (#1970); an oversized or failed diff degrades to `performFullSync`. `performFullSync` is itself authoritative for deletes — after an advancing full import it purges file-backed pages (`source_path != null` AND strategy-aware `isSyncable`) whose source file no longer exists, sparing `put_page`/manual pages (null `source_path`) and metafiles. `resolveSlugByPathOrSourcePath` at `sync.ts:267` delegates to `engine.resolveSlugsByPaths` when `sourceId` is set, keeping legacy `executeRaw` fallback for the no-sourceId path. `failedFiles` is hoisted to the top of `performSyncInner` so both delete-decompose and import loops feed the same bookmark gate. The cost gate is the shared `runInlineCostGate` (one implementation on BOTH the `--all` and single-source paths; runs at the command layer, never inside `performSync`), mode-aware via `willEmbedSynchronously` + posture-aware `shouldBlockSync` from `src/core/embedding.ts` (#2139). The DEFERRED path (v2 on, parallel) is INFORMATIONAL (embedding goes to per-source `embed-backfill` jobs with their own `$X/source/24h` cap, default $25 via `SPEND_CAP_CONFIG_KEY` from `embed-backfill-submit.ts`; prints the cap + backlog + queued-job count, NEVER exits 2). The INLINE path (v2 off, or `--serial`) gates on the DELTA estimate vs `sync.cost_gate_min_usd` (default $0.50): below floor proceeds; above floor in a TTY prompts `[y/N]`; above floor in a non-TTY/`--json` session AUTO-DEFERS embeds to capped backfill jobs and exits 0 (NEVER exit 2 — the wedged-cron fix); `spend.posture=tokenmax` makes it informational and proceeds inline. The estimate MIRRORS EXECUTION instead of pricing the whole tree: `estimateInlineNewTokens` routes through the shared `computeSyncDelta` (`src/core/sync-delta.ts`) — fetch-first against `origin/`, prices only the committed delta (a dirty-but-caught-up tree → $0), with a full-tree CEILING only on the fail-open rungs (chunker drift, first sync, git-unavailable) honestly labeled; `--full` adds the stale backlog (full sync sweeps it inline). Return shape carries `estimateKind: 'delta'|'ceiling'|'mixed'|'unchanged'` + `ceilingReasons`. Helpers `resolveCostGateFloorUsd(engine)` + `resolveBackfillCapUsd(engine)` resolve via `parseUsdLimit` (`off`/`unlimited` → `Infinity`; floor accepts `0` = block-on-any-spend). JSON envelopes carry `mode` + `gate` discriminators (`dry_run | deferred_notice | below_floor | auto_deferred_embeds | posture_tokenmax`) + a paste-ready `hint`; `Infinity` floors/caps render as the string `'unlimited'` (never raw, which JSON-serializes to `null`); `SyncStatusReportSource` gains `backfill_queued`/`backfill_active`/`backfill_last_completed_at`; cost previews read `getEmbeddingModelName()` (no hardcoded OpenAI). Format splits on the explicit `--json` flag only (human text otherwise). `SyncOpts.noSchemaPack` (CLI `--no-schema-pack`, threaded through `performSync` AND `syncOneSource`) skips `loadActivePack` so pages fall back to legacy prefix typing — an escape hatch when a suspect pack regex wedges a sync. A per-file BEGIN heartbeat `if (process.env.GBRAIN_SYNC_TRACE) serr('[sync] begin import: ')` fires BEFORE `importFile` (the `progress.tick` fires only AFTER) so a stuck file is a begin-line with no matching completion. Triage doc: `docs/architecture/serve-sync-concurrency.md` (PGLite single-writer serve↔sync contention + the `GBRAIN_SYNC_TRACE` + `--no-schema-pack` recipes). Pinned by `test/e2e/sync-status-pglite.test.ts` (IRON-RULE: PGLite seeds 2 sources × pages × chunks, soft-deletes 1 page, archives 1 source, validates the SQL excludes both AND uses the active embedding column), `test/sync-cost-gate.serial.test.ts`, `test/sync-cost-preview.test.ts`. Runaway-sync protection: `resolveSyncHardDeadline(args, {isTty, env, defaultNonTtySec?})` resolves a wall-clock hard deadline (precedence `--no-hard-deadline` > `--hard-deadline ` > `--timeout `(non-`--all`, which auto-arms the backstop) > `GBRAIN_SYNC_MAX_RUNTIME_SECONDS` env > non-TTY default 3600s > none; `HARD_DEADLINE_GRACE_SEC=30`). `src/cli.ts` installs the out-of-band watchdog (see `src/core/process-watchdog.ts`) for the sync command BEFORE `connectEngine` and disposes it in the dispatch `finally`, so even an event-loop-starved sync — or a connect-phase hang — is SIGTERM-then-SIGKILLed by the deadline instead of orphaning under cron. `runSync` registers a SIGINT handler that aborts an interrupt `AbortController` composed via `composeAbortSignals(...)` (an `AbortSignal.any` wrapper over the defined signals) with the per-source `--timeout` signal, so Ctrl-C returns a clean `partial` and releases the lock through the normal `finally` (process-cleanup.ts owns SIGTERM lock-release; the watchdog owns the hard kill). `withRefreshingLock` `unref()`s its refresh `setInterval`. The spin's own root cause is not yet pinned (leading lead is catastrophic-backtracking in a pack link-inference regex, bounded by the redos-guard); the watchdog heartbeat plus the existing `[gbrain phase]` breadcrumbs are the diagnosis surface. Pinned by `test/sync-hard-deadline.test.ts` (resolution precedence + `composeAbortSignals`). - `src/commands/import.ts` — `gbrain import` CLI + `runImport` library entrypoint. Uses a path-set checkpoint via `src/core/import-checkpoint.ts` (the walk still applies `sortNewestFirst()` for embed-cost ordering, but checkpoint correctness no longer depends on sort order). A file enters `completed: Set` only when its `processFile` returns success (including content-hash short-circuit no-ops); failed files never enter the set so the next run retries them automatically with no manual `~/.gbrain/import-checkpoint.json` delete. This closes three classes: parallel-import-with-slow-worker dropping the slow file on crash-resume (the slow file isn't in `completed` until its own `processFile` resolves), failed-file-bumps-counter-past-itself (failures don't add to `completed`), and sort-flip-drops-newest-N-on-cross-version-resume (order is no longer part of the checkpoint). Old positional checkpoints are detected and discarded with a stderr line on first resume (re-walking is cheap because `content_hash` short-circuits unchanged files). Checkpoint persists every 100 successful adds, not every 100 processed files. The `managedBookmark` opt (set by `performFullSync` when `runImport` is the full-sync engine) suppresses `runImport`'s own `sync.last_commit` advance so the shared `applySyncFailureGate` (`src/core/sync-failure-ledger.ts`) owns the bookmark + failure-ledger gating on that path — one gate decides advance/block/auto-skip across both sync paths. Pinned by `test/import-checkpoint.test.ts` + `test/import-resume.test.ts` (incl. the SLUG_MISMATCH retry regression). - `src/core/import-checkpoint.ts` — `loadCheckpoint(brainDir)`, `saveCheckpoint(brainDir, completed)`, `resumeFilter(files, completed, brainDir)`, `clearCheckpoint()`, plus the `ImportCheckpoint` type. Path-set format `{schema_version, brainDir, completed: string[]}`. Atomic write via `.tmp` + `rename()` so a mid-write crash never leaves a partial JSON. `loadCheckpoint` returns `null` on: missing file, malformed JSON, brainDir mismatch (ran against a different brain), and the old positional format (logged to stderr before discard). `resumeFilter` returns `{toProcess, skippedCount}` — pure, no I/O, deterministic. `clearCheckpoint` is no-op-on-missing for clean-exit cleanup. Honors `GBRAIN_HOME` via `gbrainPath()` so `withEnv({GBRAIN_HOME: tmpdir})` test isolation works without monkey-patching fs. Best-effort persistence — `saveCheckpoint` logs warnings on write errors but never throws. - `src/core/sort-newest-first.ts` — single source of truth for the descending-lex sort that `gbrain import` and `gbrain sync` both apply. Mutates in place (Array.prototype.sort semantics), returns the same array reference for fluent chaining. Empty/single-element inputs short-circuit. Future ordering changes flip one line here instead of touching two CLI commands. Pinned by `test/sort-newest-first.test.ts` (descending order, mixed prefixes, empty, single-element, in-place-mutation contract). @@ -322,8 +325,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/markdown.ts` — Frontmatter parsing + body splitter. `coerceFrontmatterString(v)` coerces a non-string `title`/`slug`/`type` to a deterministic string at parse time so a YAML-typed value never reaches `.toLowerCase()` and throws (the #1939 wedge: `title: 2024-06-01` parsed as a `Date`, `title: 1458` as a number, and the throw blocked the sync bookmark from advancing); a `Date` becomes its UTC ISO date (`2024-06-01`, machine-independent and matching the on-disk token, unlike `String(date)`), `null`/`undefined` become `''`, everything else uses `String()`. `splitBody` requires an explicit timeline sentinel (``, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus existing people/companies/deals/etc heuristics). - `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`. - `scripts/check-source-id-projection.sh` — CI grep guard for the multi-source bug class. Greps `src/core/postgres-engine.ts` + `src/core/pglite-engine.ts` for `SELECT.*FROM pages` projections matching the `rowToPage` feeder shape (id + slug + type + title) and fails if `source_id` is missing. `Page.source_id` is required at the type level; a projection dropping the column produces `Page` rows with `source_id: undefined` while TypeScript's `: string` lies about it. Wired into `bun run verify` + `bun run check:all`. -- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` with named volumes (`gbrain-ci-pg-data`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs unit tests with `DATABASE_URL` unset, then runs all 29 E2E files sequentially. `--diff` swaps in the diff-aware selector; `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set. -- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all 29 files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases. +- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up four `pgvector/pgvector:pg16` services (postgres-1..4) + `oven/bun:1` with named volumes (`gbrain-ci-pg-data-{1..4}`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs guards + typecheck, then the Tier 1 default: 4-shard parallel unit + E2E (`xargs -P4`, one Postgres per shard; unit phase keeps `DATABASE_URL` unset). `--no-shard` falls back to the legacy unsharded sequential flow (debug aid); `--diff` runs the diff-aware selector unsharded. Also runs a `pgbouncer` service (`edoburu/pgbouncer`, `POOL_MODE: transaction`, `AUTH_TYPE: plain` — pg16 stores SCRAM verifiers, so the userlist must hold the plaintext password; `IGNORE_STARTUP_PARAMETERS` whitelists gbrain's `statement_timeout`/`idle_in_transaction_session_timeout` startup params the way the Supabase pooler does) fronting postgres-1 on host port `GBRAIN_CI_PGBOUNCER_PORT` (default 6543); every E2E invocation exports `GBRAIN_PGBOUNCER_URL` (pooled; dedicated `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures) + `GBRAIN_PGBOUNCER_DIRECT_URL`, consumed by `test/e2e/pgbouncer-teardown.test.ts` — the transaction-mode teardown bug class (#1972/#2015/#2084) reproduced in the local gate instead of only in production. `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set. +- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases. - `scripts/run-e2e.sh` — Sequential E2E runner. Accepts an optional argv-driven file list (used by `ci:local:diff`) and a `--dry-run-list` flag that prints the resolved file list and exits (used by `ci-local.sh`'s startup smoke-test). Falls back to `test/e2e/*.test.ts` when invoked with no args. - `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output has no runtime consumer; committed for GitHub browsing and fork-safe fetching. - `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`. diff --git a/docs/guides/push-context.md b/docs/guides/push-context.md new file mode 100644 index 000000000..e4e104756 --- /dev/null +++ b/docs/guides/push-context.md @@ -0,0 +1,79 @@ +# Push-based context (#2095, v0.42.43.0) + +Retrieval used to be pull-only: the agent had to *know to ask* before the brain +contributed anything. Push-based context inverts that — the brain volunteers +relevant pages from the recent conversation, confidence-gated so push noise +never becomes worse than pull silence. + +Three channels share one zero-LLM core (`src/core/context/volunteer.ts`): + +| Channel | Surface | When to use | +|---|---|---| +| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call | +| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn | +| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out | + +## How it decides + +1. **Extract** entities across the last N turns (capitalized runs, `@handles`), + merged with recency / frequency / user-role salience. Assistant-introduced + entities and "what did she invest in?" follow-ups whose antecedent was named + in the window now resolve. +2. **Resolve** through the alias table, exact titles, and slug suffixes — each + arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6, + +0.05 when mentioned in ≥2 turns or the newest turn. +3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an + explicit lower gate), suppress pages already surfaced (slug-presence only), + cap at 3 pages (hard cap 5). + +## CLI + +```bash +# one-shot: pipe recent turns (oldest → newest) +printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \ + | gbrain volunteer-context + +# streaming: volunteered pages print as the transcript flows +some-transcript-feed | gbrain watch --json + +# the feedback loop: how often were volunteered pages actually opened? +gbrain volunteer-context --stats +``` + +Stats are **approximate** by design: "used" means `pages.last_retrieved_at > +volunteered_at` — the 5-minute last-retrieved throttle causes false negatives +and unrelated reads of the same page cause false positives. Use the per-arm +precision to tune `min_confidence`, not as an exact metric. + +**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its +connection for the whole session — a concurrent `gbrain serve` or any write +path blocks until watch exits. On a PGLite brain, run watch in bursts (piped +input exits at EOF) or use the ambient reflex channel instead, which routes +through a running serve's resolve socket rather than taking the lock. Routing +watch through that same socket is a filed follow-up (TODOS.md). Postgres +brains are unaffected. + +## Config + +| Key | Default | What it does | +|---|---|---| +| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) | +| `retrieval_reflex` | true | the ambient channel's master switch | +| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn | + +Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch` +(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch); +on the op only: `prior_context` (text whose already-surfaced slugs are suppressed), +`session_id` / `turn` attribution params (watch stamps its own per-session id and +turn numbers in the feedback log), and `days` to size the `--stats` window. + +## Storage + privacy + +Volunteered pages log to `context_volunteer_events` (migration v117): slug, +arm, confidence, channel, optional session/turn — the rationale is a +deterministic template string, never raw conversation text. Event writes are +best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal, +not an audit trail. Rows are pruned after 90 days by the dream cycle's purge +phase. Synopses always strip the takes/facts fences — the same strip `get_page` +applies to untrusted callers, applied unconditionally here so private fence rows +never reach a prompt regardless of caller trust. diff --git a/docs/operations/spend-controls.md b/docs/operations/spend-controls.md new file mode 100644 index 000000000..1e2fca9ee --- /dev/null +++ b/docs/operations/spend-controls.md @@ -0,0 +1,111 @@ +# Spend controls + +GBrain's embedding-spend gates in one place: every gate, its config key, default, +whether it blocks or just informs, how to widen or disable it, and how the +`spend.posture` switch governs all of them. + +The orienting idea: **GBrain itself is rounding error; the spend that matters is +downstream embedding.** These gates exist so a routine sync or enrich can't run up +an unexpected embedding bill, while never wedging an unattended cron. + +## `spend.posture` — one switch for "cost is not my constraint" + +```bash +gbrain config set spend.posture tokenmax # all cost gates become informational +gbrain config set spend.posture gated # default — gates enforce +``` + +| Value | Effect | +|-------|--------| +| `gated` (default) | Every cost gate enforces its limit as documented below. | +| `tokenmax` | Every cost gate prints its estimate and **proceeds** — informational only. Spend is still recorded to the ledger; posture removes the *ceiling*, not the *accounting*. | + +`spend.posture` is deliberately separate from `search.mode=tokenmax` (which governs +retrieval payload size, not embedding spend). When a gate fires and +`search.mode=tokenmax` but `spend.posture` is unset, the gate prints a one-line hint +pointing at this switch. + +**Precedence:** an explicit per-call cap (`--max-usd N`, `--max-cost N`) always wins +over posture. `tokenmax` only governs the default/absent case — it never overrides a +number you typed on the command line. + +## Off switches (`off` / `unlimited` / `none`) + +The USD-limit knobs accept `off`, `unlimited`, or `none` (case-insensitive) to mean +"no limit" — no more setting sentinel values like `100000`. + +- `0` is **not** "off". On `sync.cost_gate_min_usd`, `0` means "block on any nonzero + spend" (a real choice). On the backfill caps, `0` falls back to the default. +- Internally "no limit" is the string `unlimited` in any printed/JSON output and "no + cap" inside the budget tracker — never a raw `Infinity` (which would serialize to + `null` in ledger rows). + +## The gates + +| Gate | Config key | Default | Blocks? | Off switch | tokenmax | +|------|-----------|---------|---------|-----------|----------| +| Sync inline-embed cost gate | `sync.cost_gate_min_usd` | `0.50` | TTY prompt / non-TTY auto-defer | `off` (or `0` = block-on-any) | informational | +| Backfill 24h per-source spend cap | `embed.backfill_max_usd_per_source_24h` | `25` | refuses submission | `off` (`0` → default) | bypassed (still ledgered) | +| Backfill per-job budget | `embed.backfill_max_usd` | `10` | caps the job's tracker | `off` (`0` → default) | uncapped (still ledgered) | +| Backfill cooldown | `embed.backfill_cooldown_min` | `10` | skips re-submission inside window | — (latency knob, not spend) | **not** bypassed | +| `reindex-code` cost gate | — (preview before re-embed) | — | TTY prompt / non-TTY refuse + exit 2 | `--max-cost off` | informational | +| `enrich` / `onboard --auto` | `--max-usd` (per-call) | — | refuse without a cap (non-TTY) | `--max-usd off` | runs uncapped (still ledgered) | + +### Sync inline-embed cost gate + +Fires only when sync embeds **inline** (federated_v2 off, or `--serial` without +`--no-embed`). Under federated_v2 + parallel, embedding is deferred to capped backfill +jobs and the gate is informational. The estimate prices the **delta** — the files this +sync will actually import (fetched-first, so it sees commits the run is about to pull) — +not the whole tree. A busy brain with a dirty working tree but caught-up commits +estimates `$0`, because an attached-HEAD sync imports only the committed diff. + +Behavior above the floor: +- **TTY:** prompts `[y/N]`. +- **Non-interactive (cron/agent):** **auto-defers** embeds to capped backfill jobs and + exits 0 — it never wedges the pipeline. The backlog drains via the jobs worker or + `gbrain embed --stale`. Pass `--yes` to embed inline instead. + +Output format splits on the explicit `--json` flag: `--json` emits a structured +envelope; otherwise human text. Every gate message carries paste-ready knobs. + +`--full` re-embeds the stale backlog inline (full sync sweeps it), so a `--full` +estimate is `delta + stale backlog`, labeled as such. + +### Estimate labels + +- `~N tokens (delta: changed files since last sync)` — the precise estimate. +- `<=N tokens (full-tree ceiling for K source(s): …)` — a conservative + over-count used only when a precise delta can't be computed: a first sync, a chunker + version drift (forces a full re-chunk), or git being unavailable. Unchanged files + still skip via `content_hash` at execution, so the ceiling over-states real spend. + +## Notes & limits + +- **Pre-pull window:** the gate fetches before estimating, so it prices what the run + will pull. If a fetch fails (offline), it estimates against local HEAD and labels the + result; the bounded residual is priced on the next run. +- **Single-source `gbrain sync`** carries the same gate as `sync --all` (it previously + embedded inline with no preview). +- **Recovery under parallel:** `--skip-failed` / `--retry-failed` work under parallel + sync (the failure ledger is per-source and lock-serialized) — you no longer have to + drop to `--serial`, which is what used to arm the inline gate. + +## Escape hatches at a glance + +```bash +# Never gate this brain on cost: +gbrain config set spend.posture tokenmax + +# Widen the sync inline floor to $5: +gbrain config set sync.cost_gate_min_usd 5 + +# Disable the sync inline floor entirely: +gbrain config set sync.cost_gate_min_usd off + +# Lift the backfill 24h spend cap: +gbrain config set embed.backfill_max_usd_per_source_24h off + +# Run enrich uncapped non-interactively: +gbrain enrich --max-usd off # or: gbrain config set spend.posture tokenmax +``` diff --git a/docs/tutorials/personal-brain.md b/docs/tutorials/personal-brain.md index 3be55c8f2..8aaaca957 100644 --- a/docs/tutorials/personal-brain.md +++ b/docs/tutorials/personal-brain.md @@ -86,7 +86,7 @@ Save this token. You'll need it for the AlphaClaw setup. AlphaClaw is the setup harness that manages OpenClaw deployment. -1. Go to [alphaclaw.com](https://alphaclaw.com) +1. Go to [alphaclaw.md](https://alphaclaw.md) 2. Enter your **workspace repo** (not the brain repo): `your-org/myagent` 3. Select "Use existing" if the repo already exists 4. Enter your GitHub PAT from Step 2 diff --git a/llms-full.txt b/llms-full.txt index 8c80efc09..5c7d4546d 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -117,8 +117,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont ## Before shipping Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks, -unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a -fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the +guards + typecheck, then 4-shard parallel unit + E2E against four pgvector +containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL` +unset) and tears down. Use `bun run ci:local:diff` for the diff-aware subset during fast iteration on a focused branch. Requires Docker (Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`). @@ -186,7 +187,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP +Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -245,6 +246,8 @@ detail on demand.) | any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry | | search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` | | search modes / cost knobs | `docs/guides/search-modes.md` | +| embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` | +| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` | | schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` | | thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` | | the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry | @@ -3423,6 +3426,92 @@ the bundled resolver lives at [`skills/RESOLVER.md`](../../skills/RESOLVER.md). --- +## docs/guides/push-context.md + +Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md + +# Push-based context (#2095, v0.42.43.0) + +Retrieval used to be pull-only: the agent had to *know to ask* before the brain +contributed anything. Push-based context inverts that — the brain volunteers +relevant pages from the recent conversation, confidence-gated so push noise +never becomes worse than pull silence. + +Three channels share one zero-LLM core (`src/core/context/volunteer.ts`): + +| Channel | Surface | When to use | +|---|---|---| +| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call | +| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn | +| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out | + +## How it decides + +1. **Extract** entities across the last N turns (capitalized runs, `@handles`), + merged with recency / frequency / user-role salience. Assistant-introduced + entities and "what did she invest in?" follow-ups whose antecedent was named + in the window now resolve. +2. **Resolve** through the alias table, exact titles, and slug suffixes — each + arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6, + +0.05 when mentioned in ≥2 turns or the newest turn. +3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an + explicit lower gate), suppress pages already surfaced (slug-presence only), + cap at 3 pages (hard cap 5). + +## CLI + +```bash +# one-shot: pipe recent turns (oldest → newest) +printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \ + | gbrain volunteer-context + +# streaming: volunteered pages print as the transcript flows +some-transcript-feed | gbrain watch --json + +# the feedback loop: how often were volunteered pages actually opened? +gbrain volunteer-context --stats +``` + +Stats are **approximate** by design: "used" means `pages.last_retrieved_at > +volunteered_at` — the 5-minute last-retrieved throttle causes false negatives +and unrelated reads of the same page cause false positives. Use the per-arm +precision to tune `min_confidence`, not as an exact metric. + +**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its +connection for the whole session — a concurrent `gbrain serve` or any write +path blocks until watch exits. On a PGLite brain, run watch in bursts (piped +input exits at EOF) or use the ambient reflex channel instead, which routes +through a running serve's resolve socket rather than taking the lock. Routing +watch through that same socket is a filed follow-up (TODOS.md). Postgres +brains are unaffected. + +## Config + +| Key | Default | What it does | +|---|---|---| +| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) | +| `retrieval_reflex` | true | the ambient channel's master switch | +| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn | + +Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch` +(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch); +on the op only: `prior_context` (text whose already-surfaced slugs are suppressed), +`session_id` / `turn` attribution params (watch stamps its own per-session id and +turn numbers in the feedback log), and `days` to size the `--stats` window. + +## Storage + privacy + +Volunteered pages log to `context_volunteer_events` (migration v117): slug, +arm, confidence, channel, optional session/turn — the rationale is a +deterministic template string, never raw conversation text. Event writes are +best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal, +not an audit trail. Rows are pruned after 90 days by the dream cycle's purge +phase. Synopses always strip the takes/facts fences — the same strip `get_page` +applies to untrusted callers, applied unconditionally here so private fence rows +never reach a prompt regardless of caller trust. + +--- + ## docs/mcp/DEPLOY.md Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md diff --git a/llms.txt b/llms.txt index 8ca28d88a..34d94b6d4 100644 --- a/llms.txt +++ b/llms.txt @@ -25,6 +25,7 @@ Repo: https://github.com/garrytan/gbrain - [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist. - [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery. - [docs/guides/scaling-skills.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/scaling-skills.md): Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss. +- [docs/guides/push-context.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md): Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop. - [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment. ## AI providers diff --git a/package.json b/package.json index a345afb96..9ba4444fd 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.42.0" + "version": "0.42.45.0" } diff --git a/scripts/ci-local.sh b/scripts/ci-local.sh index b7407fa6b..06ec621d6 100755 --- a/scripts/ci-local.sh +++ b/scripts/ci-local.sh @@ -196,7 +196,10 @@ SELECTED=$(bun run scripts/select-e2e.ts) if [ -z "$SELECTED" ]; then echo "[runner] selector emitted nothing (doc-only diff); skipping E2E." else - DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test echo "$SELECTED" | xargs bash scripts/run-e2e.sh + DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \ + GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \ + GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \ + echo "$SELECTED" | xargs bash scripts/run-e2e.sh fi' else RUN_PHASES_CMD='echo "[runner] guards + typecheck" @@ -208,7 +211,10 @@ bun run typecheck echo "[runner] unit (unsharded, DATABASE_URL unset)" env -u DATABASE_URL bash scripts/run-unit-shard.sh echo "[runner] e2e (unsharded)" -DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test bash scripts/run-e2e.sh' +DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \ +GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \ +GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \ +bash scripts/run-e2e.sh' fi else # Tier 1 sharded path. Each shard runs unit+E2E sequentially against its @@ -257,10 +263,14 @@ printf '%s\\n' 1 2 3 4 | xargs -P4 -I{} sh -c ' if [ -s /tmp/e2e-selected.txt ]; then SHARD=\${shard}/4 \\ DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\ + GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\ + GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\ xargs -a /tmp/e2e-selected.txt bash scripts/run-e2e.sh >> \$log 2>&1 else SHARD=\${shard}/4 \\ DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\ + GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\ + GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\ bash scripts/run-e2e.sh >> \$log 2>&1 fi e2e_exit=\$? diff --git a/scripts/llms-config.ts b/scripts/llms-config.ts index e81154c6a..b2b338a86 100644 --- a/scripts/llms-config.ts +++ b/scripts/llms-config.ts @@ -151,6 +151,12 @@ export const SECTIONS: DocSection[] = [ "Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.", path: "docs/guides/scaling-skills.md", }, + { + title: "docs/guides/push-context.md", + description: + "Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.", + path: "docs/guides/push-context.md", + }, { title: "docs/mcp/DEPLOY.md", description: "MCP server deployment.", diff --git a/src/cli.ts b/src/cli.ts index 82b7e22a6..4cab59fbd 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -24,6 +24,7 @@ import type { GBrainConfig } from './core/config.ts'; import type { AIGatewayConfig } from './core/ai/types.ts'; import type { BrainEngine } from './core/engine.ts'; import { operations, OperationError } from './core/operations.ts'; +import { formatVolunteeredPage } from './core/context/volunteer.ts'; import type { Operation, OperationContext } from './core/operations.ts'; import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts'; import { serializeMarkdown } from './core/markdown.ts'; @@ -43,7 +44,7 @@ for (const op of operations) { } // CLI-only commands that bypass the operation layer -const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade']); +const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'watch']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -67,6 +68,8 @@ const CLI_ONLY_SELF_HELP = new Set([ 'capture', // v0.42 self-upgrade ships its own usage (flags + the agent-skill story). 'self-upgrade', + // v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol). + 'watch', // v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was // unreachable via help because the dispatcher's generic CLI-only // short-circuit fired before runSync could print its own usage block. @@ -802,8 +805,27 @@ async function makeContext(engine: BrainEngine, params: Record) }; } -function formatResult(opName: string, result: unknown): string { +// Exported for tests (same import-safety contract as cliAliases/printOpHelp). +export function formatResult(opName: string, result: unknown): string { switch (opName) { + case 'volunteer_context': { + const r = result as any; + // Stats mode (the feedback loop). + if (r && r.approximate === true && Array.isArray(r.by_arm)) { + const lines = [ + `volunteered-context precision — last ${r.days} day(s) (${r.note})`, + `total: ${r.total_volunteered} volunteered, ${r.total_used} used`, + ]; + for (const a of r.by_arm) { + lines.push(` ${a.match_arm}/${a.channel}: ${a.used}/${a.volunteered} used (precision ${a.precision})`); + } + if (!r.by_arm.length) lines.push(' (no volunteer events in the window)'); + return lines.join('\n') + '\n'; + } + const pages = (r?.pages ?? []) as any[]; + if (!pages.length) return 'Nothing volunteered (no entity cleared the confidence gate).\n'; + return pages.map((p) => formatVolunteeredPage(p)).join('\n') + '\n'; + } case 'get_page': { const r = result as any; if (r.error === 'ambiguous_slug') { @@ -925,6 +947,9 @@ function formatResult(opName: string, result: unknown): string { const THIN_CLIENT_REFUSED_COMMANDS = new Set([ 'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations', 'repair-jsonb', 'orphans', 'integrity', 'serve', + // v0.43 (#2095): watch streams against a LOCAL engine; thin clients get + // the volunteer_context MCP op instead. + 'watch', // v0.31.1 (CDX-2 op coverage matrix): more local-only commands 'dream', 'transcripts', 'storage', // v0.31.1 CDX-2 audit: takes/sources have multiple subcommands; some @@ -1156,11 +1181,14 @@ async function handleCliOnly(command: string, args: string[]) { } if (command === 'friction') { const { runFriction } = await import('./commands/friction.ts'); - process.exit(runFriction(args)); + // #2084 inner-exit sweep: verdict + return so teardown + the flush seam run. + setCliExitVerdict(runFriction(args)); + return; } if (command === 'claw-test') { const { runClawTest } = await import('./commands/claw-test.ts'); - process.exit(await runClawTest(args)); + setCliExitVerdict(await runClawTest(args)); + return; } if (command === 'report') { const { runReport } = await import('./commands/report.ts'); @@ -1269,7 +1297,7 @@ async function handleCliOnly(command: string, args: string[]) { execSync(`bash "${scriptPath}"`, { stdio: 'inherit', env: { ...process.env } }); } catch (e: any) { // Non-zero exit = some tests failed (exit code = failure count) - process.exit(e.status ?? 1); + setCliExitVerdict(e.status ?? 1); } return; } @@ -1318,7 +1346,8 @@ async function handleCliOnly(command: string, args: string[]) { // The handler self-configures the AI gateway from loadConfig() + process.env. if (command === 'eval' && args[0] === 'cross-modal') { const { runEvalCrossModal } = await import('./commands/eval-cross-modal.ts'); - process.exit(await runEvalCrossModal(args.slice(1))); + setCliExitVerdict(await runEvalCrossModal(args.slice(1))); + return; } // v0.32 EXP-5 (codex review #10): `eval takes-quality replay ` @@ -1329,7 +1358,8 @@ async function handleCliOnly(command: string, args: string[]) { // engine-required path below. if (command === 'eval' && args[0] === 'takes-quality' && args[1] === 'replay') { const { runReplayNoBrain } = await import('./commands/eval-takes-quality.ts'); - process.exit(await runReplayNoBrain(args.slice(2))); + setCliExitVerdict(await runReplayNoBrain(args.slice(2))); + return; } // v0.28.8: longmemeval brings its own in-memory PGLite. Bypassing @@ -1361,7 +1391,8 @@ async function handleCliOnly(command: string, args: string[]) { // gate runs on machines with no `~/.gbrain/config.json`. if (command === 'eval' && args[0] === 'conversation-parser') { const { runEvalConversationParser } = await import('./commands/eval-conversation-parser.ts'); - process.exit(await runEvalConversationParser(args.slice(1))); + setCliExitVerdict(await runEvalConversationParser(args.slice(1))); + return; } // v0.41.13.0: `gbrain conversation-parser list-builtins | validate @@ -1389,7 +1420,8 @@ async function handleCliOnly(command: string, args: string[]) { const cfgPre = loadConfig(); if (isThinClient(cfgPre)) { const { runEvalWhoknows } = await import('./commands/eval-whoknows.ts'); - process.exit(await runEvalWhoknows(null, args.slice(1))); + setCliExitVerdict(await runEvalWhoknows(null, args.slice(1))); + return; } } @@ -1403,7 +1435,8 @@ async function handleCliOnly(command: string, args: string[]) { if (cfgPre && isThinClient(cfgPre)) { const { runStatus } = await import('./commands/status.ts'); const result = await runStatus(null, args); - process.exit(result.exitCode); + setCliExitVerdict(result.exitCode); + return; } } @@ -1716,8 +1749,8 @@ async function handleCliOnly(command: string, args: string[]) { case 'status': { const { runStatus } = await import('./commands/status.ts'); const result = await runStatus(engine, args); - process.exit(result.exitCode); - // eslint-disable-next-line no-unreachable + // #2084 inner-exit sweep: a mid-switch exit skips the finally teardown. + setCliExitVerdict(result.exitCode); break; } // v0.38 — Capture: single human-facing entrypoint for ingestion. @@ -1887,6 +1920,15 @@ async function handleCliOnly(command: string, args: string[]) { await runQuarantine(engine, args); break; } + case 'watch': { + // v0.43 (#2095): push-based context transport. Blocks in the stdin + // iteration (interactive stays alive; piped exits at EOF), then the + // finally below runs finishCliTeardown (volunteer events drain with + // every other sink) and the import.meta.main seam flush-exits. + const { runWatch } = await import('./commands/watch.ts'); + await runWatch(engine, args); + break; + } case 'storage': { const { runStorage } = await import('./commands/storage.ts'); await runStorage(engine, args); @@ -2267,6 +2309,8 @@ ADMIN --public-url URL Public issuer URL (required behind proxy/tunnel) connect --token Wire Claude Code to a remote gbrain (bearer token) [--install] [--json] Print the paste-ready command, or --install to run it + watch [--json] Push-based context: pipe conversation turns in, + volunteered brain pages stream out (#2095) call '' Raw tool invocation version Version info --tools-json Tool discovery (JSON) diff --git a/src/commands/config.ts b/src/commands/config.ts index 075de0ca5..d4e0bb1c6 100644 --- a/src/commands/config.ts +++ b/src/commands/config.ts @@ -181,6 +181,20 @@ export async function runConfig(engine: BrainEngine, args: string[]) { const coverageOverride = args.includes('--coverage-override') || args.includes('--yes'); + // v0.42.42.0 (#2139): validate spend.posture at set time so a typo + // ('tokenMax', 'max') doesn't silently fall back to gated. + if (key === 'spend.posture') { + const { isValidSpendPosture } = await import('../core/spend-posture.ts'); + if (!isValidSpendPosture(value)) { + console.error( + `[config] spend.posture must be 'gated' or 'tokenmax' (got '${value}').\n` + + `[config] gbrain config set spend.posture tokenmax # cost gates become informational\n` + + `[config] gbrain config set spend.posture gated # default — gates enforce`, + ); + process.exit(1); + } + } + if (key === 'embedding_columns') { try { const parsed = JSON.parse(value); diff --git a/src/commands/enrich.ts b/src/commands/enrich.ts index b859f0669..99a04ed24 100644 --- a/src/commands/enrich.ts +++ b/src/commands/enrich.ts @@ -508,8 +508,13 @@ export async function runEnrichCore( // One tracker reference for both the run and the post-hoc overage check. // External tracker (cycle phase): used as-is, no withBudgetTracker wrap (that // would REPLACE not stack). Internal: capped at maxCostUsd ?? DEFAULT. + // v0.42.42.0 (#2139): Infinity = explicit "uncapped" (off / tokenmax) → pass + // `undefined` so BudgetTracker runs without a ceiling (NOT raw Infinity, which + // would serialize to null in audit rows). undefined-when-unset still → DEFAULT. + const resolvedCap = + opts.maxCostUsd === Infinity ? undefined : (opts.maxCostUsd ?? DEFAULT_MAX_COST_USD); const tracker = opts.budgetTracker ?? new BudgetTracker({ - maxCostUsd: opts.maxCostUsd ?? DEFAULT_MAX_COST_USD, + maxCostUsd: resolvedCap, label: `enrich:${sourceId}`, }); try { @@ -622,8 +627,15 @@ export function parseArgs(args: string[]): ParsedArgs { continue; } if (a === '--max-usd' || a === '--max-cost-usd') { - const n = parseFloat(args[++i] ?? ''); - if (Number.isFinite(n) && n > 0) out.maxCostUsd = n; + const raw = args[++i] ?? ''; + // v0.42.42.0 (#2139): off/unlimited/none → run uncapped (Infinity sentinel; + // mapped to "no BudgetTracker ceiling" in runEnrichCore). Spend still ledgered. + if (['off', 'unlimited', 'none'].includes(raw.trim().toLowerCase())) { + out.maxCostUsd = Infinity; + } else { + const n = parseFloat(raw); + if (Number.isFinite(n) && n > 0) out.maxCostUsd = n; + } continue; } if (a === '--min-context') { @@ -801,9 +813,24 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise posture). + const explicitOff = parsed.maxCostUsd === Infinity; + const { resolveSpendPosture } = await import('../core/spend-posture.ts'); + const posture = parsed.dryRun ? 'gated' : await resolveSpendPosture(engine); + const uncapped = + !parsed.dryRun && (explicitOff || (parsed.maxCostUsd === undefined && posture === 'tokenmax')); + if (uncapped) { + console.error(`${explicitOff ? '--max-usd off' : 'spend.posture=tokenmax'}: running uncapped, spend ledgered. docs: docs/operations/spend-controls.md`); + } + // Non-TTY execute without --max-usd or --yes is refused (cost guardrail). - if (!parsed.dryRun && parsed.maxCostUsd === undefined && !parsed.yes && !process.stdout.isTTY) { - console.error('Refusing to spend without a cap in a non-interactive context. Pass --max-usd or --yes.'); + if (!parsed.dryRun && parsed.maxCostUsd === undefined && !parsed.yes && !process.stdout.isTTY && !uncapped) { + console.error('Refusing to spend without a cap in a non-interactive context. Pass --max-usd (or `off`), --yes, or set spend.posture=tokenmax.'); process.exit(1); } @@ -812,7 +839,7 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise s.id); // Dry-run cost preview (TTY) before spending. - if (!parsed.dryRun && process.stdout.isTTY && !parsed.yes && parsed.maxCostUsd === undefined) { + if (!parsed.dryRun && process.stdout.isTTY && !parsed.yes && parsed.maxCostUsd === undefined && !uncapped) { const limit = parsed.limit ?? DEFAULT_LIMIT; const est = (limit * sourceIds.length * COST_ESTIMATE_PER_PAGE_USD).toFixed(2); console.error(`About to enrich up to ${limit} page(s) per source across ${sourceIds.length} source(s), est. ~$${est}. Re-run with --max-usd or --yes to confirm.`); @@ -834,7 +861,9 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise= 0 ? (args[maxUsdIdx + 1] ?? '').trim().toLowerCase() : ''; + const maxUsdOff = ['off', 'unlimited', 'none'].includes(maxUsdVal); const maxUsdRaw = parseFloat10(args, '--max-usd'); const maxUsd = maxUsdRaw === null ? undefined : maxUsdRaw; @@ -91,13 +98,23 @@ export async function runOnboard(engine: BrainEngine, args: string[]): Promise= 0) { const v = args[idx + 1]; + const t = (v ?? '').trim().toLowerCase(); + if (['off', 'unlimited', 'none'].includes(t)) { + maxCostUsd = undefined; // no runtime cap (reindex skips the tracker when unset) + maxCostOff = true; + break; + } const n = v ? parseFloat(v) : NaN; if (!Number.isFinite(n) || n <= 0) { - console.error(`gbrain reindex --code: ${flag} requires a positive number in USD (got ${v ?? '(missing)'})`); + console.error(`gbrain reindex --code: ${flag} requires a positive number in USD, or off/unlimited (got ${v ?? '(missing)'})`); process.exit(2); } maxCostUsd = n; @@ -493,20 +503,36 @@ export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Pr } if (!yes) { - const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); - if (!isTTY || json) { - // Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits - // on --json now (human refusal on stderr otherwise) — #1784. - const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() }); - if (refusal.stdout) console.log(refusal.stdout); - if (refusal.stderr) console.error(refusal.stderr); - process.exit(2); - } - console.log(previewMsg); - const answer = await promptYesNo('Proceed? [y/N] '); - if (!answer) { - console.log('Cancelled.'); - return; + // v0.42.42.0 (#2139): spend.posture=tokenmax makes the gate informational + // — print the estimate and proceed (the operator declared cost isn't the + // constraint). The spend is still ledgered by the runtime BudgetTracker. + const { resolveSpendPosture } = await import('../core/spend-posture.ts'); + const posture = await resolveSpendPosture(engine); + // An explicit `--max-cost off` is the same "cost isn't the constraint" + // signal as spend.posture=tokenmax — proceed past the confirmation gate. + if (posture === 'tokenmax' || maxCostOff) { + const gate = maxCostOff ? 'max_cost_off' : 'posture_tokenmax'; + if (json) { + console.log(JSON.stringify({ status: 'proceeding', gate, codePages: preview.totalPages, totalTokens: preview.totalTokens, costUsd, model: getEmbeddingModelName() })); + } else { + console.log(`${previewMsg} ${maxCostOff ? '--max-cost off' : 'spend.posture=tokenmax'}: proceeding (informational). docs: docs/operations/spend-controls.md`); + } + } else { + const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); + if (!isTTY || json) { + // Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits + // on --json now (human refusal on stderr otherwise) — #1784. + const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() }); + if (refusal.stdout) console.log(refusal.stdout); + if (refusal.stderr) console.error(refusal.stderr); + process.exit(2); + } + console.log(previewMsg); + const answer = await promptYesNo('Proceed? [y/N] '); + if (!answer) { + console.log('Cancelled.'); + return; + } } } } diff --git a/src/commands/sync.ts b/src/commands/sync.ts index 74e44a2c3..d788cd201 100644 --- a/src/commands/sync.ts +++ b/src/commands/sync.ts @@ -7,12 +7,11 @@ import { importFile } from '../core/import-file.ts'; import { collectSyncableFiles } from './import.ts'; import { createInterface } from 'readline'; import { - buildSyncManifest, isSyncable, unsyncableReason, resolveSlugForPath, unacknowledgedSyncFailures, - acknowledgeSyncFailures, + acknowledgeFailures, loadSyncFailures, formatCodeBreakdown, applySyncFailureGate, @@ -20,6 +19,16 @@ import { resolveAutoSkipThreshold, DEFAULT_SOURCE_ID, } from '../core/sync.ts'; +import { + computeSyncDelta, + buildDetachedWorkingTreeManifest, +} from '../core/sync-delta.ts'; +import { fetchRemote } from '../core/git-remote.ts'; +import { + parseUsdLimit, + formatUsdLimit, + resolveSpendPosture, +} from '../core/spend-posture.ts'; import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts'; import { estimateEmbeddingCostUsd, @@ -28,11 +37,10 @@ import { currentEmbeddingSignature, willEmbedSynchronously, shouldBlockSync, + type SyncEmbedMode, } from '../core/embedding.ts'; import { estimateCostFromChars } from '../core/embedding-pricing.ts'; -import { isSourceUnchangedSinceSync } from '../core/git-head.ts'; import { SPEND_CAP_CONFIG_KEY } from '../core/embed-backfill-submit.ts'; -import { errorFor, serializeError } from '../core/errors.ts'; import type { SyncManifest } from '../core/sync.ts'; import { createProgress } from '../core/progress.ts'; import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; @@ -217,16 +225,17 @@ export interface SyncResult { /** * Walk ONE source's working tree and sum tokens for every syncable file. - * Conservative full-tree ceiling (full file content, not the incremental - * diff) — over-counts, never under-counts, and matches the filesystem set - * `sync` actually imports (collectSyncableFiles + content_hash, NOT a git - * commit diff). Best-effort per file and per source: anything unreadable - * contributes 0 rather than blocking the preview. + * Conservative full-tree CEILING (full file content, not the incremental + * diff) — over-counts, never under-counts. Used only on the ceiling rungs of + * `estimateInlineNewTokens` (first sync, chunker drift, git-unavailable), + * where the delta is genuinely the whole tree or can't be computed. * * v0.31.2: routed through collectSyncableFiles (lstat + inode-cycle + * max-depth) so the preview walks exactly what the real sync walks. + * + * Exported (v0.42.42.0, #2139) for direct unit testing. */ -function estimateSourceTreeTokens( +export function estimateSourceTreeTokens( localPath: string, strategy: 'markdown' | 'code' | 'auto', ): { tokens: number; files: number } { @@ -251,18 +260,114 @@ function estimateSourceTreeTokens( return { tokens, files }; } +/** Sum tokens for an explicit set of repo-relative paths read at live working-tree content. */ +function estimateDeltaTokens(localPath: string, relPaths: string[]): number { + let tokens = 0; + for (const rel of relPaths) { + try { + const full = join(localPath, rel); + const stat = statSync(full); + if (stat.size > 5_000_000) continue; // skip large binaries (matches tree walk) + tokens += estimateTokens(readFileSync(full, 'utf-8')); + } catch { + // Listed in the diff but unreadable (e.g. since deleted) → 0, like the tree walk. + } + } + return tokens; +} + /** - * v0.41.31 — INLINE-path new-content estimate. Per source, contribute ZERO - * when the source is provably unchanged since its last sync (HEAD == - * last_commit AND clean working tree AND chunker_version matches CURRENT) — - * `content_hash` short-circuits every file so nothing re-embeds. Otherwise - * contribute the full-tree ceiling. The unchanged predicate mirrors - * doctor's `sync_freshness` and sync's own "do work?" gate (sync.ts:1057+ - * 1075). `isSourceUnchangedSinceSync` is fail-open (probe error → false), so - * a source we can't prove unchanged is conservatively re-estimated rather - * than silently priced at $0. + * v0.42.42.0 (#2139): resolve the commit the estimate should diff AGAINST. + * + * The cost gate runs BEFORE sync's own `git pull`, so a stale local HEAD would + * make the estimate blind to commits the run is about to pull (codex #1). So + * we FETCH first (fail-open) and target `origin/` — the estimate then + * prices exactly what this run will sync. The subsequent pull fast-forwards + * the already-fetched objects, so net new network cost ≈ 0. + * + * - detached HEAD → no upstream; target = local HEAD (+ caller merges the + * detached working-tree manifest, which sync imports on a detached repo). + * - attached + origin remote → fetch origin/ (best-effort), target = + * origin/ if resolvable, else local HEAD (offline / no upstream). + * - HEAD unresolvable (not a git repo) → null (caller treats as unavailable). + * + * NOTE: this makes `--dry-run` perform a network fetch so the preview reflects + * what a real run would pull. Fail-open: offline dry-run still previews against + * local HEAD. Uses the shared `git()` 30s budget (the fetch cost is the pull + * cost paid a few seconds early — a tighter cap would frequently fall back to + * local HEAD and underestimate the remote delta). */ -function estimateInlineNewTokens( +function resolveEstimateTarget(localPath: string): { target: string; detached: boolean } | null { + let head: string; + try { + head = git(localPath, ['rev-parse', 'HEAD']); + } catch { + return null; + } + const detached = isDetachedHead(localPath); + if (detached) return { target: head, detached: true }; + + let branch: string | null = null; + try { + branch = git(localPath, ['rev-parse', '--abbrev-ref', 'HEAD']).trim() || null; + } catch { + branch = null; + } + if (branch && branch !== 'HEAD' && hasOriginRemote(localPath)) { + try { + // v0.42.42.0 (#2139): route through the SSRF-hardened fetch (same flags + + // no-prompt env as pullRepo) — a cost preview / dry-run must NOT hit a + // remote through a less-protected path than real sync. + fetchRemote(localPath, branch, { timeoutMs: 30_000 }); + } catch { + // fail-open: offline, auth failure, no upstream — fall through to local HEAD. + } + try { + const remoteSha = git(localPath, ['rev-parse', `origin/${branch}`]); + if (remoteSha) return { target: remoteSha, detached: false }; + } catch { + // no remote-tracking ref for this branch — use local HEAD. + } + } + return { target: head, detached: false }; +} + +export type EstimateKind = 'delta' | 'ceiling' | 'mixed' | 'unchanged'; + +export interface InlineEstimate { + tokens: number; + changedSources: number; + unchangedSources: number; + estimateKind: EstimateKind; + /** Per-source ceiling reasons (chunker_drift / first_sync / git_unavailable) for honest labeling. */ + ceilingReasons: string[]; +} + +/** + * v0.42.42.0 (#2139) — INLINE-path new-content estimate. The estimate now + * MIRRORS EXECUTION instead of pricing the whole tree on every dirty sync (the + * 400x overestimate that wedged the daily cron). Per-source fail-open ladder: + * + * 1. syncEnabled === false → skip (unchanged) + * 2. chunker drift (stored !== current) → full-tree CEILING (a drift forces + * performFullSync → full re-chunk → full re-embed; a delta would + * underestimate by the whole corpus). kind: ceiling_chunker_drift + * 3. last_commit === fetch target → 0 (mirrors `up_to_date` at + * sync.ts:1402 — NO clean-working-tree requirement; a dirty tree whose + * commits are caught up imports nothing). kind: unchanged + * 4. last_commit === null (first sync) → full-tree CEILING. kind: ceiling_first_sync + * 5. computeSyncDelta ok → price added∪modified∪renamed.to + * (syncable, live working-tree content); deletes cost 0. kind: delta + * 6. computeSyncDelta unavailable → full-tree CEILING. kind: ceiling_git_unavailable + * + * The delta rung routes through the SAME `computeSyncDelta` the executor uses + * (src/core/sync-delta.ts), so the gate's dollar figure can't drift from what + * the sync imports. `--full`'s extra stale-backlog sweep is added by the gate + * (it already has `staleCostUsd`), not here — see the call site. + * + * Exported for direct unit testing. + */ +export function estimateInlineNewTokens( sources: Array<{ local_path: string | null; config: Record; @@ -270,25 +375,85 @@ function estimateInlineNewTokens( chunker_version: string | null; }>, currentChunkerVersion: string, -): { tokens: number; changedSources: number; unchangedSources: number } { +): InlineEstimate { let tokens = 0; let changedSources = 0; let unchangedSources = 0; + let hadDelta = false; + let hadCeiling = false; + const ceilingReasons: string[] = []; + + const ceiling = (localPath: string, strategy: 'markdown' | 'code' | 'auto', reason: string) => { + tokens += estimateSourceTreeTokens(localPath, strategy).tokens; + changedSources++; + hadCeiling = true; + ceilingReasons.push(reason); + }; + for (const src of sources) { if (!src.local_path) continue; const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' }; if (cfg.syncEnabled === false) continue; - const unchanged = - isSourceUnchangedSinceSync(src.local_path, src.last_commit, { requireCleanWorkingTree: true }) && - src.chunker_version === currentChunkerVersion; - if (unchanged) { + const strategy = cfg.strategy ?? 'markdown'; + const localPath = src.local_path; + + // Rung 2: chunker drift forces a full re-chunk → full re-embed. CEILING. + if (src.chunker_version !== currentChunkerVersion) { + ceiling(localPath, strategy, 'chunker_drift'); + continue; + } + + // Rung 4 (early): no bookmark → first sync imports everything. CEILING. + if (!src.last_commit) { + ceiling(localPath, strategy, 'first_sync'); + continue; + } + + const resolved = resolveEstimateTarget(localPath); + if (!resolved) { + // HEAD unresolvable (not a git repo / gone) — can't compute a delta. CEILING. + ceiling(localPath, strategy, 'git_unavailable'); + continue; + } + + // Rung 3: caught up to the fetch target AND no detached working-tree changes. + // Mirrors the executor's `up_to_date` predicate — a dirty-but-committed-current + // tree imports nothing, so it must price $0 (the heart of the false-fire fix). + const detachedManifest = resolved.detached + ? buildDetachedWorkingTreeManifest(localPath) + : null; + const detachedHasChanges = detachedManifest !== null && + (detachedManifest.added.length > 0 || + detachedManifest.modified.length > 0 || + detachedManifest.deleted.length > 0 || + detachedManifest.renamed.length > 0); + if (src.last_commit === resolved.target && !detachedHasChanges) { unchangedSources++; continue; } + + // Rung 5/6: the delta itself — SAME helper the executor diffs with. + const delta = computeSyncDelta(localPath, src.last_commit, resolved.target, { + detachedManifest, + }); + if (delta.status === 'unavailable') { + ceiling(localPath, strategy, 'git_unavailable'); + continue; + } + const syncOpts = { strategy }; + const changedPaths = unique([ + ...delta.manifest.added.filter(p => isSyncable(p, syncOpts)), + ...delta.manifest.modified.filter(p => isSyncable(p, syncOpts)), + ...delta.manifest.renamed.filter(r => isSyncable(r.to, syncOpts)).map(r => r.to), + ]); + tokens += estimateDeltaTokens(localPath, changedPaths); changedSources++; - tokens += estimateSourceTreeTokens(src.local_path, cfg.strategy ?? 'markdown').tokens; + hadDelta = true; } - return { tokens, changedSources, unchangedSources }; + + const estimateKind: EstimateKind = + hadCeiling && hadDelta ? 'mixed' : hadCeiling ? 'ceiling' : hadDelta ? 'delta' : 'unchanged'; + return { tokens, changedSources, unchangedSources, estimateKind, ceilingReasons }; } /** @@ -302,9 +467,10 @@ function estimateInlineNewTokens( async function resolveCostGateFloorUsd(engine: BrainEngine): Promise { try { const raw = await engine.getConfig('sync.cost_gate_min_usd'); - if (raw === null || raw === undefined) return 0.5; - const n = Number(raw); - return Number.isFinite(n) && n >= 0 ? n : 0.5; + // v0.42.42.0 (#2139): `0` keeps meaning "block on any nonzero spend" + // (allowZero); `off`/`unlimited`/`none` → Infinity so the gate never + // blocks (`costUsd > Infinity` is always false). + return parseUsdLimit(raw, 0.5, { allowZero: true }); } catch { return 0.5; } @@ -318,9 +484,9 @@ async function resolveCostGateFloorUsd(engine: BrainEngine): Promise { async function resolveBackfillCapUsd(engine: BrainEngine): Promise { try { const raw = await engine.getConfig(SPEND_CAP_CONFIG_KEY); - if (raw === null || raw === undefined) return 25; - const n = Number(raw); - return Number.isFinite(n) && n > 0 ? n : 25; + // v0.42.42.0 (#2139): no allowZero — `0` falls back to the default + // (off semantics ≠ 0); `off`/`unlimited`/`none` → Infinity (cap disabled). + return parseUsdLimit(raw, 25); } catch { return 25; } @@ -338,6 +504,214 @@ async function promptYesNo(question: string): Promise { }); } +// v0.42.42.0 (#2139): paste-ready knobs appended to every gate message so the +// spend-control surface is discoverable at the moment of need (humans + agents), +// not only after reading source. Closes the issue's "takes archaeology" complaint. +const SPEND_HINT = + 'widen: gbrain config set sync.cost_gate_min_usd 5 | ' + + 'never gate: gbrain config set spend.posture tokenmax | ' + + 'docs: docs/operations/spend-controls.md'; + +/** Honest token label — delta vs full-tree ceiling, with the ceiling reasons. */ +function labelEstimate(inline: InlineEstimate): string { + if (inline.estimateKind === 'unchanged') return '0 new tokens (sources caught up)'; + if (inline.estimateKind === 'delta') { + return `~${inline.tokens.toLocaleString()} new tokens (delta: changed files since last sync)`; + } + // ceiling | mixed — be explicit that this is an over-count, not the real spend. + const reasons = unique(inline.ceilingReasons).join(', ') || 'unknown'; + return ( + `<=${inline.tokens.toLocaleString()} tokens (full-tree ceiling for ${inline.changedSources} ` + + `source(s): ${reasons} — unchanged files skip via content_hash at execution)` + ); +} + +type CostGateSource = { + local_path: string | null; + config: Record; + last_commit: string | null; + chunker_version: string | null; +}; + +interface CostGateContext { + sources: CostGateSource[]; + /** Resolved embed mode. Single-source is always 'inline' (not the parallel-deferred fan-out). */ + mode: SyncEmbedMode; + dryRun: boolean; + jsonOut: boolean; + yesFlag: boolean; + full: boolean; + /** Message prefix ('sync --all' | 'sync'). */ + label: string; +} + +type CostGateOutcome = + | { action: 'proceed'; autoDeferEmbeds: boolean } + | { action: 'stop' }; + +/** + * v0.42.42.0 (#2139): the inline-embed cost gate, shared by BOTH `sync --all` + * and single-source `sync` so the spend surface is consistent. Runs at the + * COMMAND layer (never inside performSync, which `runOne` also calls — that + * would double-gate `--all`). + * + * Behavior: + * - deferred mode → FYI only, never blocks (backfill cap is the money gate). + * - inline + below floor → proceed quietly. + * - inline + spend.posture=tokenmax → informational, proceed inline. + * - inline + above floor + TTY → [y/N] prompt. + * - inline + above floor + non-TTY/--json → AUTO-DEFER embeds to capped + * backfill jobs, exit 0 (NEVER exit 2 — the wedged-cron fix). Caller sets + * effectiveNoEmbed and enqueues the backfill. + * + * Output format splits on the EXPLICIT `--json` flag only (absorbs the + * TODOS.md:340 #1784 conflation): JSON envelope iff `--json`, else human text. + */ +async function runInlineCostGate( + engine: BrainEngine, + ctx: CostGateContext, +): Promise { + const { sources, mode, dryRun, jsonOut, yesFlag, full, label } = ctx; + + // Stale backlog: cheap single SQL; fail-open to 0 so a transient DB hiccup + // never blocks the sync. Signature-aware (model/dims swap surfaces here). + let staleChars = 0; + try { + staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() }); + } catch { + staleChars = 0; + } + const staleCostUsd = estimateCostFromChars(staleChars, currentEmbeddingPricePerMTok()); + const embeddingModelName = getEmbeddingModelName(); + const floorUsd = await resolveCostGateFloorUsd(engine); + const posture = await resolveSpendPosture(engine); + + if (mode === 'deferred') { + // Deferred path: print an FYI, NEVER block. The backfill cap is the real + // money gate. + const capUsd = await resolveBackfillCapUsd(engine); + let queuedBackfills = 0; + try { + const r = await engine.executeRaw<{ n: number }>( + `SELECT COUNT(*)::int AS n FROM minion_jobs + WHERE name = 'embed-backfill' + AND status IN ('waiting','active','delayed','waiting-children')`, + ); + queuedBackfills = Number(r[0]?.n) || 0; + } catch { + queuedBackfills = 0; + } + const deferredMsg = + `${label}: embedding deferred to backfill jobs ` + + `(capped $${formatUsdLimit(capUsd)}/source/24h, not charged by this sync). ` + + `Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` + + `${embeddingModelName}) across ${sources.length} source(s); ` + + `${queuedBackfills} backfill job(s) queued.`; + if (dryRun) { + if (jsonOut) { + console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd: formatUsdLimit(capUsd), floorUsd: formatUsdLimit(floorUsd), queuedBackfills, model: embeddingModelName })); + } else { + console.log(deferredMsg); + console.log('--dry-run: exit without syncing.'); + } + return { action: 'stop' }; + } + if (jsonOut) { + console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd: formatUsdLimit(capUsd), floorUsd: formatUsdLimit(floorUsd), queuedBackfills, model: embeddingModelName })); + } else { + console.log(deferredMsg); + } + return { action: 'proceed', autoDeferEmbeds: false }; + } + + // ── Inline path ─────────────────────────────────────────────── + const inline = estimateInlineNewTokens(sources, String(CHUNKER_VERSION)); + // D7A: `--full` runs `performFullSync` → `runEmbedCore({stale:true})`, which + // sweeps the pre-existing stale backlog INLINE on top of the delta. Price it. + const costUsd = estimateEmbeddingCostUsd(inline.tokens) + (full ? staleCostUsd : 0); + const fullNote = full && staleChars > 0 + ? ` (includes ~${staleChars.toLocaleString()} stale-backlog chars swept by --full)` + : ''; + const staleNote = !full && staleChars > 0 + ? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)` + : ''; + const previewMsg = + `${label} preview (inline embed): ${inline.changedSources} changed source(s), ` + + `${inline.unchangedSources} unchanged; ${labelEstimate(inline)}, ` + + `est. $${costUsd.toFixed(2)} on ${embeddingModelName}${fullNote}${staleNote}.`; + + if (dryRun) { + if (jsonOut) { + console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, estimateKind: inline.estimateKind, staleChars, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName })); + } else { + console.log(previewMsg); + console.log('--dry-run: exit without syncing.'); + } + return { action: 'stop' }; + } + + // --yes bypasses the gate entirely (embed inline, no preview). + if (yesFlag) return { action: 'proceed', autoDeferEmbeds: false }; + + // spend.posture=tokenmax → informational, proceed INLINE (operator declared + // cost isn't the constraint; don't defer). + if (posture === 'tokenmax') { + if (jsonOut) { + console.log(JSON.stringify({ status: 'proceeding', mode, gate: 'posture_tokenmax', newTokens: inline.tokens, estimateKind: inline.estimateKind, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName, hint: SPEND_HINT })); + } else { + console.log(`${previewMsg} spend.posture=tokenmax: proceeding (informational). ${SPEND_HINT}`); + } + return { action: 'proceed', autoDeferEmbeds: false }; + } + + // Link intent: search.mode=tokenmax but spend posture unset → nudge once. + let searchModeHint = ''; + try { + const sm = await engine.getConfig('search.mode'); + if (typeof sm === 'string' && sm.trim().toLowerCase() === 'tokenmax') { + searchModeHint = + ` (search.mode=tokenmax detected — \`gbrain config set spend.posture tokenmax\` ` + + `makes cost gates informational)`; + } + } catch { + /* best-effort */ + } + + if (shouldBlockSync(costUsd, floorUsd, mode, posture)) { + const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); + if (isTTY && !jsonOut) { + // Interactive TTY: prompt [y/N]. + console.log(previewMsg + searchModeHint); + const answer = await promptYesNo('Proceed? [y/N] '); + if (!answer) { + console.log('Cancelled.'); + return { action: 'stop' }; + } + return { action: 'proceed', autoDeferEmbeds: false }; + } + // Non-TTY or --json: AUTO-DEFER embeds to capped backfill jobs. NEVER exit 2 + // (the wedged-cron fix). Format splits on the explicit --json flag only. + if (jsonOut) { + console.log(JSON.stringify({ status: 'auto_deferred', mode, gate: 'auto_deferred_embeds', newTokens: inline.tokens, estimateKind: inline.estimateKind, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName, hint: SPEND_HINT })); + } else { + console.log( + `${previewMsg} Exceeds floor $${formatUsdLimit(floorUsd)} in a non-interactive ` + + `session — importing now, deferring embeds to capped backfill jobs. ` + + `Drain: run the jobs worker or \`gbrain embed --stale\`. Pass --yes to embed inline.\n${SPEND_HINT}`, + ); + } + return { action: 'proceed', autoDeferEmbeds: true }; + } + + // Below floor → proceed without blocking (kills inline-cron noise). + if (jsonOut) { + console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, estimateKind: inline.estimateKind, staleChars, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName })); + } else { + console.log(`${previewMsg} Below cost gate floor ($${formatUsdLimit(floorUsd)}), proceeding.`); + } + return { action: 'proceed', autoDeferEmbeds: false }; +} + export interface SyncOpts { repoPath?: string; dryRun?: boolean; @@ -557,19 +931,9 @@ function unique(items: T[]): T[] { return [...new Set(items)]; } -function buildDetachedWorkingTreeManifest(repoPath: string): SyncManifest { - const manifest = buildSyncManifest(git(repoPath, ['diff', '--name-status', '-M', 'HEAD'])); - const untracked = git(repoPath, ['ls-files', '--others', '--exclude-standard']) - .split('\n') - .filter(line => line.length > 0); - - return { - added: unique([...manifest.added, ...untracked]), - modified: unique(manifest.modified), - deleted: unique(manifest.deleted), - renamed: manifest.renamed, - }; -} +// v0.42.42.0 (#2139): `buildDetachedWorkingTreeManifest` relocated to +// `src/core/sync-delta.ts` (re-imported below) so the inline cost estimator +// prices detached sources through the same code the executor imports them with. // v0.18.0 Step 5: source-scoped sync state helpers. When opts.sourceId // is set, read/write the per-source row instead of the global config @@ -1429,29 +1793,28 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise 0) { - const acked = acknowledgeSyncFailures(); - console.log(`Acknowledged ${acked.count} pre-existing failure(s).`); - } - } // v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back // to pre-v0.17 global config (sync.repo_path + sync.last_commit) when @@ -3103,6 +3459,23 @@ See also: if (nudge) process.stderr.write(nudge + '\n'); } + // --skip-failed: acknowledge pre-existing unacked failures BEFORE the sync + // runs, not only ones the current run produces. Without this, the common + // recovery flow — fix the YAML, re-run sync, then run --skip-failed to clear + // the log — fails to clear anything (no NEW failures → the inner ack path in + // performSync is never reached, and "Already up to date." leaves the log). + // + // v0.42.42.0 (#2139, D13C): scoped PER SOURCE. `--all` clears every source's + // open failures; single-source clears only its own (don't ack source B's + // failures when syncing source A). Safe under parallel — the ledger + // serializes writes via `withLedgerLock` and keys rows by `source_id` + // (#1939), which is why the old D15 "no --skip-failed under parallel" + // refusal is lifted below. + if (skipFailed) { + const acked = syncAll ? acknowledgeFailures() : acknowledgeFailures(sourceId); + if (acked.count > 0) console.log(`Acknowledged ${acked.count} pre-existing failure(s).`); + } + // v0.19.0 — `sync --all` iterates all registered sources with a // local_path. Sources are the canonical v0.18.0 abstraction: per-source // last_commit, last_sync_at, config.federated flags. Per-source @@ -3131,132 +3504,21 @@ See also: const { isFederatedV2Enabled } = await import('../core/feature-flags.ts'); const v2Enabled = await isFederatedV2Enabled(engine); - // v0.41.31 cost gate (supersedes the v0.20.0 unconditional gate). Under - // federated_v2 sync DEFERS embedding to per-source embed-backfill jobs - // that carry their own $X/source/24h spend cap, so sync itself spends - // nothing synchronously — the gate is INFORMATIONAL (never exit 2) on - // that path. The blocking ConfirmationRequired gate fires ONLY when embed - // runs INLINE (v2 off, or --serial without --no-embed) AND the estimated - // spend exceeds `sync.cost_gate_min_usd` (default $0.50). Skipped entirely - // when --no-embed is set (user opted out; will run `embed --stale` later). + // v0.42.42.0 (#2139) cost gate — shared `runInlineCostGate`. Under + // federated_v2 + parallel, embedding is DEFERRED to per-source backfill + // jobs (own spend cap) so the gate is FYI-only. Inline mode (v2 off, or + // --serial without --no-embed) gates on the DELTA estimate: below floor + // proceeds; above floor in a non-TTY/--json session AUTO-DEFERS embeds + // (exit 0, never exit 2 — the wedged-cron fix); a TTY prompts. Skipped + // entirely when --no-embed is set. + let autoDeferEmbeds = false; if (!noEmbed) { const mode = willEmbedSynchronously({ v2Enabled, serialFlag, noEmbed }); - // Stale backlog: cheap single SQL; fail-open to 0 so a transient DB - // hiccup never blocks the sync. - let staleChars = 0; - try { - // v0.41.31: signature-aware so a model/dims swap surfaces in the - // backlog estimate (NULL signature grandfathered → not counted). - staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() }); - } catch { - staleChars = 0; - } - const rate = currentEmbeddingPricePerMTok(); - const staleCostUsd = estimateCostFromChars(staleChars, rate); - const embeddingModelName = getEmbeddingModelName(); - const floorUsd = await resolveCostGateFloorUsd(engine); - - if (mode === 'deferred') { - // Deferred path: print an FYI, NEVER exit 2. The backfill cap is the - // real money gate (D1/D4). - const capUsd = await resolveBackfillCapUsd(engine); - // v0.41.31 (TODO-2): surface already-queued backfill jobs so a cron - // operator sees work is enqueued, not lost. Best-effort — minion_jobs - // may not exist on a brain that never ran a worker. - let queuedBackfills = 0; - try { - const r = await engine.executeRaw<{ n: number }>( - `SELECT COUNT(*)::int AS n FROM minion_jobs - WHERE name = 'embed-backfill' - AND status IN ('waiting','active','delayed','waiting-children')`, - ); - queuedBackfills = Number(r[0]?.n) || 0; - } catch { - queuedBackfills = 0; - } - const deferredMsg = - `sync --all: embedding deferred to backfill jobs ` + - `(capped $${capUsd}/source/24h, not charged by this sync). ` + - `Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` + - `${embeddingModelName}) across ${sources.length} source(s); ` + - `${queuedBackfills} backfill job(s) queued.`; - if (dryRun) { - if (jsonOut) { - console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName })); - } else { - console.log(deferredMsg); - console.log('--dry-run: exit without syncing.'); - } - return; - } - if (jsonOut) { - console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName })); - } else { - console.log(deferredMsg); - } - // fall through to sync — no exit 2. - } else { - // Inline path: sync embeds synchronously with no backfill cap to - // protect it, so the blocking gate applies. The BLOCKING cost is the - // new-content estimate ONLY (full-tree ceiling for changed sources; - // unchanged contribute 0) — that's what this sync actually embeds. - // The pre-existing stale backlog (NULL embeddings + signature drift) - // is NOT swept by sync; `gbrain embed --stale` clears it. So we show - // it informationally but never gate on cost this sync won't incur - // (else a model swap would block the next inline cron — F2). - const currentChunkerVersion = String(CHUNKER_VERSION); - const inline = estimateInlineNewTokens(sources, currentChunkerVersion); - const newCostUsd = estimateEmbeddingCostUsd(inline.tokens); - const costUsd = newCostUsd; - const staleNote = staleChars > 0 - ? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)` - : ''; - const previewMsg = - `sync --all preview (inline embed): ${inline.changedSources} changed source(s), ` + - `${inline.unchangedSources} unchanged; ~${inline.tokens.toLocaleString()} new tokens, ` + - `est. $${costUsd.toFixed(2)} on ${embeddingModelName}${staleNote}.`; - - if (dryRun) { - if (jsonOut) { - console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); - } else { - console.log(previewMsg); - console.log('--dry-run: exit without syncing.'); - } - return; - } - - if (!yesFlag) { - if (shouldBlockSync(costUsd, floorUsd, mode)) { - const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY); - if (!isTTY || jsonOut) { - // Agent-facing path: emit structured envelope, exit 2. - const envelope = serializeError(errorFor({ - class: 'ConfirmationRequired', - code: 'cost_preview_requires_yes', - message: previewMsg, - hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.', - })); - console.log(JSON.stringify({ error: envelope, mode, gate: 'confirmation_required', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); - process.exit(2); - } - // Interactive TTY path: prompt [y/N]. - console.log(previewMsg); - const answer = await promptYesNo('Proceed? [y/N] '); - if (!answer) { - console.log('Cancelled.'); - return; - } - } else { - // Below floor → proceed without blocking (kills inline-cron noise). - if (jsonOut) { - console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName })); - } else { - console.log(`${previewMsg} Below cost gate floor ($${floorUsd.toFixed(2)}), proceeding.`); - } - } - } - } + const gate = await runInlineCostGate(engine, { + sources, mode, dryRun, jsonOut, yesFlag, full, label: 'sync --all', + }); + if (gate.action === 'stop') return; + autoDeferEmbeds = gate.autoDeferEmbeds; } // v0.40.5.0 Federated Sync v2 (master) + v0.40.6.0 layering (this branch): @@ -3317,7 +3579,12 @@ See also: const runOne = async (src: typeof sources[number]): Promise => { const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' }; // D18: parallel path defers embed; auto-enqueue embed-backfill after. - const effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed; + // v0.42.42.0 (#2139): `autoDeferEmbeds` (the inline gate tripped in a + // non-TTY session) ALSO forces deferral — global by design (the gate's + // decision unit is the aggregate estimate; deferral strictly dominates + // the exit-2 it replaced for every source). + const effectiveNoEmbed = + (v2Enabled && !serialFlag && !noEmbed ? true : noEmbed) || autoDeferEmbeds; // v0.41.13.0 (T6 / D-V3-3 / D-V4-mech-6) — per-source AbortController. // // When the user passes --timeout, each source gets its OWN @@ -3380,8 +3647,11 @@ See also: // v0.41.13.0 (T7 / D-V3-5): partial excluded — the next clean sync // re-walks the diff and re-decides whether to enqueue embed for // pages whose content actually changed. + // v0.42.42.0 (#2139): `autoDeferEmbeds` enqueues even on the v2-OFF + // legacy path — otherwise the gate's auto-defer would strand + // NULL-embedded chunks with no queued job to embed them. if ( - v2Enabled && + (v2Enabled || autoDeferEmbeds) && !noAutoEmbed && !dryRun && result.status !== 'dry_run' && @@ -3408,18 +3678,13 @@ See also: const parallelEligible = v2Enabled && !serialFlag && engine.kind !== 'pglite' && activeSources.length > 1; - // v0.40.6.0 (D15): refuse --skip-failed / --retry-failed when running - // parallel. sync-failures.jsonl is brain-global; parallel acks race. - if (parallelEligible && (skipFailed || retryFailed)) { - const flag = skipFailed ? '--skip-failed' : '--retry-failed'; - console.error( - `Error: ${flag} is not supported under parallel sync.\n` + - ` (the sync-failures log is brain-global and parallel acks race).\n` + - ` Re-run with --serial for the recovery flow:\n` + - ` gbrain sync --all --serial ${flag}`, - ); - process.exit(1); - } + // v0.42.42.0 (#2139, D13C): the v0.40.6.0 (D15) refusal of --skip-failed / + // --retry-failed under parallel sync is LIFTED. It existed because the + // failure ledger was once brain-global with racing acks; #1939 made the + // ledger per-(source_id, path) and serialized every write through + // `withLedgerLock`, so parallel per-source acks no longer race. Lifting it + // also removes the forcing-function that pushed recovery syncs to --serial + // (and thus armed the inline cost gate) — the root cause behind #2139. // Effective parallelism — surfaced in the --json envelope so consumers // know how the run was actually dispatched. 1 in the serial fallback, @@ -3567,12 +3832,47 @@ See also: signal: composeAbortSignals(singleSourceInterrupt.signal, singleSourceController?.signal), }; + // v0.42.42.0 (#2139, Step 4b): single-source `gbrain sync` gets the SAME + // inline cost gate as `--all`. Previously single-source embedded inline with + // NO gate (only rail: the ≤100-file inline cap). Single-source always embeds + // INLINE (not the parallel-deferred fan-out), so mode is forced 'inline'. + // Skipped on --no-embed, --dry-run (performSync's own dry-run previews and + // spends nothing), and watch. Non-TTY above floor AUTO-DEFERS — so adding + // the gate can never wedge an existing cron; it converts silent ungated + // inline spend into informed inline-or-deferred spend. + let singleSourceAutoDefer = false; + if (!noEmbed && !dryRun && !watch) { + const gateRows = await engine.executeRaw<{ local_path: string | null; config: Record; last_commit: string | null; chunker_version: string | null }>( + `SELECT local_path, config, last_commit, chunker_version FROM sources WHERE id = $1`, + [sourceId], + ); + if (gateRows.length > 0) { + const gateSources = [{ + local_path: gateRows[0].local_path ?? repoPath ?? null, + config: gateRows[0].config ?? {}, + last_commit: gateRows[0].last_commit, + chunker_version: gateRows[0].chunker_version, + }]; + const gate = await runInlineCostGate(engine, { + sources: gateSources, mode: 'inline', dryRun: false, jsonOut, yesFlag, full, label: 'sync', + }); + if (gate.action === 'stop') return; + if (gate.autoDeferEmbeds) { + opts.noEmbed = true; + singleSourceAutoDefer = true; + } + } + } + // Bug 9 — --retry-failed: before running normal sync, clear acknowledgment // flags so the sync picks them up as fresh work. The actual re-attempt // happens inside the regular incremental/full loop because once the commit // pointer is behind the failures, the diff naturally revisits them. if (retryFailed) { - const failures = unacknowledgedSyncFailures(); + // v0.42.42.0 (#2139, D13C): scope the retry count to THIS source — rows + // carry source_id (#1939), so a single-source retry shouldn't report + // another source's failures. + const failures = unacknowledgedSyncFailures().filter(f => f.source_id === sourceId); if (failures.length === 0) { console.log('No unacknowledged sync failures to retry.'); } else { @@ -3615,6 +3915,27 @@ See also: manageGitignore(effectiveRepoPath, engine.kind); } } + // v0.42.42.0 (#2139, Step 4b): the inline gate auto-deferred this run's + // embeds (non-TTY, above floor) — enqueue a capped backfill job so the + // NULL-embedded chunks get embedded out of band instead of being stranded. + if ( + singleSourceAutoDefer && + result.status !== 'dry_run' && + result.status !== 'up_to_date' && + result.status !== 'partial' + ) { + try { + const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts'); + const sub = await submitEmbedBackfill(engine, sourceId, { reason: 'sync_autodefer' }); + if (sub.status === 'submitted') { + process.stderr.write(` → embed-backfill job ${sub.jobId} queued (deferred inline embed).\n`); + } else if (sub.status === 'cooldown') { + process.stderr.write(` → embed-backfill skipped (cooldown); run \`gbrain embed --stale\` to drain now.\n`); + } + } catch (e) { + process.stderr.write(` → embed-backfill submission failed: ${e instanceof Error ? e.message : String(e)}\n`); + } + } return; } diff --git a/src/commands/watch.ts b/src/commands/watch.ts new file mode 100644 index 000000000..9c990e7d2 --- /dev/null +++ b/src/commands/watch.ts @@ -0,0 +1,190 @@ +/** + * v0.43 (#2095) — `gbrain watch`: the push transport for push-based context. + * + * Reads conversation turns from stdin AS THEY ARRIVE (plain text = a user + * turn; `user:` / `assistant:` prefixed lines set the role), maintains a + * rolling window in-process, and volunteers confidence-gated brain pages to + * stdout after every turn. The consumer pipes its transcript in and reads + * volunteered pointers out — no per-entity CLI round-trips. + * + * Lifecycle: BLOCKS in the stdin iteration (like `gbrain jobs work`), so an + * interactive TTY session stays alive until Ctrl-C / Ctrl-D and piped input + * exits at EOF. Either way the handler RETURNS, the CLI_ONLY finally runs + * finishCliTeardown (volunteer events bank before teardown), and the + * entrypoint flush-exit ends the process deliberately — which is exactly why + * `watch` is NOT in DAEMON_COMMANDS: it never returns from main() while work + * is still running. SIGINT closes the stream and flows through the same + * drain path instead of killing mid-write. + * + * Session dedupe: a slug is volunteered at most once per watch session — + * already-pushed slugs ride VolunteerOpts.excludeSlugs, skipped inside the + * core's pointer loop BEFORE the confidence gate and maxPages cap (O(1) + * membership; a post-call filter would let a recurring entity starve new + * pages out of the cap — red-team finding). + * + * PGLite note: watch holds the single-writer engine for the whole session, + * so on the default engine it cannot run concurrently with `gbrain serve` + * (or any other gbrain process) against the same brain — run it against + * Postgres, or stop serve first. Routing watch through the serve resolve-IPC + * socket (like the ambient reflex) is a filed follow-up. + */ + +import { createInterface } from 'node:readline'; +import type { BrainEngine } from '../core/engine.ts'; +import { + volunteerContext, + formatVolunteeredPage, + TURN_PREFIX_RE, + VOLUNTEER_DEFAULT_MAX_PAGES, + VOLUNTEER_DEFAULT_MIN_CONFIDENCE, +} from '../core/context/volunteer.ts'; +import type { WindowTurn } from '../core/context/entity-salience.ts'; +import { DEFAULT_WINDOW_TURNS, windowTurnCount } from '../core/context/reflex.ts'; +import { loadConfig } from '../core/config.ts'; +import { logVolunteerEventsFireAndForget, volunteerEventRowsFrom } from '../core/context/volunteer-events.ts'; + +export const WATCH_HELP = `gbrain watch — push-based context: volunteer brain pages per conversation turn (#2095) + +Reads turns from stdin (one per line; 'user:' / 'assistant:' prefixes set the +role, unprefixed lines are user turns) and prints confidence-gated page +pointers with rationales after each turn. A slug is volunteered at most once +per session. Piped input exits at EOF; interactive sessions exit on Ctrl-C. + +Usage: + some-transcript-feed | gbrain watch [--json] + gbrain watch # interactive: type turns, Ctrl-C to end + +PGLite brains: watch holds the single-writer engine for the whole session — +it cannot run alongside \`gbrain serve\` (or other gbrain processes) against +the same brain. Use a Postgres brain for concurrent access. + +Flags: + --json JSONL output (one volunteered page per line) + --window-turns N rolling extraction window (default ${DEFAULT_WINDOW_TURNS}) + --max-pages N max pages volunteered per turn (default ${VOLUNTEER_DEFAULT_MAX_PAGES}, cap 5) + --min-confidence X confidence gate 0..1 (default ${VOLUNTEER_DEFAULT_MIN_CONFIDENCE}) + --source source scope (defaults to the canonical 6-tier resolution) + --help this text +`; + +function numFlag(args: string[], flag: string): number | undefined { + const i = args.indexOf(flag); + if (i < 0 || i + 1 >= args.length) return undefined; + const n = Number(args[i + 1]); + return Number.isFinite(n) ? n : undefined; +} + +function strFlag(args: string[], flag: string): string | undefined { + const i = args.indexOf(flag); + return i >= 0 && i + 1 < args.length ? args[i + 1] : undefined; +} + +export interface WatchIoDeps { + /** Injected line source for tests (defaults to readline over stdin). */ + lines?: AsyncIterable; + write?: (s: string) => void; + isTTY?: boolean; +} + +export async function runWatch(engine: BrainEngine, args: string[], deps: WatchIoDeps = {}): Promise { + if (args.includes('--help') || args.includes('-h')) { + (deps.write ?? ((s: string) => process.stdout.write(s)))(WATCH_HELP); + return; + } + + const json = args.includes('--json'); + // --window-turns wins; otherwise the same config knob the ambient reflex + // honors (retrieval_reflex_window_turns, default 4) applies here too. + // Hard cap 64: an unbounded window re-scans every retained turn on every + // turn over an hours-long session — the cost class the priorContext fix + // removed, reintroduced via a config typo (red-team finding). + const windowTurns = Math.min( + 64, + Math.max(1, Math.floor(numFlag(args, '--window-turns') ?? windowTurnCount(loadConfig()))), + ); + const maxPages = numFlag(args, '--max-pages'); + const minConfidence = numFlag(args, '--min-confidence'); + const write = deps.write ?? ((s: string) => process.stdout.write(s)); + const isTTY = deps.isTTY ?? Boolean(process.stdin.isTTY); + + const { resolveSourceId } = await import('../core/source-resolver.ts'); + const sourceId = await resolveSourceId(engine, strFlag(args, '--source') ?? null, process.cwd()); + const sourceIds = [sourceId]; + const sessionId = `watch-${process.pid}-${Date.now().toString(36)}`; + + if (!deps.lines) { + // The ready line doubles as a machine-readable readiness signal for + // scripted consumers (and the SIGINT lifecycle test): engine + source + // resolution are done, the stdin loop starts next. + process.stderr.write( + isTTY + ? `[watch] interactive session ${sessionId} ready — type turns ('assistant: ...' to set role), Ctrl-C to end\n` + : `[watch] session ${sessionId} ready\n`, + ); + } + + const rl = deps.lines + ? null + : createInterface({ input: process.stdin, crlfDelay: Infinity }); + const lines: AsyncIterable = deps.lines ?? (rl as AsyncIterable); + + // SIGINT closes the stream so the for-await ends and the normal + // drain-then-exit path runs (never a mid-write kill). + const onSigint = () => { + rl?.close(); + }; + process.on('SIGINT', onSigint); + + const window: WindowTurn[] = []; + const pushedSlugs = new Set(); // session dedupe (slug-only suppression input) + let turnNo = 0; + + try { + for await (const rawLine of lines) { + const line = rawLine.replace(/\r$/, ''); + if (!line.trim()) continue; + const m = TURN_PREFIX_RE.exec(line); + const turn: WindowTurn = m + ? { role: m[1].toLowerCase() as WindowTurn['role'], text: (m[2] ?? '').trim() } + : { role: 'user', text: line.trim() }; + if (!turn.text) continue; + turnNo++; + window.push(turn); + if (window.length > windowTurns) window.splice(0, window.length - windowTurns); + + let pages; + try { + pages = await volunteerContext(engine, [...window], { + sourceIds, + maxPages, + minConfidence, + // Session dedupe: skipped inside the core BEFORE the gate + cap + // (O(1) per pointer) so a recurring slug can't starve new pages. + excludeSlugs: pushedSlugs, + }); + } catch { + continue; // fail-open per turn: a transient DB error never kills the stream + } + if (!pages.length) continue; + + for (const p of pages) pushedSlugs.add(p.slug); + logVolunteerEventsFireAndForget( + engine, + volunteerEventRowsFrom(pages, { channel: 'watch', session_id: sessionId, turn: turnNo }), + ); + + if (json) { + for (const p of pages) { + write(JSON.stringify({ turn: turnNo, ...p }) + '\n'); + } + } else { + for (const p of pages) { + write(formatVolunteeredPage(p) + '\n'); + } + } + } + } finally { + process.off('SIGINT', onSigint); + rl?.close(); + } +} diff --git a/src/core/config.ts b/src/core/config.ts index 70ddca5b7..f62dc6294 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -169,6 +169,14 @@ export interface GBrainConfig { retrieval_reflex?: boolean; /** Max pointers injected per turn (default 3). File-plane only. */ retrieval_reflex_max_pointers?: number; + /** + * v0.43 (#2095) — how many recent turns the reflex extracts entities from + * (default 4). 1 reproduces the legacy current-turn-only behavior (and the + * legacy slug+title suppression). File-plane / env + * (GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS) only — same plane as the other + * reflex knobs. + */ + retrieval_reflex_window_turns?: number; embedding_image_ocr?: boolean; embedding_image_ocr_model?: string; @@ -533,6 +541,10 @@ export function loadConfig(): GBrainConfig | null { ...(process.env.GBRAIN_RETRIEVAL_REFLEX ? { retrieval_reflex: !(process.env.GBRAIN_RETRIEVAL_REFLEX === 'false' || process.env.GBRAIN_RETRIEVAL_REFLEX === '0') } : {}), + ...(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS && + Number.isFinite(Number(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS)) + ? { retrieval_reflex_window_turns: Number(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS) } + : {}), ...(process.env.GBRAIN_REMOTE_CLIENT_SECRET && fileConfig?.remote_mcp ? { remote_mcp: { ...fileConfig.remote_mcp, oauth_client_secret: process.env.GBRAIN_REMOTE_CLIENT_SECRET } } : {}), @@ -892,6 +904,15 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [ // Link resolution (issue #972) 'link_resolution', 'link_resolution.global_basename', + // Spend controls (v0.42.42.0, issue #2139). Previously `--force`-only — the + // operator had to discover these by reading source. Registered so `config + // set` accepts them directly. See docs/operations/spend-controls.md. + 'spend.posture', + 'sync.cost_gate_min_usd', + 'sync.federated_v2', + 'embed.backfill_cooldown_min', + 'embed.backfill_max_usd_per_source_24h', + 'embed.backfill_max_usd', ]; /** diff --git a/src/core/context-engine.ts b/src/core/context-engine.ts index 080e517da..e38114b98 100644 --- a/src/core/context-engine.ts +++ b/src/core/context-engine.ts @@ -179,6 +179,29 @@ function getLastUserText(messages: AgentMessage[]): string { return ''; } +/** + * v0.43 (#2095): the rolling extraction window — the most recent + * user/assistant turns (oldest → newest, current turn last), capped here at + * a generous max; the reflex slices to its configured + * retrieval_reflex_window_turns (default 4). + */ +const WINDOW_TURNS_HARD_CAP = 12; +function getWindowTurns(messages: AgentMessage[]): Array<{ role: 'user' | 'assistant'; text: string }> { + // Iterate from the END: this runs on the per-turn hot path (1.5s reflex + // budget) and only the last 12 turns matter — flattening every content + // block of a multi-hundred-turn session just to slice the tail would make + // the cost grow with session length. + const out: Array<{ role: 'user' | 'assistant'; text: string }> = []; + for (let i = messages.length - 1; i >= 0 && out.length < WINDOW_TURNS_HARD_CAP; i--) { + const m = messages[i]; + if (m?.role !== 'user' && m?.role !== 'assistant') continue; + const text = messageText(m.content); + if (!text) continue; + out.push({ role: m.role, text }); + } + return out.reverse(); +} + /** * Joined text of everything the agent has ALREADY seen — every message EXCEPT * the current turn (the last user message). Used for "already in context" @@ -649,6 +672,9 @@ export function createGBrainContextEngine(ctx: { workspaceDir, currentUserText: getLastUserText(messages), priorContextText: getPriorContextText(messages), + // v0.43 (#2095): rolling window — assistant-introduced entities and + // named-antecedent follow-ups from recent turns now resolve too. + windowTurns: getWindowTurns(messages), resolveEntities: ctx.resolveEntities, }); diff --git a/src/core/context/entity-salience.ts b/src/core/context/entity-salience.ts index a51d20ef2..91082a0dc 100644 --- a/src/core/context/entity-salience.ts +++ b/src/core/context/entity-salience.ts @@ -7,12 +7,15 @@ * touching the brain, so it must be fast (one regex pass) and precision-biased: * a false candidate costs a wasted resolve and, worse, a misleading pointer. * - * DELIBERATE v1 limits (documented, not bugs — see issue #1981 / eng-review): + * DELIBERATE limits (documented, not bugs — see issue #1981 / eng-review): * - Proper-case + ASCII biased. Misses lowercase names ("garry") and many * non-Latin scripts. - * - Current-user-message only. No pronoun follow-ups ("what about her?"), no - * entities the assistant introduced. - * These are TODOs, not v1 scope. Do NOT market this as full "human-like recall". + * - extractCandidates is single-turn. The v0.43 (#2095) window layer + * (extractCandidatesFromWindow) widens extraction across the last N turns + * — assistant-introduced entities and "what about her?" follow-ups whose + * antecedent was NAMED in the window now resolve; true pronoun + * coreference (never-named antecedents) remains out of scope. + * Do NOT market this as full "human-like recall". * * Resolution (alias/slug lookup) lives in retrieval-reflex.ts; this module only * decides WHAT to look up. @@ -172,3 +175,83 @@ export function extractCandidates(text: string): EntityCandidate[] { } return out; } + +// ── Rolling-window extraction (v0.43, #2095 push-based context) ────────── + +export interface WindowTurn { + role: 'user' | 'assistant'; + text: string; +} + +/** + * A window candidate with the salience metadata the volunteer layer's + * confidence boost reads: how many turns mentioned it, whether the NEWEST + * turn did, and whether the USER (vs only the assistant) ever said it. + */ +export interface WindowEntityCandidate extends EntityCandidate { + /** Number of distinct turns that mentioned this candidate. */ + occurrences: number; + /** Mentioned in the newest (last) turn of the window. */ + inNewestTurn: boolean; + /** Mentioned in at least one USER turn (assistant-only mentions rank lower). */ + userMention: boolean; +} + +/** + * Extract candidates across the last N turns (oldest → newest). Pure, + * zero-LLM: runs the per-turn extractor on each turn and merges by the + * normalizeAlias form. Ordering is salience-aware — recency of last mention, + * cross-turn frequency, and a user-role boost — so when the merged set + * exceeds MAX_CANDIDATES, the dropped tail is the stalest assistant-only + * chatter, not the entity the user just named. + */ +export function extractCandidatesFromWindow(turns: WindowTurn[]): WindowEntityCandidate[] { + if (!turns?.length) return []; + interface WAcc extends WindowEntityCandidate { + lastTurnIdx: number; + order: number; + } + const acc = new Map(); + let order = 0; + const lastIdx = turns.length - 1; + + for (let i = 0; i < turns.length; i++) { + const turn = turns[i]; + if (!turn?.text) continue; + for (const c of extractCandidates(turn.text)) { + const norm = normalizeAlias(c.query); + if (!norm) continue; + const existing = acc.get(norm); + if (existing) { + existing.occurrences += 1; + existing.lastTurnIdx = i; + existing.inNewestTurn = existing.inNewestTurn || i === lastIdx; + if (turn.role === 'user' && !existing.userMention) { + // First USER-said surface form beats an assistant-introduced one + // for the display label. + existing.display = c.display; + existing.userMention = true; + } + } else { + acc.set(norm, { + display: c.display, + query: c.query, + occurrences: 1, + lastTurnIdx: i, + inNewestTurn: i === lastIdx, + userMention: turn.role === 'user', + order: order++, + }); + } + } + } + + // Salience weight: recency dominates, then frequency, then user-role. + // Deterministic tie-break on first-seen order. + const weight = (c: WAcc) => + (c.lastTurnIdx + 1) / turns.length + Math.min(c.occurrences, 4) * 0.1 + (c.userMention ? 0.15 : 0); + return Array.from(acc.values()) + .sort((a, b) => weight(b) - weight(a) || a.order - b.order) + .slice(0, MAX_CANDIDATES) + .map(({ lastTurnIdx: _l, order: _o, ...rest }) => rest); +} diff --git a/src/core/context/reflex.ts b/src/core/context/reflex.ts index afec4054a..188dbea21 100644 --- a/src/core/context/reflex.ts +++ b/src/core/context/reflex.ts @@ -20,7 +20,12 @@ import { join } from 'node:path'; import { mkdirSync, appendFileSync } from 'node:fs'; import { loadConfig, type GBrainConfig } from '../config.ts'; import type { BrainEngine } from '../engine.ts'; -import { extractCandidates, type EntityCandidate } from './entity-salience.ts'; +import { + extractCandidates, + extractCandidatesFromWindow, + type EntityCandidate, + type WindowTurn, +} from './entity-salience.ts'; import { resolveEntitiesToPointers, DEFAULT_MAX_POINTERS, @@ -28,22 +33,69 @@ import { } from './retrieval-reflex.ts'; import { resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } from './resolve-ipc.ts'; -/** Host capability shape (D1=A): candidates in, pointers out. Narrow by design. */ +/** Per-turn resolver options shared by every rung of the ladder. */ +export interface ResolveEntitiesOpts { + priorContextText?: string; + maxPointers?: number; + /** v0.43 (#2095): 'slug-only' under windowing — see ResolvePointersOpts. */ + suppression?: 'slug-and-title' | 'slug-only'; +} + +/** + * Host capability shape (D1=A): candidates in, pointers out. Narrow by design. + * + * CONTRACT (red-team): a host resolver MUST honor `opts.suppression`. Under + * windowing the orchestrator passes 'slug-only' — a resolver that keeps + * applying the legacy title-whole-word rule will suppress every entity merely + * mentioned in a prior window turn and silently disable the feature. Hosts + * built against the pre-window contract should be upgraded or pinned to + * `retrieval_reflex_window_turns: 1`. (A capability/version gate so the + * orchestrator can detect a stale host is a filed TODO.) + */ export type ResolveEntitiesFn = ( candidates: EntityCandidate[], - opts: { priorContextText?: string; maxPointers?: number }, + opts: ResolveEntitiesOpts, ) => Promise; export interface ReflexParams { workspaceDir: string; - /** The current turn's user text (drives extraction). */ + /** The current turn's user text (drives extraction when no window is given). */ currentUserText: string; /** Joined PRIOR turns + loaded page bodies — EXCLUDES the current turn (suppression). */ priorContextText: string; + /** + * v0.43 (#2095): recent turns (oldest → newest, current turn last). When + * present and the configured window is > 1, extraction widens to the last + * N turns (assistant-introduced entities + named-antecedent follow-ups now + * resolve) and suppression switches to slug-only (codex D7 — the title rule + * would suppress every entity merely mentioned in a prior window turn). + */ + windowTurns?: WindowTurn[]; /** Host-provided resolver, if the OpenClaw plugin contract supplied one. */ resolveEntities?: ResolveEntitiesFn; } +/** Default extraction window (turns). 1 = legacy current-turn-only. */ +export const DEFAULT_WINDOW_TURNS = 4; + +export function windowTurnCount(cfg: GBrainConfig | null): number { + // Env plane is read DIRECTLY here (mirroring reflexEnabled's direct + // process.env read), not just via loadConfig's env→config mapping. When + // there's no config file AND no DATABASE_URL, loadConfig() returns null and + // drops that mapping entirely — so without this, the documented + // GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS escape hatch would be silently + // ignored and the window would fall back to the default of 4 (a real + // config-less-environment bug, e.g. a clean CI shard with no brain). + const env = process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; + if (env != null && env !== '') { + const e = Number(env); + if (Number.isFinite(e) && e >= 1) return Math.floor(e); + } + const n = cfg?.retrieval_reflex_window_turns; + if (typeof n === 'number' && Number.isFinite(n) && n >= 1) return Math.floor(n); + return DEFAULT_WINDOW_TURNS; +} + const TIMEOUT_MS = 1500; // generous per-turn ceiling; the work is usually <100ms const HEARTBEAT_PATH = join(homedir(), '.gbrain', 'integrations', 'retrieval-reflex', 'heartbeat.jsonl'); @@ -68,14 +120,37 @@ export async function buildReflexAddition(params: ReflexParams): Promise 1. Window=1 reproduces the legacy + // current-turn-only behavior exactly (including suppression mode). + const windowN = windowTurnCount(cfg); + const windowed = windowN > 1 && (params.windowTurns?.length ?? 0) > 0; + const candidates: EntityCandidate[] = windowed + ? extractCandidatesFromWindow(params.windowTurns!.slice(-windowN)) + : extractCandidates(params.currentUserText); + // Zero-candidate fast path: regex passes only, no brain touch. if (!candidates.length) return null; - const opts = { priorContextText: params.priorContextText, maxPointers: maxPointers(cfg) }; + const opts: ResolveEntitiesOpts = { + priorContextText: params.priorContextText, + maxPointers: maxPointers(cfg), + suppression: windowed ? 'slug-only' : 'slug-and-title', + }; const block = await withTimeout(resolve(params, cfg, candidates, opts), TIMEOUT_MS); if (!block || !block.pointers.length) return null; + // Accept-side reflex-channel logging (red-team): the block survived the + // per-turn timeout, so these pointers ARE being injected. Only the + // direct-Postgres rung has an engine here; the IPC rung logs server-side + // at delivery; host-injected resolvers can't log (documented gap). + if (!params.resolveEntities && isPostgres(cfg)) { + const engine = await getPostgresEngine(cfg); + if (engine) { + const { logDeliveredReflexPointers } = await import('./retrieval-reflex.ts'); + logDeliveredReflexPointers(engine, block.pointers); + } + } + writeHeartbeat(cfg, block.pointers.length); return block.text; } catch { @@ -87,7 +162,7 @@ async function resolve( params: ReflexParams, cfg: GBrainConfig | null, candidates: EntityCandidate[], - opts: { priorContextText?: string; maxPointers?: number }, + opts: ResolveEntitiesOpts, ): Promise { // 1. Host capability (any engine). if (params.resolveEntities) { diff --git a/src/core/context/resolve-ipc.ts b/src/core/context/resolve-ipc.ts index 3955ea3e1..55e705e93 100644 --- a/src/core/context/resolve-ipc.ts +++ b/src/core/context/resolve-ipc.ts @@ -35,6 +35,8 @@ export interface ResolveRequest { priorContextText?: string; maxPointers?: number; sourceId?: string; + /** v0.43 (#2095, codex D7): suppression mode — 'slug-only' under windowing. */ + suppression?: 'slug-and-title' | 'slug-only'; } export type ResolveHandler = (req: ResolveRequest) => Promise; @@ -98,6 +100,13 @@ export async function resolveViaIpc( export async function startResolveIpcServer( socketPath: string, handler: ResolveHandler, + /** + * v0.43 (#2095, red-team): fired ONLY after the response was successfully + * written to the client — the accept-side seam for reflex-channel feedback + * logging. A block the client never received (timeout, dead socket) was + * never injected into a prompt and must not count as "volunteered". + */ + onDelivered?: (block: PointerBlock, req: ResolveRequest) => void, ): Promise { // Remove a stale socket file if present (a previous serve that didn't clean up). cleanupStaleSocket(socketPath); @@ -113,14 +122,23 @@ export async function startResolveIpcServer( if (nl < 0) return; const line = buf.slice(0, nl); let resp: string; + let delivered: { block: PointerBlock; req: ResolveRequest } | null = null; try { const req = JSON.parse(line) as ResolveRequest; const block = await handler(req); resp = JSON.stringify({ ok: true, block }); + if (block) delivered = { block, req }; } catch (e) { resp = JSON.stringify({ ok: false, error: (e as Error).message }); } - try { conn.write(resp + '\n'); } catch { /* client gone */ } + try { + conn.write(resp + '\n'); + // Write accepted — the client (250ms budget) may still have hung + // up, but this is the closest observable delivery point. + if (delivered && onDelivered) { + try { onDelivered(delivered.block, delivered.req); } catch { /* telemetry only */ } + } + } catch { /* client gone — do NOT log undelivered pointers */ } conn.end(); }); conn.on('error', () => { try { conn.destroy(); } catch { /* noop */ } }); diff --git a/src/core/context/retrieval-reflex.ts b/src/core/context/retrieval-reflex.ts index d1cb8e3af..1f9b3d70f 100644 --- a/src/core/context/retrieval-reflex.ts +++ b/src/core/context/retrieval-reflex.ts @@ -34,10 +34,40 @@ import type { EntityCandidate } from './entity-salience.ts'; export const DEFAULT_MAX_POINTERS = 3; const SYNOPSIS_MAX = 160; +/** Which resolution arm produced a pointer (provenance → honest confidence). */ +export type ResolveArm = 'alias' | 'title' | 'slug-suffix'; + +/** + * v0.43 (#2095) — arm → confidence. Lives HERE, next to the arm definitions, + * so arm identity and its score can't drift apart (eng-review note). The + * volunteer layer imports these; small deterministic boosts (multi-turn / + * newest-turn mention) are added on top there. + */ +export const ARM_CONFIDENCE: Record = { + alias: 0.9, + title: 0.8, + 'slug-suffix': 0.6, +}; + export interface ReflexPointer { display: string; slug: string; + /** Which brain source the page lives in (federated callers need it for dedup). */ + source_id: string; synopsis: string; + /** Resolution provenance (v0.43 #2095). */ + arm: ResolveArm; + /** Base arm confidence (ARM_CONFIDENCE[arm]); callers may boost. */ + confidence: number; + /** + * normalizeAlias form of the CANDIDATE that resolved this pointer (v0.43 + * #2095) — lets the volunteer layer join pointers back to window-salience + * metadata without guessing from the display label (which falls back to the + * page title when the candidate surface differs, e.g. alias "Swami" → + * title "Swami X"). Absent only when suffix classification couldn't + * recover the source candidate. + */ + matchedNorm?: string; } export interface PointerBlock { @@ -55,10 +85,30 @@ export interface ResolvePointersOpts { * message's own mention would suppress every pointer (eng-review/Codex fix). */ priorContextText?: string; + /** + * v0.43 (#2095, codex D7) — suppression mode. + * 'slug-and-title' (default, window=1 legacy): suppress when the slug OR + * the page title appears whole-word in prior context. + * 'slug-only' (REQUIRED under multi-turn windowing): suppress on slug + * presence only. Slugs only enter context when a pointer block or page + * body was actually injected; a bare mention of "Alice Example" in a + * prior turn never contains `people/alice-example`. The title rule + * would suppress every entity merely MENTIONED in a prior window turn + * — breaking window extraction by construction. + */ + suppression?: 'slug-and-title' | 'slug-only'; + /** + * v0.43 (#2095) — federated scope: resolve across these sources instead of + * the single positional sourceId. Precedence mirrors sourceScopeOpts + * (federated array > scalar). Alias arm loops per source; the title/slug + * arm uses source_id = ANY(...) in one query. + */ + sourceIds?: string[]; } interface PageRow { slug: string; + source_id: string; title: string; type: string | null; frontmatter: Record | null; @@ -88,40 +138,63 @@ export async function resolveEntitiesToPointers( const titlesLc: string[] = []; const exactSlugs: string[] = []; const slugSuffixes: string[] = []; + // Reverse maps for arm-2 provenance (which candidate produced a row) — + // populated in this same pass so the derivations happen exactly once. + const titleToNorm = new Map(); + const slugToNorm = new Map(); for (const c of candidates) { const norm = normalizeAlias(c.query); if (!norm) continue; if (!displayByNorm.has(norm)) displayByNorm.set(norm, c.display); aliasNorms.push(norm); - titlesLc.push(c.query.toLowerCase()); + const tl = c.query.toLowerCase(); + titlesLc.push(tl); + if (!titleToNorm.has(tl)) titleToNorm.set(tl, norm); const s = slugify(c.query); if (s) { exactSlugs.push(s); slugSuffixes.push(`%/${s}`); + if (!slugToNorm.has(s)) slugToNorm.set(s, norm); } } if (!aliasNorms.length) return null; - // Ordered set of resolved slugs (alias hits first → higher confidence). - const resolvedSlugs: string[] = []; + // Federated scope (v0.43 #2095): explicit sourceIds win over the scalar. + const sourceIds = opts.sourceIds?.length ? opts.sourceIds : [sourceId]; + + // Ordered set of resolved (source, slug) pairs with arm provenance — + // alias hits pushed first → higher confidence. + const resolved: Array<{ slug: string; source_id: string; arm: ResolveArm; matchedNorm?: string }> = []; const seen = new Set(); - const pushSlug = (slug: string) => { - if (slug && !seen.has(slug)) { - seen.add(slug); - resolvedSlugs.push(slug); + // Neither source ids nor slugs contain spaces, so a space separator is safe. + const keyOf = (src: string, slug: string) => `${src} ${slug}`; + const push = (slug: string, src: string, arm: ResolveArm, matchedNorm?: string) => { + if (!slug) return; + const k = keyOf(src, slug); + if (!seen.has(k)) { + seen.add(k); + resolved.push({ slug, source_id: src, arm, matchedNorm }); } }; - - // Arm 1 — alias-first. Unambiguous single-slug hits only. Guarded: pre-v110 - // brains throw "relation page_aliases does not exist" — swallow and continue. - try { - const aliasMap = await engine.resolveAliases(aliasNorms, { sourceId }); + // Arm 1 — alias-first. Unambiguous single-slug hits only, per source (no + // engine-interface change for federation). Guarded: pre-v110 brains throw + // "relation page_aliases does not exist" — swallow and continue. + // Per-source lookups are independent — run them concurrently so a + // federated caller (M granted sources) pays one RTT, not M sequential + // ones (~71ms each cross-region; the reflex runs under a 1.5s budget). + // Results are folded back in sourceIds order so pointer ordering stays + // deterministic. Per-source failures degrade independently (pre-v110 + // brains have no page_aliases table). + const aliasResults = await Promise.allSettled( + sourceIds.map((src) => engine.resolveAliases(aliasNorms, { sourceId: src })), + ); + for (let i = 0; i < sourceIds.length; i++) { + const r = aliasResults[i]; + if (r.status !== 'fulfilled') continue; for (const norm of aliasNorms) { - const hits = aliasMap.get(norm); - if (hits && hits.length === 1) pushSlug(hits[0].slug); + const hits = r.value.get(norm); + if (hits && hits.length === 1) push(hits[0].slug, sourceIds[i], 'alias', norm); } - } catch { - /* no page_aliases table (pre-v110) — degrade to the title/slug arm */ } // Arm 2 — exact title OR slug-suffix. This is the recall fix: a bare "Alice @@ -130,53 +203,72 @@ export async function resolveEntitiesToPointers( let rows: PageRow[] = []; try { rows = await engine.executeRaw( - `SELECT slug, title, type, frontmatter, compiled_truth + `SELECT slug, source_id, title, type, frontmatter, compiled_truth FROM pages WHERE deleted_at IS NULL - AND source_id = $1 + AND source_id = ANY($1::text[]) AND ( lower(title) = ANY($2::text[]) OR slug = ANY($3::text[]) OR slug LIKE ANY($4::text[]) )`, - [sourceId, titlesLc, exactSlugs, slugSuffixes], + [sourceIds, titlesLc, exactSlugs, slugSuffixes], ); } catch { rows = []; } - // Hydrate alias-resolved slugs too (their bodies for the synopsis) if not in rows. - const rowBySlug = new Map(); - for (const r of rows) rowBySlug.set(r.slug, r); - const aliasOnly = resolvedSlugs.filter((s) => !rowBySlug.has(s)); + // Hydrate alias-resolved pages too (their bodies for the synopsis) if not in rows. + const rowByKey = new Map(); + for (const r of rows) rowByKey.set(keyOf(r.source_id, r.slug), r); + const aliasOnly = resolved.filter((p) => !rowByKey.has(keyOf(p.source_id, p.slug))); if (aliasOnly.length) { try { const extra = await engine.executeRaw( - `SELECT slug, title, type, frontmatter, compiled_truth + `SELECT slug, source_id, title, type, frontmatter, compiled_truth FROM pages - WHERE deleted_at IS NULL AND source_id = $1 AND slug = ANY($2::text[])`, - [sourceId, aliasOnly], + WHERE deleted_at IS NULL AND source_id = ANY($1::text[]) AND slug = ANY($2::text[])`, + [sourceIds, aliasOnly.map((p) => p.slug)], ); - for (const r of extra) rowBySlug.set(r.slug, r); + for (const r of extra) rowByKey.set(keyOf(r.source_id, r.slug), r); } catch { /* ignore — alias slug may be stale */ } } // Title/slug matches that weren't alias hits, appended after alias hits. - for (const r of rows) pushSlug(r.slug); + // Arm provenance per row is classified in JS (codex D8) — the combined OR + // can't report which predicate matched: an exact lower(title) hit is the + // 'title' arm; anything else got in via slug / slug-suffix. + const titleSet = new Set(titlesLc); + for (const r of rows) { + const titleLc = (r.title ?? '').toLowerCase(); + if (titleSet.has(titleLc)) { + push(r.slug, r.source_id, 'title', titleToNorm.get(titleLc)); + } else { + // Slug arm: exact slugified-candidate match, else suffix scan. + const tail = r.slug.includes('/') ? r.slug.slice(r.slug.lastIndexOf('/') + 1) : r.slug; + push(r.slug, r.source_id, 'slug-suffix', slugToNorm.get(r.slug) ?? slugToNorm.get(tail)); + } + } // Build pointers in confidence order, applying suppression + cap. + const suppression = opts.suppression ?? 'slug-and-title'; const pointers: ReflexPointer[] = []; - for (const slug of resolvedSlugs) { - const row = rowBySlug.get(slug); + for (const { slug, source_id, arm, matchedNorm } of resolved) { + const row = rowByKey.get(keyOf(source_id, slug)); if (!row) continue; - // Suppression: already present in PRIOR context (slug or title). The current - // turn is deliberately excluded from priorContextText. + // Suppression: already present in PRIOR context. The current turn is + // deliberately excluded from priorContextText. Under windowing + // ('slug-only', codex D7) only the slug counts — a slug appears in prior + // context only when a pointer/page was actually surfaced there, while a + // title appears on any bare mention. if (priorLc) { - const titleLc = (row.title ?? '').toLowerCase(); if (priorLc.includes(slug.toLowerCase())) continue; - if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue; + if (suppression === 'slug-and-title') { + const titleLc = (row.title ?? '').toLowerCase(); + if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue; + } } const display = displayForRow(row, displayByNorm); const synopsis = safeSynopsis(row); - pointers.push({ display, slug, synopsis }); + pointers.push({ display, slug, source_id, synopsis, arm, confidence: ARM_CONFIDENCE[arm], matchedNorm }); if (pointers.length >= maxPointers) break; } @@ -246,3 +338,30 @@ export function renderPointerBlock(pointers: ReflexPointer[]): string { } return lines.join('\n'); } + +/** + * v0.43 (#2095, codex D11 + red-team) — ambient-channel feedback logging, + * ACCEPT-SIDE ONLY. Called by the delivery points (the serve IPC server after + * a successful write; the direct-Postgres reflex rung after its per-turn + * timeout admitted the block) — never inside the resolver itself, because a + * pointer block that timed out client-side was NEVER injected into a prompt, + * and logging it would inflate "volunteered" counts and drag the measured + * precision toward zero (corrupting the exact stats users tune + * min_confidence with). + */ +export function logDeliveredReflexPointers(engine: BrainEngine, pointers: ReflexPointer[]): void { + if (!pointers.length) return; + void import('./volunteer-events.ts') + .then(({ logVolunteerEventsFireAndForget, volunteerEventRowsFrom }) => { + logVolunteerEventsFireAndForget( + engine, + volunteerEventRowsFrom( + pointers.map((p) => ({ ...p, rationale: `${p.arm} match "${p.display}"` })), + { channel: 'reflex' }, + ), + ); + }) + .catch(() => { + /* telemetry only */ + }); +} diff --git a/src/core/context/volunteer-events.ts b/src/core/context/volunteer-events.ts new file mode 100644 index 000000000..2ec14a7b6 --- /dev/null +++ b/src/core/context/volunteer-events.ts @@ -0,0 +1,186 @@ +/** + * v0.43 (#2095) — context_volunteer_events: the feedback-loop log behind + * push-based context. One row per page the brain VOLUNTEERED, written + * fire-and-forget by the volunteer_context op, the retrieval-reflex pointer + * path (channel 'reflex'), and `gbrain watch` (channel 'watch'). + * + * "Used" is DERIVED, never written: a volunteered page counts as used when + * pages.last_retrieved_at > volunteered_at (the existing bumpLastRetrievedAt + * write-back on get_page/search/query is the open/cite signal). The join is + * approximate by design — last-retrieved is 5-min throttled (false negatives) + * and unrelated reads match too (false positives); stats output carries the + * caveat. + * + * Retention: rows older than VOLUNTEER_EVENTS_TTL_DAYS are pruned by the + * dream cycle's purge phase so conversation-adjacent telemetry never grows + * unbounded. rationale is a deterministic template that may embed the matched + * entity's surface form (which by construction resolved to an existing + * alias/title/slug) — never free conversation text. + */ + +import type { BrainEngine } from './../engine.ts'; +import { registerBackgroundWorkDrainer } from '../background-work.ts'; + +export const VOLUNTEER_EVENTS_TTL_DAYS = 90; + +export type VolunteerChannel = 'op' | 'reflex' | 'watch'; + +/** + * Map volunteered pages to event rows for one channel — the ONE place the + * VolunteerEventRow shape is assembled (op / reflex / watch all call this, + * so adding a column is a one-site change). + */ +export function volunteerEventRowsFrom( + pages: Array<{ source_id: string; slug: string; confidence: number; arm: string; rationale: string }>, + opts: { channel: VolunteerChannel; session_id?: string | null; turn?: number | null }, +): VolunteerEventRow[] { + return pages.map((p) => ({ + source_id: p.source_id, + slug: p.slug, + confidence: p.confidence, + match_arm: p.arm, + rationale: p.rationale, + channel: opts.channel, + session_id: opts.session_id ?? null, + turn: opts.turn ?? null, + })); +} + +export interface VolunteerEventRow { + source_id: string; + slug: string; + confidence: number; + match_arm: string; + rationale: string; + channel: VolunteerChannel; + session_id?: string | null; + turn?: number | null; +} + +/** + * ONE multi-row parameterized INSERT for a batch of volunteered pages (max 5 + * per call by the volunteer cap) — never per-row awaited INSERTs (up to 5 + * RTTs ≈ 355ms on a cross-region deployment; eng-review D4). Throws on + * failure; callers run it through the volunteer-events background-work sink + * with try/catch so logging can never fail the op. + */ +export async function insertVolunteerEvents( + engine: BrainEngine, + rows: VolunteerEventRow[], +): Promise { + if (!rows.length) return; + const params: unknown[] = []; + const tuples = rows.map((r) => { + const base = params.length; + params.push( + r.source_id, + r.slug, + r.confidence, + r.match_arm, + r.rationale, + r.channel, + r.session_id ?? null, + r.turn ?? null, + ); + const ph = Array.from({ length: 8 }, (_, i) => `$${base + i + 1}`); + return `(${ph.join(', ')})`; + }); + await engine.executeRaw( + `INSERT INTO context_volunteer_events + (source_id, slug, confidence, match_arm, rationale, channel, session_id, turn) + VALUES ${tuples.join(', ')}`, + params, + ); +} + +// ── Fire-and-forget sink (eng-review D4) ───────────────────────────────── +// Mirrors src/core/last-retrieved.ts: track every dangling INSERT promise in +// a module Set, register a drainer so finishCliTeardown settles them +// against a live engine before teardown on EVERY CLI exit path (the commit-1 +// drain hoist). Logging failure never fails the caller. + +const pendingVolunteerEventWrites = new Set>(); + +/** + * Log volunteered pages without blocking the hot path. The batched INSERT + * runs as a tracked dangling promise; errors are swallowed (pre-v117 brains, + * transient DB failures — the volunteer result is unaffected). + */ +export function logVolunteerEventsFireAndForget( + engine: BrainEngine, + rows: VolunteerEventRow[], +): void { + if (!rows.length) return; + const p = insertVolunteerEvents(engine, rows).catch(() => { + /* best-effort telemetry — never surfaces */ + }); + pendingVolunteerEventWrites.add(p); + void p.finally(() => pendingVolunteerEventWrites.delete(p)); +} + +/** Drain pending event writes (bounded). Same snapshot semantics as last-retrieved. */ +export async function awaitPendingVolunteerEventWrites( + timeoutMs = 5_000, +): Promise<{ unfinished: number }> { + if (pendingVolunteerEventWrites.size === 0) return { unfinished: 0 }; + const snapshot = Array.from(pendingVolunteerEventWrites); + let timer: ReturnType | undefined; + const timeout = new Promise<'timeout'>((resolve) => { + timer = setTimeout(() => resolve('timeout'), timeoutMs); + }); + const drain = Promise.allSettled(snapshot).then(() => 'drained' as const); + const outcome = await Promise.race([drain, timeout]); + if (timer) clearTimeout(timer); + if (outcome === 'timeout') { + const unfinished = pendingVolunteerEventWrites.size; + // Drop the snapshot so a long-lived process (`gbrain watch`) doesn't + // accumulate references to forever-pending work (last-retrieved C1). + for (const p of snapshot) pendingVolunteerEventWrites.delete(p); + return { unfinished }; + } + return { unfinished: 0 }; +} + +// Registered in the enqueue-owning module (background-work contract): module +// not imported ⇒ nothing enqueued ⇒ nothing to drain. Order 4 — after facts / +// last-retrieved / search-cache / eval-capture; bare INSERTs, no abort. +registerBackgroundWorkDrainer({ + name: 'volunteer-events', + order: 4, + drain: (ms) => awaitPendingVolunteerEventWrites(ms), +}); + +/** Test seam — clears the pending set so each test starts clean. */ +export function _resetPendingVolunteerEventWritesForTests(): void { + pendingVolunteerEventWrites.clear(); +} + +/** Test seam — peek the current pending count. */ +export function _peekPendingVolunteerEventWritesForTests(): number { + return pendingVolunteerEventWrites.size; +} + +/** + * 90-day GC, called from the dream cycle's purge phase (mirrors + * purgeStaleCheckpoints). Best-effort: returns 0 on any failure (pre-v117 + * brains have no table yet). + */ +export async function purgeStaleVolunteerEvents( + engine: BrainEngine, + ttlDays = VOLUNTEER_EVENTS_TTL_DAYS, +): Promise { + try { + const rows = await engine.executeRaw<{ count: string | number }>( + `WITH deleted AS ( + DELETE FROM context_volunteer_events + WHERE volunteered_at < now() - ($1 || ' days')::interval + RETURNING 1 + ) + SELECT count(*)::text AS count FROM deleted`, + [String(ttlDays)], + ); + return Number(rows[0]?.count ?? 0); + } catch { + return 0; + } +} diff --git a/src/core/context/volunteer.ts b/src/core/context/volunteer.ts new file mode 100644 index 000000000..dcdd721ad --- /dev/null +++ b/src/core/context/volunteer.ts @@ -0,0 +1,268 @@ +/** + * v0.43 (#2095) — push-based context: the brain VOLUNTEERS relevant pages + * from a rolling conversation window instead of waiting to be asked. + * + * window text ─ parseWindow() ─→ turns[] ─ extractCandidatesFromWindow() + * │ (recency + frequency + + * │ user-role weights) + * ▼ + * resolveEntitiesToPointers(sourceIds[]) alias 0.9 / title 0.8 / + * │ slug-suffix 0.6 (+0.05 boost + * ▼ for ≥2-turn or newest-turn + * gate min_confidence (default 0.7) → mentions) + * suppression (slug-only — windowing) → + * cap (3 default / 5 max) + * + * Zero-LLM, deterministic, precision-biased: push noise is worse than pull + * silence (#2095). At the default gate, slug-suffix matches (0.6+0.05 < 0.7) + * never volunteer — they need an explicit lower min_confidence. + * + * Consumed by three channels: the volunteer_context op, the retrieval-reflex + * window path, and `gbrain watch`. Event logging lives in + * volunteer-events.ts; usage stats here derive "used" from + * pages.last_retrieved_at — APPROXIMATE by design (the 5-min last-retrieved + * throttle causes false negatives; unrelated reads cause false positives). + */ + +import type { BrainEngine } from '../engine.ts'; +import { normalizeAlias } from '../search/alias-normalize.ts'; +import { + extractCandidatesFromWindow, + type WindowTurn, + type WindowEntityCandidate, +} from './entity-salience.ts'; +import { + resolveEntitiesToPointers, + ARM_CONFIDENCE, + type ResolveArm, +} from './retrieval-reflex.ts'; + +export const VOLUNTEER_DEFAULT_MAX_PAGES = 3; +export const VOLUNTEER_MAX_PAGES_CAP = 5; +export const VOLUNTEER_DEFAULT_MIN_CONFIDENCE = 0.7; +/** Deterministic boost for ≥2-turn or newest-turn mentions. */ +export const VOLUNTEER_SALIENCE_BOOST = 0.05; + +export interface VolunteeredPage { + slug: string; + source_id: string; + display: string; + confidence: number; + arm: ResolveArm; + /** Deterministic template string — never raw conversation text. */ + rationale: string; + synopsis: string; +} + +export interface VolunteerOpts { + /** Resolved source scope (federated array > scalar — sourceScopeOpts shape). */ + sourceIds: string[]; + /** Prior context (already-surfaced pointers/pages) for slug-only suppression. */ + priorContext?: string; + /** + * Slugs to skip BEFORE the confidence gate and the maxPages cap (O(1) + * membership). `gbrain watch` passes its session-dedupe set here — a + * post-call filter would let a recurring already-pushed entity consume cap + * slots every turn and starve new pages behind it (red-team finding). + */ + excludeSlugs?: ReadonlySet; + maxPages?: number; + minConfidence?: number; +} + +/** Shared wire protocol for window turns — watch.ts imports this so the two + * channels can never desynchronize on the prefix grammar. */ +export const TURN_PREFIX_RE = /^(user|assistant)\s*:\s?(.*)$/i; + +/** + * Lenient window parser: `user:` / `assistant:` line prefixes start a new + * turn (oldest → newest); unprefixed lines continue the current turn; input + * with no prefixes at all is ONE user turn (so `echo "..." | volunteer-context` + * just works). CRLF-tolerant. Empty/whitespace input → []. + */ +export function parseWindow(text: string): WindowTurn[] { + if (!text || !text.trim()) return []; + const turns: WindowTurn[] = []; + let current: WindowTurn | null = null; + for (const rawLine of text.split(/\r?\n/)) { + const m = TURN_PREFIX_RE.exec(rawLine); + if (m) { + if (current) turns.push(current); + current = { role: m[1].toLowerCase() as WindowTurn['role'], text: m[2] ?? '' }; + } else if (current) { + current.text += (current.text ? '\n' : '') + rawLine; + } else if (rawLine.trim()) { + // No prefix seen yet — accumulate into an implicit user turn. + current = { role: 'user', text: rawLine }; + } + } + if (current) turns.push(current); + // Trim trailing whitespace-only turn bodies. + return turns + .map((t) => ({ role: t.role, text: t.text.trim() })) + .filter((t) => t.text.length > 0); +} + +function clampMaxPages(n: number | undefined): number { + if (typeof n !== 'number' || !Number.isFinite(n) || n < 1) return VOLUNTEER_DEFAULT_MAX_PAGES; + return Math.min(Math.floor(n), VOLUNTEER_MAX_PAGES_CAP); +} + +function rationaleFor(arm: ResolveArm, display: string, c: WindowEntityCandidate | undefined, windowSize: number): string { + const armText = + arm === 'alias' ? `alias match "${display}"` + : arm === 'title' ? `exact title match "${display}"` + : `slug match "${display}"`; + if (!c) return armText; + const parts = [armText]; + if (c.occurrences >= 2) parts.push(`mentioned in ${c.occurrences} of last ${windowSize} turns`); + else if (c.inNewestTurn) parts.push('mentioned in the newest turn'); + if (!c.userMention) parts.push('assistant-introduced'); + return parts.join('; '); +} + +/** + * Volunteer confidence-gated pages for a conversation window. Pure read — + * event logging is the CALLER's job (through the volunteer-events sink). + * Non-relational, zero-LLM; returns [] when nothing clears the gate. + */ +export async function volunteerContext( + engine: BrainEngine, + turns: WindowTurn[], + opts: VolunteerOpts, +): Promise { + if (!turns.length || !opts.sourceIds?.length) return []; + const candidates = extractCandidatesFromWindow(turns); + if (!candidates.length) return []; + const byNorm = new Map(); + for (const c of candidates) { + const norm = normalizeAlias(c.query); + if (norm && !byNorm.has(norm)) byNorm.set(norm, c); + } + + const maxPages = clampMaxPages(opts.maxPages); + const minConfidence = + typeof opts.minConfidence === 'number' && opts.minConfidence >= 0 && opts.minConfidence <= 1 + ? opts.minConfidence + : VOLUNTEER_DEFAULT_MIN_CONFIDENCE; + + // Resolve up to the hard cap so the confidence gate sees the full pool — + // a gated-out alias hit must not shadow a passing title hit behind it. + const block = await resolveEntitiesToPointers(engine, opts.sourceIds[0], candidates, { + sourceIds: opts.sourceIds, + priorContextText: opts.priorContext, + suppression: 'slug-only', + maxPointers: VOLUNTEER_MAX_PAGES_CAP * 2, + }); + if (!block) return []; + + const out: VolunteeredPage[] = []; + for (const p of block.pointers) { + if (opts.excludeSlugs?.has(p.slug)) continue; // before gate + cap — see VolunteerOpts + // matchedNorm is the resolver's provenance join-key (the candidate that + // resolved the pointer); display-based lookup is the fallback for the + // rare suffix rows where provenance couldn't be recovered. + const cand = (p.matchedNorm ? byNorm.get(p.matchedNorm) : undefined) ?? byNorm.get(normalizeAlias(p.display)); + const boost = cand && (cand.occurrences >= 2 || cand.inNewestTurn) ? VOLUNTEER_SALIENCE_BOOST : 0; + const confidence = Math.min(0.99, ARM_CONFIDENCE[p.arm] + boost); + if (confidence < minConfidence) continue; + out.push({ + slug: p.slug, + source_id: p.source_id, + display: p.display, + confidence, + arm: p.arm, + rationale: rationaleFor(p.arm, p.display, cand, turns.length), + synopsis: p.synopsis, + }); + if (out.length >= maxPages) break; + } + return out; +} + +/** + * Canonical human rendering of one volunteered page — shared by + * `gbrain volunteer-context` (cli.ts formatResult) and `gbrain watch` so the + * two surfaces can't drift. + */ +export function formatVolunteeredPage(p: VolunteeredPage): string { + return ( + `${p.display} → ${p.slug} (${p.confidence.toFixed(2)}, ${p.arm}) — ${p.rationale}` + + (p.synopsis ? `\n ${p.synopsis}` : '') + ); +} + +// ── Usage stats (the feedback loop) ────────────────────────────────────── + +export interface VolunteerArmStats { + match_arm: string; + channel: string; + volunteered: number; + used: number; + /** used / volunteered, 0 when nothing volunteered. */ + precision: number; +} + +export interface VolunteerUsageStats { + days: number; + /** The join is approximate — see the note (false +/- documented in #2095 D9). */ + approximate: true; + note: string; + total_volunteered: number; + total_used: number; + by_arm: VolunteerArmStats[]; +} + +export const VOLUNTEER_STATS_NOTE = + 'approximate: "used" = pages.last_retrieved_at > volunteered_at. The 5-min ' + + 'last-retrieved throttle causes false negatives; unrelated reads of the same ' + + 'page cause false positives.'; + +/** + * Per-arm/channel precision over the last N days, source-scoped. Read-only; + * returns zeroed stats on pre-v117 brains (no table). + */ +export async function volunteerUsageStats( + engine: BrainEngine, + sourceIds: string[], + days = 30, +): Promise { + const safeDays = Number.isFinite(days) && days > 0 ? Math.floor(days) : 30; + let rows: Array<{ match_arm: string; channel: string; volunteered: string | number; used: string | number }> = []; + try { + rows = await engine.executeRaw( + `SELECT e.match_arm, e.channel, + count(*)::text AS volunteered, + count(*) FILTER (WHERE p.last_retrieved_at > e.volunteered_at)::text AS used + FROM context_volunteer_events e + LEFT JOIN pages p + ON p.source_id = e.source_id AND p.slug = e.slug AND p.deleted_at IS NULL + WHERE e.source_id = ANY($1::text[]) + AND e.volunteered_at > now() - ($2 || ' days')::interval + GROUP BY e.match_arm, e.channel + ORDER BY e.match_arm, e.channel`, + [sourceIds, String(safeDays)], + ); + } catch { + rows = []; // pre-v117 brain — table doesn't exist yet + } + const by_arm: VolunteerArmStats[] = rows.map((r) => { + const volunteered = Number(r.volunteered); + const used = Number(r.used); + return { + match_arm: r.match_arm, + channel: r.channel, + volunteered, + used, + precision: volunteered > 0 ? Number((used / volunteered).toFixed(3)) : 0, + }; + }); + return { + days: safeDays, + approximate: true, + note: VOLUNTEER_STATS_NOTE, + total_volunteered: by_arm.reduce((s, a) => s + a.volunteered, 0), + total_used: by_arm.reduce((s, a) => s + a.used, 0), + by_arm, + }; +} diff --git a/src/core/cycle.ts b/src/core/cycle.ts index 116e5420f..17a31bccb 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -1285,6 +1285,16 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise parseUsdLimit(raw, DEFAULT_SPEND_CAP_USD))(await engine.getConfig(SPEND_CAP_CONFIG_KEY)); + const posture = opts.postureOverride ?? (await resolveSpendPosture(engine)); // ── Source-level cooldown ───────────────────────────────────── // Block re-submission if (a) an embed-backfill is currently active for this @@ -172,9 +185,14 @@ export async function submitEmbedBackfill( } // ── 24h rolling spend cap ───────────────────────────────────── + // v0.42.42.0 (#2139): `spend.posture=tokenmax` waves past the cap (the + // operator declared cost isn't the constraint). The per-job BudgetTracker + // still ledgers the spend — posture removes the ceiling, not the accounting. + // An `off`/`unlimited` cap (Infinity) is likewise never tripped. const spend24hFn = opts.spend24hFn ?? defaultSpend24hForSource; const spend24h = await spend24hFn(engine, sourceId); - if (spend24h >= spendCap) { + const spendCapBypassed = posture === 'tokenmax' && spend24h >= spendCap; + if (spend24h >= spendCap && !spendCapBypassed) { return { status: 'spend_capped', spend24hUsd: spend24h, @@ -194,7 +212,9 @@ export async function submitEmbedBackfill( }, ); - return { status: 'submitted', jobId: job.id }; + return spendCapBypassed + ? { status: 'submitted', jobId: job.id, spendCapBypassed: true, spend24hUsd: spend24h } + : { status: 'submitted', jobId: job.id }; } /** Round timestamp down to the nearest `bucketMs` boundary. */ diff --git a/src/core/embedding.ts b/src/core/embedding.ts index 20515d313..108b854f8 100644 --- a/src/core/embedding.ts +++ b/src/core/embedding.ts @@ -218,16 +218,23 @@ export function willEmbedSynchronously(opts: { } /** - * Pure cost-gate decision. The gate BLOCKS (prompt in TTY, exit 2 envelope - * in non-TTY) only when embed runs inline AND the estimated spend exceeds - * the floor. Deferred mode NEVER blocks — the backfill cap is the real - * money gate, and blocking the cheap markdown import for cost the import - * doesn't synchronously incur is the bug this fix removes. + * Pure cost-gate decision. The gate BLOCKS (prompt in TTY, auto-defer in + * non-TTY) only when embed runs inline AND the estimated spend exceeds the + * floor. Deferred mode NEVER blocks — the backfill cap is the real money gate, + * and blocking the cheap markdown import for cost the import doesn't + * synchronously incur is the bug this fix removes. + * + * v0.42.42.0 (#2139): `spend.posture=tokenmax` makes the gate INFORMATIONAL — + * the operator has declared cost isn't the constraint, so it never blocks + * (the caller prints the estimate and proceeds inline). An `off`/`unlimited` + * floor (Infinity) is likewise never exceeded. */ export function shouldBlockSync( costUsd: number, floorUsd: number, mode: SyncEmbedMode, + posture: 'gated' | 'tokenmax' = 'gated', ): boolean { + if (posture === 'tokenmax') return false; return mode === 'inline' && costUsd > floorUsd; } diff --git a/src/core/git-remote.ts b/src/core/git-remote.ts index dbbc339d1..43f8515eb 100644 --- a/src/core/git-remote.ts +++ b/src/core/git-remote.ts @@ -131,7 +131,7 @@ export interface CloneOpts { export class GitOperationError extends Error { constructor( - public op: 'clone' | 'pull' | 'remote_get_url', + public op: 'clone' | 'pull' | 'fetch' | 'remote_get_url', message: string, public cause?: unknown, ) { @@ -217,6 +217,31 @@ export function pullRepo(repoPath: string, opts: { timeoutMs?: number } = {}): v } } +/** + * Fetch a single remote branch with the SAME SSRF-defensive flags + no-prompt + * env as cloneRepo/pullRepo (GIT_SSRF_FLAGS, --no-recurse-submodules, + * GIT_TERMINAL_PROMPT=0). Used by the sync cost-estimator's fetch-first path + * (#2139) so a cost preview / dry-run never hits a remote through a + * less-protected route than real sync. Throws GitOperationError on failure; + * the estimator catches and falls back to local HEAD. + */ +export function fetchRemote(repoPath: string, branch: string, opts: { timeoutMs?: number } = {}): void { + const args: string[] = ['-C', repoPath, ...GIT_SSRF_FLAGS, 'fetch', ...GIT_SSRF_SUBCOMMAND_FLAGS, 'origin', branch]; + try { + execFileSync('git', args, { + stdio: ['ignore', 'pipe', 'pipe'], + timeout: opts.timeoutMs ?? 30_000, + env: { ...process.env, ...GIT_ENV }, + }); + } catch (e) { + throw new GitOperationError( + 'fetch', + `git fetch failed in ${repoPath}: ${(e as Error).message}`, + e, + ); + } +} + export type RepoState = | 'healthy' | 'missing' diff --git a/src/core/migrate.ts b/src/core/migrate.ts index 31d3a10c1..019f02ea4 100644 --- a/src/core/migrate.ts +++ b/src/core/migrate.ts @@ -5232,6 +5232,44 @@ export const MIGRATIONS: Migration[] = [ ON code_edges_chunk (from_symbol_qualified); `, }, + { + // Renumbered 116 -> 117 at merge: master's v0.42.41.0 triage wave + // claimed v116 (code_edges_source_backfill_and_callee_index) first. + version: 117, + name: 'context_volunteer_events_table', + // #2095 push-based context: feedback-loop log of every page the brain + // VOLUNTEERED (via the volunteer_context op, the retrieval-reflex pointer + // path, or `gbrain watch`). "Used" is derived later by joining + // pages.last_retrieved_at > volunteered_at — no second write path. + // session_id/turn are caller-supplied attribution (nullable). rationale is + // a deterministic template string, NEVER raw conversation text. Rows are + // pruned past 90 days by the dream cycle's purge phase + // (purgeStaleVolunteerEvents). No ::jsonb anywhere. RLS: covered by the + // v35 auto_rls_on_create_table event trigger on Postgres (same as + // v110/v115 tables); pinned by the volunteer-context Postgres e2e. + // Created empty; plain CREATE INDEX is instant — no CONCURRENTLY needed. + // Keep in sync with src/schema.sql, src/core/pglite-schema.ts, + // src/core/schema-embedded.ts. + idempotent: true, + sql: ` + CREATE TABLE IF NOT EXISTS context_volunteer_events ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + slug TEXT NOT NULL, + confidence DOUBLE PRECISION NOT NULL, + match_arm TEXT NOT NULL, + rationale TEXT NOT NULL DEFAULT '', + channel TEXT NOT NULL DEFAULT 'op', + session_id TEXT, + turn INTEGER, + volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx + ON context_volunteer_events (source_id, volunteered_at DESC); + CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx + ON context_volunteer_events (source_id, slug); + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/minions/handlers/embed-backfill.ts b/src/core/minions/handlers/embed-backfill.ts index 362ea470f..0eea0b484 100644 --- a/src/core/minions/handlers/embed-backfill.ts +++ b/src/core/minions/handlers/embed-backfill.ts @@ -40,6 +40,7 @@ import { type DbPacer, createDbPacer, createNoopPacer } from '../../db-pacer.ts' import { resolvePaceMode, loadPaceModeConfig, readPaceEnv } from '../../pace-mode.ts'; import type { BrainEngine } from '../../engine.ts'; import type { MinionJobContext } from '../types.ts'; +import { parseUsdLimit, usdLimitToCap, resolveSpendPosture } from '../../spend-posture.ts'; import { embedBackfillLockId, EMBED_BACKFILL_LOCK_TTL_MIN } from '../../embed-backfill-lock.ts'; @@ -95,12 +96,21 @@ async function resolveBackfillPacer( } } -/** Read embed.backfill_max_usd config or default. */ -async function readMaxUsd(engine: BrainEngine): Promise { +/** + * Resolve the per-job budget cap (USD) for the BudgetTracker. + * + * v0.42.42.0 (#2139): returns `undefined` = "no cap" (which BudgetTracker + * treats as cap-absent) when the config is `off`/`unlimited`/`none` OR when + * `spend.posture=tokenmax`. NEVER returns Infinity (that would pass through as + * a real ceiling and serialize to `null` in audit rows). Spend is still + * ledgered by the tracker either way — posture removes the ceiling, not the + * accounting. `0`/garbage fall back to the $10 default. + */ +async function readMaxUsd(engine: BrainEngine): Promise { + const posture = await resolveSpendPosture(engine); + if (posture === 'tokenmax') return undefined; const raw = await engine.getConfig('embed.backfill_max_usd'); - if (raw === null || raw === undefined) return DEFAULT_MAX_USD_PER_JOB; - const n = Number(raw); - return Number.isFinite(n) && n > 0 ? n : DEFAULT_MAX_USD_PER_JOB; + return usdLimitToCap(parseUsdLimit(raw, DEFAULT_MAX_USD_PER_JOB)); } /** Validate + extract typed job params. Throws on malformed input. */ diff --git a/src/core/operations.ts b/src/core/operations.ts index aa6f41e41..eb6adc70f 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -3145,6 +3145,99 @@ const get_recent_salience: Operation = { cliHints: { name: 'salience' }, }; +// --- v0.43 (#2095): push-based context — the brain volunteers pages --- + +const volunteer_context: Operation = { + name: 'volunteer_context', + description: + 'Push-based context: volunteer brain pages relevant to a rolling conversation window ' + + 'WITHOUT being asked. Zero-LLM, confidence-gated (alias 0.9 / exact-title 0.8 / ' + + 'slug-suffix 0.6, +0.05 for multi-turn or newest-turn mentions; default gate 0.7), ' + + 'capped at 3 pages (max 5). Returns pointers with one-line rationales + synopses — ' + + 'open the page (get_page) before relying on details. Pass stats: true for the ' + + 'approximate volunteered-vs-used precision summary (the feedback loop).', + scope: 'read', + params: { + window: { + type: 'string', + description: + "Recent conversation turns, oldest → newest, as 'user:' / 'assistant:' prefixed " + + 'lines (unprefixed text = one user turn). Required unless stats: true. ' + + 'CLI: piped stdin fills this.', + }, + prior_context: { + type: 'string', + description: + 'Already-surfaced context (pointer blocks / opened page bodies). Pages whose slug ' + + 'appears here are not re-volunteered.', + }, + max_pages: { type: 'number', description: 'Max pages to volunteer (default 3, hard cap 5).' }, + min_confidence: { + type: 'number', + description: + 'Confidence gate 0..1 (default 0.7 — slug-suffix matches need an explicit lower gate).', + }, + session_id: { type: 'string', description: 'Optional caller session id, logged for attribution.' }, + turn: { type: 'number', description: 'Optional caller turn number, logged for attribution.' }, + stats: { + type: 'boolean', + description: + 'Return the volunteered-vs-used precision summary instead of volunteering. ' + + 'APPROXIMATE: "used" = pages.last_retrieved_at > volunteered_at.', + }, + days: { type: 'number', description: 'Stats window in days (default 30; stats mode only).' }, + }, + handler: async (ctx, p) => { + const { parseWindow, volunteerContext, volunteerUsageStats } = await import('./context/volunteer.ts'); + const scope = sourceScopeOpts(ctx); + const sourceIds = scope.sourceIds ?? (scope.sourceId ? [scope.sourceId] : ['default']); + + if (p.stats === true) { + return volunteerUsageStats(ctx.engine, sourceIds, typeof p.days === 'number' ? p.days : undefined); + } + + if (typeof p.window !== 'string' || !p.window.trim()) { + throw new OperationError( + 'invalid_params', + 'window is required unless stats: true', + 'Pass the recent turns as a string (CLI: pipe them on stdin), or use --stats.', + ); + } + const turns = parseWindow(p.window); + const pages = await volunteerContext(ctx.engine, turns, { + sourceIds, + priorContext: typeof p.prior_context === 'string' ? p.prior_context : undefined, + maxPages: typeof p.max_pages === 'number' ? p.max_pages : undefined, + minConfidence: typeof p.min_confidence === 'number' ? p.min_confidence : undefined, + }); + + // Feedback-loop logging: fire-and-forget batched INSERT through the + // volunteer-events sink (drained at exit). Never fails the op. + if (pages.length) { + try { + const { logVolunteerEventsFireAndForget, volunteerEventRowsFrom } = await import('./context/volunteer-events.ts'); + // Trust-boundary clamps (remote MCP callers): cap session_id length so + // a read-scoped token can't bank unbounded TEXT per request, and only + // log integer turns — a non-integer would throw inside the single + // multi-row INSERT and silently drop the whole batch. + const sessionId = typeof p.session_id === 'string' ? p.session_id.slice(0, 256) : null; + const turn = + typeof p.turn === 'number' && Number.isInteger(p.turn) && Math.abs(p.turn) <= 2_147_483_647 + ? p.turn + : null; + logVolunteerEventsFireAndForget( + ctx.engine, + volunteerEventRowsFrom(pages, { channel: 'op', session_id: sessionId, turn }), + ); + } catch { + /* telemetry only */ + } + } + return { pages, count: pages.length, window_turns: turns.length }; + }, + cliHints: { name: 'volunteer-context', stdin: 'window' }, +}; + const find_anomalies: Operation = { name: 'find_anomalies', description: FIND_ANOMALIES_DESCRIPTION, @@ -4856,6 +4949,8 @@ export const operations: Operation[] = [ whoami, sources_add, sources_list, sources_remove, sources_status, // v0.29: Salience + anomalies + recent transcripts get_recent_salience, find_anomalies, get_recent_transcripts, + // v0.43 (#2095): push-based context + volunteer_context, // v0.31: hot memory (facts table) extract_facts, recall, forget_fact, // v0.32.6: contradiction probe MCP surface (M3) diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 8808efec5..6fdb57d07 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -264,6 +264,12 @@ export class PGLiteEngine implements BrainEngine { } } + // NOTE (#2084): PGLite's Emscripten runtime writes the WASM backend's + // proc_exit status into `process.exitCode` (initdb here at create-time, + // the postmaster at close-time), and the writes land asynchronously — + // a snapshot/restore around these awaits does NOT contain them. That is + // why the CLI's exit paths read gbrain's own verdict + // (cli-force-exit.ts currentExitCode), never ambient process.exitCode. try { this._db = await preservingProcessExitCode(() => PGlite.create({ diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index 44c3cc418..b0d7da947 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -945,6 +945,27 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths ( FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE ); +-- #2095 push-based context: feedback-loop log of volunteered pages. +-- Mirrors migration v117 + src/schema.sql. "Used" derives from +-- pages.last_retrieved_at > volunteered_at; 90-day prune in the dream +-- cycle's purge phase. +CREATE TABLE IF NOT EXISTS context_volunteer_events ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + slug TEXT NOT NULL, + confidence DOUBLE PRECISION NOT NULL, + match_arm TEXT NOT NULL, + rationale TEXT NOT NULL DEFAULT '', + channel TEXT NOT NULL DEFAULT 'op', + session_id TEXT, + turn INTEGER, + volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx + ON context_volunteer_events (source_id, volunteered_at DESC); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx + ON context_volunteer_events (source_id, slug); + -- ============================================================ -- migration_impact_log (v0.41.18.0 — gbrain onboard wave) -- ============================================================ diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index 2e30b49f1..e0f313dc8 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -711,6 +711,28 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths ( FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE ); +-- #2095 push-based context: feedback-loop log of volunteered pages +-- (volunteer_context op / retrieval-reflex pointers / gbrain watch). +-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is +-- a deterministic template string, never raw conversation text. 90-day prune +-- in the dream cycle's purge phase. Mirrors migration v117. +CREATE TABLE IF NOT EXISTS context_volunteer_events ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + slug TEXT NOT NULL, + confidence DOUBLE PRECISION NOT NULL, + match_arm TEXT NOT NULL, + rationale TEXT NOT NULL DEFAULT '', + channel TEXT NOT NULL DEFAULT 'op', + session_id TEXT, + turn INTEGER, + volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx + ON context_volunteer_events (source_id, volunteered_at DESC); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx + ON context_volunteer_events (source_id, slug); + -- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676) -- because its \`job_id BIGINT REFERENCES minion_jobs(id)\` FK requires -- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix. diff --git a/src/core/spend-posture.ts b/src/core/spend-posture.ts new file mode 100644 index 000000000..f06d125c3 --- /dev/null +++ b/src/core/spend-posture.ts @@ -0,0 +1,106 @@ +/** + * Spend posture + USD-limit parsing — the single spend-control surface for + * gbrain's cost gates (issue #2139). Two concerns live here: + * + * 1. `spend.posture` (DB-plane config): `'gated'` (default) makes every cost + * gate behave as before; `'tokenmax'` makes them INFORMATIONAL — print the + * estimate, proceed, and keep ledgering spend. The operator who sets + * `tokenmax` has declared "cost is not my constraint." Posture removes the + * CEILING, never the ACCOUNTING (the spend ledger still records every + * dollar). It is deliberately SEPARATE from `search.mode=tokenmax` (which + * governs retrieval payload size, not embedding spend); the gate prints a + * hint linking the two when only the search mode is set. + * + * 2. `parseUsdLimit` / `formatUsdLimit`: first-class `off` / `unlimited` / + * `none` on the USD gate knobs (so operators stop setting sentinel values + * like `100000`). The parse layer represents "no limit" as `Infinity` so + * comparisons stay special-case-free (`cost > Infinity` is never true). + * `formatUsdLimit` renders it as the string `'unlimited'` — NEVER serialize + * raw `Infinity`, because `JSON.stringify(Infinity)` emits `null`, which is + * ambiguous in audit/ledger rows. At the budget-machinery boundary, callers + * convert `Infinity` → `undefined` ("no cap"), which `BudgetTracker` already + * treats as cap-absent. + * + * LEAF module: imports only the engine type so any cost-gate site can pull it + * in without a circular dependency. + */ +import type { BrainEngine } from './engine.ts'; + +export const SPEND_POSTURE_CONFIG_KEY = 'spend.posture'; + +export type SpendPosture = 'gated' | 'tokenmax'; + +/** + * Resolve `spend.posture` from DB-plane config. Fail-open to `'gated'` on a + * missing/unknown value or a config-read error — a posture probe must never + * crash a sync, and an unrecognized value must never silently disable gates. + */ +export async function resolveSpendPosture(engine: BrainEngine): Promise { + try { + const raw = await engine.getConfig(SPEND_POSTURE_CONFIG_KEY); + return normalizeSpendPosture(raw); + } catch { + return 'gated'; + } +} + +/** Pure normalizer (testable without an engine). Anything but `tokenmax` → `gated`. */ +export function normalizeSpendPosture(raw: unknown): SpendPosture { + if (typeof raw === 'string' && raw.trim().toLowerCase() === 'tokenmax') return 'tokenmax'; + return 'gated'; +} + +/** True iff `raw` is a valid `spend.posture` value (for `config set` validation). */ +export function isValidSpendPosture(raw: unknown): boolean { + return typeof raw === 'string' && ['gated', 'tokenmax'].includes(raw.trim().toLowerCase()); +} + +const OFF_TOKENS = new Set(['off', 'unlimited', 'none']); + +/** + * Parse a USD-limit config value. + * - `'off'` / `'unlimited'` / `'none'` (case-insensitive) → `Infinity` (no limit) + * - a finite positive number (or `0` when `allowZero`) → that number + * - anything else (garbage, negative, empty, NaN) → `def` + * + * `allowZero` distinguishes the two knob semantics: + * - `sync.cost_gate_min_usd` uses `allowZero: true` — `0` means "block on any + * nonzero spend" (a real operator choice). `off` is the no-limit escape. + * - the backfill caps reject `0` (fall back to the default); only `off` + * disables them. `off` semantics ≠ `0` (issue #2139). + */ +export function parseUsdLimit( + raw: unknown, + def: number, + opts: { allowZero?: boolean } = {}, +): number { + if (raw === null || raw === undefined) return def; + if (typeof raw === 'string') { + const t = raw.trim().toLowerCase(); + if (t === '') return def; + if (OFF_TOKENS.has(t)) return Infinity; + } + const n = Number(raw); + if (!Number.isFinite(n)) return def; + if (n < 0) return def; + if (n === 0) return opts.allowZero ? 0 : def; + return n; +} + +/** + * Render a USD limit for human/JSON output. `Infinity` → `'unlimited'` (NEVER + * the raw value — `JSON.stringify(Infinity)` is `null`). Finite values are + * returned as-is so callers can `$${formatUsdLimit(x)}` or embed in JSON. + */ +export function formatUsdLimit(n: number): string | number { + return Number.isFinite(n) ? n : 'unlimited'; +} + +/** + * Convert a parsed USD limit to the budget-machinery cap representation: + * `Infinity` → `undefined` ("no cap", which `BudgetTracker` treats as + * cap-absent), finite → the number. Keeps `null` out of ledger rows. + */ +export function usdLimitToCap(n: number): number | undefined { + return Number.isFinite(n) ? n : undefined; +} diff --git a/src/core/sync-delta.ts b/src/core/sync-delta.ts new file mode 100644 index 000000000..9ceb51835 --- /dev/null +++ b/src/core/sync-delta.ts @@ -0,0 +1,151 @@ +/** + * Shared sync-delta machinery — ONE implementation of "what changed since + * last_commit" consumed by BOTH the sync executor (`performSyncInner` in + * `src/commands/sync.ts`) and the inline-embed cost estimator + * (`src/core/sync-cost-estimate.ts`). Before this module the executor diffed + * `last_commit..pin` while the estimator priced the entire tree, so the + * gate's dollar figure had no relationship to what the sync actually embedded + * (issue #2139: a 400x overestimate that wedged the daily cron). Routing both + * through `computeSyncDelta` makes diff/manifest drift between estimate and + * execution structurally impossible. + * + * Shell-injection safe: `execFileSync` with array args (no `/bin/sh -c`), so a + * `sources.local_path` containing shell metacharacters can never escape — same + * posture documented at `git-head.ts:14-19`. + * + * Fail-open ladder (never throws): + * + * computeSyncDelta(repo, from, to) + * │ + * ├─ `git cat-file -t ` throws → { unavailable, anchor_missing } + * │ (bookmark object gc'd after a history rewrite — nothing to diff; + * │ caller falls back to a full reconcile / full-tree ceiling) + * │ + * ├─ `git diff --name-status -M from..to` throws → { unavailable, diff_failed } + * │ (oversized post-rewrite diff exceeds the 30s / 100 MiB budget) + * │ + * └─ ok → { ok, manifest } (+ detached working-tree manifest merged in) + * + * NOTE: a present-but-non-ancestor `from` (force-push, squash, master→main) is + * still diffable — `git diff A..B` is an endpoint-tree comparison and does NOT + * require A to be an ancestor of B (unlike a rev-walk or `A...B` merge-base). + * That is the #1970 property this module preserves. + */ +import { execFileSync } from 'node:child_process'; +import { buildSyncManifest, type SyncManifest } from './sync.ts'; + +/** Runs a git subcommand in `repoPath` and returns trimmed stdout (throws on failure). */ +export type GitRunner = (repoPath: string, args: string[]) => string; + +// Mirrors `git()` + `buildGitInvocation()` in commands/sync.ts: `core.quotepath=false` +// so non-ASCII (CJK) paths arrive as UTF-8; 30s timeout; 100 MiB maxBuffer (a +// 100K-file `--name-status` diff tops out ~10-20 MiB — Node's 1 MiB default +// would ENOBUFS-crash the sync with no log line). +const DEFAULT_GIT_RUNNER: GitRunner = (repoPath, args) => + execFileSync('git', ['-c', 'core.quotepath=false', '-C', repoPath, ...args], { + encoding: 'utf-8', + timeout: 30_000, + maxBuffer: 100 * 1024 * 1024, + }).trim(); + +let _gitRunner: GitRunner = DEFAULT_GIT_RUNNER; + +/** + * Test seam (probe-seam pattern, matches `git-head.ts:_setGitHeadProbeForTests`) + * so tests drive `computeSyncDelta` without mocking child_process or routing + * through `mock.module` (R2-compliant). Pass `null` to restore the default. + */ +export function _setGitRunnerForTests(fn: GitRunner | null): void { + _gitRunner = fn ?? DEFAULT_GIT_RUNNER; +} + +function unique(items: T[]): T[] { + return [...new Set(items)]; +} + +/** + * Working-tree manifest for a DETACHED HEAD (relocated from sync.ts:557 so the + * estimator can price detached sources identically to how the executor imports + * them). On a detached HEAD, sync syncs from the live working tree: tracked + * changes (`git diff --name-status -M HEAD`) PLUS untracked files (`ls-files + * --others --exclude-standard`). Attached HEADs never call this — their + * incremental path imports ONLY the commit diff (untracked/dirty files are not + * imported), which is why the estimator must not price dirty files on an + * attached repo (issue #2139 phantom-cost class). + */ +export function buildDetachedWorkingTreeManifest( + repoPath: string, + run: GitRunner = _gitRunner, +): SyncManifest { + const manifest = buildSyncManifest(run(repoPath, ['diff', '--name-status', '-M', 'HEAD'])); + const untracked = run(repoPath, ['ls-files', '--others', '--exclude-standard']) + .split('\n') + .filter(line => line.length > 0); + return { + added: unique([...manifest.added, ...untracked]), + modified: unique(manifest.modified), + deleted: unique(manifest.deleted), + renamed: manifest.renamed, + }; +} + +export type SyncDeltaResult = + | { status: 'ok'; manifest: SyncManifest } + | { status: 'unavailable'; reason: 'anchor_missing' | 'diff_failed' }; + +export interface ComputeSyncDeltaOpts { + /** + * Pre-computed detached working-tree manifest to merge into the commit diff + * (the executor already builds one for its `up_to_date` gate; pass it to + * avoid a redundant `git diff HEAD` + `ls-files`). When omitted and + * `detached` is true, this module builds it. + */ + detachedManifest?: SyncManifest | null; + /** Build the detached manifest internally (estimator path). Ignored if `detachedManifest` is provided. */ + detached?: boolean; +} + +/** + * The single source of truth for "what changed between two commits in this + * repo." Returns the RAW merged manifest (added/modified/deleted/renamed) — + * callers apply their own `isSyncable` filtering + side effects. + */ +export function computeSyncDelta( + repoPath: string, + fromCommit: string, + toCommit: string, + opts: ComputeSyncDeltaOpts = {}, +): SyncDeltaResult { + const run = _gitRunner; + + // Reachability: a gc'd bookmark object can't be diffed (#1970). + try { + run(repoPath, ['cat-file', '-t', fromCommit]); + } catch { + return { status: 'unavailable', reason: 'anchor_missing' }; + } + + let diffOutput: string; + try { + diffOutput = run(repoPath, ['diff', '--name-status', '-M', `${fromCommit}..${toCommit}`]); + } catch { + return { status: 'unavailable', reason: 'diff_failed' }; + } + + const manifest = buildSyncManifest(diffOutput); + + const detached = + opts.detachedManifest !== undefined && opts.detachedManifest !== null + ? opts.detachedManifest + : opts.detached + ? buildDetachedWorkingTreeManifest(repoPath, run) + : null; + if (detached) { + manifest.added = unique([...manifest.added, ...detached.added]); + manifest.modified = unique([...manifest.modified, ...detached.modified]); + manifest.deleted = unique([...manifest.deleted, ...detached.deleted]); + manifest.renamed = [...manifest.renamed, ...detached.renamed]; + } + + return { status: 'ok', manifest }; +} diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 9ea0a7470..ae5ab4c3f 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -13,7 +13,7 @@ import { startResolveIpcServer, cleanupStaleSocket, } from '../core/context/resolve-ipc.ts'; -import { resolveEntitiesToPointers } from '../core/context/retrieval-reflex.ts'; +import { resolveEntitiesToPointers, logDeliveredReflexPointers } from '../core/context/retrieval-reflex.ts'; export async function startMcpServer(engine: BrainEngine) { const server = new Server( @@ -68,13 +68,24 @@ export async function startMcpServer(engine: BrainEngine) { if (cfg?.engine === 'pglite' && cfg.database_path) { resolveSocket = resolveSocketPath(cfg.database_path); const defaultSource = process.env.GBRAIN_SOURCE || 'default'; - resolveServer = await startResolveIpcServer(resolveSocket, (req) => - resolveEntitiesToPointers( - engine, - req.sourceId || defaultSource, - req.candidates ?? [], - { priorContextText: req.priorContextText, maxPointers: req.maxPointers }, - ), + resolveServer = await startResolveIpcServer( + resolveSocket, + (req) => + resolveEntitiesToPointers( + engine, + req.sourceId || defaultSource, + req.candidates ?? [], + { + priorContextText: req.priorContextText, + maxPointers: req.maxPointers, + suppression: req.suppression, + }, + ), + // The IPC resolve path IS the ambient reflex channel. Logging happens + // at DELIVERY (post-write), not inside the resolver — a block the + // client's 250ms budget abandoned was never injected, and counting it + // would corrupt the volunteered-vs-used precision stats (red-team). + (block) => logDeliveredReflexPointers(engine, block.pointers), ); } } catch { diff --git a/src/schema.sql b/src/schema.sql index 3a5f32ef7..d05b45117 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -707,6 +707,28 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths ( FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE ); +-- #2095 push-based context: feedback-loop log of volunteered pages +-- (volunteer_context op / retrieval-reflex pointers / gbrain watch). +-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is +-- a deterministic template string, never raw conversation text. 90-day prune +-- in the dream cycle's purge phase. Mirrors migration v117. +CREATE TABLE IF NOT EXISTS context_volunteer_events ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + slug TEXT NOT NULL, + confidence DOUBLE PRECISION NOT NULL, + match_arm TEXT NOT NULL, + rationale TEXT NOT NULL DEFAULT '', + channel TEXT NOT NULL DEFAULT 'op', + session_id TEXT, + turn INTEGER, + volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx + ON context_volunteer_events (source_id, volunteered_at DESC); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx + ON context_volunteer_events (source_id, slug); + -- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676) -- because its `job_id BIGINT REFERENCES minion_jobs(id)` FK requires -- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix. diff --git a/test/cli-exit-verdict-pin.test.ts b/test/cli-exit-verdict-pin.test.ts new file mode 100644 index 000000000..75322ca47 --- /dev/null +++ b/test/cli-exit-verdict-pin.test.ts @@ -0,0 +1,34 @@ +/** + * #2084 class pin — every exit-code write in src/ routes through + * setCliExitVerdict. + * + * A RAW `process.exitCode = N` is silently ZEROED by the deliberate + * flush-exit: currentExitCode() reads only gbrain's owned verdict (the + * PGLite-Emscripten-pollution defense), so a command that bypasses the + * setter reports success on failure. Caught live twice in one day: doctor's + * FAIL path exited 0 after a merge introduced a raw write. Runtime variants + * are pinned in test/cli-finish-teardown.test.ts; this is the structural + * guard that catches the NEXT raw write at review time. + */ + +import { describe, test, expect } from 'bun:test'; +import { execSync } from 'child_process'; + +describe('exit-verdict ownership — no raw process.exitCode assignments', () => { + test('every exit-code write in src/ routes through setCliExitVerdict', () => { + // Two legitimate writers are exempt: + // - cli-force-exit.ts: setCliExitVerdict's own mirror-write. + // - pglite-engine.ts preservingProcessExitCode: #2141's containment + // RESTORE around PGlite.create() — it keeps the GLOBAL tidy for + // external readers and is explicitly not a verdict write (the owned + // channel never reads process.exitCode). + // Whitespace/operator-tolerant: catches `process.exitCode=1`, + // `process.exitCode ??= 1`, `process.exitCode ||= 1`, and the bracket + // form — every shape is equally zeroed by the owned-verdict read. + const hits = execSync( + String.raw`grep -rnE "process(\.|\[')exitCode('\])?[[:space:]]*([?|&]{2})?=[^=]" src --include='*.ts' | grep -v "core/cli-force-exit.ts" | grep -v "core/pglite-engine.ts" || true`, + { encoding: 'utf-8', cwd: new URL('..', import.meta.url).pathname }, + ).trim(); + expect(hits).toBe(''); + }); +}); diff --git a/test/cli-format-volunteer.test.ts b/test/cli-format-volunteer.test.ts new file mode 100644 index 000000000..3e7f24226 --- /dev/null +++ b/test/cli-format-volunteer.test.ts @@ -0,0 +1,67 @@ +/** + * v0.43 (#2095) — formatResult's volunteer_context human rendering + * (ship coverage G3): pointer lines with confidence/arm/rationale, + * the empty-result message, and the approximate stats summary. + */ +import { describe, test, expect } from 'bun:test'; +import { formatResult } from '../src/cli.ts'; + +describe('formatResult — volunteer_context', () => { + test('renders pointer lines with confidence, arm, rationale, and synopsis', () => { + const out = formatResult('volunteer_context', { + pages: [ + { + slug: 'people/alice-example', + source_id: 'default', + display: 'Alice Example', + confidence: 0.85, + arm: 'title', + rationale: 'exact title match "Alice Example"; mentioned in the newest turn', + synopsis: 'Alice is an early founder.', + }, + ], + count: 1, + window_turns: 3, + }); + expect(out).toContain('Alice Example → people/alice-example (0.85, title)'); + expect(out).toContain('exact title match'); + expect(out).toContain('Alice is an early founder.'); + }); + + test('empty result explains the confidence gate', () => { + const out = formatResult('volunteer_context', { pages: [], count: 0, window_turns: 1 }); + expect(out).toContain('Nothing volunteered'); + expect(out).toContain('confidence gate'); + }); + + test('stats mode renders totals, per-arm precision, and the approximate note', () => { + const out = formatResult('volunteer_context', { + days: 30, + approximate: true, + note: 'approximate: "used" = pages.last_retrieved_at > volunteered_at.', + total_volunteered: 4, + total_used: 3, + by_arm: [ + { match_arm: 'alias', channel: 'reflex', volunteered: 2, used: 2, precision: 1 }, + { match_arm: 'title', channel: 'op', volunteered: 2, used: 1, precision: 0.5 }, + ], + }); + expect(out).toContain('last 30 day(s)'); + expect(out).toContain('approximate'); + expect(out).toContain('total: 4 volunteered, 3 used'); + expect(out).toContain('alias/reflex: 2/2 used (precision 1)'); + expect(out).toContain('title/op: 1/2 used (precision 0.5)'); + }); + + test('stats mode with zero events prints the empty-window line', () => { + const out = formatResult('volunteer_context', { + days: 7, + approximate: true, + note: 'approximate', + total_volunteered: 0, + total_used: 0, + by_arm: [], + }); + expect(out).toContain('no volunteer events in the window'); + }); +}); diff --git a/test/cli-pipe-truncation.test.ts b/test/cli-pipe-truncation.test.ts new file mode 100644 index 000000000..bea683ef7 --- /dev/null +++ b/test/cli-pipe-truncation.test.ts @@ -0,0 +1,47 @@ +/** + * v0.43 (#2084) — real-CLI pipe completeness pin (the incident #1959 class). + * + * The synthetic flush-mechanism coverage lives in + * test/flush-then-exit-harness.test.ts (4MB late-reader byte-complete pin). + * This file keeps the IMPLEMENTATION-AGNOSTIC check: the actual CLI, run the + * way agents run it (piped stdout), produces complete, parseable, byte-stable + * output and exits deliberately — well under the teardown backstop. + */ + +import { describe, test, expect } from 'bun:test'; +import { spawnSync } from 'child_process'; +import { join, resolve } from 'path'; + +const REPO = resolve(import.meta.dir, '..'); +const CLI = join(REPO, 'src', 'cli.ts'); + +describe('cli pipe completeness — deliberate exit never truncates piped stdout (#2084)', () => { + test('real CLI: --tools-json over a pipe is complete, parseable, byte-stable, and prompt', () => { + const run = () => { + const t0 = Date.now(); + const res = spawnSync('bun', [CLI, '--tools-json'], { + stdio: ['ignore', 'pipe', 'pipe'], + encoding: 'utf-8', + timeout: 60_000, + env: { ...process.env, GBRAIN_SKIP_STARTUP_HOOKS: '1' }, + maxBuffer: 64 * 1024 * 1024, + }); + return { stdout: res.stdout ?? '', stderr: res.stderr ?? '', status: res.status, ms: Date.now() - t0 }; + }; + const first = run(); + expect(first.status).toBe(0); + expect(Buffer.byteLength(first.stdout, 'utf-8')).toBeGreaterThan(16 * 1024); + // Truncated JSON does not parse — the strongest single-run completeness check. + const parsed = JSON.parse(first.stdout); + expect(Array.isArray(parsed)).toBe(true); + // Deliberate exit, not the teardown backstop. A wall-clock bound is flaky + // on cold CI (bun parse alone runs 10-20s there) — the backstop's banner + // is the truthful signal, same assertion the pgbouncer e2e uses. + expect(first.stderr).not.toContain('force-exiting'); + expect(first.stderr).not.toContain('did not return within'); + + const second = run(); + expect(second.status).toBe(0); + expect(second.stdout).toBe(first.stdout); + }, 180_000); +}); diff --git a/test/cli-should-force-exit.test.ts b/test/cli-should-force-exit.test.ts index 9831c7325..1cc5f8a48 100644 --- a/test/cli-should-force-exit.test.ts +++ b/test/cli-should-force-exit.test.ts @@ -82,4 +82,20 @@ describe('shouldForceExitAfterMain — daemon survival gate', () => { expect(shouldForceExitAfterMain(['serves'])).toBe(true); expect(shouldForceExitAfterMain(['serve-cluster'])).toBe(true); }); + + test('awaited long-runners exit deliberately when their handler resolves', () => { + // `jobs work`, `jobs watch --follow`, `autopilot`, and `gbrain watch` + // (#2095) all BLOCK inside their awaited handler until done — when + // main() resolves for them, the work is over and the deliberate exit is + // correct (v0.43 #2084 contract). Only commands that RETURN from main() + // while the event loop carries the daemon (`serve`) belong in + // DAEMON_COMMANDS — `watch` blocks in its stdin iteration, so piped EOF + // must flow through the flush-exit instead of hanging on lingering + // sockets. + expect(shouldForceExitAfterMain(['jobs', 'work'])).toBe(true); + expect(shouldForceExitAfterMain(['jobs', 'watch', '--follow'])).toBe(true); + expect(shouldForceExitAfterMain(['autopilot'])).toBe(true); + expect(shouldForceExitAfterMain(['watch'])).toBe(true); + expect(shouldForceExitAfterMain(['watch', '--json'])).toBe(true); + }); }); diff --git a/test/config-set.test.ts b/test/config-set.test.ts index 9857baf46..8d353c838 100644 --- a/test/config-set.test.ts +++ b/test/config-set.test.ts @@ -29,6 +29,15 @@ describe('KNOWN_CONFIG_KEYS', () => { expect(KNOWN_CONFIG_KEYS).toContain('models.tier.subagent'); }); + test('contains the spend-control keys (v0.42.42.0, #2139) — no --force archaeology', () => { + expect(KNOWN_CONFIG_KEYS).toContain('spend.posture'); + expect(KNOWN_CONFIG_KEYS).toContain('sync.cost_gate_min_usd'); + expect(KNOWN_CONFIG_KEYS).toContain('sync.federated_v2'); + expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_cooldown_min'); + expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_max_usd_per_source_24h'); + expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_max_usd'); + }); + test('no duplicate entries', () => { const set = new Set(KNOWN_CONFIG_KEYS); expect(set.size).toBe(KNOWN_CONFIG_KEYS.length); diff --git a/test/context/resolve-ipc.test.ts b/test/context/resolve-ipc.test.ts index c9e83f8f0..a9687b88d 100644 --- a/test/context/resolve-ipc.test.ts +++ b/test/context/resolve-ipc.test.ts @@ -26,7 +26,10 @@ describe('resolve IPC', () => { test('round-trip: client gets the pointer block the server returns', async () => { const dir = tmpDir(); const sock = resolveSocketPath(dir); - const block: PointerBlock = { pointers: [{ display: 'Alice', slug: 'people/alice', synopsis: 'x' }], text: 'BLOCK' }; + const block: PointerBlock = { + pointers: [{ display: 'Alice', slug: 'people/alice', source_id: 'default', synopsis: 'x', arm: 'alias', confidence: 0.9 }], + text: 'BLOCK', + }; const server = await startResolveIpcServer(sock, async (req) => { expect(req.candidates[0].query).toBe('Alice'); return block; diff --git a/test/doctor-federation-health.test.ts b/test/doctor-federation-health.test.ts index 091ef92f9..71a808fd8 100644 --- a/test/doctor-federation-health.test.ts +++ b/test/doctor-federation-health.test.ts @@ -8,11 +8,24 @@ */ import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { configureGateway } from '../src/core/ai/gateway.ts'; import { checkFederationHealth } from '../src/commands/doctor.ts'; let engine: PGLiteEngine; beforeAll(async () => { + // Pin the legacy OpenAI/1536 embedding shape BEFORE initSchema builds the + // content_chunks vector column. beforeAll runs before the legacy-embedding + // preload's restoring beforeEach fires, so the gateway here is whatever the + // PRIOR file in this shard left it — if that file configured a 1280-d model + // and didn't reset, initSchema would build a vector(1280) column and the + // 1536-d coverage fixture below would fail CheckExpectedDim. Establish the + // dimension this file's fixtures assume rather than inheriting it. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { ...process.env }, + }); engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); diff --git a/test/e2e/helpers.ts b/test/e2e/helpers.ts index 683a83dd3..56cec2ff5 100644 --- a/test/e2e/helpers.ts +++ b/test/e2e/helpers.ts @@ -49,6 +49,9 @@ const ALL_TABLES = [ 'page_versions', 'ingest_log', 'files', + // v0.43 (#2095): volunteered-context feedback log — no FK to pages (slug + // join), but stale rows poison stats/count assertions across runs. + 'context_volunteer_events', 'pages', // last because of foreign keys 'config', 'minion_attachments', diff --git a/test/e2e/pgbouncer-teardown.test.ts b/test/e2e/pgbouncer-teardown.test.ts new file mode 100644 index 000000000..29b3feb9e --- /dev/null +++ b/test/e2e/pgbouncer-teardown.test.ts @@ -0,0 +1,148 @@ +/** + * v0.43 (#2084 / eng-review TD1) — PgBouncer transaction-mode teardown E2E. + * + * Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs + * that were verified only against one production deployment, because CI had + * no transaction-mode pooler. This file pins the bug CLASS, not exact + * timings: a CLI op against a txn-mode pooled URL must + * + * 1. exit zero with intact stdout, and + * 2. NOT ride the 10s hard-deadline backstop (the + * "[cli] engine.disconnect() did not return within 10000ms" banner is + * the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). + * + * Topology: docker-compose.ci.yml runs `pgbouncer` (transaction mode) in + * front of postgres-1. The test uses a DEDICATED database + * (`gbrain_pgbouncer`) created via the direct URL, so it never races the + * TRUNCATE-based fixtures any shard runs against `gbrain_test`. + * + * Gated by GBRAIN_PGBOUNCER_URL + GBRAIN_PGBOUNCER_DIRECT_URL — skips + * gracefully outside the docker CI gate. Run manually: + * + * GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@localhost:6543/gbrain_pgbouncer \ + * GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test \ + * bun test test/e2e/pgbouncer-teardown.test.ts + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs'; +import { join, resolve } from 'path'; +import { tmpdir } from 'os'; +import postgres from 'postgres'; +import { PostgresEngine } from '../../src/core/postgres-engine.ts'; + +const POOLED_URL = process.env.GBRAIN_PGBOUNCER_URL; +const DIRECT_ADMIN_URL = process.env.GBRAIN_PGBOUNCER_DIRECT_URL; +const SKIP = !POOLED_URL || !DIRECT_ADMIN_URL; +const describePooled = SKIP ? describe.skip : describe; + +const REPO = resolve(import.meta.dir, '..', '..'); +const TEST_DB = 'gbrain_pgbouncer'; +const SLUG = 'test/pgbouncer-teardown-fixture'; +const MARKER = 'pgbouncer-teardown-marker-content-7c4f'; + +/** Direct URL pointing at the dedicated test database (same server). */ +function directTestDbUrl(): string { + const u = new URL(DIRECT_ADMIN_URL!); + u.pathname = `/${TEST_DB}`; + return u.toString(); +} + +async function runCli( + args: string[], + env: Record, + timeoutMs: number, +): Promise<{ exitCode: number; stdout: string; stderr: string; wallMs: number }> { + const t0 = Date.now(); + const proc = Bun.spawn(['bun', 'run', join(REPO, 'src', 'cli.ts'), ...args], { + cwd: REPO, + env: { ...process.env, ...env, GBRAIN_SKIP_STARTUP_HOOKS: '1' }, + stdout: 'pipe', + stderr: 'pipe', + }); + const killer = setTimeout(() => { + try { proc.kill('SIGKILL'); } catch { /* already dead */ } + }, timeoutMs); + try { + const [stdout, stderr, exitCode] = await Promise.all([ + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + proc.exited, + ]); + return { exitCode, stdout, stderr, wallMs: Date.now() - t0 }; + } finally { + clearTimeout(killer); + } +} + +describePooled('pgbouncer txn-mode teardown (#2084 / TD1)', () => { + let home: string; + + beforeAll(async () => { + // Dedicated database on the same server, created via the DIRECT url + // (CREATE DATABASE cannot run through a transaction-mode pooler). + const admin = postgres(DIRECT_ADMIN_URL!, { max: 1 }); + try { + await admin.unsafe(`DROP DATABASE IF EXISTS ${TEST_DB} WITH (FORCE)`); + await admin.unsafe(`CREATE DATABASE ${TEST_DB}`); + } finally { + await admin.end({ timeout: 5 }); + } + + // Schema + fixture via the direct connection (DDL stays off the pooler, + // matching the production split-pool discipline). + const eng = new PostgresEngine(); + await eng.connect({ engine: 'postgres', database_url: directTestDbUrl() }); + await eng.initSchema(); + await eng.putPage(SLUG, { + type: 'note', + title: 'PgBouncer teardown fixture', + compiled_truth: MARKER, + timeline: '', + }); + await eng.disconnect(); + + // Brain config pointing the CLI at the POOLED url. + home = mkdtempSync(join(tmpdir(), 'gbrain-pgbouncer-')); + mkdirSync(join(home, '.gbrain'), { recursive: true }); + const pooled = new URL(POOLED_URL!); + pooled.pathname = `/${TEST_DB}`; + writeFileSync( + join(home, '.gbrain', 'config.json'), + JSON.stringify({ engine: 'postgres', database_url: pooled.toString() }) + '\n', + ); + }, 240_000); + + afterAll(() => { + try { rmSync(home, { recursive: true, force: true }); } catch { /* best effort */ } + }); + + test('op against the pooled URL exits clean — output intact, no force-exit banner', async () => { + const env = { HOME: home, GBRAIN_HOME: home }; + const res = await runCli(['get', SLUG], env, 90_000); + + if (res.exitCode !== 0 || /force-exiting/.test(res.stderr)) { + console.error('--- stdout ---\n' + res.stdout); + console.error('--- stderr ---\n' + res.stderr); + } + expect(res.exitCode).toBe(0); + // Output is complete — the #1959 truncation class. + expect(res.stdout).toContain(MARKER); + // The smoking gun: pre-#2084 the hard-deadline banner printed every time + // a query-shaped op ran against a txn-mode pooler. + expect(res.stderr).not.toMatch(/force-exiting/); + expect(res.stderr).not.toMatch(/did not return within/); + // Generous CLASS bound (cold bun parse on CI is 10-20s): the op itself is + // milliseconds; anything that ALSO waited out a 10s teardown backstop + // lands well past this. + expect(res.wallMs).toBeLessThan(60_000); + }, 120_000); + + test('second run (warm schema probe) also exits clean through the pooler', async () => { + const env = { HOME: home, GBRAIN_HOME: home }; + const res = await runCli(['get', SLUG], env, 90_000); + expect(res.exitCode).toBe(0); + expect(res.stdout).toContain(MARKER); + expect(res.stderr).not.toMatch(/force-exiting/); + }, 120_000); +}); diff --git a/test/e2e/volunteer-context-postgres.test.ts b/test/e2e/volunteer-context-postgres.test.ts new file mode 100644 index 000000000..a516fdd53 --- /dev/null +++ b/test/e2e/volunteer-context-postgres.test.ts @@ -0,0 +1,97 @@ +/** + * v0.43 (#2095) — volunteer_context on REAL Postgres (engine parity beyond + * PGLite) + the RLS pin for the v117 table. + * + * The unit suite covers the volunteer core hermetically on PGLite; this file + * proves the same op handler against pgvector Postgres: resolution arms, + * the fire-and-forget event sink landing rows, the stats join, and that + * context_volunteer_events (migration v117) has ROW LEVEL SECURITY enabled (the v35 + * auto_rls_on_create_table event trigger covers migration-created tables — + * this is the assertion that keeps that mechanism honest for new tables). + * + * Gated by DATABASE_URL — skips gracefully without real Postgres. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts'; +import { operationsByName } from '../../src/core/operations.ts'; +import { + awaitPendingVolunteerEventWrites, + _resetPendingVolunteerEventWritesForTests, +} from '../../src/core/context/volunteer-events.ts'; + +const SKIP = !hasDatabase(); +const describePg = SKIP ? describe.skip : describe; + +function mkCtx(engine: unknown, overrides: Record = {}) { + return { + engine, + config: {} as never, + logger: { info: () => {}, warn: () => {}, error: () => {} } as never, + dryRun: false, + remote: false, + sourceId: 'default', + ...overrides, + } as never; +} + +describePg('volunteer_context on real Postgres (#2095)', () => { + beforeAll(async () => { + await setupDB(); + const engine = getEngine(); + await engine.putPage('people/alice-example', { + type: 'person', + title: 'Alice Example', + compiled_truth: 'Alice is an early founder.', + timeline: '', + }); + }, 240_000); + + afterAll(async () => { + await teardownDB(); + }); + + test('op volunteers, logs through the sink, and stats join works', async () => { + _resetPendingVolunteerEventWritesForTests(); + const engine = getEngine(); + const op = operationsByName.volunteer_context; + + const result = (await op.handler(mkCtx(engine), { + window: 'user: who knows the seed market?\nassistant: Alice Example does.\nuser: ask her', + session_id: 'e2e-pg', + })) as any; + expect(result.count).toBe(1); + expect(result.pages[0].slug).toBe('people/alice-example'); + expect(result.pages[0].arm).toBe('title'); + + const { unfinished } = await awaitPendingVolunteerEventWrites(10_000); + expect(unfinished).toBe(0); + + const rows = await engine.executeRaw<{ slug: string; session_id: string }>( + `SELECT slug, session_id FROM context_volunteer_events WHERE session_id = 'e2e-pg'`, + [], + ); + expect(rows.length).toBe(1); + + // Mark the page as opened after volunteering → stats counts it used. + await engine.executeRaw( + `UPDATE pages SET last_retrieved_at = now() + interval '1 minute' WHERE slug = 'people/alice-example'`, + [], + ); + const stats = (await op.handler(mkCtx(engine), { stats: true })) as any; + expect(stats.approximate).toBe(true); + expect(stats.total_volunteered).toBeGreaterThanOrEqual(1); + expect(stats.total_used).toBeGreaterThanOrEqual(1); + }, 120_000); + + test('RLS is enabled on context_volunteer_events (auto-RLS covers v117)', async () => { + const engine = getEngine(); + const rows = await engine.executeRaw<{ relrowsecurity: boolean }>( + `SELECT relrowsecurity FROM pg_class + WHERE oid = 'public.context_volunteer_events'::regclass`, + [], + ); + expect(rows.length).toBe(1); + expect(rows[0].relrowsecurity).toBe(true); + }, 60_000); +}); diff --git a/test/embed-backfill-submit.test.ts b/test/embed-backfill-submit.test.ts index 9a19a0d30..3903ceb94 100644 --- a/test/embed-backfill-submit.test.ts +++ b/test/embed-backfill-submit.test.ts @@ -155,6 +155,54 @@ describe('submitEmbedBackfill — 24h spend cap', () => { expect(result.status).toBe('spend_capped'); expect(result.spendCapUsd).toBe(5); }); + + // v0.42.42.0 (#2139): off-switch + tokenmax bypass. + test('cap "off" → submits even at huge spend (Infinity cap never tripped)', async () => { + await engine.setConfig(SPEND_CAP_CONFIG_KEY, 'off'); + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + spend24hFn: async () => 1e9, + }); + expect(result.status).toBe('submitted'); + }); + + test('0 falls back to the default cap (off semantics ≠ 0)', async () => { + await engine.setConfig(SPEND_CAP_CONFIG_KEY, '0'); + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + spend24hFn: async () => 25, // == default $25 → capped + }); + expect(result.status).toBe('spend_capped'); + expect(result.spendCapUsd).toBe(25); + }); + + test('spend.posture=tokenmax bypasses the cap, marks spendCapBypassed', async () => { + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + postureOverride: 'tokenmax', + spendCapUsdOverride: 25, + spend24hFn: async () => 100, // way over cap + }); + expect(result.status).toBe('submitted'); + expect(result.spendCapBypassed).toBe(true); + expect(result.spend24hUsd).toBe(100); + }); + + test('tokenmax does NOT bypass the cooldown (axis split — churn protection stays)', async () => { + const queue = new MinionQueue(engine); + const job = await queue.add('embed-backfill', { sourceId: 'default' }, {}); + await engine.executeRaw( + `UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '1 minute' WHERE id=$1`, + [job.id], + ); + const result = await submitEmbedBackfill(engine, 'default', { + reason: 'unit', + postureOverride: 'tokenmax', + cooldownMinOverride: 10, + spend24hFn: async () => 1e9, + }); + expect(result.status).toBe('cooldown'); // posture lifts the cap, NOT the cooldown + }); }); describe('submitEmbedBackfill — source isolation', () => { diff --git a/test/fix-wave-structural.test.ts b/test/fix-wave-structural.test.ts index e2b668e70..1b734617c 100644 --- a/test/fix-wave-structural.test.ts +++ b/test/fix-wave-structural.test.ts @@ -250,3 +250,20 @@ describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => { expect(src).toMatch(/buildPgliteInitErrorMessage\(verdict, original\)/); }); }); + +describe('v0.42.43.0 #2095 — volunteer-events sink + cycle purge wiring (structural pins)', () => { + test('volunteer-events registers a background-work drainer (order 4)', () => { + // Deleting this registration would silently drop volunteer events on + // every CLI exit with no behavioral test failing — same pin class as the + // other four sinks above. + const src = readFileSync('src/core/context/volunteer-events.ts', 'utf8'); + expect(src).toMatch(/registerBackgroundWorkDrainer\(\{[\s\S]*?name:\s*'volunteer-events'/); + expect(src).toMatch(/order:\s*4/); + }); + + test("the dream cycle's purge phase invokes purgeStaleVolunteerEvents and reports the count", () => { + const src = readFileSync('src/core/cycle.ts', 'utf8'); + expect(src).toMatch(/purgeStaleVolunteerEvents\(engine\)/); + expect(src).toMatch(/purged_volunteer_events_count/); + }); +}); diff --git a/test/migrate.test.ts b/test/migrate.test.ts index 67fd83248..13067b2c0 100644 --- a/test/migrate.test.ts +++ b/test/migrate.test.ts @@ -2179,3 +2179,67 @@ describe('v112 — pages_links_extracted_at', () => { }); }); + +// v0.43 (#2095): context_volunteer_events — push-based-context feedback log. +describe('v117 — context_volunteer_events_table', () => { + let engine: PGLiteEngine; + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }, 60_000); + afterAll(async () => { if (engine) await engine.disconnect(); }, 60_000); + + test('v117 entry exists, named + idempotent', () => { + const m = MIGRATIONS.find(x => x.version === 117); + expect(m).toBeDefined(); + expect(m!.name).toBe('context_volunteer_events_table'); + expect(m!.idempotent).toBe(true); + }); + + test('LATEST_VERSION is at or above 117', () => { + expect(LATEST_VERSION).toBeGreaterThanOrEqual(117); + }); + + test('table exists after initSchema with the documented columns', async () => { + const rows = await engine.executeRaw<{ column_name: string; is_nullable: string }>( + `SELECT column_name, is_nullable FROM information_schema.columns + WHERE table_name = 'context_volunteer_events' ORDER BY ordinal_position`, [], + ); + const byName = new Map(rows.map(r => [r.column_name, r.is_nullable])); + for (const required of ['source_id', 'slug', 'confidence', 'match_arm', 'rationale', 'channel', 'volunteered_at']) { + expect(byName.get(required)).toBe('NO'); + } + // Caller-supplied attribution columns are nullable by contract (codex D9). + expect(byName.get('session_id')).toBe('YES'); + expect(byName.get('turn')).toBe('YES'); + }); + + test('both source-scoped indexes exist after initSchema', async () => { + const rows = await engine.executeRaw<{ indexname: string }>( + `SELECT indexname FROM pg_indexes WHERE tablename = 'context_volunteer_events'`, [], + ); + const names = rows.map(r => r.indexname); + expect(names).toContain('context_volunteer_events_src_time_idx'); + expect(names).toContain('context_volunteer_events_src_slug_idx'); + }); + + test('insert + 90-day purge round-trip (purgeStaleVolunteerEvents)', async () => { + const { insertVolunteerEvents, purgeStaleVolunteerEvents } = await import('../src/core/context/volunteer-events.ts'); + await insertVolunteerEvents(engine, [ + { source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'alias match', channel: 'op' }, + { source_id: 'default', slug: 'projects/acme-example', confidence: 0.8, match_arm: 'title', rationale: 'exact title', channel: 'reflex', session_id: 's1', turn: 3 }, + ]); + // Age one row past the TTL, leave the other fresh. + await engine.executeRaw( + `UPDATE context_volunteer_events SET volunteered_at = now() - interval '120 days' + WHERE slug = 'projects/acme-example'`, [], + ); + const purged = await purgeStaleVolunteerEvents(engine, 90); + expect(purged).toBe(1); + const left = await engine.executeRaw<{ slug: string }>( + `SELECT slug FROM context_volunteer_events`, [], + ); + expect(left.map(r => r.slug)).toEqual(['people/alice-example']); + }); +}); diff --git a/test/retrieval-reflex.test.ts b/test/retrieval-reflex.test.ts index 385af152b..f79e0c04c 100644 --- a/test/retrieval-reflex.test.ts +++ b/test/retrieval-reflex.test.ts @@ -182,3 +182,258 @@ describe('context-engine assemble() — Retrieval Reflex integration', () => { }); }); }); + +describe('v0.43 (#2095) — rolling window extraction through assemble()', () => { + const REFLEX_ON = { GBRAIN_RETRIEVAL_REFLEX: 'true' }; + + test('entity named ONLY in a previous assistant turn yields a pointer now', async () => { + await withEnv(REFLEX_ON, async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w1', + resolveEntities: (candidates, opts) => + resolveEntitiesToPointers(engine, 'default', candidates, opts), + }); + // Current turn is a pronoun follow-up; the antecedent was NAMED two + // turns back by the ASSISTANT. Pre-window this never fired. + const res = await ce.assemble({ + sessionId: 'w1', + messages: [ + { role: 'user', content: 'who should I talk to about the seed round?' }, + { role: 'assistant', content: 'Alice Example led a similar round last year.' }, + { role: 'user', content: 'what did she invest in?' }, + ], + }); + expect(res.systemPromptAddition).toContain('people/alice-example'); + }); + }); + + test('window=1 reproduces the legacy current-turn-only behavior', async () => { + await withEnv({ ...REFLEX_ON, GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '1' }, async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w2', + resolveEntities: (candidates, opts) => + resolveEntitiesToPointers(engine, 'default', candidates, opts), + }); + const res = await ce.assemble({ + sessionId: 'w2', + messages: [ + { role: 'assistant', content: 'Alice Example led a similar round last year.' }, + { role: 'user', content: 'what did she invest in?' }, + ], + }); + // Current turn has no extractable entity; window=1 must NOT widen. + expect(res.systemPromptAddition).not.toContain('people/alice-example'); + }); + }); + + test('windowed suppression is slug-only: a prior-turn MENTION does not suppress (codex D7)', async () => { + await withEnv(REFLEX_ON, async () => { + await seed('people/alice-example', 'Alice Example', 'A founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w3', + resolveEntities: (candidates, opts) => + resolveEntitiesToPointers(engine, 'default', candidates, opts), + }); + // "Alice Example" appears in a PRIOR turn (a bare mention — prior + // context contains the TITLE). Under the legacy title rule the pointer + // would be suppressed; slug-only windowing must still fire. + const res = await ce.assemble({ + sessionId: 'w3', + messages: [ + { role: 'user', content: 'I met Alice Example yesterday' }, + { role: 'assistant', content: 'How did the meeting with Alice Example go?' }, + { role: 'user', content: 'she wants to invest — thoughts?' }, + ], + }); + expect(res.systemPromptAddition).toContain('people/alice-example'); + }); + }); + + test('windowed suppression still drops an already-surfaced page (slug in prior context)', async () => { + await withEnv(REFLEX_ON, async () => { + await seed('people/alice-example', 'Alice Example', 'A founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w4', + resolveEntities: (candidates, opts) => + resolveEntitiesToPointers(engine, 'default', candidates, opts), + }); + const res = await ce.assemble({ + sessionId: 'w4', + messages: [ + { role: 'assistant', content: 'Pointer: **Alice Example** → `people/alice-example` (use get_page)' }, + { role: 'user', content: 'tell me more about Alice Example' }, + ], + }); + expect(res.systemPromptAddition).not.toContain('Brain pages mentioned this turn'); + }); + }); + + test('fail-open: a throwing resolver under windowing never breaks the turn', async () => { + await withEnv(REFLEX_ON, async () => { + await seed('people/alice-example', 'Alice Example', 'A founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w5', + resolveEntities: async () => { throw new Error('resolver exploded'); }, + }); + const res = await ce.assemble({ + sessionId: 'w5', + messages: [ + { role: 'assistant', content: 'Alice Example is relevant here.' }, + { role: 'user', content: 'ok tell me about her' }, + ], + }); + expect(res.systemPromptAddition).toContain('Live Context'); + expect(res.systemPromptAddition).not.toContain('Brain pages mentioned this turn'); + }); + }); +}); + +describe('ambient-channel event logging (codex D11 — accept-side logDeliveredReflexPointers)', () => { + test('logDeliveredReflexPointers logs channel=reflex events through the drained sink', async () => { + const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts'); + const { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } = + await import('../src/core/context/volunteer-events.ts'); + _resetPendingVolunteerEventWritesForTests(); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await seed('people/alice-example', 'Alice Example', 'A founder.'); + + const block = await resolveEntitiesToPointers( + engine, + 'default', + extractCandidates('what do you think about Alice Example?'), + {}, + ); + expect(block).not.toBeNull(); + logDeliveredReflexPointers(engine, block!.pointers); + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); + const rows = await engine.executeRaw<{ channel: string; slug: string; match_arm: string }>( + 'SELECT channel, slug, match_arm FROM context_volunteer_events', + [], + ); + expect(rows.length).toBe(1); + expect(rows[0].channel).toBe('reflex'); + expect(rows[0].slug).toBe('people/alice-example'); + expect(rows[0].match_arm).toBe('title'); + }); + + test('the bare resolver logs nothing — delivery is the only logging seam', async () => { + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await seed('people/alice-example', 'Alice Example', 'A founder.'); + const block = await resolveEntitiesToPointers( + engine, + 'default', + extractCandidates('about Alice Example'), + {}, + ); + expect(block).not.toBeNull(); + const { awaitPendingVolunteerEventWrites } = await import('../src/core/context/volunteer-events.ts'); + await awaitPendingVolunteerEventWrites(5_000); + const rows = await engine.executeRaw<{ channel: string }>('SELECT channel FROM context_volunteer_events', []); + expect(rows.length).toBe(0); + }); + + test('logDeliveredReflexPointers with an empty pointer list is a no-op', async () => { + const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts'); + const { awaitPendingVolunteerEventWrites } = await import('../src/core/context/volunteer-events.ts'); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + logDeliveredReflexPointers(engine, []); + await awaitPendingVolunteerEventWrites(5_000); + const rows = await engine.executeRaw<{ channel: string }>('SELECT channel FROM context_volunteer_events', []); + expect(rows.length).toBe(0); + }); +}); + +describe('serve IPC wiring — suppression passthrough + reflex-channel logging (review hardening)', () => { + test('the IPC round-trip honors slug-only suppression and logs channel=reflex', async () => { + const { startResolveIpcServer, resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } = + await import('../src/core/context/resolve-ipc.ts'); + const { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } = + await import('../src/core/context/volunteer-events.ts'); + const { mkdtempSync, rmSync } = await import('fs'); + const { join } = await import('path'); + const { tmpdir } = await import('os'); + + _resetPendingVolunteerEventWritesForTests(); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await seed('people/alice-example', 'Alice Example', 'A founder.'); + + const dir = mkdtempSync(join(tmpdir(), 'rr-ipc-')); + const sock = resolveSocketPath(dir); + // The SAME wiring shape src/mcp/server.ts uses for serve: forwards + // suppression from the request; logging happens at DELIVERY via the + // onDelivered hook (post-write), never inside the resolver. + const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts'); + const server = await startResolveIpcServer( + sock, + (req) => + resolveEntitiesToPointers(engine, req.sourceId || 'default', req.candidates ?? [], { + priorContextText: req.priorContextText, + maxPointers: req.maxPointers, + suppression: req.suppression, + }), + (block) => logDeliveredReflexPointers(engine, block.pointers), + ); + expect(server).not.toBeNull(); + try { + // slug-only suppression: a TITLE mention in prior context must NOT + // suppress (the windowing contract), and the resolve must log. + const block = await resolveViaIpc(sock, { + candidates: extractCandidates('tell me about Alice Example'), + priorContextText: 'earlier turn merely mentioned Alice Example', + suppression: 'slug-only', + }); + expect(block).not.toBe(IPC_UNAVAILABLE); + expect(block).not.toBeNull(); + expect((block as { pointers: Array<{ slug: string }> }).pointers[0].slug).toBe('people/alice-example'); + + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); + const rows = await engine.executeRaw<{ channel: string }>( + 'SELECT channel FROM context_volunteer_events', [], + ); + expect(rows.length).toBe(1); + expect(rows[0].channel).toBe('reflex'); + } finally { + server!.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +describe('windowTurnCount — knob edge semantics', () => { + test('0, negative, NaN, and absent all fall back to the default of 4 (1 = legacy off)', async () => { + const { windowTurnCount, DEFAULT_WINDOW_TURNS } = await import('../src/core/context/reflex.ts'); + expect(DEFAULT_WINDOW_TURNS).toBe(4); + expect(windowTurnCount(null)).toBe(4); + expect(windowTurnCount({ retrieval_reflex_window_turns: 0 } as never)).toBe(4); + expect(windowTurnCount({ retrieval_reflex_window_turns: -3 } as never)).toBe(4); + expect(windowTurnCount({ retrieval_reflex_window_turns: Number.NaN } as never)).toBe(4); + // The documented "off" switch is 1 (legacy single-turn), not 0. + expect(windowTurnCount({ retrieval_reflex_window_turns: 1 } as never)).toBe(1); + expect(windowTurnCount({ retrieval_reflex_window_turns: 6.9 } as never)).toBe(6); + }); + + test('the env escape hatch is honored even when config is null (no config file / DB)', async () => { + const { windowTurnCount } = await import('../src/core/context/reflex.ts'); + // loadConfig() returns null in a config-less environment (clean CI shard, + // no brain) and drops its env→config mapping — windowTurnCount must still + // read the env var directly, or the documented escape hatch is dead and + // the window silently defaults to 4. withEnv() (not raw process.env + // mutation) keeps the linter + isolation guard happy. + await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '1' }, async () => { + expect(windowTurnCount(null)).toBe(1); + }); + await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '7' }, async () => { + expect(windowTurnCount(null)).toBe(7); + // Env wins over a config value too (env is the higher-precedence plane). + expect(windowTurnCount({ retrieval_reflex_window_turns: 3 } as never)).toBe(7); + }); + await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: 'not-a-number' }, async () => { + // Garbage env falls through to config / default, not a crash. + expect(windowTurnCount(null)).toBe(4); + }); + }); +}); diff --git a/test/spend-off-switch.test.ts b/test/spend-off-switch.test.ts new file mode 100644 index 000000000..2eb40c1dd --- /dev/null +++ b/test/spend-off-switch.test.ts @@ -0,0 +1,53 @@ +/** + * v0.42.42.0 (#2139) — `--max-usd off` / `--max-cost off` uncapped-switch pins + * across enrich / onboard / reindex (the T6 secondary cost gates). + * + * The enrich arg parser carries the real logic (off → Infinity sentinel, mapped + * to "no BudgetTracker ceiling" in runEnrichCore), so it gets a direct unit test. + * reindex + onboard detect `off` inline at the CLI dispatch and proceed past the + * confirmation / missing-cap refusal; those are pinned as source-level regression + * guards (the codex pre-landing review found all three half-built — these keep + * them from silently regressing without standing up full CLI+gateway harnesses). + */ +import { describe, test, expect } from 'bun:test'; +import { parseArgs } from '../src/commands/enrich.ts'; + +describe('enrich parseArgs — --max-usd off → uncapped (Infinity sentinel)', () => { + test('off / unlimited / none (case-insensitive) → Infinity', () => { + for (const v of ['off', 'OFF', 'unlimited', 'none', 'None']) { + expect(parseArgs(['--max-usd', v]).maxCostUsd).toBe(Infinity); + } + // --max-cost-usd alias too. + expect(parseArgs(['--max-cost-usd', 'off']).maxCostUsd).toBe(Infinity); + }); + test('finite positive number passes through; absent → undefined; garbage → undefined', () => { + expect(parseArgs(['--max-usd', '5']).maxCostUsd).toBe(5); + expect(parseArgs([]).maxCostUsd).toBeUndefined(); + expect(parseArgs(['--max-usd', 'abc']).maxCostUsd).toBeUndefined(); + expect(parseArgs(['--max-usd', '0']).maxCostUsd).toBeUndefined(); // non-positive ignored + }); +}); + +describe('reindex / onboard off-switch dispatch (regression guards)', () => { + test('reindex-code: --max-cost off proceeds past the confirmation gate', async () => { + const src = await Bun.file(new URL('../src/commands/reindex-code.ts', import.meta.url)).text(); + // off sets maxCostOff and the gate proceeds when (tokenmax || maxCostOff). + expect(src).toMatch(/maxCostOff\s*=\s*true/); + expect(src).toMatch(/posture === 'tokenmax' \|\| maxCostOff/); + }); + + test('onboard: --max-usd off lifts the --auto missing-cap refusal', async () => { + const src = await Bun.file(new URL('../src/commands/onboard.ts', import.meta.url)).text(); + expect(src).toMatch(/maxUsdOff\s*=/); + // refusal skipped when (maxUsdOff || tokenmax). + expect(src).toMatch(/maxUsdOff \|\| tokenmax/); + }); + + test('enrich: uncapped path maps the Infinity sentinel to no BudgetTracker ceiling', async () => { + const src = await Bun.file(new URL('../src/commands/enrich.ts', import.meta.url)).text(); + // The sentinel must become `undefined` at the tracker (never raw Infinity, + // which serializes to null in audit rows). + expect(src).toMatch(/opts\.maxCostUsd === Infinity \? undefined/); + expect(src).toMatch(/uncapped \? Infinity : parsed\.maxCostUsd/); + }); +}); diff --git a/test/sync-cost-estimate.test.ts b/test/sync-cost-estimate.test.ts new file mode 100644 index 000000000..cc53947ce --- /dev/null +++ b/test/sync-cost-estimate.test.ts @@ -0,0 +1,140 @@ +/** + * v0.42.42.0 (#2139) — estimateInlineNewTokens ladder coverage. + * + * The inline cost estimator now MIRRORS EXECUTION (delta, not full-tree + * ceiling) so the gate's dollar figure stops being a ~400x phantom on a busy + * brain. Real temp git repos, no PGLite. estimateInlineNewTokens is exported + * from commands/sync.ts; CHUNKER_VERSION is the live chunker version. + */ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { estimateInlineNewTokens } from '../src/commands/sync.ts'; +import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts'; + +const CURRENT = String(CHUNKER_VERSION); +let repo: string; + +function commitAll(msg: string): string { + execSync('git add -A', { cwd: repo, stdio: 'pipe' }); + execSync(`git commit -m "${msg}"`, { cwd: repo, stdio: 'pipe' }); + return execSync('git rev-parse HEAD', { cwd: repo, stdio: 'pipe' }).toString().trim(); +} +function src(over: Partial<{ last_commit: string | null; chunker_version: string | null; config: Record }> = {}) { + return { + local_path: repo, + config: over.config ?? {}, + last_commit: over.last_commit ?? null, + chunker_version: over.chunker_version ?? null, + }; +} + +beforeEach(() => { + repo = mkdtempSync(join(tmpdir(), 'gbrain-est-')); + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "t@t.com"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' }); + mkdirSync(join(repo, 'topics'), { recursive: true }); +}); +afterEach(() => { + if (repo) rmSync(repo, { recursive: true, force: true }); +}); + +describe('estimateInlineNewTokens — ladder', () => { + test('chunker drift → full-tree ceiling even with an empty git delta', () => { + writeFileSync(join(repo, 'topics/a.md'), 'some body content here'); + const head = commitAll('base'); + // git unchanged (last_commit == HEAD) but chunker drifted → must NOT be 0. + const r = estimateInlineNewTokens([src({ last_commit: head, chunker_version: 'STALE-0' })], CURRENT); + expect(r.tokens).toBeGreaterThan(0); + expect(r.estimateKind).toBe('ceiling'); + expect(r.ceilingReasons).toContain('chunker_drift'); + expect(r.changedSources).toBe(1); + }); + + test('[D2A headline] HEAD==last_commit + current chunker + DIRTY tree → 0 (unchanged)', () => { + writeFileSync(join(repo, 'topics/a.md'), 'body'); + const head = commitAll('base'); + // Dirty the tree (untracked scratch + uncommitted edit) — attached sync + // imports nothing, so the estimate must be 0. This is the exact pre-fix + // false-fire shape. + writeFileSync(join(repo, 'scratch.tmp'), 'agent scratch'); + writeFileSync(join(repo, 'topics/a.md'), 'uncommitted edit'); + const r = estimateInlineNewTokens([src({ last_commit: head, chunker_version: CURRENT })], CURRENT); + expect(r.tokens).toBe(0); + expect(r.estimateKind).toBe('unchanged'); + expect(r.unchangedSources).toBe(1); + }); + + test('first sync (last_commit null) → full-tree ceiling', () => { + writeFileSync(join(repo, 'topics/a.md'), 'body content'); + commitAll('base'); + const r = estimateInlineNewTokens([src({ last_commit: null, chunker_version: CURRENT })], CURRENT); + expect(r.tokens).toBeGreaterThan(0); + expect(r.estimateKind).toBe('ceiling'); + expect(r.ceilingReasons).toContain('first_sync'); + }); + + test('delta rung: only changed committed files priced; deletes cost 0', () => { + writeFileSync(join(repo, 'topics/a.md'), 'a'.repeat(400)); + writeFileSync(join(repo, 'topics/b.md'), 'b'.repeat(400)); + const base = commitAll('base'); + writeFileSync(join(repo, 'topics/a.md'), 'a'.repeat(800)); // modify + rmSync(join(repo, 'topics/b.md')); // delete → 0 + commitAll('change'); + + const r = estimateInlineNewTokens([src({ last_commit: base, chunker_version: CURRENT })], CURRENT); + expect(r.estimateKind).toBe('delta'); + expect(r.tokens).toBeGreaterThan(0); // a.md priced + // A pure-delete delta would be 0; here a.md modify keeps it > 0. Sanity: + // the magnitude is delta-scale (one ~800-char file), not full-tree. + }); + + test('delta rung: non-syncable changed file is filtered out (markdown strategy)', () => { + writeFileSync(join(repo, 'topics/a.md'), 'md'); + const base = commitAll('base'); + writeFileSync(join(repo, 'notes.txt'), 'x'.repeat(4000)); // .txt under markdown strategy → not syncable + commitAll('add txt'); + + const r = estimateInlineNewTokens([src({ last_commit: base, chunker_version: CURRENT })], CURRENT); + // No syncable changes → delta priced at 0 tokens (but the source changed). + expect(r.tokens).toBe(0); + expect(r.estimateKind).toBe('delta'); + }); + + test('syncEnabled:false sources are skipped (neither changed nor unchanged)', () => { + writeFileSync(join(repo, 'topics/a.md'), 'body'); + commitAll('base'); + const r = estimateInlineNewTokens([src({ last_commit: null, config: { syncEnabled: false } })], CURRENT); + expect(r.tokens).toBe(0); + expect(r.changedSources).toBe(0); + expect(r.unchangedSources).toBe(0); + }); + + test('mixed: one ceiling source + one unchanged source → estimateKind mixed-or-ceiling, reasons captured', () => { + writeFileSync(join(repo, 'topics/a.md'), 'body'); + const head = commitAll('base'); + const r = estimateInlineNewTokens( + [ + src({ last_commit: null, chunker_version: CURRENT }), // first_sync ceiling + src({ last_commit: head, chunker_version: CURRENT }), // unchanged + ], + CURRENT, + ); + expect(r.ceilingReasons).toContain('first_sync'); + expect(r.unchangedSources).toBe(1); + // hadCeiling true, hadDelta false → 'ceiling' aggregate. + expect(r.estimateKind).toBe('ceiling'); + }); + + test('missing local_path source contributes nothing', () => { + const r = estimateInlineNewTokens( + [{ local_path: null, config: {}, last_commit: null, chunker_version: CURRENT }], + CURRENT, + ); + expect(r.tokens).toBe(0); + expect(r.changedSources).toBe(0); + }); +}); diff --git a/test/sync-cost-gate.serial.test.ts b/test/sync-cost-gate.serial.test.ts index 48d7b057d..1371fd01f 100644 --- a/test/sync-cost-gate.serial.test.ts +++ b/test/sync-cost-gate.serial.test.ts @@ -1,15 +1,17 @@ /** - * v0.41.31 — `gbrain sync --all` cost-gate wiring regressions (PGLite). + * `gbrain sync` cost-gate wiring regressions (PGLite). * - * Pure shouldBlockSync / willEmbedSynchronously logic is pinned in - * test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in - * runSync's --all path: + * Pure shouldBlockSync / willEmbedSynchronously / parseUsdLimit logic is pinned + * in test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in + * runSync's --all AND single-source paths: * * R-1 (headline): deferred-embed sync --all, non-TTY, with backlog → - * emits gate:'deferred_notice' and NEVER exit 2 (the cron-blocking - * bug this release fixes). - * R-2 (protection): inline-embed sync --all (--serial), non-TTY, above - * floor → still exit 2 with gate:'confirmation_required'. + * emits gate:'deferred_notice' and NEVER exit 2. + * R-2 (v0.42.42.0, #2139): inline sync --all (--serial), non-TTY, above floor + * → AUTO-DEFERS (exit 0, gate:'auto_deferred_embeds') and enqueues an + * embed-backfill job. The exit-2 wedge is gone. + * R-3: chunker drift → full-tree CEILING estimate, auto-defers (not exit 2). + * + posture tokenmax, off-switch, format split (#1784/D3A), single-source gate. * * Serial-quarantined: stubs process.exit + console.log (process-global). */ @@ -21,10 +23,18 @@ import { join } from 'path'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; import { runSources } from '../src/commands/sources.ts'; import { resetPgliteState } from './helpers/reset-pglite.ts'; -import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts'; +import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts'; import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts'; import type { ChunkInput } from '../src/core/types.ts'; +/** Offline embed stub so inline-proceed paths (posture tokenmax) don't network. */ +function stubOfflineEmbed(): void { + __setEmbedTransportForTests(async ({ values }: any) => ({ + embeddings: values.map(() => new Array(1536).fill(0)), + usage: { tokens: 0 }, + }) as any); +} + let engine: PGLiteEngine; let repoPath: string; let headSha: string; @@ -64,6 +74,7 @@ beforeEach(async () => { }); afterEach(() => { + __setEmbedTransportForTests(null); resetGateway(); if (repoPath) rmSync(repoPath, { recursive: true, force: true }); }); @@ -115,39 +126,43 @@ describe('v0.41.31 — sync --all cost gate wiring', () => { expect(stdout).toContain('"gate":"deferred_notice"'); }, 60_000); - test('R-2: inline sync --all (--serial) above floor still exit 2', async () => { + test('R-2 (#2139): inline sync --all (--serial) above floor AUTO-DEFERS (exit 0, never exit 2) + enqueues backfill', async () => { await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); - // Floor 0 → any nonzero inline cost blocks. Source is unsynced - // (last_commit NULL) so estimateInlineNewTokens sees it as changed → - // full-tree tokens > 0 → costUsd > 0 > floor. + // Floor 0 → any nonzero inline cost trips the gate. Source is unsynced + // (last_commit NULL) → first-sync ceiling > 0 > floor. await engine.setConfig('sync.cost_gate_min_usd', '0'); - // --serial forces inline even with v2 on. --json → non-TTY exit-2 path. + // --serial forces inline even with v2 on. --json → non-TTY path → AUTO-DEFER. const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); - expect(exitCode).toBe(2); - expect(stdout).toContain('"gate":"confirmation_required"'); + expect(exitCode).not.toBe(2); + expect(stdout).toContain('"gate":"auto_deferred_embeds"'); + expect(stdout).not.toContain('"gate":"confirmation_required"'); + // The run PROCEEDED to import (the wedge is gone) — embeds were deferred, + // not blocked. (The embed-backfill enqueue wiring + its graceful + // missing-table tolerance is pinned in embed-backfill-submit.test.ts; the + // minion_jobs table isn't provisioned in this gate-wiring harness.) + expect(stdout).toContain('"sync_status":"first_sync"'); }, 60_000); - test('R-3: inline, git-unchanged source but STALE chunker_version still estimates (not $0)', async () => { - // The unchanged-source short-circuit requires HEAD==last_commit AND clean - // tree AND chunker_version == current. Here git is unchanged but the - // chunker drifted, so the source must NOT be treated as 0 — sync would - // re-chunk + re-embed everything. floor=0 so any nonzero cost blocks. + test('R-3 (#2139): chunker drift → full-tree CEILING estimate, auto-defers (not exit 2)', async () => { + // git unchanged (HEAD==last_commit) but chunker drifted → the source must + // NOT price $0 (sync would re-chunk + re-embed everything). The estimate is + // the full-tree ceiling; the gate auto-defers rather than wedging. await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, 'STALE-0']); await engine.setConfig('sync.cost_gate_min_usd', '0'); const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); - expect(exitCode).toBe(2); - expect(stdout).toContain('"gate":"confirmation_required"'); + expect(exitCode).not.toBe(2); + expect(stdout).toContain('"gate":"auto_deferred_embeds"'); + expect(stdout).toContain('"estimateKind":"ceiling"'); }, 60_000); - test('R-3 control: inline, git-unchanged + CURRENT chunker_version short-circuits to $0 (no exit 2)', async () => { - // Same setup but chunker_version matches current → the source IS unchanged - // → contributes 0 new-content tokens → below floor → proceeds (no block). - // Proves the short-circuit fires when (and only when) everything matches. + test('R-3 control: git-unchanged + CURRENT chunker → $0 estimate, below floor (no auto-defer)', async () => { + // Mirrors the executor's up_to_date predicate: HEAD==last_commit AND chunker + // matches → 0 new tokens → below floor → proceeds without deferring. await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]); await engine.setConfig('sync.cost_gate_min_usd', '0'); @@ -155,6 +170,78 @@ describe('v0.41.31 — sync --all cost gate wiring', () => { const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); expect(exitCode).not.toBe(2); - expect(stdout).not.toContain('"gate":"confirmation_required"'); + expect(stdout).not.toContain('"gate":"auto_deferred_embeds"'); + expect(stdout).toContain('"estimateKind":"unchanged"'); + }, 60_000); + + test('headline regression: HEAD==last_commit + DIRTY untracked file → $0, no gate (the false-fire)', async () => { + // The exact pre-fix false-fire: a busy brain's working tree is never + // git-clean, but the commits are caught up. The OLD estimator priced the + // whole tree (158M-token phantom); the new one mirrors execution → $0. + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]); + // Dirty the tree with an untracked non-syncable scratch file (agents/crons + // write constantly) — attached-HEAD sync never imports it. + writeFileSync(join(repoPath, 'scratch.tmp'), 'uncommitted agent scratch'); + writeFileSync(join(repoPath, 'topics/foo.md'), 'uncommitted edit, not staged'); + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).not.toBe(2); + expect(stdout).not.toContain('"gate":"auto_deferred_embeds"'); + expect(stdout).toContain('"estimateKind":"unchanged"'); + }, 60_000); + + test('spend.posture=tokenmax → proceeds inline, gate:posture_tokenmax (informational)', async () => { + stubOfflineEmbed(); // inline embed proceeds — keep it off the network. + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.setConfig('sync.cost_gate_min_usd', '0'); + await engine.setConfig('spend.posture', 'tokenmax'); + + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).not.toBe(2); + expect(stdout).toContain('"gate":"posture_tokenmax"'); + expect(stdout).not.toContain('"gate":"auto_deferred_embeds"'); + }, 60_000); + + test('sync.cost_gate_min_usd=off → floor renders "unlimited", never blocks', async () => { + stubOfflineEmbed(); + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.setConfig('sync.cost_gate_min_usd', 'off'); + + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']); + + expect(exitCode).not.toBe(2); + expect(stdout).toContain('"floorUsd":"unlimited"'); + expect(stdout).not.toContain('"gate":"auto_deferred_embeds"'); + }, 60_000); + + test('format split (#1784/D3A): non-TTY WITHOUT --json emits human text, no JSON envelope', async () => { + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + // No --json: above floor in a non-TTY session → human auto-defer text. + const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--no-pull']); + + expect(exitCode).not.toBe(2); + expect(stdout).not.toContain('"gate":'); // no JSON envelope without --json + expect(stdout.toLowerCase()).toContain('deferring embeds'); + expect(stdout).toContain('spend.posture'); // self-describing hint present + }, 60_000); + + test('single-source sync gets the same gate (auto-defers above floor, exit 0)', async () => { + await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']); + await engine.setConfig('sync.cost_gate_min_usd', '0'); + + // Single-source (no --all): unsynced → ceiling > 0 → non-TTY auto-defer. + const { exitCode, stdout } = await runSyncCaptured(['--source', 'vault', '--json', '--no-pull']); + + expect(exitCode).not.toBe(2); + expect(stdout).toContain('"gate":"auto_deferred_embeds"'); + // The gate now exists on the single-source path (was ungated before + // #2139) and proceeds to import rather than blocking. + expect(stdout.toLowerCase()).toContain('imported'); }, 60_000); }); diff --git a/test/sync-cost-preview.test.ts b/test/sync-cost-preview.test.ts index 7f492c032..79ef708dc 100644 --- a/test/sync-cost-preview.test.ts +++ b/test/sync-cost-preview.test.ts @@ -13,7 +13,7 @@ * envelope paths don't depend on DB state. */ -import { describe, test, expect } from 'bun:test'; +import { describe, test, expect, beforeEach } from 'bun:test'; import { EMBEDDING_COST_PER_1K_TOKENS, estimateEmbeddingCostUsd, @@ -21,9 +21,27 @@ import { shouldBlockSync, } from '../src/core/embedding.ts'; import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts'; +import { resetGateway } from '../src/core/ai/gateway.ts'; import { estimateTokens } from '../src/core/chunkers/code.ts'; +import { + parseUsdLimit, + formatUsdLimit, + usdLimitToCap, + normalizeSpendPosture, + isValidSpendPosture, +} from '../src/core/spend-posture.ts'; describe('Layer 8 D1 — embedding cost model', () => { + // The estimateEmbeddingCostUsd tests assert the gateway-UNCONFIGURED OpenAI + // fallback ($0.13/Mtok). That fallback only fires when no gateway is + // configured — so guarantee that precondition rather than depending on + // ambient state. Without this, a sibling test file in the same shard that + // configured (and didn't reset) a cheaper model leaks its rate in here and + // these assertions flip (the legacy-embedding preload only restores when the + // gateway slot is EMPTY, so a non-empty foreign config survives). + beforeEach(() => resetGateway()); + + test('EMBEDDING_COST_PER_1K_TOKENS back-compat constant is the OpenAI 3-large rate', () => { // Retained only for back-compat imports. Live cost math now resolves the // CONFIGURED model's rate via embedding-pricing.ts (see model-aware test @@ -116,6 +134,73 @@ describe('v0.41.31 — shouldBlockSync (cost-gate decision)', () => { expect(shouldBlockSync(0.0001, 0, 'inline')).toBe(true); expect(shouldBlockSync(0, 0, 'inline')).toBe(false); }); + + // v0.42.42.0 (#2139): posture + Infinity-floor behavior. + test('spend.posture=tokenmax never blocks, even above floor', () => { + expect(shouldBlockSync(999, 0.5, 'inline', 'tokenmax')).toBe(false); + expect(shouldBlockSync(999, 0, 'inline', 'tokenmax')).toBe(false); + }); + test('default posture (gated) preserves the legacy decision', () => { + expect(shouldBlockSync(0.51, 0.5, 'inline', 'gated')).toBe(true); + expect(shouldBlockSync(0.03, 0.5, 'inline', 'gated')).toBe(false); + }); + test('off/unlimited floor (Infinity) is never exceeded → never blocks', () => { + expect(shouldBlockSync(999, Infinity, 'inline')).toBe(false); + expect(shouldBlockSync(1e9, Infinity, 'inline', 'gated')).toBe(false); + }); +}); + +describe('v0.42.42.0 (#2139) — spend-posture USD-limit parsing', () => { + test('off / unlimited / none (case-insensitive) → Infinity', () => { + for (const v of ['off', 'OFF', 'unlimited', 'Unlimited', 'none', 'NONE', ' off ']) { + expect(parseUsdLimit(v, 25)).toBe(Infinity); + } + }); + test('finite positive numbers pass through', () => { + expect(parseUsdLimit('5', 25)).toBe(5); + expect(parseUsdLimit(0.5, 25)).toBe(0.5); + expect(parseUsdLimit('100000', 25)).toBe(100000); + }); + test('0 falls back to default unless allowZero', () => { + expect(parseUsdLimit('0', 25)).toBe(25); // backfill cap: off ≠ 0 + expect(parseUsdLimit('0', 0.5, { allowZero: true })).toBe(0); // floor: 0 = block-on-any + }); + test('garbage / negative / empty / null → default', () => { + expect(parseUsdLimit('abc', 25)).toBe(25); + expect(parseUsdLimit('-3', 25)).toBe(25); + expect(parseUsdLimit('', 25)).toBe(25); + expect(parseUsdLimit(null, 25)).toBe(25); + expect(parseUsdLimit(undefined, 0.5)).toBe(0.5); + }); + test('formatUsdLimit: Infinity → "unlimited" (never the JSON.stringify=null trap), finite passthrough', () => { + expect(formatUsdLimit(Infinity)).toBe('unlimited'); + expect(formatUsdLimit(5)).toBe(5); + expect(formatUsdLimit(0)).toBe(0); + // The trap this guards: raw Infinity serializes to null. + expect(JSON.stringify({ cap: Infinity })).toBe('{"cap":null}'); + expect(JSON.stringify({ cap: formatUsdLimit(Infinity) })).toBe('{"cap":"unlimited"}'); + }); + test('usdLimitToCap: Infinity → undefined (no cap), finite passthrough', () => { + expect(usdLimitToCap(Infinity)).toBeUndefined(); + expect(usdLimitToCap(10)).toBe(10); + }); + test('normalizeSpendPosture: only tokenmax is tokenmax; everything else gated', () => { + expect(normalizeSpendPosture('tokenmax')).toBe('tokenmax'); + expect(normalizeSpendPosture('TokenMax')).toBe('tokenmax'); + expect(normalizeSpendPosture('gated')).toBe('gated'); + expect(normalizeSpendPosture('max')).toBe('gated'); + expect(normalizeSpendPosture('')).toBe('gated'); + expect(normalizeSpendPosture(null)).toBe('gated'); + expect(normalizeSpendPosture(42)).toBe('gated'); + }); + test('isValidSpendPosture accepts gated/tokenmax (case-insensitive), rejects the rest', () => { + expect(isValidSpendPosture('gated')).toBe(true); + expect(isValidSpendPosture('tokenmax')).toBe(true); + expect(isValidSpendPosture('TokenMax')).toBe(true); // normalized lowercase + expect(isValidSpendPosture('max')).toBe(false); + expect(isValidSpendPosture('')).toBe(false); + expect(isValidSpendPosture(7)).toBe(false); + }); }); describe('Layer 8 D1 — estimateTokens (exported from chunkers/code.ts)', () => { diff --git a/test/sync-delta.test.ts b/test/sync-delta.test.ts new file mode 100644 index 000000000..20f1ffff8 --- /dev/null +++ b/test/sync-delta.test.ts @@ -0,0 +1,154 @@ +/** + * v0.42.42.0 (#2139) — computeSyncDelta unit coverage. + * + * The shared diff/manifest helper that BOTH the sync executor and the inline + * cost estimator route through (so the gate's dollar figure can't drift from + * what the sync imports). Real temp git repos; no PGLite, no env writes + * (R1/R2-clean). The git-runner seam drives the unavailable branches. + */ +import { describe, test, expect, beforeEach, afterEach } from 'bun:test'; +import { mkdtempSync, writeFileSync, rmSync, mkdirSync, renameSync } from 'fs'; +import { execSync } from 'child_process'; +import { tmpdir } from 'os'; +import { join } from 'path'; +import { + computeSyncDelta, + buildDetachedWorkingTreeManifest, + _setGitRunnerForTests, +} from '../src/core/sync-delta.ts'; + +let repo: string; + +function git(args: string): string { + return execSync(`git ${args}`, { cwd: repo, stdio: 'pipe' }).toString().trim(); +} +function commitAll(msg: string): string { + execSync('git add -A', { cwd: repo, stdio: 'pipe' }); + execSync(`git commit -m "${msg}"`, { cwd: repo, stdio: 'pipe' }); + return git('rev-parse HEAD'); +} + +beforeEach(() => { + repo = mkdtempSync(join(tmpdir(), 'gbrain-delta-')); + execSync('git init', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.email "t@t.com"', { cwd: repo, stdio: 'pipe' }); + execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' }); + mkdirSync(join(repo, 'topics'), { recursive: true }); +}); + +afterEach(() => { + _setGitRunnerForTests(null); + if (repo) rmSync(repo, { recursive: true, force: true }); +}); + +describe('computeSyncDelta — commit diff', () => { + test('A/M/D classified; only committed changes in the manifest', () => { + writeFileSync(join(repo, 'topics/a.md'), 'a'); + writeFileSync(join(repo, 'topics/b.md'), 'b'); + const base = commitAll('base'); + writeFileSync(join(repo, 'topics/a.md'), 'a-edited'); // modify + writeFileSync(join(repo, 'topics/c.md'), 'c'); // add + rmSync(join(repo, 'topics/b.md')); // delete + const head = commitAll('change'); + + const r = computeSyncDelta(repo, base, head); + expect(r.status).toBe('ok'); + if (r.status !== 'ok') return; + expect(r.manifest.modified).toContain('topics/a.md'); + expect(r.manifest.added).toContain('topics/c.md'); + expect(r.manifest.deleted).toContain('topics/b.md'); + }); + + test('rename → destination path on the renamed list', () => { + writeFileSync(join(repo, 'topics/old.md'), 'x'.repeat(200)); + const base = commitAll('base'); + renameSync(join(repo, 'topics/old.md'), join(repo, 'topics/new.md')); + const head = commitAll('rename'); + + const r = computeSyncDelta(repo, base, head); + expect(r.status).toBe('ok'); + if (r.status !== 'ok') return; + expect(r.manifest.renamed.map(x => x.to)).toContain('topics/new.md'); + }); + + test('[D2A] attached HEAD: dirty tracked + untracked files are NOT in the manifest', () => { + writeFileSync(join(repo, 'topics/a.md'), 'a'); + const base = commitAll('base'); + const head = git('rev-parse HEAD'); // HEAD == base, no new commits + // Dirty the tree: an uncommitted edit + an untracked scratch file. + writeFileSync(join(repo, 'topics/a.md'), 'uncommitted edit'); + writeFileSync(join(repo, 'scratch.tmp'), 'untracked'); + + const r = computeSyncDelta(repo, base, head); // not detached → commit diff only + expect(r.status).toBe('ok'); + if (r.status !== 'ok') return; + expect(r.manifest.added).toHaveLength(0); + expect(r.manifest.modified).toHaveLength(0); + expect(r.manifest.deleted).toHaveLength(0); + }); +}); + +describe('computeSyncDelta — detached HEAD merges the working-tree manifest', () => { + test('detached + working-tree changes → merged into the manifest', () => { + writeFileSync(join(repo, 'topics/a.md'), 'a'); + const base = commitAll('base'); + // Detach HEAD and dirty the tree. + execSync(`git checkout --detach ${base}`, { cwd: repo, stdio: 'pipe' }); + writeFileSync(join(repo, 'topics/a.md'), 'detached edit'); // tracked modify + writeFileSync(join(repo, 'topics/new.md'), 'new'); // untracked add + + const r = computeSyncDelta(repo, base, base, { detached: true }); + expect(r.status).toBe('ok'); + if (r.status !== 'ok') return; + expect(r.manifest.modified).toContain('topics/a.md'); + expect(r.manifest.added).toContain('topics/new.md'); // untracked picked up on detached + }); + + test('buildDetachedWorkingTreeManifest: clean detached tree → empty manifest', () => { + writeFileSync(join(repo, 'topics/a.md'), 'a'); + const base = commitAll('base'); + execSync(`git checkout --detach ${base}`, { cwd: repo, stdio: 'pipe' }); + const m = buildDetachedWorkingTreeManifest(repo); + expect(m.added).toHaveLength(0); + expect(m.modified).toHaveLength(0); + }); +}); + +describe('computeSyncDelta — fail-open ladder', () => { + test('bogus anchor SHA → unavailable: anchor_missing', () => { + writeFileSync(join(repo, 'topics/a.md'), 'a'); + const head = commitAll('base'); + const r = computeSyncDelta(repo, 'deadbeefdeadbeefdeadbeefdeadbeefdeadbeef', head); + expect(r.status).toBe('unavailable'); + if (r.status === 'unavailable') expect(r.reason).toBe('anchor_missing'); + }); + + test('non-ancestor anchor still diffs (the #1970 property)', () => { + // git diff A..B is endpoint-tree, no ancestry requirement. + writeFileSync(join(repo, 'topics/a.md'), 'a'); + const base = commitAll('base'); + // Rewrite history: amend creates a new commit not descended from `base`, + // but `base` is still on disk (reflog) → diffable. + writeFileSync(join(repo, 'topics/a.md'), 'rewritten'); + execSync('git add -A && git commit --amend -m rewritten', { cwd: repo, stdio: 'pipe' }); + const head = git('rev-parse HEAD'); + expect(head).not.toBe(base); + + const r = computeSyncDelta(repo, base, head); + expect(r.status).toBe('ok'); // orphaned-but-present anchor is still diffable + }); + + test('injected git failure on the diff → unavailable: diff_failed', () => { + writeFileSync(join(repo, 'topics/a.md'), 'a'); + const base = commitAll('base'); + const head = git('rev-parse HEAD'); + _setGitRunnerForTests((_repo, args) => { + if (args[0] === 'cat-file') return 'commit'; // anchor reachable + if (args[0] === 'diff') throw new Error('simulated oversized diff / timeout'); + return ''; + }); + const r = computeSyncDelta(repo, base, head); + expect(r.status).toBe('unavailable'); + if (r.status === 'unavailable') expect(r.reason).toBe('diff_failed'); + }); +}); diff --git a/test/sync-failures.test.ts b/test/sync-failures.test.ts index 63ccb6a29..b1b762fc8 100644 --- a/test/sync-failures.test.ts +++ b/test/sync-failures.test.ts @@ -170,10 +170,10 @@ describe('Bug 9 — sync.ts CLI flag wiring', () => { // performSync's inner ack path only fires when failedFiles.length > 0 // in the current run. This test pins the up-front ack at the top of // runSync so the flag means "ack whatever is currently flagged". + // v0.42.42.0 (#2139, D13C): the pre-ack is now scoped PER SOURCE — `--all` + // acks every source, single-source acks only its own. const source = await Bun.file(new URL('../src/commands/sync.ts', import.meta.url)).text(); - // Ensure the up-front check exists before the syncAll / performSync - // dispatch, gated on skipFailed. - expect(source).toMatch(/if \(skipFailed\) \{[\s\S]*?unacknowledgedSyncFailures\(\)[\s\S]*?acknowledgeSyncFailures\(\)/); + expect(source).toMatch(/if \(skipFailed\) \{[\s\S]*?syncAll \? acknowledgeFailures\(\) : acknowledgeFailures\(sourceId\)/); }); test('acknowledgeSyncFailures clears stale failures end-to-end', async () => { diff --git a/test/volunteer-context.test.ts b/test/volunteer-context.test.ts new file mode 100644 index 000000000..a972ef4f9 --- /dev/null +++ b/test/volunteer-context.test.ts @@ -0,0 +1,483 @@ +/** + * v0.43 (#2095) — push-based context core: window parsing, multi-turn + * extraction, confidence-gated volunteering, slug-only suppression, privacy, + * and the usage-stats join. Hermetic in-memory PGLite (no file lock), + * modeled on test/retrieval-reflex.test.ts. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { normalizeAlias } from '../src/core/search/alias-normalize.ts'; +import { + extractCandidatesFromWindow, + MAX_CANDIDATES, + type WindowTurn, +} from '../src/core/context/entity-salience.ts'; +import { resolveEntitiesToPointers } from '../src/core/context/retrieval-reflex.ts'; +import { + parseWindow, + volunteerContext, + volunteerUsageStats, + VOLUNTEER_DEFAULT_MIN_CONFIDENCE, +} from '../src/core/context/volunteer.ts'; +import { insertVolunteerEvents } from '../src/core/context/volunteer-events.ts'; +import { TAKES_FENCE_BEGIN, TAKES_FENCE_END } from '../src/core/takes-fence.ts'; + +let engine: PGLiteEngine; + +async function seed(slug: string, title: string, body: string, source = 'default') { + if (source !== 'default') { + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2) + ON CONFLICT (id) DO NOTHING`, + [source, `/tmp/${source}`], + ); + } + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES ($1, $2, 'person', $3, $4, '')`, + [slug, source, title, body], + ); +} + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 60_000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await engine.executeRaw('DELETE FROM page_aliases').catch(() => {}); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await engine.executeRaw('DELETE FROM pages'); +}); + +describe('parseWindow', () => { + test('role prefixes split turns oldest → newest', () => { + const turns = parseWindow('user: ask Alice about the deal\nassistant: noted\nuser: what did she say?'); + expect(turns).toEqual([ + { role: 'user', text: 'ask Alice about the deal' }, + { role: 'assistant', text: 'noted' }, + { role: 'user', text: 'what did she say?' }, + ]); + }); + + test('CRLF input parses identically', () => { + const turns = parseWindow('user: hello\r\nassistant: hi\r\n'); + expect(turns).toEqual([ + { role: 'user', text: 'hello' }, + { role: 'assistant', text: 'hi' }, + ]); + }); + + test('unprefixed text is ONE user turn (echo | volunteer-context just works)', () => { + const turns = parseWindow('met with Alice Example today\nshe asked about acme'); + expect(turns).toEqual([{ role: 'user', text: 'met with Alice Example today\nshe asked about acme' }]); + }); + + test('continuation lines append to the open turn', () => { + const turns = parseWindow('user: first line\nsecond line\nassistant: ok'); + expect(turns[0]).toEqual({ role: 'user', text: 'first line\nsecond line' }); + expect(turns[1]).toEqual({ role: 'assistant', text: 'ok' }); + }); + + test('empty / whitespace input → []', () => { + expect(parseWindow('')).toEqual([]); + expect(parseWindow(' \n \n')).toEqual([]); + }); +}); + +describe('extractCandidatesFromWindow', () => { + test('merges across turns with occurrence + newest-turn metadata', () => { + const turns: WindowTurn[] = [ + { role: 'user', text: 'ask Alice Example about the deal' }, + { role: 'assistant', text: 'Alice Example said she will follow up' }, + { role: 'user', text: 'and ping Bob Sample too' }, + ]; + const cands = extractCandidatesFromWindow(turns); + const alice = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Alice Example')); + const bob = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Bob Sample')); + expect(alice).toBeDefined(); + expect(alice!.occurrences).toBe(2); + expect(alice!.inNewestTurn).toBe(false); + expect(alice!.userMention).toBe(true); + expect(bob).toBeDefined(); + expect(bob!.inNewestTurn).toBe(true); + }); + + test('assistant-only mention is flagged (userMention=false)', () => { + const cands = extractCandidatesFromWindow([ + { role: 'assistant', text: 'You should talk to Charlie Demo about this' }, + { role: 'user', text: 'good idea' }, + ]); + const charlie = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Charlie Demo')); + expect(charlie).toBeDefined(); + expect(charlie!.userMention).toBe(false); + }); + + test('cap holds across a noisy window', () => { + const noisy = Array.from({ length: 30 }, (_, i) => `Entity Number${i} did something.`).join(' '); + const cands = extractCandidatesFromWindow([{ role: 'user', text: noisy }]); + expect(cands.length).toBeLessThanOrEqual(MAX_CANDIDATES); + }); +}); + +describe('volunteerContext', () => { + test('assistant-introduced entity two turns back resolves on a pronoun follow-up', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is an early founder.'); + const turns = parseWindow( + 'user: who should I talk to about the seed round?\n' + + 'assistant: Alice Example led a similar round last year.\n' + + 'user: what did she invest in?', + ); + const pages = await volunteerContext(engine, turns, { sourceIds: ['default'] }); + expect(pages.length).toBe(1); + expect(pages[0].slug).toBe('people/alice-example'); + expect(pages[0].arm).toBe('title'); + expect(pages[0].rationale).toContain('assistant-introduced'); + }); + + test('excludeSlugs skips BEFORE the cap: an excluded entity never starves a fresh page', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.'); + const turns = parseWindow('user: intro Alice Example to Bob Sample'); + const pages = await volunteerContext(engine, turns, { + sourceIds: ['default'], + maxPages: 1, + excludeSlugs: new Set(['people/alice-example']), + }); + // A post-call filter would return [] here (Alice burns the single cap + // slot, then gets filtered). The pre-cap exclusion hands Bob the slot. + expect(pages.length).toBe(1); + expect(pages[0].slug).toBe('people/bob-sample'); + }); + + test('confidence gate drops slug-suffix matches at the default threshold', async () => { + // Page resolvable ONLY via slug-suffix: title differs from the mention. + await seed('projects/widget-co', 'The Widget Company Project', 'A project page.'); + const turns = parseWindow('user: any updates on Widget-Co this week?'); + const gated = await volunteerContext(engine, turns, { sourceIds: ['default'] }); + expect(gated).toEqual([]); + // Lowering min_confidence lets it through with honest provenance. + const loose = await volunteerContext(engine, turns, { sourceIds: ['default'], minConfidence: 0.5 }); + expect(loose.length).toBe(1); + expect(loose[0].arm).toBe('slug-suffix'); + expect(loose[0].confidence).toBeLessThan(VOLUNTEER_DEFAULT_MIN_CONFIDENCE); + }); + + test('alias arm volunteers with boost when mentioned in the newest turn', async () => { + await seed('people/swami-x', 'Swami X', 'A close friend.'); + await engine.setPageAliases('people/swami-x', 'default', [normalizeAlias('Swami')]); + const pages = await volunteerContext( + engine, + parseWindow('user: Spoke with Swami today'), + { sourceIds: ['default'] }, + ); + expect(pages.length).toBe(1); + expect(pages[0].arm).toBe('alias'); + expect(pages[0].confidence).toBeCloseTo(0.95, 5); // 0.9 + newest-turn boost + expect(pages[0].rationale).toContain('alias match'); + }); + + test('suppression is slug-only under windowing: a prior-turn MENTION does not suppress', async () => { + await seed('people/alice-example', 'Alice Example', 'A founder.'); + // Prior context contains the TITLE (a bare mention from an earlier turn) + // but NOT the slug — the page was never actually surfaced. + const pages = await volunteerContext( + engine, + parseWindow('user: ping Alice Example again'), + { sourceIds: ['default'], priorContext: 'earlier the user said: tell Alice Example the news' }, + ); + expect(pages.length).toBe(1); + // A slug in prior context (the page WAS surfaced) does suppress. + const suppressed = await volunteerContext( + engine, + parseWindow('user: ping Alice Example again'), + { sourceIds: ['default'], priorContext: 'pointer: people/alice-example was injected last turn' }, + ); + expect(suppressed).toEqual([]); + }); + + test('privacy: takes-fence content never leaks into the synopsis', async () => { + const body = `${TAKES_FENCE_BEGIN}\nSECRET_HUNCH_DO_NOT_LEAK\n${TAKES_FENCE_END}\nAlice is a founder.`; + await seed('people/alice-example', 'Alice Example', body); + const pages = await volunteerContext(engine, parseWindow('user: about Alice Example'), { sourceIds: ['default'] }); + expect(pages.length).toBe(1); + expect(JSON.stringify(pages)).not.toContain('SECRET_HUNCH_DO_NOT_LEAK'); + }); + + test('multi-source scope: resolves from both sources, never outside the scope', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.', 'default'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.', 'team-brain'); + await seed('people/eve-other', 'Eve Other', 'Out of scope.', 'private-brain'); + const turns = parseWindow('user: intro Alice Example to Bob Sample and Eve Other'); + const pages = await volunteerContext(engine, turns, { sourceIds: ['default', 'team-brain'] }); + const keys = pages.map((p) => `${p.source_id}:${p.slug}`).sort(); + expect(keys).toEqual(['default:people/alice-example', 'team-brain:people/bob-sample']); + }); + + test('zero-candidate fast path: no entities → [] without touching resolution', async () => { + const pages = await volunteerContext(engine, parseWindow('user: ok thanks, sounds good'), { + sourceIds: ['default'], + }); + expect(pages).toEqual([]); + }); + + test('max_pages caps at 5 even when asked for more', async () => { + for (let i = 0; i < 8; i++) { + await seed(`people/person-${i}`, `Person Alpha${i}`, 'A person.'); + } + const text = Array.from({ length: 8 }, (_, i) => `Person Alpha${i}`).join(' and '); + const pages = await volunteerContext(engine, parseWindow(`user: intro ${text}`), { + sourceIds: ['default'], + maxPages: 50, + }); + expect(pages.length).toBeLessThanOrEqual(5); + }); +}); + +describe('volunteerUsageStats', () => { + test('join math: used = last_retrieved_at > volunteered_at, labeled approximate', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.'); + await insertVolunteerEvents(engine, [ + { source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'op' }, + { source_id: 'default', slug: 'people/bob-sample', confidence: 0.8, match_arm: 'title', rationale: 'r', channel: 'op' }, + ]); + // Alice was opened AFTER being volunteered; Bob never was. + await engine.executeRaw( + `UPDATE pages SET last_retrieved_at = now() + interval '1 minute' WHERE slug = 'people/alice-example'`, [], + ); + const stats = await volunteerUsageStats(engine, ['default'], 30); + expect(stats.approximate).toBe(true); + expect(stats.note).toContain('approximate'); + expect(stats.total_volunteered).toBe(2); + expect(stats.total_used).toBe(1); + const alias = stats.by_arm.find((a) => a.match_arm === 'alias')!; + expect(alias.used).toBe(1); + expect(alias.precision).toBe(1); + const title = stats.by_arm.find((a) => a.match_arm === 'title')!; + expect(title.used).toBe(0); + }); + + test('zero events → zeroed stats, not an error', async () => { + const stats = await volunteerUsageStats(engine, ['default'], 7); + expect(stats.total_volunteered).toBe(0); + expect(stats.by_arm).toEqual([]); + }); +}); + +describe('resolveEntitiesToPointers — new provenance surface (back-compat)', () => { + test('pointers carry arm + confidence + source_id', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const block = await resolveEntitiesToPointers( + engine, + 'default', + [{ display: 'Alice Example', query: 'Alice Example' }], + {}, + ); + expect(block).not.toBeNull(); + expect(block!.pointers[0].arm).toBe('title'); + expect(block!.pointers[0].confidence).toBe(0.8); + expect(block!.pointers[0].source_id).toBe('default'); + }); + + test('legacy suppression (slug-and-title) still drops a title mention in prior context', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const block = await resolveEntitiesToPointers( + engine, + 'default', + [{ display: 'Alice Example', query: 'Alice Example' }], + { priorContextText: 'we discussed Alice Example earlier' }, + ); + expect(block).toBeNull(); + }); +}); + +describe('volunteer_context op (contract surface)', () => { + const { operationsByName } = require('../src/core/operations.ts') as typeof import('../src/core/operations.ts'); + const { + awaitPendingVolunteerEventWrites, + _resetPendingVolunteerEventWritesForTests, + } = require('../src/core/context/volunteer-events.ts') as typeof import('../src/core/context/volunteer-events.ts'); + + function mkCtx(overrides: Record = {}) { + return { + engine, + config: {} as never, + logger: { info: () => {}, warn: () => {}, error: () => {} } as never, + dryRun: false, + remote: false, + sourceId: 'default', + ...overrides, + } as never; + } + + test('registered, read-scope, not localOnly, stdin cliHint on window', () => { + const op = operationsByName.volunteer_context; + expect(op).toBeDefined(); + expect(op.scope).toBe('read'); + expect(op.localOnly).toBeFalsy(); + expect(op.cliHints?.name).toBe('volunteer-context'); + expect(op.cliHints?.stdin).toBe('window'); + }); + + test('window required unless stats: true', async () => { + const op = operationsByName.volunteer_context; + await expect(op.handler(mkCtx(), {})).rejects.toThrow(/window is required/); + const stats = (await op.handler(mkCtx(), { stats: true })) as any; + expect(stats.approximate).toBe(true); + }); + + test('volunteers pages and logs events through the drained sink', async () => { + _resetPendingVolunteerEventWritesForTests(); + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const op = operationsByName.volunteer_context; + const result = (await op.handler(mkCtx(), { + window: 'user: ping Alice Example about the deal', + session_id: 's-42', + turn: 7, + })) as any; + expect(result.count).toBe(1); + expect(result.pages[0].slug).toBe('people/alice-example'); + expect(result.window_turns).toBe(1); + // Event row lands via the fire-and-forget sink once drained. + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); + const rows = await engine.executeRaw<{ slug: string; channel: string; session_id: string; turn: number }>( + `SELECT slug, channel, session_id, turn FROM context_volunteer_events`, [], + ); + expect(rows.length).toBe(1); + expect(rows[0].slug).toBe('people/alice-example'); + expect(rows[0].channel).toBe('op'); + expect(rows[0].session_id).toBe('s-42'); + expect(Number(rows[0].turn)).toBe(7); + }); + + test('event-log failure never fails the op (failing engine injected for the INSERT)', async () => { + _resetPendingVolunteerEventWritesForTests(); + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const op = operationsByName.volunteer_context; + // Wrap the engine: reads pass through, INSERTs into the events table throw. + const failingEngine = new Proxy(engine, { + get(target, prop, receiver) { + if (prop === 'executeRaw') { + return (sql: string, params: unknown[]) => { + if (/INSERT INTO context_volunteer_events/.test(sql)) { + return Promise.reject(new Error('telemetry db down')); + } + return target.executeRaw(sql, params); + }; + } + return Reflect.get(target, prop, receiver); + }, + }); + const result = (await op.handler(mkCtx({ engine: failingEngine }), { + window: 'user: ping Alice Example', + })) as any; + expect(result.count).toBe(1); // the volunteer result is unaffected + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); // failed write settled (swallowed), not stuck + }); + + test('federated grant scopes the volunteer (allowedSources)', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.', 'default'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.', 'grant-brain'); + await seed('people/eve-other', 'Eve Other', 'Out of grant.', 'secret-brain'); + const op = operationsByName.volunteer_context; + const result = (await op.handler( + mkCtx({ remote: true, auth: { allowedSources: ['grant-brain'] } }), + { window: 'user: intro Alice Example to Bob Sample and Eve Other' }, + )) as any; + expect(result.pages.map((p: any) => p.slug)).toEqual(['people/bob-sample']); + }); + + test('stats mode is source-scoped and returns the approximate note', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await insertVolunteerEvents(engine, [ + { source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'watch' }, + ]); + const op = operationsByName.volunteer_context; + const stats = (await op.handler(mkCtx(), { stats: true, days: 7 })) as any; + expect(stats.days).toBe(7); + expect(stats.total_volunteered).toBe(1); + expect(stats.by_arm[0].channel).toBe('watch'); + expect(stats.note).toContain('approximate'); + }); +}); + +describe('knob clamps — untrusted MCP caller inputs (review hardening)', () => { + test('minConfidence outside [0,1] (or NaN) falls back to the 0.7 default gate', async () => { + await seed('projects/widget-co', 'The Widget Company Project', 'A project.'); + const turns = parseWindow('user: updates on Widget-Co?'); + for (const bad of [5, -1, Number.NaN]) { + const pages = await volunteerContext(engine, turns, { sourceIds: ['default'], minConfidence: bad }); + expect(pages).toEqual([]); // slug-suffix (0.6+boost) stays gated at the default 0.7 + } + }); + + test('maxPages 0 / negative / NaN fall back to the 3-page default', async () => { + for (let i = 0; i < 5; i++) await seed(`people/person-${i}`, `Person Alpha${i}`, 'A person.'); + const text = Array.from({ length: 5 }, (_, i) => `Person Alpha${i}`).join(' and '); + for (const bad of [0, -3, Number.NaN]) { + const pages = await volunteerContext(engine, parseWindow(`user: intro ${text}`), { + sourceIds: ['default'], + maxPages: bad, + }); + expect(pages.length).toBeLessThanOrEqual(3); + expect(pages.length).toBeGreaterThan(0); + } + }); + + test('stats days <= 0 / NaN falls back to 30', async () => { + const stats = await volunteerUsageStats(engine, ['default'], -5); + expect(stats.days).toBe(30); + const stats2 = await volunteerUsageStats(engine, ['default'], Number.NaN); + expect(stats2.days).toBe(30); + }); +}); + +describe('window-cap ordering — the newest user mention survives the cap', () => { + test('stale assistant-only chatter is dropped before a newest-turn user entity', async () => { + const { extractCandidatesFromWindow: extract, MAX_CANDIDATES: CAP } = await import('../src/core/context/entity-salience.ts'); + // 14 stale assistant-introduced entities in turn 1, then the user names + // ONE entity in the newest turn. The cap (12) must keep the user's. + const stale = Array.from({ length: 14 }, (_, i) => `Stale Chatter${i}`).join(', '); + const cands = extract([ + { role: 'assistant', text: `consider ${stale}.` }, + { role: 'user', text: 'actually ask Alice Example first' }, + ]); + expect(cands.length).toBeLessThanOrEqual(CAP); + const alice = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Alice Example')); + expect(alice).toBeDefined(); + // Recency + user-role weighting puts the newest user mention FIRST. + expect(normalizeAlias(cands[0].query)).toBe(normalizeAlias('Alice Example')); + }); +}); + +describe('volunteer-events sink — timeout branch (long-lived process safety)', () => { + test('a hung write reports unfinished and drops the snapshot (no ghost references)', async () => { + const { + logVolunteerEventsFireAndForget, + awaitPendingVolunteerEventWrites, + _resetPendingVolunteerEventWritesForTests, + _peekPendingVolunteerEventWritesForTests, + } = await import('../src/core/context/volunteer-events.ts'); + _resetPendingVolunteerEventWritesForTests(); + const hangingEngine = { + executeRaw: () => new Promise(() => { /* never settles */ }), + } as never; + logVolunteerEventsFireAndForget(hangingEngine, [ + { source_id: 'default', slug: 'people/x', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'watch' }, + ]); + const { unfinished } = await awaitPendingVolunteerEventWrites(20); + expect(unfinished).toBe(1); + // Snapshot dropped so a long-lived `gbrain watch` never accumulates + // references to forever-pending work (the last-retrieved C1 class). + expect(_peekPendingVolunteerEventWritesForTests()).toBe(0); + _resetPendingVolunteerEventWritesForTests(); + }); +}); diff --git a/test/watch-command.test.ts b/test/watch-command.test.ts new file mode 100644 index 000000000..b29801b0b --- /dev/null +++ b/test/watch-command.test.ts @@ -0,0 +1,231 @@ +/** + * v0.43 (#2095) — `gbrain watch` push transport: streaming loop, rolling + * window, session dedupe, --json shape, event logging on channel 'watch', + * and clean EOF return. Hermetic PGLite + injected line/write deps (no + * subprocess, no real stdin). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { runWatch, WATCH_HELP } from '../src/commands/watch.ts'; +import { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } from '../src/core/context/volunteer-events.ts'; + +let engine: PGLiteEngine; + +async function seed(slug: string, title: string, body: string) { + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES ($1, 'default', 'person', $2, $3, '')`, + [slug, title, body], + ); +} + +async function* feed(lines: string[]): AsyncGenerator { + for (const l of lines) yield l; +} + +async function watchRun(lines: string[], extraArgs: string[] = []): Promise { + const out: string[] = []; + await runWatch(engine, ['--source', 'default', ...extraArgs], { + lines: feed(lines), + write: (s) => out.push(s), + isTTY: false, + }); + return out; +} + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 60_000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + _resetPendingVolunteerEventWritesForTests(); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await engine.executeRaw('DELETE FROM pages'); +}); + +describe('gbrain watch (#2095)', () => { + test('--help prints WATCH_HELP and touches nothing', async () => { + const out = await watchRun([], ['--help']); + expect(out.join('')).toBe(WATCH_HELP); + }); + + test('volunteers per turn and returns cleanly at EOF', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun(['ping Alice Example about the deal']); + const text = out.join(''); + expect(text).toContain('people/alice-example'); + expect(text).toContain('exact title match'); + // runWatch RESOLVED — the EOF → clean-return contract (the entrypoint + // flush-exit + finally drain handle the rest in the real CLI). + }); + + test('rolling window: assistant-introduced entity fires on the pronoun follow-up turn', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun([ + 'user: who should I ask about the round?', + 'assistant: Alice Example led one last year.', + 'user: what did she invest in?', + ]); + expect(out.join('')).toContain('people/alice-example'); + }); + + test('session dedupe: a slug is volunteered at most once per session', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun([ + 'user: ping Alice Example', + 'user: ok', + 'user: Alice Example again please', + ]); + const hits = out.join('').split('people/alice-example').length - 1; + expect(hits).toBe(1); + }); + + test('--json emits one JSONL row per volunteered page with turn attribution', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun(['user: hello there', 'user: ping Alice Example'], ['--json']); + expect(out.length).toBe(1); + const row = JSON.parse(out[0]); + expect(row.slug).toBe('people/alice-example'); + expect(row.turn).toBe(2); + expect(row.arm).toBe('title'); + expect(typeof row.confidence).toBe('number'); + }); + + test('events land on channel watch with session_id + turn (drained sink)', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + await watchRun(['user: ping Alice Example']); + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); + const rows = await engine.executeRaw<{ channel: string; session_id: string; turn: number }>( + `SELECT channel, session_id, turn FROM context_volunteer_events`, [], + ); + expect(rows.length).toBe(1); + expect(rows[0].channel).toBe('watch'); + expect(rows[0].session_id).toMatch(/^watch-/); + expect(Number(rows[0].turn)).toBe(1); + }); + + test('min-confidence flag gates exactly like the op', async () => { + await seed('projects/widget-co', 'The Widget Company Project', 'A project.'); + const gated = await watchRun(['user: updates on Widget-Co?']); + expect(gated.join('')).toBe(''); + const loose = await watchRun(['user: updates on Widget-Co?'], ['--min-confidence', '0.5']); + expect(loose.join('')).toContain('projects/widget-co'); + }); + + test('blank lines and CRLF are tolerated; no turn, no volunteer', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun(['', ' ', 'user: ping Alice Example\r']); + expect(out.join('')).toContain('people/alice-example'); + }); +}); + +describe('gbrain watch — window + cap flags (ship coverage G4)', () => { + test('--window-turns 1: fires on the mention turn itself; the pronoun follow-up adds nothing', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun( + [ + 'assistant: Alice Example led one last year.', + 'user: what did she invest in?', + ], + ['--window-turns', '1', '--json'], + ); + // Watch volunteers per turn: the entity fires at turn 1 (its mention + // turn). With window=1 the follow-up turn extracts nothing, so exactly + // one row exists and it carries turn 1 attribution. + expect(out.length).toBe(1); + expect(JSON.parse(out[0]).turn).toBe(1); + }); + + test('--max-pages 1 caps a multi-entity turn to one volunteered page', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.'); + const out = await watchRun( + ['user: intro Alice Example to Bob Sample'], + ['--max-pages', '1', '--json'], + ); + expect(out.length).toBe(1); + }); +}); + +describe('gbrain watch — red-team hardening', () => { + test('starvation guard: an already-pushed slug never burns the --max-pages cap slot', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.'); + const out = await watchRun( + [ + 'user: ping Alice Example about the round', + 'user: also intro Alice Example to Bob Sample', + ], + ['--max-pages', '1', '--json'], + ); + // Turn 1: Alice takes the single cap slot. Turn 2: Alice is excluded + // BEFORE the cap (not filtered after), so Bob gets the slot instead of + // the turn volunteering nothing forever. + expect(out.length).toBe(2); + expect(JSON.parse(out[0]).slug).toBe('people/alice-example'); + expect(JSON.parse(out[1]).slug).toBe('people/bob-sample'); + }); + + test('--window-turns clamps: 0 and negatives behave as window 1', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const out = await watchRun( + [ + 'assistant: Alice Example led one last year.', + 'user: what did she invest in?', + ], + ['--window-turns', '0', '--json'], + ); + // Clamped to 1: fires on the mention turn only — identical to the + // --window-turns 1 contract above, not a crash or an unbounded window. + expect(out.length).toBe(1); + expect(JSON.parse(out[0]).turn).toBe(1); + }); + + test('--window-turns absurdly large still runs (hard cap, no unbounded re-scan)', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const filler = Array.from({ length: 70 }, (_, i) => `user: filler line ${i}`); + const out = await watchRun( + [...filler, 'user: ping Alice Example'], + ['--window-turns', '999999', '--json'], + ); + expect(out.length).toBe(1); + expect(JSON.parse(out[0]).slug).toBe('people/alice-example'); + }); +}); + +describe('gbrain watch — per-turn fail-open (review hardening)', () => { + test('a transient DB error on one turn never kills the stream', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + // Fail the FIRST resolver query against pages (turn 1's resolution); + // everything else — incl. resolveSourceId's pre-loop check — passes. + let pagesQueries = 0; + const flaky = new Proxy(engine, { + get(target, prop, receiver) { + if (prop === 'executeRaw') { + return (sql: string, params: unknown[]) => { + if (/FROM pages/.test(sql) && ++pagesQueries === 1) { + return Promise.reject(new Error('transient db hiccup')); + } + return target.executeRaw(sql, params); + }; + } + return Reflect.get(target, prop, receiver); + }, + }); + const out: string[] = []; + await runWatch(flaky as never, ['--source', 'default'], { + lines: feed(['user: ping Alice Example', 'user: ping Alice Example please']), + write: (s) => out.push(s), + isTTY: false, + }); + // Turn 1 failed open; turn 2 volunteered. The stream survived. + expect(out.join('')).toContain('people/alice-example'); + }); +}); diff --git a/test/watch-sigint.serial.test.ts b/test/watch-sigint.serial.test.ts new file mode 100644 index 000000000..62edfe713 --- /dev/null +++ b/test/watch-sigint.serial.test.ts @@ -0,0 +1,75 @@ +/** + * v0.43 (#2095) — `gbrain watch` SIGINT lifecycle. SERIAL: spawns a real CLI + * subprocess with a tmpdir brain (the parallel unit shards flake on + * concurrent subprocess spawns — same isolation rationale as + * apply-migrations-pglite-spawn.serial.test.ts). + */ +import { describe, test, expect } from 'bun:test'; + +describe('gbrain watch — SIGINT lifecycle (real subprocess)', () => { + test('SIGINT mid-stream closes the stream and exits cleanly (drain path, exit 0)', async () => { + const { mkdtempSync, mkdirSync, writeFileSync, rmSync } = await import('fs'); + const { join, resolve } = await import('path'); + const { tmpdir } = await import('os'); + const REPO = resolve(import.meta.dir, '..'); + const home = mkdtempSync(join(tmpdir(), 'gbrain-watch-sigint-')); + try { + mkdirSync(join(home, '.gbrain'), { recursive: true }); + writeFileSync( + join(home, '.gbrain', 'config.json'), + JSON.stringify({ + engine: 'pglite', + database_path: join(home, '.gbrain', 'brain.pglite'), + embedding_dimensions: 1536, + }) + '\n', + ); + // Piped stdin that NEVER reaches EOF — only SIGINT can end the stream. + const proc = Bun.spawn(['bun', 'run', join(REPO, 'src', 'cli.ts'), 'watch'], { + cwd: REPO, + env: { ...process.env, HOME: home, GBRAIN_HOME: home, GBRAIN_SKIP_STARTUP_HOOKS: '1' }, + stdin: 'pipe', + stdout: 'pipe', + stderr: 'pipe', + }); + proc.stdin.write('user: nothing relevant here\n'); + await proc.stdin.flush(); + // Readiness probe: watch prints "[watch] session ready" on stderr + // once engine + source resolution are done and the stdin loop is live. + // A fixed sleep raced cold PGLite init (other tests budget 60s for it) + // — SIGINT before the handler registers means default-disposition kill. + const stderrChunks: string[] = []; + const reader = (proc.stderr as ReadableStream).getReader(); + const decoder = new TextDecoder(); + const deadline = Date.now() + 90_000; + let ready = false; + while (Date.now() < deadline) { + const { value, done } = await reader.read(); + if (done) break; + stderrChunks.push(decoder.decode(value, { stream: true })); + if (stderrChunks.join('').includes('ready')) { ready = true; break; } + } + expect(ready).toBe(true); + // Brief settle so the first turn's volunteerContext round-trip is in + // flight or done, then interrupt mid-stream. + await new Promise((r) => setTimeout(r, 500)); + proc.kill('SIGINT'); + const killer = setTimeout(() => { + try { proc.kill('SIGKILL'); } catch { /* already dead */ } + }, 30_000); + const exitCode = await proc.exited; + clearTimeout(killer); + // Drain the rest of stderr for the banner assertion. + while (true) { + const { value, done } = await reader.read(); + if (done) break; + stderrChunks.push(decoder.decode(value, { stream: true })); + } + const stderr = stderrChunks.join(''); + // Clean drain-then-exit: no force-exit banner, no SIGKILL (137), exit 0. + expect(stderr).not.toContain('force-exiting'); + expect(exitCode).toBe(0); + } finally { + rmSync(home, { recursive: true, force: true }); + } + }, 120_000); +});