Merge remote-tracking branch 'origin/master' into garrytan/paced-backfill-internals

This commit is contained in:
Garry Tan
2026-06-16 22:08:47 -07:00
69 changed files with 4983 additions and 389 deletions
+3 -2
View File
@@ -104,8 +104,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
## Before shipping
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL`
unset) and tears down. Use `bun run ci:local:diff` for the
diff-aware subset during fast iteration on a focused branch. Requires Docker
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
+49
View File
@@ -2,6 +2,55 @@
All notable changes to GBrain will be documented in this file.
## [0.42.45.0] - 2026-06-13
**The daily sync cron stops wedging on cost, and the embedding-spend estimate finally matches what a sync actually does (gbrain#2139).** On an active brain the inline-embed cost gate priced the *entire* corpus every time the working tree was dirty — which is always, since agents and crons write to it constantly — so a routine daily sync estimated ~158M tokens / ~$8 when the real delta was a few hundred files / ~$0.04, then blocked the cron with a confirmation it could never answer. Embeds silently stalled until someone noticed. The estimate now mirrors execution: it fetches first and prices only the files this run will pull and import, through the same diff machinery the sync itself uses. A brain whose commits are caught up but whose tree is dirty estimates $0, because an attached-HEAD sync imports only the committed diff.
When the gate does fire in a non-interactive session, it no longer exits with an error — it imports now and defers embedding to capped background jobs (which drain via the jobs worker or `gbrain embed --stale`), so a cron is never wedged again. Operators who have decided cost isn't the constraint get one switch — `gbrain config set spend.posture tokenmax` — that makes every cost gate informational across sync, reindex, enrich, and onboard (spend is still recorded; the switch removes the ceiling, not the accounting). The USD knobs accept `off` / `unlimited`, and every gate message now carries paste-ready commands so the controls are discoverable at the moment they fire.
This release also lifts the rule that blocked `--skip-failed` / `--retry-failed` under parallel sync — failure recovery no longer has to drop to `--serial` (which is what armed the inline gate in the first place).
### Added
- **`spend.posture` config** — `tokenmax` makes every embedding-cost gate informational (print the estimate, proceed, keep the ledger); `gated` (default) enforces as before. Documented end-to-end in `docs/operations/spend-controls.md`.
- **First-class off switches**`sync.cost_gate_min_usd`, `embed.backfill_max_usd_per_source_24h`, `embed.backfill_max_usd`, and `reindex --max-cost` / `enrich --max-usd` accept `off` / `unlimited` / `none`. No more sentinel values like `100000`.
- **Single-source `gbrain sync` cost preview** — plain `gbrain sync` previously embedded inline with no preview; it now carries the same gate as `sync --all` (auto-defers in non-TTY sessions, never blocks).
- **Self-describing gate messages** — every cost-gate / FYI line ends with the exact `gbrain config set` commands to widen, disable, or switch posture, plus a docs pointer.
### Changed
- **`gbrain sync --all` is no longer blocked by the cost gate in cron/agent contexts.** Above the floor in a non-interactive session it auto-defers embeds (exit 0) instead of emitting a `cost_preview_requires_yes` envelope and exiting 2. Cron wrappers that branched on exit 2 now see exit 0 with `status: "auto_deferred"`. A TTY still prompts `[y/N]`; `--yes` still embeds inline.
- **`--skip-failed` / `--retry-failed` now work under parallel sync.** The failure ledger is per-source and lock-serialized, so the previous "not supported under parallel — re-run with --serial" refusal is retired.
- **The six spend-control config keys are now first-class** (`gbrain config set` accepts them without `--force`).
### To take advantage of v0.42.45.0
`gbrain upgrade`. Nothing to configure for the headline fix — the daily sync cron stops wedging and the estimate is accurate out of the box. If you run a high-volume brain where cost genuinely isn't the constraint, `gbrain config set spend.posture tokenmax` makes every gate informational. To widen or disable a specific gate instead, see the table in `docs/operations/spend-controls.md` (e.g. `gbrain config set sync.cost_gate_min_usd off`). Failure-recovery syncs can now stay parallel: `gbrain sync --all --skip-failed` no longer forces `--serial`.
## [0.42.44.0] - 2026-06-13
### Fixed
- **Personal-brain tutorial points at the correct AlphaClaw site.** Step 4 of `docs/tutorials/personal-brain.md` ("Deploy via AlphaClaw on Render") linked to the wrong top-level domain, sending readers to a site that isn't the official AlphaClaw. The link now resolves to the right destination, so the deploy step works as written (gbrain#2165).
## [0.42.43.0] - 2026-06-12
**The brain now volunteers relevant pages instead of waiting to be asked (gbrain#2095).** Retrieval used to be pull-only: a deep session could run for hours with zero brain contributions — not because the brain had nothing, but because nothing prompted the agent to ask, and pages stored under coined names were missed by literal-string queries. Push-based context inverts that, on three channels sharing one zero-LLM, confidence-gated core: the ambient retrieval reflex now reads the last few conversation turns (an entity your assistant introduced two turns ago resolves on the "what did she invest in?" follow-up), a new `volunteer_context` operation gives any agent a per-turn volunteer surface over CLI stdin or MCP, and `gbrain watch` streams volunteered pages as a transcript flows through it.
Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got retrieved afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the retrieval signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves.
This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in the local CI gate (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure.
### Added
- **`volunteer_context` operation** (CLI: `gbrain volunteer-context`, MCP tool) — pipe recent turns in (`user:` / `assistant:` prefixed lines, or plain text), get confidence-gated page pointers with rationales and synopses out. `--stats` returns the volunteered-vs-used precision summary. Per-call knobs: `max_pages`, `min_confidence`, `session_id`/`turn` attribution.
- **`gbrain watch`** — the streaming push transport: feed a transcript on stdin, volunteered pages stream out (`--json` for JSONL), each slug at most once per session. Piped input exits cleanly at end-of-input; interactive sessions run until Ctrl-C.
- **Rolling-window retrieval reflex** — the ambient channel extracts entities from the last 4 turns (configurable via `retrieval_reflex_window_turns` / `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`; 1 restores the previous single-turn behavior). Assistant-introduced entities and named-antecedent follow-ups now surface pointers with zero agent-initiated queries.
- **Volunteered-context feedback log** — volunteered pages are recorded best-effort (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses always strip the takes/facts privacy fences before reaching a prompt.
- **Transaction-mode pooler in the local CI gate**`bun run ci:local` now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after.
### Fixed
- **`gbrain doctor` exits 1 on FAIL again on every engine.** Its verdict write predated the v0.42.42.0 exit-verdict channel and was being silently zeroed; swept, plus a structural test that fails CI on the next raw exit-code write anywhere in the CLI.
- **Reflex pointer suppression under multi-turn windows** distinguishes "page already surfaced" from "entity merely mentioned earlier" — without this, window extraction would have suppressed every prior-turn entity by construction.
### To take advantage of v0.42.43.0
`gbrain upgrade` (applies the new feedback-log migration automatically). The wider reflex window is on by default for context-engine hosts — set `retrieval_reflex_window_turns: 1` in `~/.gbrain/config.json` to restore single-turn behavior. Agents without the context engine: call `volunteer_context` per turn (window in, pointers out), or pipe a transcript through `gbrain watch --json`. After a few days, `gbrain volunteer-context --stats` shows which resolution arms are earning their keep; raise or lower `min_confidence` accordingly.
## [0.42.42.0] - 2026-06-12
**`gbrain query` no longer pays a flat 10-second exit tax on managed Postgres behind a transaction-mode pooler — and CLI exit codes finally tell the truth on PGLite.** On deployments where the pooler holds sockets open past the bounded pool drain (gbrain#2084, a residual of gbrain#1972), every query printed its results and then sat for 10 seconds until the force-exit banner fired. The cause was two-layered: the hard-deadline timer was armed *before* the operation handler, so a multi-second search on a large brain burned the teardown budget (and any operation slower than 10 seconds was silently killed mid-run with exit 0 and truncated output); and the CLI never exited explicitly on success — it waited for Bun's event loop to drain, which a stuck pooler socket can hold open forever.
+3 -1
View File
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -97,6 +97,8 @@ detail on demand.)
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
| search modes / cost knobs | `docs/guides/search-modes.md` |
| embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` |
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
+90 -5
View File
@@ -1,5 +1,88 @@
# TODOS
## Spend-controls wave follow-ups (filed v0.42.45.0, #2139)
Deferred from the #2139 delta-estimator wave. See plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-lovely-balloon.md`.
- [ ] **P3 — Measured post-import chunk-count gating (#2139 proposal 2b).**
**What:** Gate the inline cost decision on the actual chunk count sync produced
(known after import, before embedding) instead of the pre-sync token estimate.
**Why:** A fully execution-accurate gate with zero estimate error. **Context:**
After v0.42.42.0 the estimator already mirrors execution (fetch-first delta via the
shared `computeSyncDelta`, `--full`=delta+stale, dirty-tree→$0). This is the
belt-and-suspenders fallback if a future case still drifts. **Trigger:** only if the
delta estimator proves insufficient in practice. **Start:** the gate call site in
`src/commands/sync.ts` (`runInlineCostGate`), gate on post-import `chunksCreated`.
- [ ] **P3 — Per-source defer granularity (#2139, D8A road-not-taken).**
**What:** When the aggregate inline gate trips in a non-TTY session, defer embeds
only for sources above a per-source floor; let cheap sources keep embedding inline.
**Why:** Cheap sources would get embeddings minutes sooner instead of waiting for a
backfill-worker drain. **Context:** v0.42.42.0 chose GLOBAL defer (one flag, strictly
dominates the exit-2 it replaced). This is the granularity upgrade. **Trigger:** a
filed embedding-latency-by-minutes complaint. **Start:** thread per-source estimates
through `runOne` (`src/commands/sync.ts`); design worked out at D8A in the plan.
## gbrain#2095 push-based context follow-ups (v0.43+)
Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`).
Deliberately scoped OUT of v1 per the eng-review scope decision (success criteria
are the bar). Plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-cheerful-elephant.md`.
- [ ] **P3 — SSE/HTTP push channel via serve-http.** The op + `gbrain watch` cover
pull-per-turn and stdin streaming; a serve-http SSE feed would push volunteered
pages to remote agents without a local CLI. **Why:** thin-client/remote-MCP
deployments get push too. **Cons:** async plumbing + auth scoping; no consumer
wired today. **Where:** `src/commands/serve-http.ts` + `src/core/context/volunteer.ts`.
**Blocked by:** a real consumer (revisit when one exists).
- [ ] **P3 — policy skill + doctor check for push-context.** The ambient reflex
needed doctor visibility because silent failure was invisible; volunteer is
invoked-on-demand so v1 skipped it. If `volunteer-context --stats` adoption shows
agents not discovering the surface, ship a `push-context` recipe (mirror
`recipes/retrieval-reflex/`) + a doctor check reading the events table.
**Where:** `recipes/`, `src/commands/doctor.ts`.
- [ ] **P3 — structured `messages[]` param for volunteer_context.** v1 takes a
string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract.
If MCP callers accumulate parsing bugs, add a structured array param beside it.
**Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`.
- [ ] **P3 — index shapes for the per-turn resolver query.** The arm-2 resolver
(`retrieval-reflex.ts`: `lower(title) = ANY() OR slug = ANY() OR slug LIKE
ANY('%/...')`) predates #2095 but now runs per turn on three channels
(reflex window, volunteer_context, watch) federated across sources. Neither
the leading-wildcard suffix arm nor `lower(title)` is index-served. If
per-turn latency telemetry on large brains comes back hot: add
`(source_id, lower(title))` btree + a reverse(slug) text_pattern_ops (or
gin_trgm) index, or split the OR into three index-friendly queries.
**Where:** `src/core/context/retrieval-reflex.ts`, migration.
- [ ] **P3 — batch the volunteer-events pruner's first run after a long gap.**
`purgeStaleVolunteerEvents` is one unbatched DELETE with a bare
`volunteered_at` predicate (full scan; fine for a TTL-bounded table). Edge:
a brain whose dream cycle was off for months could hit the pooler's ~2min
statement_timeout on the first prune, get swallowed by the catch, and never
make progress. If observed: id-batched chunks (`DELETE ... WHERE id IN
(SELECT ... LIMIT 10000)` looped). **Where:**
`src/core/context/volunteer-events.ts:purgeStaleVolunteerEvents`.
- [ ] **P3 — route `gbrain watch` through the serve resolve-IPC on PGLite.**
`watch` connects directly, so on a PGLite brain it monopolizes the single
connection for its whole (potentially hours-long) session — a concurrent
`gbrain serve` or any write path blocks on the lock until watch exits.
WATCH_HELP documents the monopoly; the fix is an IPC rung in watch's
resolver (reuse `resolveViaIpc` like the ambient reflex's ladder) so a
running serve answers and watch never takes the lock. **Why:** watch +
serve concurrently is the natural agent topology. **Where:**
`src/commands/watch.ts`, `src/core/context/resolve-ipc.ts` (red-team RT2).
- [ ] **P3 — capability/version gate for host-injected reflex resolvers.**
Windowing switched the orchestrator's suppression request to 'slug-only';
a host resolver built against the pre-window contract that still applies
title-whole-word suppression silently self-suppresses every windowed
entity. The contract is documented at `ResolveEntitiesFn` (reflex.ts), but
nothing detects a stale host. Add a capability handshake (e.g. resolver
advertises `supportsSuppressionModes`) and fall back to
`window_turns: 1` semantics when absent. **Where:**
`src/core/context/reflex.ts:ResolveEntitiesFn` + the OpenClaw plugin
contract (red-team RT4).
## gbrain triage wave follow-ups (filed v0.42.41.0)
Deferred from the v0.42.41.0 fix wave (eng-reviewed as separate scope, not hotfixes).
@@ -32,11 +115,13 @@ Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extra
is deterministic + precision-biased. See plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-wild-yeti.md`.
- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The v1 extractor
(`src/core/context/entity-salience.ts`) misses lowercase names, many non-Latin
scripts, pronoun follow-ups ("what about her?"), and assistant-introduced entities.
These need conversation state or an LLM pass. **Why:** higher recall on the read
side. **Where:** `entity-salience.ts` + the orchestrator's `priorContextText`.
- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The extractor
(`src/core/context/entity-salience.ts`) misses lowercase names and many non-Latin
scripts; these need an LLM pass or script-aware heuristics. **Why:** higher recall
on the read side. **Where:** `entity-salience.ts`. *(Partially done by the #2095
wave: `extractCandidatesFromWindow` now covers assistant-introduced entities and
pronoun follow-ups whose antecedent was NAMED in the rolling window; true pronoun
coreference for never-named antecedents remains with the LLM-pass idea.)*
- [ ] **P3 — recall knob: optional fuzzy/prefix-expansion resolution.** The resolver
(`src/core/context/retrieval-reflex.ts`) is exact-only (alias + title + slug-suffix)
for precision. Revisit adding `resolveEntitySlug`'s trgm-fuzzy / prefix-expansion
+1 -1
View File
@@ -1 +1 @@
0.42.42.0
0.42.45.0
+36
View File
@@ -85,6 +85,40 @@ services:
volumes:
- gbrain-ci-pg-data-4:/var/lib/postgresql/data
# v0.43 (#2084 / eng-review TD1): PgBouncer in TRANSACTION pooling mode
# fronting postgres-1 — the production topology (Supabase direct :5432 +
# pooled :6543) behind three consecutive pooler-teardown waves
# (#1972 → #2015 → #2084) that CI could never reproduce.
# test/e2e/pgbouncer-teardown.test.ts uses a DEDICATED database
# (gbrain_pgbouncer) on postgres-1 so it never races shard 1's
# TRUNCATE-based fixtures; pgbouncer's wildcard [databases] section
# forwards any dbname to DB_HOST.
pgbouncer:
image: edoburu/pgbouncer:latest
environment:
DB_HOST: postgres-1
DB_PORT: "5432"
DB_USER: postgres
DB_PASSWORD: postgres
POOL_MODE: transaction
# plain (CI-only): pg16 stores SCRAM verifiers, and pgbouncer can only
# answer the server's SCRAM challenge when its userlist holds the
# PLAINTEXT password — an md5-hashed userlist fails with
# "server login failed: wrong password type".
AUTH_TYPE: plain
MAX_CLIENT_CONN: "200"
DEFAULT_POOL_SIZE: "10"
# gbrain's client sets statement_timeout + idle_in_transaction_session_timeout
# as startup parameters (db.ts buildConnectionParams); the Supabase pooler
# whitelists them, so this pooler must too or every connection is refused
# before the teardown path is even reached.
IGNORE_STARTUP_PARAMETERS: extra_float_digits,statement_timeout,idle_in_transaction_session_timeout,search_path
ports:
- "${GBRAIN_CI_PGBOUNCER_PORT:-6543}:5432"
depends_on:
postgres-1:
condition: service_healthy
runner:
image: oven/bun:1
working_dir: /app
@@ -97,6 +131,8 @@ services:
condition: service_healthy
postgres-4:
condition: service_healthy
pgbouncer:
condition: service_started
# No global DATABASE_URL — scripts/ci-local.sh sets per-shard URL via -e.
# Unit phase explicitly unsets DATABASE_URL so test/e2e/* gracefully skip.
volumes:
+10 -8
View File
@@ -12,15 +12,17 @@ Before shipping (/ship) or reviewing (/review), always run the full test suite.
Two equivalent paths:
**Path A — local CI gate (recommended, v0.23.1+):**
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit
tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a
fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to
what nightly Tier 1 catches. Spins up + tears down postgres automatically via
`docker-compose.ci.yml`. Override the host port with
`GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host),
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
containers plus a transaction-mode PgBouncer service (unit phase keeps
`DATABASE_URL` unset; `--no-shard` for the legacy sequential flow). Stronger
than PR CI's 2-file Tier 1 set; closer to what nightly Tier 1 catches. Spins
up + tears down postgres automatically via `docker-compose.ci.yml`. Override
the host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides.
- `bun run ci:local:diff` runs only the E2E files matched by the diff selector
(`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or
schema/skills/package.json changes. Fast iteration during a focused branch.
(`scripts/select-e2e.ts`), falling back to ALL E2E files on unmapped src/
paths or schema/skills/package.json changes. Fast iteration during a focused
branch.
**Path B — manual lifecycle (still supported):**
- `bun test` — unit tests (no database required)
+10 -2
View File
@@ -15,7 +15,7 @@ Seven test command tiers, each with a clear scope:
| `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. |
| `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. |
| `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. |
| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. |
| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. |
### CI vs local: intentionally divergent file sets
@@ -126,6 +126,12 @@ Unit tests and what they cover:
- `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override).
- `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling).
- `test/cli-should-force-exit.test.ts``shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command).
- `test/cli-exit-verdict-pin.test.ts`#2084 structural class pin: greps `src/` so the NEXT raw `process.exitCode =` write fails CI (a raw write bypasses the gbrain-owned verdict channel and gets silently zeroed by the deliberate flush-exit — the bug that made doctor's FAIL path exit 0). Runtime variants live in `test/cli-finish-teardown.test.ts`; this is the review-time guard.
- `test/cli-pipe-truncation.test.ts` — real-CLI pipe completeness (the #1959 incident class), implementation-agnostic: the actual CLI run the way agents run it (piped stdout) produces complete, parseable, byte-stable `--tools-json` output and exits deliberately, well under the teardown backstop. Synthetic flush-mechanism coverage stays in `test/flush-then-exit-harness.test.ts`.
- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join.
- `test/watch-command.test.ts``gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin).
- `test/watch-sigint.serial.test.ts``gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`).
- `test/cli-format-volunteer.test.ts``formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary.
- `test/config.test.ts` — config redaction.
- `test/files.test.ts` — MIME/hash.
- `test/import-file.test.ts` — import pipeline.
@@ -133,7 +139,7 @@ Unit tests and what they cover:
- `test/file-migration.test.ts` — file migration.
- `test/file-resolver.test.ts` — file resolution.
- `test/import-resume.test.ts` — import checkpoints.
- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion.
- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion; v117 `context_volunteer_events` (named + idempotent entry, documented columns + both source-scoped indexes after `initSchema`, insert + 90-day `purgeStaleVolunteerEvents` round-trip).
- `test/bootstrap.test.ts` — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on a simulated legacy brain, fresh-install regression guard, legacy `links` shape coverage.
- `test/schema-bootstrap-coverage.test.ts` — CI guard. `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in `PGLITE_SCHEMA_SQL`; the test fails loudly if `applyForwardReferenceBootstrap` skips one (extend both arrays when adding a column-with-index to the embedded schema blob). Also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`) and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies — catching the column-only forward-reference class (e.g. `sources.archived`, `oauth_clients.source_id`) that a CREATE INDEX parser alone can't see. `parseBaseTableColumns` strips SQL line + block comments before identifying column names so commented-out lines don't hide adjacent columns.
- `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each, snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via `PGLITE_SCHEMA_SQL` or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skips without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`.
@@ -223,6 +229,8 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D
- `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required).
- `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use.
- `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner.
- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972#2015#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures.
- `test/e2e/volunteer-context-postgres.test.ts``volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated.
- `test/e2e/openclaw-reference-compat.test.ts``check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape.
- `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `<fork>/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory.
- `test/e2e/search-exclude.test.ts``test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths.
File diff suppressed because one or more lines are too long
+79
View File
@@ -0,0 +1,79 @@
# Push-based context (#2095, v0.42.43.0)
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
contributed anything. Push-based context inverts that — the brain volunteers
relevant pages from the recent conversation, confidence-gated so push noise
never becomes worse than pull silence.
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
| Channel | Surface | When to use |
|---|---|---|
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
## How it decides
1. **Extract** entities across the last N turns (capitalized runs, `@handles`),
merged with recency / frequency / user-role salience. Assistant-introduced
entities and "what did she invest in?" follow-ups whose antecedent was named
in the window now resolve.
2. **Resolve** through the alias table, exact titles, and slug suffixes — each
arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6,
+0.05 when mentioned in ≥2 turns or the newest turn.
3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an
explicit lower gate), suppress pages already surfaced (slug-presence only),
cap at 3 pages (hard cap 5).
## CLI
```bash
# one-shot: pipe recent turns (oldest → newest)
printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \
| gbrain volunteer-context
# streaming: volunteered pages print as the transcript flows
some-transcript-feed | gbrain watch --json
# the feedback loop: how often were volunteered pages actually opened?
gbrain volunteer-context --stats
```
Stats are **approximate** by design: "used" means `pages.last_retrieved_at >
volunteered_at` — the 5-minute last-retrieved throttle causes false negatives
and unrelated reads of the same page cause false positives. Use the per-arm
precision to tune `min_confidence`, not as an exact metric.
**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its
connection for the whole session — a concurrent `gbrain serve` or any write
path blocks until watch exits. On a PGLite brain, run watch in bursts (piped
input exits at EOF) or use the ambient reflex channel instead, which routes
through a running serve's resolve socket rather than taking the lock. Routing
watch through that same socket is a filed follow-up (TODOS.md). Postgres
brains are unaffected.
## Config
| Key | Default | What it does |
|---|---|---|
| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) |
| `retrieval_reflex` | true | the ambient channel's master switch |
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch`
(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch);
on the op only: `prior_context` (text whose already-surfaced slugs are suppressed),
`session_id` / `turn` attribution params (watch stamps its own per-session id and
turn numbers in the feedback log), and `days` to size the `--stats` window.
## Storage + privacy
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
arm, confidence, channel, optional session/turn — the rationale is a
deterministic template string, never raw conversation text. Event writes are
best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal,
not an audit trail. Rows are pruned after 90 days by the dream cycle's purge
phase. Synopses always strip the takes/facts fences — the same strip `get_page`
applies to untrusted callers, applied unconditionally here so private fence rows
never reach a prompt regardless of caller trust.
+111
View File
@@ -0,0 +1,111 @@
# Spend controls
GBrain's embedding-spend gates in one place: every gate, its config key, default,
whether it blocks or just informs, how to widen or disable it, and how the
`spend.posture` switch governs all of them.
The orienting idea: **GBrain itself is rounding error; the spend that matters is
downstream embedding.** These gates exist so a routine sync or enrich can't run up
an unexpected embedding bill, while never wedging an unattended cron.
## `spend.posture` — one switch for "cost is not my constraint"
```bash
gbrain config set spend.posture tokenmax # all cost gates become informational
gbrain config set spend.posture gated # default — gates enforce
```
| Value | Effect |
|-------|--------|
| `gated` (default) | Every cost gate enforces its limit as documented below. |
| `tokenmax` | Every cost gate prints its estimate and **proceeds** — informational only. Spend is still recorded to the ledger; posture removes the *ceiling*, not the *accounting*. |
`spend.posture` is deliberately separate from `search.mode=tokenmax` (which governs
retrieval payload size, not embedding spend). When a gate fires and
`search.mode=tokenmax` but `spend.posture` is unset, the gate prints a one-line hint
pointing at this switch.
**Precedence:** an explicit per-call cap (`--max-usd N`, `--max-cost N`) always wins
over posture. `tokenmax` only governs the default/absent case — it never overrides a
number you typed on the command line.
## Off switches (`off` / `unlimited` / `none`)
The USD-limit knobs accept `off`, `unlimited`, or `none` (case-insensitive) to mean
"no limit" — no more setting sentinel values like `100000`.
- `0` is **not** "off". On `sync.cost_gate_min_usd`, `0` means "block on any nonzero
spend" (a real choice). On the backfill caps, `0` falls back to the default.
- Internally "no limit" is the string `unlimited` in any printed/JSON output and "no
cap" inside the budget tracker — never a raw `Infinity` (which would serialize to
`null` in ledger rows).
## The gates
| Gate | Config key | Default | Blocks? | Off switch | tokenmax |
|------|-----------|---------|---------|-----------|----------|
| Sync inline-embed cost gate | `sync.cost_gate_min_usd` | `0.50` | TTY prompt / non-TTY auto-defer | `off` (or `0` = block-on-any) | informational |
| Backfill 24h per-source spend cap | `embed.backfill_max_usd_per_source_24h` | `25` | refuses submission | `off` (`0` → default) | bypassed (still ledgered) |
| Backfill per-job budget | `embed.backfill_max_usd` | `10` | caps the job's tracker | `off` (`0` → default) | uncapped (still ledgered) |
| Backfill cooldown | `embed.backfill_cooldown_min` | `10` | skips re-submission inside window | — (latency knob, not spend) | **not** bypassed |
| `reindex-code` cost gate | — (preview before re-embed) | — | TTY prompt / non-TTY refuse + exit 2 | `--max-cost off` | informational |
| `enrich` / `onboard --auto` | `--max-usd` (per-call) | — | refuse without a cap (non-TTY) | `--max-usd off` | runs uncapped (still ledgered) |
### Sync inline-embed cost gate
Fires only when sync embeds **inline** (federated_v2 off, or `--serial` without
`--no-embed`). Under federated_v2 + parallel, embedding is deferred to capped backfill
jobs and the gate is informational. The estimate prices the **delta** — the files this
sync will actually import (fetched-first, so it sees commits the run is about to pull) —
not the whole tree. A busy brain with a dirty working tree but caught-up commits
estimates `$0`, because an attached-HEAD sync imports only the committed diff.
Behavior above the floor:
- **TTY:** prompts `[y/N]`.
- **Non-interactive (cron/agent):** **auto-defers** embeds to capped backfill jobs and
exits 0 — it never wedges the pipeline. The backlog drains via the jobs worker or
`gbrain embed --stale`. Pass `--yes` to embed inline instead.
Output format splits on the explicit `--json` flag: `--json` emits a structured
envelope; otherwise human text. Every gate message carries paste-ready knobs.
`--full` re-embeds the stale backlog inline (full sync sweeps it), so a `--full`
estimate is `delta + stale backlog`, labeled as such.
### Estimate labels
- `~N tokens (delta: changed files since last sync)` — the precise estimate.
- `<=N tokens (full-tree ceiling for K source(s): <reasons> …)` — a conservative
over-count used only when a precise delta can't be computed: a first sync, a chunker
version drift (forces a full re-chunk), or git being unavailable. Unchanged files
still skip via `content_hash` at execution, so the ceiling over-states real spend.
## Notes & limits
- **Pre-pull window:** the gate fetches before estimating, so it prices what the run
will pull. If a fetch fails (offline), it estimates against local HEAD and labels the
result; the bounded residual is priced on the next run.
- **Single-source `gbrain sync`** carries the same gate as `sync --all` (it previously
embedded inline with no preview).
- **Recovery under parallel:** `--skip-failed` / `--retry-failed` work under parallel
sync (the failure ledger is per-source and lock-serialized) — you no longer have to
drop to `--serial`, which is what used to arm the inline gate.
## Escape hatches at a glance
```bash
# Never gate this brain on cost:
gbrain config set spend.posture tokenmax
# Widen the sync inline floor to $5:
gbrain config set sync.cost_gate_min_usd 5
# Disable the sync inline floor entirely:
gbrain config set sync.cost_gate_min_usd off
# Lift the backfill 24h spend cap:
gbrain config set embed.backfill_max_usd_per_source_24h off
# Run enrich uncapped non-interactively:
gbrain enrich --max-usd off # or: gbrain config set spend.posture tokenmax
```
+1 -1
View File
@@ -86,7 +86,7 @@ Save this token. You'll need it for the AlphaClaw setup.
AlphaClaw is the setup harness that manages OpenClaw deployment.
1. Go to [alphaclaw.com](https://alphaclaw.com)
1. Go to [alphaclaw.md](https://alphaclaw.md)
2. Enter your **workspace repo** (not the brain repo): `your-org/myagent`
3. Select "Use existing" if the repo already exists
4. Enter your GitHub PAT from Step 2
+92 -3
View File
@@ -117,8 +117,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont
## Before shipping
Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks,
unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a
fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the
guards + typecheck, then 4-shard parallel unit + E2E against four pgvector
containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL`
unset) and tears down. Use `bun run ci:local:diff` for the
diff-aware subset during fast iteration on a focused branch. Requires Docker
(Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`).
@@ -186,7 +187,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -245,6 +246,8 @@ detail on demand.)
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
| search modes / cost knobs | `docs/guides/search-modes.md` |
| embedding spend gates / cost gate / `spend.posture` / off switches | `docs/operations/spend-controls.md` |
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
@@ -3423,6 +3426,92 @@ the bundled resolver lives at [`skills/RESOLVER.md`](../../skills/RESOLVER.md).
---
## docs/guides/push-context.md
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md
# Push-based context (#2095, v0.42.43.0)
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
contributed anything. Push-based context inverts that — the brain volunteers
relevant pages from the recent conversation, confidence-gated so push noise
never becomes worse than pull silence.
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
| Channel | Surface | When to use |
|---|---|---|
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
## How it decides
1. **Extract** entities across the last N turns (capitalized runs, `@handles`),
merged with recency / frequency / user-role salience. Assistant-introduced
entities and "what did she invest in?" follow-ups whose antecedent was named
in the window now resolve.
2. **Resolve** through the alias table, exact titles, and slug suffixes — each
arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6,
+0.05 when mentioned in ≥2 turns or the newest turn.
3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an
explicit lower gate), suppress pages already surfaced (slug-presence only),
cap at 3 pages (hard cap 5).
## CLI
```bash
# one-shot: pipe recent turns (oldest → newest)
printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \
| gbrain volunteer-context
# streaming: volunteered pages print as the transcript flows
some-transcript-feed | gbrain watch --json
# the feedback loop: how often were volunteered pages actually opened?
gbrain volunteer-context --stats
```
Stats are **approximate** by design: "used" means `pages.last_retrieved_at >
volunteered_at` — the 5-minute last-retrieved throttle causes false negatives
and unrelated reads of the same page cause false positives. Use the per-arm
precision to tune `min_confidence`, not as an exact metric.
**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its
connection for the whole session — a concurrent `gbrain serve` or any write
path blocks until watch exits. On a PGLite brain, run watch in bursts (piped
input exits at EOF) or use the ambient reflex channel instead, which routes
through a running serve's resolve socket rather than taking the lock. Routing
watch through that same socket is a filed follow-up (TODOS.md). Postgres
brains are unaffected.
## Config
| Key | Default | What it does |
|---|---|---|
| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) |
| `retrieval_reflex` | true | the ambient channel's master switch |
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch`
(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch);
on the op only: `prior_context` (text whose already-surfaced slugs are suppressed),
`session_id` / `turn` attribution params (watch stamps its own per-session id and
turn numbers in the feedback log), and `days` to size the `--stats` window.
## Storage + privacy
Volunteered pages log to `context_volunteer_events` (migration v117): slug,
arm, confidence, channel, optional session/turn — the rationale is a
deterministic template string, never raw conversation text. Event writes are
best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal,
not an audit trail. Rows are pruned after 90 days by the dream cycle's purge
phase. Synopses always strip the takes/facts fences — the same strip `get_page`
applies to untrusted callers, applied unconditionally here so private fence rows
never reach a prompt regardless of caller trust.
---
## docs/mcp/DEPLOY.md
Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md
+1
View File
@@ -25,6 +25,7 @@ Repo: https://github.com/garrytan/gbrain
- [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist.
- [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery.
- [docs/guides/scaling-skills.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/scaling-skills.md): Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.
- [docs/guides/push-context.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md): Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.
- [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment.
## AI providers
+1 -1
View File
@@ -143,5 +143,5 @@
"bun": ">=1.3.10"
},
"license": "MIT",
"version": "0.42.42.0"
"version": "0.42.45.0"
}
+12 -2
View File
@@ -196,7 +196,10 @@ SELECTED=$(bun run scripts/select-e2e.ts)
if [ -z "$SELECTED" ]; then
echo "[runner] selector emitted nothing (doc-only diff); skipping E2E."
else
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test echo "$SELECTED" | xargs bash scripts/run-e2e.sh
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
echo "$SELECTED" | xargs bash scripts/run-e2e.sh
fi'
else
RUN_PHASES_CMD='echo "[runner] guards + typecheck"
@@ -208,7 +211,10 @@ bun run typecheck
echo "[runner] unit (unsharded, DATABASE_URL unset)"
env -u DATABASE_URL bash scripts/run-unit-shard.sh
echo "[runner] e2e (unsharded)"
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test bash scripts/run-e2e.sh'
DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \
bash scripts/run-e2e.sh'
fi
else
# Tier 1 sharded path. Each shard runs unit+E2E sequentially against its
@@ -257,10 +263,14 @@ printf '%s\\n' 1 2 3 4 | xargs -P4 -I{} sh -c '
if [ -s /tmp/e2e-selected.txt ]; then
SHARD=\${shard}/4 \\
DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\
xargs -a /tmp/e2e-selected.txt bash scripts/run-e2e.sh >> \$log 2>&1
else
SHARD=\${shard}/4 \\
DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\
GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\
GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\
bash scripts/run-e2e.sh >> \$log 2>&1
fi
e2e_exit=\$?
+6
View File
@@ -151,6 +151,12 @@ export const SECTIONS: DocSection[] = [
"Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.",
path: "docs/guides/scaling-skills.md",
},
{
title: "docs/guides/push-context.md",
description:
"Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.",
path: "docs/guides/push-context.md",
},
{
title: "docs/mcp/DEPLOY.md",
description: "MCP server deployment.",
+56 -12
View File
@@ -24,6 +24,7 @@ import type { GBrainConfig } from './core/config.ts';
import type { AIGatewayConfig } from './core/ai/types.ts';
import type { BrainEngine } from './core/engine.ts';
import { operations, OperationError } from './core/operations.ts';
import { formatVolunteeredPage } from './core/context/volunteer.ts';
import type { Operation, OperationContext } from './core/operations.ts';
import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts';
import { serializeMarkdown } from './core/markdown.ts';
@@ -43,7 +44,7 @@ for (const op of operations) {
}
// CLI-only commands that bypass the operation layer
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade']);
const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'watch']);
// CLI-only commands whose handlers print their own --help text. These are
// excluded from the generic short-circuit so detailed per-command and
// per-subcommand usage stays reachable.
@@ -67,6 +68,8 @@ const CLI_ONLY_SELF_HELP = new Set([
'capture',
// v0.42 self-upgrade ships its own usage (flags + the agent-skill story).
'self-upgrade',
// v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol).
'watch',
// v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was
// unreachable via help because the dispatcher's generic CLI-only
// short-circuit fired before runSync could print its own usage block.
@@ -802,8 +805,27 @@ async function makeContext(engine: BrainEngine, params: Record<string, unknown>)
};
}
function formatResult(opName: string, result: unknown): string {
// Exported for tests (same import-safety contract as cliAliases/printOpHelp).
export function formatResult(opName: string, result: unknown): string {
switch (opName) {
case 'volunteer_context': {
const r = result as any;
// Stats mode (the feedback loop).
if (r && r.approximate === true && Array.isArray(r.by_arm)) {
const lines = [
`volunteered-context precision — last ${r.days} day(s) (${r.note})`,
`total: ${r.total_volunteered} volunteered, ${r.total_used} used`,
];
for (const a of r.by_arm) {
lines.push(` ${a.match_arm}/${a.channel}: ${a.used}/${a.volunteered} used (precision ${a.precision})`);
}
if (!r.by_arm.length) lines.push(' (no volunteer events in the window)');
return lines.join('\n') + '\n';
}
const pages = (r?.pages ?? []) as any[];
if (!pages.length) return 'Nothing volunteered (no entity cleared the confidence gate).\n';
return pages.map((p) => formatVolunteeredPage(p)).join('\n') + '\n';
}
case 'get_page': {
const r = result as any;
if (r.error === 'ambiguous_slug') {
@@ -925,6 +947,9 @@ function formatResult(opName: string, result: unknown): string {
const THIN_CLIENT_REFUSED_COMMANDS = new Set([
'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations',
'repair-jsonb', 'orphans', 'integrity', 'serve',
// v0.43 (#2095): watch streams against a LOCAL engine; thin clients get
// the volunteer_context MCP op instead.
'watch',
// v0.31.1 (CDX-2 op coverage matrix): more local-only commands
'dream', 'transcripts', 'storage',
// v0.31.1 CDX-2 audit: takes/sources have multiple subcommands; some
@@ -1156,11 +1181,14 @@ async function handleCliOnly(command: string, args: string[]) {
}
if (command === 'friction') {
const { runFriction } = await import('./commands/friction.ts');
process.exit(runFriction(args));
// #2084 inner-exit sweep: verdict + return so teardown + the flush seam run.
setCliExitVerdict(runFriction(args));
return;
}
if (command === 'claw-test') {
const { runClawTest } = await import('./commands/claw-test.ts');
process.exit(await runClawTest(args));
setCliExitVerdict(await runClawTest(args));
return;
}
if (command === 'report') {
const { runReport } = await import('./commands/report.ts');
@@ -1269,7 +1297,7 @@ async function handleCliOnly(command: string, args: string[]) {
execSync(`bash "${scriptPath}"`, { stdio: 'inherit', env: { ...process.env } });
} catch (e: any) {
// Non-zero exit = some tests failed (exit code = failure count)
process.exit(e.status ?? 1);
setCliExitVerdict(e.status ?? 1);
}
return;
}
@@ -1318,7 +1346,8 @@ async function handleCliOnly(command: string, args: string[]) {
// The handler self-configures the AI gateway from loadConfig() + process.env.
if (command === 'eval' && args[0] === 'cross-modal') {
const { runEvalCrossModal } = await import('./commands/eval-cross-modal.ts');
process.exit(await runEvalCrossModal(args.slice(1)));
setCliExitVerdict(await runEvalCrossModal(args.slice(1)));
return;
}
// v0.32 EXP-5 (codex review #10): `eval takes-quality replay <receipt>`
@@ -1329,7 +1358,8 @@ async function handleCliOnly(command: string, args: string[]) {
// engine-required path below.
if (command === 'eval' && args[0] === 'takes-quality' && args[1] === 'replay') {
const { runReplayNoBrain } = await import('./commands/eval-takes-quality.ts');
process.exit(await runReplayNoBrain(args.slice(2)));
setCliExitVerdict(await runReplayNoBrain(args.slice(2)));
return;
}
// v0.28.8: longmemeval brings its own in-memory PGLite. Bypassing
@@ -1361,7 +1391,8 @@ async function handleCliOnly(command: string, args: string[]) {
// gate runs on machines with no `~/.gbrain/config.json`.
if (command === 'eval' && args[0] === 'conversation-parser') {
const { runEvalConversationParser } = await import('./commands/eval-conversation-parser.ts');
process.exit(await runEvalConversationParser(args.slice(1)));
setCliExitVerdict(await runEvalConversationParser(args.slice(1)));
return;
}
// v0.41.13.0: `gbrain conversation-parser list-builtins | validate
@@ -1389,7 +1420,8 @@ async function handleCliOnly(command: string, args: string[]) {
const cfgPre = loadConfig();
if (isThinClient(cfgPre)) {
const { runEvalWhoknows } = await import('./commands/eval-whoknows.ts');
process.exit(await runEvalWhoknows(null, args.slice(1)));
setCliExitVerdict(await runEvalWhoknows(null, args.slice(1)));
return;
}
}
@@ -1403,7 +1435,8 @@ async function handleCliOnly(command: string, args: string[]) {
if (cfgPre && isThinClient(cfgPre)) {
const { runStatus } = await import('./commands/status.ts');
const result = await runStatus(null, args);
process.exit(result.exitCode);
setCliExitVerdict(result.exitCode);
return;
}
}
@@ -1716,8 +1749,8 @@ async function handleCliOnly(command: string, args: string[]) {
case 'status': {
const { runStatus } = await import('./commands/status.ts');
const result = await runStatus(engine, args);
process.exit(result.exitCode);
// eslint-disable-next-line no-unreachable
// #2084 inner-exit sweep: a mid-switch exit skips the finally teardown.
setCliExitVerdict(result.exitCode);
break;
}
// v0.38 — Capture: single human-facing entrypoint for ingestion.
@@ -1887,6 +1920,15 @@ async function handleCliOnly(command: string, args: string[]) {
await runQuarantine(engine, args);
break;
}
case 'watch': {
// v0.43 (#2095): push-based context transport. Blocks in the stdin
// iteration (interactive stays alive; piped exits at EOF), then the
// finally below runs finishCliTeardown (volunteer events drain with
// every other sink) and the import.meta.main seam flush-exits.
const { runWatch } = await import('./commands/watch.ts');
await runWatch(engine, args);
break;
}
case 'storage': {
const { runStorage } = await import('./commands/storage.ts');
await runStorage(engine, args);
@@ -2267,6 +2309,8 @@ ADMIN
--public-url URL Public issuer URL (required behind proxy/tunnel)
connect <mcp-url> --token <t> Wire Claude Code to a remote gbrain (bearer token)
[--install] [--json] Print the paste-ready command, or --install to run it
watch [--json] Push-based context: pipe conversation turns in,
volunteered brain pages stream out (#2095)
call <tool> '<json>' Raw tool invocation
version Version info
--tools-json Tool discovery (JSON)
+14
View File
@@ -181,6 +181,20 @@ export async function runConfig(engine: BrainEngine, args: string[]) {
const coverageOverride =
args.includes('--coverage-override') || args.includes('--yes');
// v0.42.42.0 (#2139): validate spend.posture at set time so a typo
// ('tokenMax', 'max') doesn't silently fall back to gated.
if (key === 'spend.posture') {
const { isValidSpendPosture } = await import('../core/spend-posture.ts');
if (!isValidSpendPosture(value)) {
console.error(
`[config] spend.posture must be 'gated' or 'tokenmax' (got '${value}').\n` +
`[config] gbrain config set spend.posture tokenmax # cost gates become informational\n` +
`[config] gbrain config set spend.posture gated # default — gates enforce`,
);
process.exit(1);
}
}
if (key === 'embedding_columns') {
try {
const parsed = JSON.parse(value);
+36 -7
View File
@@ -508,8 +508,13 @@ export async function runEnrichCore(
// One tracker reference for both the run and the post-hoc overage check.
// External tracker (cycle phase): used as-is, no withBudgetTracker wrap (that
// would REPLACE not stack). Internal: capped at maxCostUsd ?? DEFAULT.
// v0.42.42.0 (#2139): Infinity = explicit "uncapped" (off / tokenmax) → pass
// `undefined` so BudgetTracker runs without a ceiling (NOT raw Infinity, which
// would serialize to null in audit rows). undefined-when-unset still → DEFAULT.
const resolvedCap =
opts.maxCostUsd === Infinity ? undefined : (opts.maxCostUsd ?? DEFAULT_MAX_COST_USD);
const tracker = opts.budgetTracker ?? new BudgetTracker({
maxCostUsd: opts.maxCostUsd ?? DEFAULT_MAX_COST_USD,
maxCostUsd: resolvedCap,
label: `enrich:${sourceId}`,
});
try {
@@ -622,8 +627,15 @@ export function parseArgs(args: string[]): ParsedArgs {
continue;
}
if (a === '--max-usd' || a === '--max-cost-usd') {
const n = parseFloat(args[++i] ?? '');
if (Number.isFinite(n) && n > 0) out.maxCostUsd = n;
const raw = args[++i] ?? '';
// v0.42.42.0 (#2139): off/unlimited/none → run uncapped (Infinity sentinel;
// mapped to "no BudgetTracker ceiling" in runEnrichCore). Spend still ledgered.
if (['off', 'unlimited', 'none'].includes(raw.trim().toLowerCase())) {
out.maxCostUsd = Infinity;
} else {
const n = parseFloat(raw);
if (Number.isFinite(n) && n > 0) out.maxCostUsd = n;
}
continue;
}
if (a === '--min-context') {
@@ -801,9 +813,24 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
process.exit(1);
}
// v0.42.42.0 (#2139, D15A): enrich runs UNCAPPED when the operator says cost
// isn't the constraint — either explicit `--max-usd off` (parsed to Infinity)
// or `spend.posture=tokenmax` with no per-call cap. Uncapped → the missing-cap
// refusals lift AND runEnrichCore passes no ceiling to the BudgetTracker (spend
// still ledgered; posture removes the ceiling, not the accounting). An explicit
// finite --max-usd always wins (precedence: per-call > posture).
const explicitOff = parsed.maxCostUsd === Infinity;
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
const posture = parsed.dryRun ? 'gated' : await resolveSpendPosture(engine);
const uncapped =
!parsed.dryRun && (explicitOff || (parsed.maxCostUsd === undefined && posture === 'tokenmax'));
if (uncapped) {
console.error(`${explicitOff ? '--max-usd off' : 'spend.posture=tokenmax'}: running uncapped, spend ledgered. docs: docs/operations/spend-controls.md`);
}
// Non-TTY execute without --max-usd or --yes is refused (cost guardrail).
if (!parsed.dryRun && parsed.maxCostUsd === undefined && !parsed.yes && !process.stdout.isTTY) {
console.error('Refusing to spend without a cap in a non-interactive context. Pass --max-usd <FLOAT> or --yes.');
if (!parsed.dryRun && parsed.maxCostUsd === undefined && !parsed.yes && !process.stdout.isTTY && !uncapped) {
console.error('Refusing to spend without a cap in a non-interactive context. Pass --max-usd <FLOAT> (or `off`), --yes, or set spend.posture=tokenmax.');
process.exit(1);
}
@@ -812,7 +839,7 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
: (await listSources(engine)).map((s) => s.id);
// Dry-run cost preview (TTY) before spending.
if (!parsed.dryRun && process.stdout.isTTY && !parsed.yes && parsed.maxCostUsd === undefined) {
if (!parsed.dryRun && process.stdout.isTTY && !parsed.yes && parsed.maxCostUsd === undefined && !uncapped) {
const limit = parsed.limit ?? DEFAULT_LIMIT;
const est = (limit * sourceIds.length * COST_ESTIMATE_PER_PAGE_USD).toFixed(2);
console.error(`About to enrich up to ${limit} page(s) per source across ${sourceIds.length} source(s), est. ~$${est}. Re-run with --max-usd or --yes to confirm.`);
@@ -834,7 +861,9 @@ export async function runEnrich(engine: BrainEngine, args: string[]): Promise<vo
limit: parsed.limit,
workers: parsed.workers,
model: parsed.model,
maxCostUsd: parsed.maxCostUsd,
// uncapped (off / tokenmax) → Infinity sentinel; runEnrichCore maps it
// to "no BudgetTracker ceiling".
maxCostUsd: uncapped ? Infinity : parsed.maxCostUsd,
minContextChars: parsed.minContextChars,
thinThreshold: parsed.thinThreshold,
reenrichAfterMs: parsed.reenrichAfterMs,
+23 -6
View File
@@ -45,6 +45,13 @@ export async function runOnboard(engine: BrainEngine, args: string[]): Promise<v
// No-op without a pack_upgrade_available finding.
const explain = args.includes('--explain');
const targetScore = parseInt10(args, '--target-score') ?? 90;
// v0.42.42.0 (#2139): `--max-usd off`/`unlimited`/`none` → run uncapped. maxUsd
// stays undefined, which runRemediation already treats as no ceiling (skips the
// est-cost refusal + BudgetTracker runs uncapped); `maxUsdOff` lifts the
// missing-cap refusal below. Spend is still ledgered.
const maxUsdIdx = args.indexOf('--max-usd');
const maxUsdVal = maxUsdIdx >= 0 ? (args[maxUsdIdx + 1] ?? '').trim().toLowerCase() : '';
const maxUsdOff = ['off', 'unlimited', 'none'].includes(maxUsdVal);
const maxUsdRaw = parseFloat10(args, '--max-usd');
const maxUsd = maxUsdRaw === null ? undefined : maxUsdRaw;
@@ -91,13 +98,23 @@ export async function runOnboard(engine: BrainEngine, args: string[]): Promise<v
}
// --auto refuses without --max-usd (cron-safety per A12 + A20).
// v0.42.42.0 (#2139, D15A): spend.posture=tokenmax lifts the refusal —
// --auto runs UNCAPPED (maxUsd stays undefined), spend still ledgered by the
// remediation budget tracker. An explicit --max-usd always wins.
if (auto && maxUsd === undefined) {
process.stderr.write(
`gbrain onboard --auto refuses without --max-usd N.\n` +
`Set a cap to avoid surprise spend:\n` +
` gbrain onboard --auto --max-usd 5\n`,
);
process.exit(2);
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
const tokenmax = (await resolveSpendPosture(engine)) === 'tokenmax';
if (maxUsdOff || tokenmax) {
process.stderr.write(`${maxUsdOff ? '--max-usd off' : 'spend.posture=tokenmax'}: onboard --auto running uncapped, spend ledgered. docs: docs/operations/spend-controls.md\n`);
} else {
process.stderr.write(
`gbrain onboard --auto refuses without --max-usd N.\n` +
`Set a cap to avoid surprise spend:\n` +
` gbrain onboard --auto --max-usd 5\n` +
`Or pass --max-usd off / set spend.posture=tokenmax to run uncapped. docs: docs/operations/spend-controls.md\n`,
);
process.exit(2);
}
}
// Build the plan: T4 checks supply extra remediations on top of T3's
+41 -15
View File
@@ -444,14 +444,24 @@ export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Pr
}
// F3: --max-cost / --max-cost-usd both accepted for symmetry with brainstorm.
// v0.42.42.0 (#2139): `off`/`unlimited`/`none` → no runtime cap AND an explicit
// "cost isn't the constraint" decision that proceeds past the confirmation gate
// (like --yes). Numeric must be positive; `0`/garbage is rejected.
let maxCostUsd: number | undefined;
let maxCostOff = false;
for (const flag of ['--max-cost', '--max-cost-usd']) {
const idx = args.indexOf(flag);
if (idx >= 0) {
const v = args[idx + 1];
const t = (v ?? '').trim().toLowerCase();
if (['off', 'unlimited', 'none'].includes(t)) {
maxCostUsd = undefined; // no runtime cap (reindex skips the tracker when unset)
maxCostOff = true;
break;
}
const n = v ? parseFloat(v) : NaN;
if (!Number.isFinite(n) || n <= 0) {
console.error(`gbrain reindex --code: ${flag} requires a positive number in USD (got ${v ?? '(missing)'})`);
console.error(`gbrain reindex --code: ${flag} requires a positive number in USD, or off/unlimited (got ${v ?? '(missing)'})`);
process.exit(2);
}
maxCostUsd = n;
@@ -493,20 +503,36 @@ export async function runReindexCodeCli(engine: BrainEngine, args: string[]): Pr
}
if (!yes) {
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
if (!isTTY || json) {
// Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits
// on --json now (human refusal on stderr otherwise) — #1784.
const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() });
if (refusal.stdout) console.log(refusal.stdout);
if (refusal.stderr) console.error(refusal.stderr);
process.exit(2);
}
console.log(previewMsg);
const answer = await promptYesNo('Proceed? [y/N] ');
if (!answer) {
console.log('Cancelled.');
return;
// v0.42.42.0 (#2139): spend.posture=tokenmax makes the gate informational
// — print the estimate and proceed (the operator declared cost isn't the
// constraint). The spend is still ledgered by the runtime BudgetTracker.
const { resolveSpendPosture } = await import('../core/spend-posture.ts');
const posture = await resolveSpendPosture(engine);
// An explicit `--max-cost off` is the same "cost isn't the constraint"
// signal as spend.posture=tokenmax — proceed past the confirmation gate.
if (posture === 'tokenmax' || maxCostOff) {
const gate = maxCostOff ? 'max_cost_off' : 'posture_tokenmax';
if (json) {
console.log(JSON.stringify({ status: 'proceeding', gate, codePages: preview.totalPages, totalTokens: preview.totalTokens, costUsd, model: getEmbeddingModelName() }));
} else {
console.log(`${previewMsg} ${maxCostOff ? '--max-cost off' : 'spend.posture=tokenmax'}: proceeding (informational). docs: docs/operations/spend-controls.md`);
}
} else {
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
if (!isTTY || json) {
// Guardrail unchanged: refuse + exit 2, no spend. Only the FORMAT splits
// on --json now (human refusal on stderr otherwise) — #1784.
const refusal = buildCostRefusal({ json, previewMsg, preview, costUsd, model: getEmbeddingModelName() });
if (refusal.stdout) console.log(refusal.stdout);
if (refusal.stderr) console.error(refusal.stderr);
process.exit(2);
}
console.log(previewMsg);
const answer = await promptYesNo('Proceed? [y/N] ');
if (!answer) {
console.log('Cancelled.');
return;
}
}
}
}
+530 -209
View File
@@ -7,12 +7,11 @@ import { importFile } from '../core/import-file.ts';
import { collectSyncableFiles } from './import.ts';
import { createInterface } from 'readline';
import {
buildSyncManifest,
isSyncable,
unsyncableReason,
resolveSlugForPath,
unacknowledgedSyncFailures,
acknowledgeSyncFailures,
acknowledgeFailures,
loadSyncFailures,
formatCodeBreakdown,
applySyncFailureGate,
@@ -20,6 +19,16 @@ import {
resolveAutoSkipThreshold,
DEFAULT_SOURCE_ID,
} from '../core/sync.ts';
import {
computeSyncDelta,
buildDetachedWorkingTreeManifest,
} from '../core/sync-delta.ts';
import { fetchRemote } from '../core/git-remote.ts';
import {
parseUsdLimit,
formatUsdLimit,
resolveSpendPosture,
} from '../core/spend-posture.ts';
import { estimateTokens, CHUNKER_VERSION } from '../core/chunkers/code.ts';
import {
estimateEmbeddingCostUsd,
@@ -28,11 +37,10 @@ import {
currentEmbeddingSignature,
willEmbedSynchronously,
shouldBlockSync,
type SyncEmbedMode,
} from '../core/embedding.ts';
import { estimateCostFromChars } from '../core/embedding-pricing.ts';
import { isSourceUnchangedSinceSync } from '../core/git-head.ts';
import { SPEND_CAP_CONFIG_KEY } from '../core/embed-backfill-submit.ts';
import { errorFor, serializeError } from '../core/errors.ts';
import type { SyncManifest } from '../core/sync.ts';
import { createProgress } from '../core/progress.ts';
import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts';
@@ -217,16 +225,17 @@ export interface SyncResult {
/**
* Walk ONE source's working tree and sum tokens for every syncable file.
* Conservative full-tree ceiling (full file content, not the incremental
* diff) — over-counts, never under-counts, and matches the filesystem set
* `sync` actually imports (collectSyncableFiles + content_hash, NOT a git
* commit diff). Best-effort per file and per source: anything unreadable
* contributes 0 rather than blocking the preview.
* Conservative full-tree CEILING (full file content, not the incremental
* diff) — over-counts, never under-counts. Used only on the ceiling rungs of
* `estimateInlineNewTokens` (first sync, chunker drift, git-unavailable),
* where the delta is genuinely the whole tree or can't be computed.
*
* v0.31.2: routed through collectSyncableFiles (lstat + inode-cycle +
* max-depth) so the preview walks exactly what the real sync walks.
*
* Exported (v0.42.42.0, #2139) for direct unit testing.
*/
function estimateSourceTreeTokens(
export function estimateSourceTreeTokens(
localPath: string,
strategy: 'markdown' | 'code' | 'auto',
): { tokens: number; files: number } {
@@ -251,18 +260,114 @@ function estimateSourceTreeTokens(
return { tokens, files };
}
/** Sum tokens for an explicit set of repo-relative paths read at live working-tree content. */
function estimateDeltaTokens(localPath: string, relPaths: string[]): number {
let tokens = 0;
for (const rel of relPaths) {
try {
const full = join(localPath, rel);
const stat = statSync(full);
if (stat.size > 5_000_000) continue; // skip large binaries (matches tree walk)
tokens += estimateTokens(readFileSync(full, 'utf-8'));
} catch {
// Listed in the diff but unreadable (e.g. since deleted) → 0, like the tree walk.
}
}
return tokens;
}
/**
* v0.41.31 — INLINE-path new-content estimate. Per source, contribute ZERO
* when the source is provably unchanged since its last sync (HEAD ==
* last_commit AND clean working tree AND chunker_version matches CURRENT) —
* `content_hash` short-circuits every file so nothing re-embeds. Otherwise
* contribute the full-tree ceiling. The unchanged predicate mirrors
* doctor's `sync_freshness` and sync's own "do work?" gate (sync.ts:1057+
* 1075). `isSourceUnchangedSinceSync` is fail-open (probe error → false), so
* a source we can't prove unchanged is conservatively re-estimated rather
* than silently priced at $0.
* v0.42.42.0 (#2139): resolve the commit the estimate should diff AGAINST.
*
* The cost gate runs BEFORE sync's own `git pull`, so a stale local HEAD would
* make the estimate blind to commits the run is about to pull (codex #1). So
* we FETCH first (fail-open) and target `origin/<branch>` — the estimate then
* prices exactly what this run will sync. The subsequent pull fast-forwards
* the already-fetched objects, so net new network cost ≈ 0.
*
* - detached HEAD → no upstream; target = local HEAD (+ caller merges the
* detached working-tree manifest, which sync imports on a detached repo).
* - attached + origin remote → fetch origin/<branch> (best-effort), target =
* origin/<branch> if resolvable, else local HEAD (offline / no upstream).
* - HEAD unresolvable (not a git repo) → null (caller treats as unavailable).
*
* NOTE: this makes `--dry-run` perform a network fetch so the preview reflects
* what a real run would pull. Fail-open: offline dry-run still previews against
* local HEAD. Uses the shared `git()` 30s budget (the fetch cost is the pull
* cost paid a few seconds early — a tighter cap would frequently fall back to
* local HEAD and underestimate the remote delta).
*/
function estimateInlineNewTokens(
function resolveEstimateTarget(localPath: string): { target: string; detached: boolean } | null {
let head: string;
try {
head = git(localPath, ['rev-parse', 'HEAD']);
} catch {
return null;
}
const detached = isDetachedHead(localPath);
if (detached) return { target: head, detached: true };
let branch: string | null = null;
try {
branch = git(localPath, ['rev-parse', '--abbrev-ref', 'HEAD']).trim() || null;
} catch {
branch = null;
}
if (branch && branch !== 'HEAD' && hasOriginRemote(localPath)) {
try {
// v0.42.42.0 (#2139): route through the SSRF-hardened fetch (same flags +
// no-prompt env as pullRepo) — a cost preview / dry-run must NOT hit a
// remote through a less-protected path than real sync.
fetchRemote(localPath, branch, { timeoutMs: 30_000 });
} catch {
// fail-open: offline, auth failure, no upstream — fall through to local HEAD.
}
try {
const remoteSha = git(localPath, ['rev-parse', `origin/${branch}`]);
if (remoteSha) return { target: remoteSha, detached: false };
} catch {
// no remote-tracking ref for this branch — use local HEAD.
}
}
return { target: head, detached: false };
}
export type EstimateKind = 'delta' | 'ceiling' | 'mixed' | 'unchanged';
export interface InlineEstimate {
tokens: number;
changedSources: number;
unchangedSources: number;
estimateKind: EstimateKind;
/** Per-source ceiling reasons (chunker_drift / first_sync / git_unavailable) for honest labeling. */
ceilingReasons: string[];
}
/**
* v0.42.42.0 (#2139) — INLINE-path new-content estimate. The estimate now
* MIRRORS EXECUTION instead of pricing the whole tree on every dirty sync (the
* 400x overestimate that wedged the daily cron). Per-source fail-open ladder:
*
* 1. syncEnabled === false → skip (unchanged)
* 2. chunker drift (stored !== current) → full-tree CEILING (a drift forces
* performFullSync → full re-chunk → full re-embed; a delta would
* underestimate by the whole corpus). kind: ceiling_chunker_drift
* 3. last_commit === fetch target → 0 (mirrors `up_to_date` at
* sync.ts:1402 — NO clean-working-tree requirement; a dirty tree whose
* commits are caught up imports nothing). kind: unchanged
* 4. last_commit === null (first sync) → full-tree CEILING. kind: ceiling_first_sync
* 5. computeSyncDelta ok → price addedmodifiedrenamed.to
* (syncable, live working-tree content); deletes cost 0. kind: delta
* 6. computeSyncDelta unavailable → full-tree CEILING. kind: ceiling_git_unavailable
*
* The delta rung routes through the SAME `computeSyncDelta` the executor uses
* (src/core/sync-delta.ts), so the gate's dollar figure can't drift from what
* the sync imports. `--full`'s extra stale-backlog sweep is added by the gate
* (it already has `staleCostUsd`), not here — see the call site.
*
* Exported for direct unit testing.
*/
export function estimateInlineNewTokens(
sources: Array<{
local_path: string | null;
config: Record<string, unknown>;
@@ -270,25 +375,85 @@ function estimateInlineNewTokens(
chunker_version: string | null;
}>,
currentChunkerVersion: string,
): { tokens: number; changedSources: number; unchangedSources: number } {
): InlineEstimate {
let tokens = 0;
let changedSources = 0;
let unchangedSources = 0;
let hadDelta = false;
let hadCeiling = false;
const ceilingReasons: string[] = [];
const ceiling = (localPath: string, strategy: 'markdown' | 'code' | 'auto', reason: string) => {
tokens += estimateSourceTreeTokens(localPath, strategy).tokens;
changedSources++;
hadCeiling = true;
ceilingReasons.push(reason);
};
for (const src of sources) {
if (!src.local_path) continue;
const cfg = (src.config || {}) as { syncEnabled?: boolean; strategy?: 'markdown' | 'code' | 'auto' };
if (cfg.syncEnabled === false) continue;
const unchanged =
isSourceUnchangedSinceSync(src.local_path, src.last_commit, { requireCleanWorkingTree: true }) &&
src.chunker_version === currentChunkerVersion;
if (unchanged) {
const strategy = cfg.strategy ?? 'markdown';
const localPath = src.local_path;
// Rung 2: chunker drift forces a full re-chunk → full re-embed. CEILING.
if (src.chunker_version !== currentChunkerVersion) {
ceiling(localPath, strategy, 'chunker_drift');
continue;
}
// Rung 4 (early): no bookmark → first sync imports everything. CEILING.
if (!src.last_commit) {
ceiling(localPath, strategy, 'first_sync');
continue;
}
const resolved = resolveEstimateTarget(localPath);
if (!resolved) {
// HEAD unresolvable (not a git repo / gone) — can't compute a delta. CEILING.
ceiling(localPath, strategy, 'git_unavailable');
continue;
}
// Rung 3: caught up to the fetch target AND no detached working-tree changes.
// Mirrors the executor's `up_to_date` predicate — a dirty-but-committed-current
// tree imports nothing, so it must price $0 (the heart of the false-fire fix).
const detachedManifest = resolved.detached
? buildDetachedWorkingTreeManifest(localPath)
: null;
const detachedHasChanges = detachedManifest !== null &&
(detachedManifest.added.length > 0 ||
detachedManifest.modified.length > 0 ||
detachedManifest.deleted.length > 0 ||
detachedManifest.renamed.length > 0);
if (src.last_commit === resolved.target && !detachedHasChanges) {
unchangedSources++;
continue;
}
// Rung 5/6: the delta itself — SAME helper the executor diffs with.
const delta = computeSyncDelta(localPath, src.last_commit, resolved.target, {
detachedManifest,
});
if (delta.status === 'unavailable') {
ceiling(localPath, strategy, 'git_unavailable');
continue;
}
const syncOpts = { strategy };
const changedPaths = unique([
...delta.manifest.added.filter(p => isSyncable(p, syncOpts)),
...delta.manifest.modified.filter(p => isSyncable(p, syncOpts)),
...delta.manifest.renamed.filter(r => isSyncable(r.to, syncOpts)).map(r => r.to),
]);
tokens += estimateDeltaTokens(localPath, changedPaths);
changedSources++;
tokens += estimateSourceTreeTokens(src.local_path, cfg.strategy ?? 'markdown').tokens;
hadDelta = true;
}
return { tokens, changedSources, unchangedSources };
const estimateKind: EstimateKind =
hadCeiling && hadDelta ? 'mixed' : hadCeiling ? 'ceiling' : hadDelta ? 'delta' : 'unchanged';
return { tokens, changedSources, unchangedSources, estimateKind, ceilingReasons };
}
/**
@@ -302,9 +467,10 @@ function estimateInlineNewTokens(
async function resolveCostGateFloorUsd(engine: BrainEngine): Promise<number> {
try {
const raw = await engine.getConfig('sync.cost_gate_min_usd');
if (raw === null || raw === undefined) return 0.5;
const n = Number(raw);
return Number.isFinite(n) && n >= 0 ? n : 0.5;
// v0.42.42.0 (#2139): `0` keeps meaning "block on any nonzero spend"
// (allowZero); `off`/`unlimited`/`none` → Infinity so the gate never
// blocks (`costUsd > Infinity` is always false).
return parseUsdLimit(raw, 0.5, { allowZero: true });
} catch {
return 0.5;
}
@@ -318,9 +484,9 @@ async function resolveCostGateFloorUsd(engine: BrainEngine): Promise<number> {
async function resolveBackfillCapUsd(engine: BrainEngine): Promise<number> {
try {
const raw = await engine.getConfig(SPEND_CAP_CONFIG_KEY);
if (raw === null || raw === undefined) return 25;
const n = Number(raw);
return Number.isFinite(n) && n > 0 ? n : 25;
// v0.42.42.0 (#2139): no allowZero — `0` falls back to the default
// (off semantics ≠ 0); `off`/`unlimited`/`none` → Infinity (cap disabled).
return parseUsdLimit(raw, 25);
} catch {
return 25;
}
@@ -338,6 +504,214 @@ async function promptYesNo(question: string): Promise<boolean> {
});
}
// v0.42.42.0 (#2139): paste-ready knobs appended to every gate message so the
// spend-control surface is discoverable at the moment of need (humans + agents),
// not only after reading source. Closes the issue's "takes archaeology" complaint.
const SPEND_HINT =
'widen: gbrain config set sync.cost_gate_min_usd 5 | ' +
'never gate: gbrain config set spend.posture tokenmax | ' +
'docs: docs/operations/spend-controls.md';
/** Honest token label — delta vs full-tree ceiling, with the ceiling reasons. */
function labelEstimate(inline: InlineEstimate): string {
if (inline.estimateKind === 'unchanged') return '0 new tokens (sources caught up)';
if (inline.estimateKind === 'delta') {
return `~${inline.tokens.toLocaleString()} new tokens (delta: changed files since last sync)`;
}
// ceiling | mixed — be explicit that this is an over-count, not the real spend.
const reasons = unique(inline.ceilingReasons).join(', ') || 'unknown';
return (
`<=${inline.tokens.toLocaleString()} tokens (full-tree ceiling for ${inline.changedSources} ` +
`source(s): ${reasons} — unchanged files skip via content_hash at execution)`
);
}
type CostGateSource = {
local_path: string | null;
config: Record<string, unknown>;
last_commit: string | null;
chunker_version: string | null;
};
interface CostGateContext {
sources: CostGateSource[];
/** Resolved embed mode. Single-source is always 'inline' (not the parallel-deferred fan-out). */
mode: SyncEmbedMode;
dryRun: boolean;
jsonOut: boolean;
yesFlag: boolean;
full: boolean;
/** Message prefix ('sync --all' | 'sync'). */
label: string;
}
type CostGateOutcome =
| { action: 'proceed'; autoDeferEmbeds: boolean }
| { action: 'stop' };
/**
* v0.42.42.0 (#2139): the inline-embed cost gate, shared by BOTH `sync --all`
* and single-source `sync` so the spend surface is consistent. Runs at the
* COMMAND layer (never inside performSync, which `runOne` also calls — that
* would double-gate `--all`).
*
* Behavior:
* - deferred mode → FYI only, never blocks (backfill cap is the money gate).
* - inline + below floor → proceed quietly.
* - inline + spend.posture=tokenmax → informational, proceed inline.
* - inline + above floor + TTY → [y/N] prompt.
* - inline + above floor + non-TTY/--json → AUTO-DEFER embeds to capped
* backfill jobs, exit 0 (NEVER exit 2 — the wedged-cron fix). Caller sets
* effectiveNoEmbed and enqueues the backfill.
*
* Output format splits on the EXPLICIT `--json` flag only (absorbs the
* TODOS.md:340 #1784 conflation): JSON envelope iff `--json`, else human text.
*/
async function runInlineCostGate(
engine: BrainEngine,
ctx: CostGateContext,
): Promise<CostGateOutcome> {
const { sources, mode, dryRun, jsonOut, yesFlag, full, label } = ctx;
// Stale backlog: cheap single SQL; fail-open to 0 so a transient DB hiccup
// never blocks the sync. Signature-aware (model/dims swap surfaces here).
let staleChars = 0;
try {
staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() });
} catch {
staleChars = 0;
}
const staleCostUsd = estimateCostFromChars(staleChars, currentEmbeddingPricePerMTok());
const embeddingModelName = getEmbeddingModelName();
const floorUsd = await resolveCostGateFloorUsd(engine);
const posture = await resolveSpendPosture(engine);
if (mode === 'deferred') {
// Deferred path: print an FYI, NEVER block. The backfill cap is the real
// money gate.
const capUsd = await resolveBackfillCapUsd(engine);
let queuedBackfills = 0;
try {
const r = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM minion_jobs
WHERE name = 'embed-backfill'
AND status IN ('waiting','active','delayed','waiting-children')`,
);
queuedBackfills = Number(r[0]?.n) || 0;
} catch {
queuedBackfills = 0;
}
const deferredMsg =
`${label}: embedding deferred to backfill jobs ` +
`(capped $${formatUsdLimit(capUsd)}/source/24h, not charged by this sync). ` +
`Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` +
`${embeddingModelName}) across ${sources.length} source(s); ` +
`${queuedBackfills} backfill job(s) queued.`;
if (dryRun) {
if (jsonOut) {
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd: formatUsdLimit(capUsd), floorUsd: formatUsdLimit(floorUsd), queuedBackfills, model: embeddingModelName }));
} else {
console.log(deferredMsg);
console.log('--dry-run: exit without syncing.');
}
return { action: 'stop' };
}
if (jsonOut) {
console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd: formatUsdLimit(capUsd), floorUsd: formatUsdLimit(floorUsd), queuedBackfills, model: embeddingModelName }));
} else {
console.log(deferredMsg);
}
return { action: 'proceed', autoDeferEmbeds: false };
}
// ── Inline path ───────────────────────────────────────────────
const inline = estimateInlineNewTokens(sources, String(CHUNKER_VERSION));
// D7A: `--full` runs `performFullSync` → `runEmbedCore({stale:true})`, which
// sweeps the pre-existing stale backlog INLINE on top of the delta. Price it.
const costUsd = estimateEmbeddingCostUsd(inline.tokens) + (full ? staleCostUsd : 0);
const fullNote = full && staleChars > 0
? ` (includes ~${staleChars.toLocaleString()} stale-backlog chars swept by --full)`
: '';
const staleNote = !full && staleChars > 0
? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)`
: '';
const previewMsg =
`${label} preview (inline embed): ${inline.changedSources} changed source(s), ` +
`${inline.unchangedSources} unchanged; ${labelEstimate(inline)}, ` +
`est. $${costUsd.toFixed(2)} on ${embeddingModelName}${fullNote}${staleNote}.`;
if (dryRun) {
if (jsonOut) {
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, estimateKind: inline.estimateKind, staleChars, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName }));
} else {
console.log(previewMsg);
console.log('--dry-run: exit without syncing.');
}
return { action: 'stop' };
}
// --yes bypasses the gate entirely (embed inline, no preview).
if (yesFlag) return { action: 'proceed', autoDeferEmbeds: false };
// spend.posture=tokenmax → informational, proceed INLINE (operator declared
// cost isn't the constraint; don't defer).
if (posture === 'tokenmax') {
if (jsonOut) {
console.log(JSON.stringify({ status: 'proceeding', mode, gate: 'posture_tokenmax', newTokens: inline.tokens, estimateKind: inline.estimateKind, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName, hint: SPEND_HINT }));
} else {
console.log(`${previewMsg} spend.posture=tokenmax: proceeding (informational). ${SPEND_HINT}`);
}
return { action: 'proceed', autoDeferEmbeds: false };
}
// Link intent: search.mode=tokenmax but spend posture unset → nudge once.
let searchModeHint = '';
try {
const sm = await engine.getConfig('search.mode');
if (typeof sm === 'string' && sm.trim().toLowerCase() === 'tokenmax') {
searchModeHint =
` (search.mode=tokenmax detected — \`gbrain config set spend.posture tokenmax\` ` +
`makes cost gates informational)`;
}
} catch {
/* best-effort */
}
if (shouldBlockSync(costUsd, floorUsd, mode, posture)) {
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
if (isTTY && !jsonOut) {
// Interactive TTY: prompt [y/N].
console.log(previewMsg + searchModeHint);
const answer = await promptYesNo('Proceed? [y/N] ');
if (!answer) {
console.log('Cancelled.');
return { action: 'stop' };
}
return { action: 'proceed', autoDeferEmbeds: false };
}
// Non-TTY or --json: AUTO-DEFER embeds to capped backfill jobs. NEVER exit 2
// (the wedged-cron fix). Format splits on the explicit --json flag only.
if (jsonOut) {
console.log(JSON.stringify({ status: 'auto_deferred', mode, gate: 'auto_deferred_embeds', newTokens: inline.tokens, estimateKind: inline.estimateKind, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName, hint: SPEND_HINT }));
} else {
console.log(
`${previewMsg} Exceeds floor $${formatUsdLimit(floorUsd)} in a non-interactive ` +
`session — importing now, deferring embeds to capped backfill jobs. ` +
`Drain: run the jobs worker or \`gbrain embed --stale\`. Pass --yes to embed inline.\n${SPEND_HINT}`,
);
}
return { action: 'proceed', autoDeferEmbeds: true };
}
// Below floor → proceed without blocking (kills inline-cron noise).
if (jsonOut) {
console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, estimateKind: inline.estimateKind, staleChars, costUsd, floorUsd: formatUsdLimit(floorUsd), model: embeddingModelName }));
} else {
console.log(`${previewMsg} Below cost gate floor ($${formatUsdLimit(floorUsd)}), proceeding.`);
}
return { action: 'proceed', autoDeferEmbeds: false };
}
export interface SyncOpts {
repoPath?: string;
dryRun?: boolean;
@@ -557,19 +931,9 @@ function unique<T>(items: T[]): T[] {
return [...new Set(items)];
}
function buildDetachedWorkingTreeManifest(repoPath: string): SyncManifest {
const manifest = buildSyncManifest(git(repoPath, ['diff', '--name-status', '-M', 'HEAD']));
const untracked = git(repoPath, ['ls-files', '--others', '--exclude-standard'])
.split('\n')
.filter(line => line.length > 0);
return {
added: unique([...manifest.added, ...untracked]),
modified: unique(manifest.modified),
deleted: unique(manifest.deleted),
renamed: manifest.renamed,
};
}
// v0.42.42.0 (#2139): `buildDetachedWorkingTreeManifest` relocated to
// `src/core/sync-delta.ts` (re-imported below) so the inline cost estimator
// prices detached sources through the same code the executor imports them with.
// v0.18.0 Step 5: source-scoped sync state helpers. When opts.sourceId
// is set, read/write the per-source row instead of the global config
@@ -1429,29 +1793,28 @@ async function performSyncInner(engine: BrainEngine, opts: SyncOpts): Promise<Sy
// both endpoints are stable across every resume, so the manifest is
// deterministic and resumeFilter maps cleanly onto completed paths.
//
// v0.42.42.0 (#2139): the diff + detached-working-tree merge now route
// through `computeSyncDelta` (src/core/sync-delta.ts) — the SAME helper the
// inline cost estimator uses, so the gate's dollar figure can't drift from
// what this sync actually imports. `detachedWorkingTreeManifest` (computed
// above for the `up_to_date` gate) is passed through to avoid recomputing it.
//
// #1970 (F-B): a non-ancestor diff against a wildly divergent tree (e.g. a
// force-push to unrelated history) can exceed git()'s 30s timeout / 100 MiB
// buffer. On failure, fall back to the authoritative full reconcile instead
// of throwing — a slow correct reconcile beats a hard error or a silent walk.
let diffOutput: string;
try {
diffOutput = git(repoPath, ['diff', '--name-status', '-M', `${lastCommit}..${pin}`]);
} catch (e) {
const msg = e instanceof Error ? e.message : String(e);
// buffer, and a gc'd anchor object can't be diffed at all. On either
// `unavailable`, fall back to the authoritative full reconcile instead of
// throwing — a slow correct reconcile beats a hard error or a silent walk.
const delta = computeSyncDelta(repoPath, lastCommit, pin, {
detachedManifest: detachedWorkingTreeManifest,
});
if (delta.status === 'unavailable') {
serr(
`[sync] git diff ${lastCommit.slice(0, 8)}..${pin.slice(0, 8)} failed ` +
`(${msg.slice(0, 80)}) — likely an oversized post-rewrite diff; ` +
`falling back to full reconcile.`,
`[sync] delta ${lastCommit.slice(0, 8)}..${pin.slice(0, 8)} unavailable ` +
`(${delta.reason}) — falling back to full reconcile.`,
);
return performFullSync(engine, repoPath, headCommit, opts);
}
const manifest = buildSyncManifest(diffOutput);
if (detachedWorkingTreeManifest) {
manifest.added = unique([...manifest.added, ...detachedWorkingTreeManifest.added]);
manifest.modified = unique([...manifest.modified, ...detachedWorkingTreeManifest.modified]);
manifest.deleted = unique([...manifest.deleted, ...detachedWorkingTreeManifest.deleted]);
manifest.renamed = [...manifest.renamed, ...detachedWorkingTreeManifest.renamed];
}
const manifest = delta.manifest;
// Filter to syncable files (strategy-aware)
const syncOpts = opts.strategy ? { strategy: opts.strategy } : undefined;
@@ -3071,13 +3434,6 @@ See also:
// never reached, and "Already up to date." leaves the log untouched. Both
// doctor and printSyncResult instruct users to run --skip-failed in
// exactly this case, so the flag has to handle stale entries up-front.
if (skipFailed) {
const stale = unacknowledgedSyncFailures();
if (stale.length > 0) {
const acked = acknowledgeSyncFailures();
console.log(`Acknowledged ${acked.count} pre-existing failure(s).`);
}
}
// v0.18.0 Step 5: --source resolves to a sources(id) row. Falls back
// to pre-v0.17 global config (sync.repo_path + sync.last_commit) when
@@ -3103,6 +3459,23 @@ See also:
if (nudge) process.stderr.write(nudge + '\n');
}
// --skip-failed: acknowledge pre-existing unacked failures BEFORE the sync
// runs, not only ones the current run produces. Without this, the common
// recovery flow — fix the YAML, re-run sync, then run --skip-failed to clear
// the log — fails to clear anything (no NEW failures → the inner ack path in
// performSync is never reached, and "Already up to date." leaves the log).
//
// v0.42.42.0 (#2139, D13C): scoped PER SOURCE. `--all` clears every source's
// open failures; single-source clears only its own (don't ack source B's
// failures when syncing source A). Safe under parallel — the ledger
// serializes writes via `withLedgerLock` and keys rows by `source_id`
// (#1939), which is why the old D15 "no --skip-failed under parallel"
// refusal is lifted below.
if (skipFailed) {
const acked = syncAll ? acknowledgeFailures() : acknowledgeFailures(sourceId);
if (acked.count > 0) console.log(`Acknowledged ${acked.count} pre-existing failure(s).`);
}
// v0.19.0 — `sync --all` iterates all registered sources with a
// local_path. Sources are the canonical v0.18.0 abstraction: per-source
// last_commit, last_sync_at, config.federated flags. Per-source
@@ -3131,132 +3504,21 @@ See also:
const { isFederatedV2Enabled } = await import('../core/feature-flags.ts');
const v2Enabled = await isFederatedV2Enabled(engine);
// v0.41.31 cost gate (supersedes the v0.20.0 unconditional gate). Under
// federated_v2 sync DEFERS embedding to per-source embed-backfill jobs
// that carry their own $X/source/24h spend cap, so sync itself spends
// nothing synchronously — the gate is INFORMATIONAL (never exit 2) on
// that path. The blocking ConfirmationRequired gate fires ONLY when embed
// runs INLINE (v2 off, or --serial without --no-embed) AND the estimated
// spend exceeds `sync.cost_gate_min_usd` (default $0.50). Skipped entirely
// when --no-embed is set (user opted out; will run `embed --stale` later).
// v0.42.42.0 (#2139) cost gate — shared `runInlineCostGate`. Under
// federated_v2 + parallel, embedding is DEFERRED to per-source backfill
// jobs (own spend cap) so the gate is FYI-only. Inline mode (v2 off, or
// --serial without --no-embed) gates on the DELTA estimate: below floor
// proceeds; above floor in a non-TTY/--json session AUTO-DEFERS embeds
// (exit 0, never exit 2 — the wedged-cron fix); a TTY prompts. Skipped
// entirely when --no-embed is set.
let autoDeferEmbeds = false;
if (!noEmbed) {
const mode = willEmbedSynchronously({ v2Enabled, serialFlag, noEmbed });
// Stale backlog: cheap single SQL; fail-open to 0 so a transient DB
// hiccup never blocks the sync.
let staleChars = 0;
try {
// v0.41.31: signature-aware so a model/dims swap surfaces in the
// backlog estimate (NULL signature grandfathered → not counted).
staleChars = await engine.sumStaleChunkChars({ signature: currentEmbeddingSignature() });
} catch {
staleChars = 0;
}
const rate = currentEmbeddingPricePerMTok();
const staleCostUsd = estimateCostFromChars(staleChars, rate);
const embeddingModelName = getEmbeddingModelName();
const floorUsd = await resolveCostGateFloorUsd(engine);
if (mode === 'deferred') {
// Deferred path: print an FYI, NEVER exit 2. The backfill cap is the
// real money gate (D1/D4).
const capUsd = await resolveBackfillCapUsd(engine);
// v0.41.31 (TODO-2): surface already-queued backfill jobs so a cron
// operator sees work is enqueued, not lost. Best-effort — minion_jobs
// may not exist on a brain that never ran a worker.
let queuedBackfills = 0;
try {
const r = await engine.executeRaw<{ n: number }>(
`SELECT COUNT(*)::int AS n FROM minion_jobs
WHERE name = 'embed-backfill'
AND status IN ('waiting','active','delayed','waiting-children')`,
);
queuedBackfills = Number(r[0]?.n) || 0;
} catch {
queuedBackfills = 0;
}
const deferredMsg =
`sync --all: embedding deferred to backfill jobs ` +
`(capped $${capUsd}/source/24h, not charged by this sync). ` +
`Current backlog ~${staleChars.toLocaleString()} chars (~$${staleCostUsd.toFixed(2)} on ` +
`${embeddingModelName}) across ${sources.length} source(s); ` +
`${queuedBackfills} backfill job(s) queued.`;
if (dryRun) {
if (jsonOut) {
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName }));
} else {
console.log(deferredMsg);
console.log('--dry-run: exit without syncing.');
}
return;
}
if (jsonOut) {
console.log(JSON.stringify({ status: 'deferred', mode, gate: 'deferred_notice', staleChars, staleCostUsd, capUsd, floorUsd, queuedBackfills, model: embeddingModelName }));
} else {
console.log(deferredMsg);
}
// fall through to sync — no exit 2.
} else {
// Inline path: sync embeds synchronously with no backfill cap to
// protect it, so the blocking gate applies. The BLOCKING cost is the
// new-content estimate ONLY (full-tree ceiling for changed sources;
// unchanged contribute 0) — that's what this sync actually embeds.
// The pre-existing stale backlog (NULL embeddings + signature drift)
// is NOT swept by sync; `gbrain embed --stale` clears it. So we show
// it informationally but never gate on cost this sync won't incur
// (else a model swap would block the next inline cron — F2).
const currentChunkerVersion = String(CHUNKER_VERSION);
const inline = estimateInlineNewTokens(sources, currentChunkerVersion);
const newCostUsd = estimateEmbeddingCostUsd(inline.tokens);
const costUsd = newCostUsd;
const staleNote = staleChars > 0
? ` (plus ~${staleChars.toLocaleString()} stale-backlog chars pending \`gbrain embed --stale\`)`
: '';
const previewMsg =
`sync --all preview (inline embed): ${inline.changedSources} changed source(s), ` +
`${inline.unchangedSources} unchanged; ~${inline.tokens.toLocaleString()} new tokens, ` +
`est. $${costUsd.toFixed(2)} on ${embeddingModelName}${staleNote}.`;
if (dryRun) {
if (jsonOut) {
console.log(JSON.stringify({ status: 'dry_run', mode, gate: 'dry_run', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
} else {
console.log(previewMsg);
console.log('--dry-run: exit without syncing.');
}
return;
}
if (!yesFlag) {
if (shouldBlockSync(costUsd, floorUsd, mode)) {
const isTTY = Boolean(process.stdout.isTTY) && Boolean(process.stdin.isTTY);
if (!isTTY || jsonOut) {
// Agent-facing path: emit structured envelope, exit 2.
const envelope = serializeError(errorFor({
class: 'ConfirmationRequired',
code: 'cost_preview_requires_yes',
message: previewMsg,
hint: 'Pass --yes to proceed, or --dry-run to see the preview and exit 0.',
}));
console.log(JSON.stringify({ error: envelope, mode, gate: 'confirmation_required', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
process.exit(2);
}
// Interactive TTY path: prompt [y/N].
console.log(previewMsg);
const answer = await promptYesNo('Proceed? [y/N] ');
if (!answer) {
console.log('Cancelled.');
return;
}
} else {
// Below floor → proceed without blocking (kills inline-cron noise).
if (jsonOut) {
console.log(JSON.stringify({ status: 'below_floor', mode, gate: 'below_floor', newTokens: inline.tokens, staleChars, costUsd, floorUsd, model: embeddingModelName }));
} else {
console.log(`${previewMsg} Below cost gate floor ($${floorUsd.toFixed(2)}), proceeding.`);
}
}
}
}
const gate = await runInlineCostGate(engine, {
sources, mode, dryRun, jsonOut, yesFlag, full, label: 'sync --all',
});
if (gate.action === 'stop') return;
autoDeferEmbeds = gate.autoDeferEmbeds;
}
// v0.40.5.0 Federated Sync v2 (master) + v0.40.6.0 layering (this branch):
@@ -3317,7 +3579,12 @@ See also:
const runOne = async (src: typeof sources[number]): Promise<SyncResult> => {
const cfg = (src.config || {}) as { strategy?: 'markdown' | 'code' | 'auto' };
// D18: parallel path defers embed; auto-enqueue embed-backfill after.
const effectiveNoEmbed = v2Enabled && !serialFlag && !noEmbed ? true : noEmbed;
// v0.42.42.0 (#2139): `autoDeferEmbeds` (the inline gate tripped in a
// non-TTY session) ALSO forces deferral — global by design (the gate's
// decision unit is the aggregate estimate; deferral strictly dominates
// the exit-2 it replaced for every source).
const effectiveNoEmbed =
(v2Enabled && !serialFlag && !noEmbed ? true : noEmbed) || autoDeferEmbeds;
// v0.41.13.0 (T6 / D-V3-3 / D-V4-mech-6) — per-source AbortController.
//
// When the user passes --timeout, each source gets its OWN
@@ -3380,8 +3647,11 @@ See also:
// v0.41.13.0 (T7 / D-V3-5): partial excluded — the next clean sync
// re-walks the diff and re-decides whether to enqueue embed for
// pages whose content actually changed.
// v0.42.42.0 (#2139): `autoDeferEmbeds` enqueues even on the v2-OFF
// legacy path — otherwise the gate's auto-defer would strand
// NULL-embedded chunks with no queued job to embed them.
if (
v2Enabled &&
(v2Enabled || autoDeferEmbeds) &&
!noAutoEmbed &&
!dryRun &&
result.status !== 'dry_run' &&
@@ -3408,18 +3678,13 @@ See also:
const parallelEligible =
v2Enabled && !serialFlag && engine.kind !== 'pglite' && activeSources.length > 1;
// v0.40.6.0 (D15): refuse --skip-failed / --retry-failed when running
// parallel. sync-failures.jsonl is brain-global; parallel acks race.
if (parallelEligible && (skipFailed || retryFailed)) {
const flag = skipFailed ? '--skip-failed' : '--retry-failed';
console.error(
`Error: ${flag} is not supported under parallel sync.\n` +
` (the sync-failures log is brain-global and parallel acks race).\n` +
` Re-run with --serial for the recovery flow:\n` +
` gbrain sync --all --serial ${flag}`,
);
process.exit(1);
}
// v0.42.42.0 (#2139, D13C): the v0.40.6.0 (D15) refusal of --skip-failed /
// --retry-failed under parallel sync is LIFTED. It existed because the
// failure ledger was once brain-global with racing acks; #1939 made the
// ledger per-(source_id, path) and serialized every write through
// `withLedgerLock`, so parallel per-source acks no longer race. Lifting it
// also removes the forcing-function that pushed recovery syncs to --serial
// (and thus armed the inline cost gate) — the root cause behind #2139.
// Effective parallelism — surfaced in the --json envelope so consumers
// know how the run was actually dispatched. 1 in the serial fallback,
@@ -3567,12 +3832,47 @@ See also:
signal: composeAbortSignals(singleSourceInterrupt.signal, singleSourceController?.signal),
};
// v0.42.42.0 (#2139, Step 4b): single-source `gbrain sync` gets the SAME
// inline cost gate as `--all`. Previously single-source embedded inline with
// NO gate (only rail: the ≤100-file inline cap). Single-source always embeds
// INLINE (not the parallel-deferred fan-out), so mode is forced 'inline'.
// Skipped on --no-embed, --dry-run (performSync's own dry-run previews and
// spends nothing), and watch. Non-TTY above floor AUTO-DEFERS — so adding
// the gate can never wedge an existing cron; it converts silent ungated
// inline spend into informed inline-or-deferred spend.
let singleSourceAutoDefer = false;
if (!noEmbed && !dryRun && !watch) {
const gateRows = await engine.executeRaw<{ local_path: string | null; config: Record<string, unknown>; last_commit: string | null; chunker_version: string | null }>(
`SELECT local_path, config, last_commit, chunker_version FROM sources WHERE id = $1`,
[sourceId],
);
if (gateRows.length > 0) {
const gateSources = [{
local_path: gateRows[0].local_path ?? repoPath ?? null,
config: gateRows[0].config ?? {},
last_commit: gateRows[0].last_commit,
chunker_version: gateRows[0].chunker_version,
}];
const gate = await runInlineCostGate(engine, {
sources: gateSources, mode: 'inline', dryRun: false, jsonOut, yesFlag, full, label: 'sync',
});
if (gate.action === 'stop') return;
if (gate.autoDeferEmbeds) {
opts.noEmbed = true;
singleSourceAutoDefer = true;
}
}
}
// Bug 9 — --retry-failed: before running normal sync, clear acknowledgment
// flags so the sync picks them up as fresh work. The actual re-attempt
// happens inside the regular incremental/full loop because once the commit
// pointer is behind the failures, the diff naturally revisits them.
if (retryFailed) {
const failures = unacknowledgedSyncFailures();
// v0.42.42.0 (#2139, D13C): scope the retry count to THIS source — rows
// carry source_id (#1939), so a single-source retry shouldn't report
// another source's failures.
const failures = unacknowledgedSyncFailures().filter(f => f.source_id === sourceId);
if (failures.length === 0) {
console.log('No unacknowledged sync failures to retry.');
} else {
@@ -3615,6 +3915,27 @@ See also:
manageGitignore(effectiveRepoPath, engine.kind);
}
}
// v0.42.42.0 (#2139, Step 4b): the inline gate auto-deferred this run's
// embeds (non-TTY, above floor) — enqueue a capped backfill job so the
// NULL-embedded chunks get embedded out of band instead of being stranded.
if (
singleSourceAutoDefer &&
result.status !== 'dry_run' &&
result.status !== 'up_to_date' &&
result.status !== 'partial'
) {
try {
const { submitEmbedBackfill } = await import('../core/embed-backfill-submit.ts');
const sub = await submitEmbedBackfill(engine, sourceId, { reason: 'sync_autodefer' });
if (sub.status === 'submitted') {
process.stderr.write(` → embed-backfill job ${sub.jobId} queued (deferred inline embed).\n`);
} else if (sub.status === 'cooldown') {
process.stderr.write(` → embed-backfill skipped (cooldown); run \`gbrain embed --stale\` to drain now.\n`);
}
} catch (e) {
process.stderr.write(` → embed-backfill submission failed: ${e instanceof Error ? e.message : String(e)}\n`);
}
}
return;
}
+190
View File
@@ -0,0 +1,190 @@
/**
* v0.43 (#2095) — `gbrain watch`: the push transport for push-based context.
*
* Reads conversation turns from stdin AS THEY ARRIVE (plain text = a user
* turn; `user:` / `assistant:` prefixed lines set the role), maintains a
* rolling window in-process, and volunteers confidence-gated brain pages to
* stdout after every turn. The consumer pipes its transcript in and reads
* volunteered pointers out — no per-entity CLI round-trips.
*
* Lifecycle: BLOCKS in the stdin iteration (like `gbrain jobs work`), so an
* interactive TTY session stays alive until Ctrl-C / Ctrl-D and piped input
* exits at EOF. Either way the handler RETURNS, the CLI_ONLY finally runs
* finishCliTeardown (volunteer events bank before teardown), and the
* entrypoint flush-exit ends the process deliberately — which is exactly why
* `watch` is NOT in DAEMON_COMMANDS: it never returns from main() while work
* is still running. SIGINT closes the stream and flows through the same
* drain path instead of killing mid-write.
*
* Session dedupe: a slug is volunteered at most once per watch session —
* already-pushed slugs ride VolunteerOpts.excludeSlugs, skipped inside the
* core's pointer loop BEFORE the confidence gate and maxPages cap (O(1)
* membership; a post-call filter would let a recurring entity starve new
* pages out of the cap — red-team finding).
*
* PGLite note: watch holds the single-writer engine for the whole session,
* so on the default engine it cannot run concurrently with `gbrain serve`
* (or any other gbrain process) against the same brain — run it against
* Postgres, or stop serve first. Routing watch through the serve resolve-IPC
* socket (like the ambient reflex) is a filed follow-up.
*/
import { createInterface } from 'node:readline';
import type { BrainEngine } from '../core/engine.ts';
import {
volunteerContext,
formatVolunteeredPage,
TURN_PREFIX_RE,
VOLUNTEER_DEFAULT_MAX_PAGES,
VOLUNTEER_DEFAULT_MIN_CONFIDENCE,
} from '../core/context/volunteer.ts';
import type { WindowTurn } from '../core/context/entity-salience.ts';
import { DEFAULT_WINDOW_TURNS, windowTurnCount } from '../core/context/reflex.ts';
import { loadConfig } from '../core/config.ts';
import { logVolunteerEventsFireAndForget, volunteerEventRowsFrom } from '../core/context/volunteer-events.ts';
export const WATCH_HELP = `gbrain watch — push-based context: volunteer brain pages per conversation turn (#2095)
Reads turns from stdin (one per line; 'user:' / 'assistant:' prefixes set the
role, unprefixed lines are user turns) and prints confidence-gated page
pointers with rationales after each turn. A slug is volunteered at most once
per session. Piped input exits at EOF; interactive sessions exit on Ctrl-C.
Usage:
some-transcript-feed | gbrain watch [--json]
gbrain watch # interactive: type turns, Ctrl-C to end
PGLite brains: watch holds the single-writer engine for the whole session —
it cannot run alongside \`gbrain serve\` (or other gbrain processes) against
the same brain. Use a Postgres brain for concurrent access.
Flags:
--json JSONL output (one volunteered page per line)
--window-turns N rolling extraction window (default ${DEFAULT_WINDOW_TURNS})
--max-pages N max pages volunteered per turn (default ${VOLUNTEER_DEFAULT_MAX_PAGES}, cap 5)
--min-confidence X confidence gate 0..1 (default ${VOLUNTEER_DEFAULT_MIN_CONFIDENCE})
--source <id> source scope (defaults to the canonical 6-tier resolution)
--help this text
`;
function numFlag(args: string[], flag: string): number | undefined {
const i = args.indexOf(flag);
if (i < 0 || i + 1 >= args.length) return undefined;
const n = Number(args[i + 1]);
return Number.isFinite(n) ? n : undefined;
}
function strFlag(args: string[], flag: string): string | undefined {
const i = args.indexOf(flag);
return i >= 0 && i + 1 < args.length ? args[i + 1] : undefined;
}
export interface WatchIoDeps {
/** Injected line source for tests (defaults to readline over stdin). */
lines?: AsyncIterable<string>;
write?: (s: string) => void;
isTTY?: boolean;
}
export async function runWatch(engine: BrainEngine, args: string[], deps: WatchIoDeps = {}): Promise<void> {
if (args.includes('--help') || args.includes('-h')) {
(deps.write ?? ((s: string) => process.stdout.write(s)))(WATCH_HELP);
return;
}
const json = args.includes('--json');
// --window-turns wins; otherwise the same config knob the ambient reflex
// honors (retrieval_reflex_window_turns, default 4) applies here too.
// Hard cap 64: an unbounded window re-scans every retained turn on every
// turn over an hours-long session — the cost class the priorContext fix
// removed, reintroduced via a config typo (red-team finding).
const windowTurns = Math.min(
64,
Math.max(1, Math.floor(numFlag(args, '--window-turns') ?? windowTurnCount(loadConfig()))),
);
const maxPages = numFlag(args, '--max-pages');
const minConfidence = numFlag(args, '--min-confidence');
const write = deps.write ?? ((s: string) => process.stdout.write(s));
const isTTY = deps.isTTY ?? Boolean(process.stdin.isTTY);
const { resolveSourceId } = await import('../core/source-resolver.ts');
const sourceId = await resolveSourceId(engine, strFlag(args, '--source') ?? null, process.cwd());
const sourceIds = [sourceId];
const sessionId = `watch-${process.pid}-${Date.now().toString(36)}`;
if (!deps.lines) {
// The ready line doubles as a machine-readable readiness signal for
// scripted consumers (and the SIGINT lifecycle test): engine + source
// resolution are done, the stdin loop starts next.
process.stderr.write(
isTTY
? `[watch] interactive session ${sessionId} ready — type turns ('assistant: ...' to set role), Ctrl-C to end\n`
: `[watch] session ${sessionId} ready\n`,
);
}
const rl = deps.lines
? null
: createInterface({ input: process.stdin, crlfDelay: Infinity });
const lines: AsyncIterable<string> = deps.lines ?? (rl as AsyncIterable<string>);
// SIGINT closes the stream so the for-await ends and the normal
// drain-then-exit path runs (never a mid-write kill).
const onSigint = () => {
rl?.close();
};
process.on('SIGINT', onSigint);
const window: WindowTurn[] = [];
const pushedSlugs = new Set<string>(); // session dedupe (slug-only suppression input)
let turnNo = 0;
try {
for await (const rawLine of lines) {
const line = rawLine.replace(/\r$/, '');
if (!line.trim()) continue;
const m = TURN_PREFIX_RE.exec(line);
const turn: WindowTurn = m
? { role: m[1].toLowerCase() as WindowTurn['role'], text: (m[2] ?? '').trim() }
: { role: 'user', text: line.trim() };
if (!turn.text) continue;
turnNo++;
window.push(turn);
if (window.length > windowTurns) window.splice(0, window.length - windowTurns);
let pages;
try {
pages = await volunteerContext(engine, [...window], {
sourceIds,
maxPages,
minConfidence,
// Session dedupe: skipped inside the core BEFORE the gate + cap
// (O(1) per pointer) so a recurring slug can't starve new pages.
excludeSlugs: pushedSlugs,
});
} catch {
continue; // fail-open per turn: a transient DB error never kills the stream
}
if (!pages.length) continue;
for (const p of pages) pushedSlugs.add(p.slug);
logVolunteerEventsFireAndForget(
engine,
volunteerEventRowsFrom(pages, { channel: 'watch', session_id: sessionId, turn: turnNo }),
);
if (json) {
for (const p of pages) {
write(JSON.stringify({ turn: turnNo, ...p }) + '\n');
}
} else {
for (const p of pages) {
write(formatVolunteeredPage(p) + '\n');
}
}
}
} finally {
process.off('SIGINT', onSigint);
rl?.close();
}
}
+21
View File
@@ -169,6 +169,14 @@ export interface GBrainConfig {
retrieval_reflex?: boolean;
/** Max pointers injected per turn (default 3). File-plane only. */
retrieval_reflex_max_pointers?: number;
/**
* v0.43 (#2095) — how many recent turns the reflex extracts entities from
* (default 4). 1 reproduces the legacy current-turn-only behavior (and the
* legacy slug+title suppression). File-plane / env
* (GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS) only — same plane as the other
* reflex knobs.
*/
retrieval_reflex_window_turns?: number;
embedding_image_ocr?: boolean;
embedding_image_ocr_model?: string;
@@ -533,6 +541,10 @@ export function loadConfig(): GBrainConfig | null {
...(process.env.GBRAIN_RETRIEVAL_REFLEX
? { retrieval_reflex: !(process.env.GBRAIN_RETRIEVAL_REFLEX === 'false' || process.env.GBRAIN_RETRIEVAL_REFLEX === '0') }
: {}),
...(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS &&
Number.isFinite(Number(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS))
? { retrieval_reflex_window_turns: Number(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS) }
: {}),
...(process.env.GBRAIN_REMOTE_CLIENT_SECRET && fileConfig?.remote_mcp
? { remote_mcp: { ...fileConfig.remote_mcp, oauth_client_secret: process.env.GBRAIN_REMOTE_CLIENT_SECRET } }
: {}),
@@ -892,6 +904,15 @@ export const KNOWN_CONFIG_KEYS: readonly string[] = [
// Link resolution (issue #972)
'link_resolution',
'link_resolution.global_basename',
// Spend controls (v0.42.42.0, issue #2139). Previously `--force`-only — the
// operator had to discover these by reading source. Registered so `config
// set` accepts them directly. See docs/operations/spend-controls.md.
'spend.posture',
'sync.cost_gate_min_usd',
'sync.federated_v2',
'embed.backfill_cooldown_min',
'embed.backfill_max_usd_per_source_24h',
'embed.backfill_max_usd',
];
/**
+26
View File
@@ -179,6 +179,29 @@ function getLastUserText(messages: AgentMessage[]): string {
return '';
}
/**
* v0.43 (#2095): the rolling extraction window — the most recent
* user/assistant turns (oldest → newest, current turn last), capped here at
* a generous max; the reflex slices to its configured
* retrieval_reflex_window_turns (default 4).
*/
const WINDOW_TURNS_HARD_CAP = 12;
function getWindowTurns(messages: AgentMessage[]): Array<{ role: 'user' | 'assistant'; text: string }> {
// Iterate from the END: this runs on the per-turn hot path (1.5s reflex
// budget) and only the last 12 turns matter — flattening every content
// block of a multi-hundred-turn session just to slice the tail would make
// the cost grow with session length.
const out: Array<{ role: 'user' | 'assistant'; text: string }> = [];
for (let i = messages.length - 1; i >= 0 && out.length < WINDOW_TURNS_HARD_CAP; i--) {
const m = messages[i];
if (m?.role !== 'user' && m?.role !== 'assistant') continue;
const text = messageText(m.content);
if (!text) continue;
out.push({ role: m.role, text });
}
return out.reverse();
}
/**
* Joined text of everything the agent has ALREADY seen — every message EXCEPT
* the current turn (the last user message). Used for "already in context"
@@ -649,6 +672,9 @@ export function createGBrainContextEngine(ctx: {
workspaceDir,
currentUserText: getLastUserText(messages),
priorContextText: getPriorContextText(messages),
// v0.43 (#2095): rolling window — assistant-introduced entities and
// named-antecedent follow-ups from recent turns now resolve too.
windowTurns: getWindowTurns(messages),
resolveEntities: ctx.resolveEntities,
});
+87 -4
View File
@@ -7,12 +7,15 @@
* touching the brain, so it must be fast (one regex pass) and precision-biased:
* a false candidate costs a wasted resolve and, worse, a misleading pointer.
*
* DELIBERATE v1 limits (documented, not bugs — see issue #1981 / eng-review):
* DELIBERATE limits (documented, not bugs — see issue #1981 / eng-review):
* - Proper-case + ASCII biased. Misses lowercase names ("garry") and many
* non-Latin scripts.
* - Current-user-message only. No pronoun follow-ups ("what about her?"), no
* entities the assistant introduced.
* These are TODOs, not v1 scope. Do NOT market this as full "human-like recall".
* - extractCandidates is single-turn. The v0.43 (#2095) window layer
* (extractCandidatesFromWindow) widens extraction across the last N turns
* — assistant-introduced entities and "what about her?" follow-ups whose
* antecedent was NAMED in the window now resolve; true pronoun
* coreference (never-named antecedents) remains out of scope.
* Do NOT market this as full "human-like recall".
*
* Resolution (alias/slug lookup) lives in retrieval-reflex.ts; this module only
* decides WHAT to look up.
@@ -172,3 +175,83 @@ export function extractCandidates(text: string): EntityCandidate[] {
}
return out;
}
// ── Rolling-window extraction (v0.43, #2095 push-based context) ──────────
export interface WindowTurn {
role: 'user' | 'assistant';
text: string;
}
/**
* A window candidate with the salience metadata the volunteer layer's
* confidence boost reads: how many turns mentioned it, whether the NEWEST
* turn did, and whether the USER (vs only the assistant) ever said it.
*/
export interface WindowEntityCandidate extends EntityCandidate {
/** Number of distinct turns that mentioned this candidate. */
occurrences: number;
/** Mentioned in the newest (last) turn of the window. */
inNewestTurn: boolean;
/** Mentioned in at least one USER turn (assistant-only mentions rank lower). */
userMention: boolean;
}
/**
* Extract candidates across the last N turns (oldest → newest). Pure,
* zero-LLM: runs the per-turn extractor on each turn and merges by the
* normalizeAlias form. Ordering is salience-aware — recency of last mention,
* cross-turn frequency, and a user-role boost — so when the merged set
* exceeds MAX_CANDIDATES, the dropped tail is the stalest assistant-only
* chatter, not the entity the user just named.
*/
export function extractCandidatesFromWindow(turns: WindowTurn[]): WindowEntityCandidate[] {
if (!turns?.length) return [];
interface WAcc extends WindowEntityCandidate {
lastTurnIdx: number;
order: number;
}
const acc = new Map<string, WAcc>();
let order = 0;
const lastIdx = turns.length - 1;
for (let i = 0; i < turns.length; i++) {
const turn = turns[i];
if (!turn?.text) continue;
for (const c of extractCandidates(turn.text)) {
const norm = normalizeAlias(c.query);
if (!norm) continue;
const existing = acc.get(norm);
if (existing) {
existing.occurrences += 1;
existing.lastTurnIdx = i;
existing.inNewestTurn = existing.inNewestTurn || i === lastIdx;
if (turn.role === 'user' && !existing.userMention) {
// First USER-said surface form beats an assistant-introduced one
// for the display label.
existing.display = c.display;
existing.userMention = true;
}
} else {
acc.set(norm, {
display: c.display,
query: c.query,
occurrences: 1,
lastTurnIdx: i,
inNewestTurn: i === lastIdx,
userMention: turn.role === 'user',
order: order++,
});
}
}
}
// Salience weight: recency dominates, then frequency, then user-role.
// Deterministic tie-break on first-seen order.
const weight = (c: WAcc) =>
(c.lastTurnIdx + 1) / turns.length + Math.min(c.occurrences, 4) * 0.1 + (c.userMention ? 0.15 : 0);
return Array.from(acc.values())
.sort((a, b) => weight(b) - weight(a) || a.order - b.order)
.slice(0, MAX_CANDIDATES)
.map(({ lastTurnIdx: _l, order: _o, ...rest }) => rest);
}
+83 -8
View File
@@ -20,7 +20,12 @@ import { join } from 'node:path';
import { mkdirSync, appendFileSync } from 'node:fs';
import { loadConfig, type GBrainConfig } from '../config.ts';
import type { BrainEngine } from '../engine.ts';
import { extractCandidates, type EntityCandidate } from './entity-salience.ts';
import {
extractCandidates,
extractCandidatesFromWindow,
type EntityCandidate,
type WindowTurn,
} from './entity-salience.ts';
import {
resolveEntitiesToPointers,
DEFAULT_MAX_POINTERS,
@@ -28,22 +33,69 @@ import {
} from './retrieval-reflex.ts';
import { resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } from './resolve-ipc.ts';
/** Host capability shape (D1=A): candidates in, pointers out. Narrow by design. */
/** Per-turn resolver options shared by every rung of the ladder. */
export interface ResolveEntitiesOpts {
priorContextText?: string;
maxPointers?: number;
/** v0.43 (#2095): 'slug-only' under windowing — see ResolvePointersOpts. */
suppression?: 'slug-and-title' | 'slug-only';
}
/**
* Host capability shape (D1=A): candidates in, pointers out. Narrow by design.
*
* CONTRACT (red-team): a host resolver MUST honor `opts.suppression`. Under
* windowing the orchestrator passes 'slug-only' — a resolver that keeps
* applying the legacy title-whole-word rule will suppress every entity merely
* mentioned in a prior window turn and silently disable the feature. Hosts
* built against the pre-window contract should be upgraded or pinned to
* `retrieval_reflex_window_turns: 1`. (A capability/version gate so the
* orchestrator can detect a stale host is a filed TODO.)
*/
export type ResolveEntitiesFn = (
candidates: EntityCandidate[],
opts: { priorContextText?: string; maxPointers?: number },
opts: ResolveEntitiesOpts,
) => Promise<PointerBlock | null>;
export interface ReflexParams {
workspaceDir: string;
/** The current turn's user text (drives extraction). */
/** The current turn's user text (drives extraction when no window is given). */
currentUserText: string;
/** Joined PRIOR turns + loaded page bodies — EXCLUDES the current turn (suppression). */
priorContextText: string;
/**
* v0.43 (#2095): recent turns (oldest → newest, current turn last). When
* present and the configured window is > 1, extraction widens to the last
* N turns (assistant-introduced entities + named-antecedent follow-ups now
* resolve) and suppression switches to slug-only (codex D7 — the title rule
* would suppress every entity merely mentioned in a prior window turn).
*/
windowTurns?: WindowTurn[];
/** Host-provided resolver, if the OpenClaw plugin contract supplied one. */
resolveEntities?: ResolveEntitiesFn;
}
/** Default extraction window (turns). 1 = legacy current-turn-only. */
export const DEFAULT_WINDOW_TURNS = 4;
export function windowTurnCount(cfg: GBrainConfig | null): number {
// Env plane is read DIRECTLY here (mirroring reflexEnabled's direct
// process.env read), not just via loadConfig's env→config mapping. When
// there's no config file AND no DATABASE_URL, loadConfig() returns null and
// drops that mapping entirely — so without this, the documented
// GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS escape hatch would be silently
// ignored and the window would fall back to the default of 4 (a real
// config-less-environment bug, e.g. a clean CI shard with no brain).
const env = process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS;
if (env != null && env !== '') {
const e = Number(env);
if (Number.isFinite(e) && e >= 1) return Math.floor(e);
}
const n = cfg?.retrieval_reflex_window_turns;
if (typeof n === 'number' && Number.isFinite(n) && n >= 1) return Math.floor(n);
return DEFAULT_WINDOW_TURNS;
}
const TIMEOUT_MS = 1500; // generous per-turn ceiling; the work is usually <100ms
const HEARTBEAT_PATH = join(homedir(), '.gbrain', 'integrations', 'retrieval-reflex', 'heartbeat.jsonl');
@@ -68,14 +120,37 @@ export async function buildReflexAddition(params: ReflexParams): Promise<string
const cfg = loadConfig();
if (!reflexEnabled(cfg)) return null;
// Zero-candidate fast path: one regex pass, no brain touch.
const candidates = extractCandidates(params.currentUserText);
// v0.43 (#2095): widen extraction across the last N turns when a window
// is supplied and configured > 1. Window=1 reproduces the legacy
// current-turn-only behavior exactly (including suppression mode).
const windowN = windowTurnCount(cfg);
const windowed = windowN > 1 && (params.windowTurns?.length ?? 0) > 0;
const candidates: EntityCandidate[] = windowed
? extractCandidatesFromWindow(params.windowTurns!.slice(-windowN))
: extractCandidates(params.currentUserText);
// Zero-candidate fast path: regex passes only, no brain touch.
if (!candidates.length) return null;
const opts = { priorContextText: params.priorContextText, maxPointers: maxPointers(cfg) };
const opts: ResolveEntitiesOpts = {
priorContextText: params.priorContextText,
maxPointers: maxPointers(cfg),
suppression: windowed ? 'slug-only' : 'slug-and-title',
};
const block = await withTimeout(resolve(params, cfg, candidates, opts), TIMEOUT_MS);
if (!block || !block.pointers.length) return null;
// Accept-side reflex-channel logging (red-team): the block survived the
// per-turn timeout, so these pointers ARE being injected. Only the
// direct-Postgres rung has an engine here; the IPC rung logs server-side
// at delivery; host-injected resolvers can't log (documented gap).
if (!params.resolveEntities && isPostgres(cfg)) {
const engine = await getPostgresEngine(cfg);
if (engine) {
const { logDeliveredReflexPointers } = await import('./retrieval-reflex.ts');
logDeliveredReflexPointers(engine, block.pointers);
}
}
writeHeartbeat(cfg, block.pointers.length);
return block.text;
} catch {
@@ -87,7 +162,7 @@ async function resolve(
params: ReflexParams,
cfg: GBrainConfig | null,
candidates: EntityCandidate[],
opts: { priorContextText?: string; maxPointers?: number },
opts: ResolveEntitiesOpts,
): Promise<PointerBlock | null> {
// 1. Host capability (any engine).
if (params.resolveEntities) {
+19 -1
View File
@@ -35,6 +35,8 @@ export interface ResolveRequest {
priorContextText?: string;
maxPointers?: number;
sourceId?: string;
/** v0.43 (#2095, codex D7): suppression mode — 'slug-only' under windowing. */
suppression?: 'slug-and-title' | 'slug-only';
}
export type ResolveHandler = (req: ResolveRequest) => Promise<PointerBlock | null>;
@@ -98,6 +100,13 @@ export async function resolveViaIpc(
export async function startResolveIpcServer(
socketPath: string,
handler: ResolveHandler,
/**
* v0.43 (#2095, red-team): fired ONLY after the response was successfully
* written to the client — the accept-side seam for reflex-channel feedback
* logging. A block the client never received (timeout, dead socket) was
* never injected into a prompt and must not count as "volunteered".
*/
onDelivered?: (block: PointerBlock, req: ResolveRequest) => void,
): Promise<net.Server | null> {
// Remove a stale socket file if present (a previous serve that didn't clean up).
cleanupStaleSocket(socketPath);
@@ -113,14 +122,23 @@ export async function startResolveIpcServer(
if (nl < 0) return;
const line = buf.slice(0, nl);
let resp: string;
let delivered: { block: PointerBlock; req: ResolveRequest } | null = null;
try {
const req = JSON.parse(line) as ResolveRequest;
const block = await handler(req);
resp = JSON.stringify({ ok: true, block });
if (block) delivered = { block, req };
} catch (e) {
resp = JSON.stringify({ ok: false, error: (e as Error).message });
}
try { conn.write(resp + '\n'); } catch { /* client gone */ }
try {
conn.write(resp + '\n');
// Write accepted — the client (250ms budget) may still have hung
// up, but this is the closest observable delivery point.
if (delivered && onDelivered) {
try { onDelivered(delivered.block, delivered.req); } catch { /* telemetry only */ }
}
} catch { /* client gone — do NOT log undelivered pointers */ }
conn.end();
});
conn.on('error', () => { try { conn.destroy(); } catch { /* noop */ } });
+154 -35
View File
@@ -34,10 +34,40 @@ import type { EntityCandidate } from './entity-salience.ts';
export const DEFAULT_MAX_POINTERS = 3;
const SYNOPSIS_MAX = 160;
/** Which resolution arm produced a pointer (provenance → honest confidence). */
export type ResolveArm = 'alias' | 'title' | 'slug-suffix';
/**
* v0.43 (#2095) — arm → confidence. Lives HERE, next to the arm definitions,
* so arm identity and its score can't drift apart (eng-review note). The
* volunteer layer imports these; small deterministic boosts (multi-turn /
* newest-turn mention) are added on top there.
*/
export const ARM_CONFIDENCE: Record<ResolveArm, number> = {
alias: 0.9,
title: 0.8,
'slug-suffix': 0.6,
};
export interface ReflexPointer {
display: string;
slug: string;
/** Which brain source the page lives in (federated callers need it for dedup). */
source_id: string;
synopsis: string;
/** Resolution provenance (v0.43 #2095). */
arm: ResolveArm;
/** Base arm confidence (ARM_CONFIDENCE[arm]); callers may boost. */
confidence: number;
/**
* normalizeAlias form of the CANDIDATE that resolved this pointer (v0.43
* #2095) — lets the volunteer layer join pointers back to window-salience
* metadata without guessing from the display label (which falls back to the
* page title when the candidate surface differs, e.g. alias "Swami" →
* title "Swami X"). Absent only when suffix classification couldn't
* recover the source candidate.
*/
matchedNorm?: string;
}
export interface PointerBlock {
@@ -55,10 +85,30 @@ export interface ResolvePointersOpts {
* message's own mention would suppress every pointer (eng-review/Codex fix).
*/
priorContextText?: string;
/**
* v0.43 (#2095, codex D7) — suppression mode.
* 'slug-and-title' (default, window=1 legacy): suppress when the slug OR
* the page title appears whole-word in prior context.
* 'slug-only' (REQUIRED under multi-turn windowing): suppress on slug
* presence only. Slugs only enter context when a pointer block or page
* body was actually injected; a bare mention of "Alice Example" in a
* prior turn never contains `people/alice-example`. The title rule
* would suppress every entity merely MENTIONED in a prior window turn
* — breaking window extraction by construction.
*/
suppression?: 'slug-and-title' | 'slug-only';
/**
* v0.43 (#2095) — federated scope: resolve across these sources instead of
* the single positional sourceId. Precedence mirrors sourceScopeOpts
* (federated array > scalar). Alias arm loops per source; the title/slug
* arm uses source_id = ANY(...) in one query.
*/
sourceIds?: string[];
}
interface PageRow {
slug: string;
source_id: string;
title: string;
type: string | null;
frontmatter: Record<string, unknown> | null;
@@ -88,40 +138,63 @@ export async function resolveEntitiesToPointers(
const titlesLc: string[] = [];
const exactSlugs: string[] = [];
const slugSuffixes: string[] = [];
// Reverse maps for arm-2 provenance (which candidate produced a row) —
// populated in this same pass so the derivations happen exactly once.
const titleToNorm = new Map<string, string>();
const slugToNorm = new Map<string, string>();
for (const c of candidates) {
const norm = normalizeAlias(c.query);
if (!norm) continue;
if (!displayByNorm.has(norm)) displayByNorm.set(norm, c.display);
aliasNorms.push(norm);
titlesLc.push(c.query.toLowerCase());
const tl = c.query.toLowerCase();
titlesLc.push(tl);
if (!titleToNorm.has(tl)) titleToNorm.set(tl, norm);
const s = slugify(c.query);
if (s) {
exactSlugs.push(s);
slugSuffixes.push(`%/${s}`);
if (!slugToNorm.has(s)) slugToNorm.set(s, norm);
}
}
if (!aliasNorms.length) return null;
// Ordered set of resolved slugs (alias hits first → higher confidence).
const resolvedSlugs: string[] = [];
// Federated scope (v0.43 #2095): explicit sourceIds win over the scalar.
const sourceIds = opts.sourceIds?.length ? opts.sourceIds : [sourceId];
// Ordered set of resolved (source, slug) pairs with arm provenance —
// alias hits pushed first → higher confidence.
const resolved: Array<{ slug: string; source_id: string; arm: ResolveArm; matchedNorm?: string }> = [];
const seen = new Set<string>();
const pushSlug = (slug: string) => {
if (slug && !seen.has(slug)) {
seen.add(slug);
resolvedSlugs.push(slug);
// Neither source ids nor slugs contain spaces, so a space separator is safe.
const keyOf = (src: string, slug: string) => `${src} ${slug}`;
const push = (slug: string, src: string, arm: ResolveArm, matchedNorm?: string) => {
if (!slug) return;
const k = keyOf(src, slug);
if (!seen.has(k)) {
seen.add(k);
resolved.push({ slug, source_id: src, arm, matchedNorm });
}
};
// Arm 1 — alias-first. Unambiguous single-slug hits only. Guarded: pre-v110
// brains throw "relation page_aliases does not exist" — swallow and continue.
try {
const aliasMap = await engine.resolveAliases(aliasNorms, { sourceId });
// Arm 1 — alias-first. Unambiguous single-slug hits only, per source (no
// engine-interface change for federation). Guarded: pre-v110 brains throw
// "relation page_aliases does not exist" — swallow and continue.
// Per-source lookups are independent — run them concurrently so a
// federated caller (M granted sources) pays one RTT, not M sequential
// ones (~71ms each cross-region; the reflex runs under a 1.5s budget).
// Results are folded back in sourceIds order so pointer ordering stays
// deterministic. Per-source failures degrade independently (pre-v110
// brains have no page_aliases table).
const aliasResults = await Promise.allSettled(
sourceIds.map((src) => engine.resolveAliases(aliasNorms, { sourceId: src })),
);
for (let i = 0; i < sourceIds.length; i++) {
const r = aliasResults[i];
if (r.status !== 'fulfilled') continue;
for (const norm of aliasNorms) {
const hits = aliasMap.get(norm);
if (hits && hits.length === 1) pushSlug(hits[0].slug);
const hits = r.value.get(norm);
if (hits && hits.length === 1) push(hits[0].slug, sourceIds[i], 'alias', norm);
}
} catch {
/* no page_aliases table (pre-v110) — degrade to the title/slug arm */
}
// Arm 2 — exact title OR slug-suffix. This is the recall fix: a bare "Alice
@@ -130,53 +203,72 @@ export async function resolveEntitiesToPointers(
let rows: PageRow[] = [];
try {
rows = await engine.executeRaw<PageRow>(
`SELECT slug, title, type, frontmatter, compiled_truth
`SELECT slug, source_id, title, type, frontmatter, compiled_truth
FROM pages
WHERE deleted_at IS NULL
AND source_id = $1
AND source_id = ANY($1::text[])
AND ( lower(title) = ANY($2::text[])
OR slug = ANY($3::text[])
OR slug LIKE ANY($4::text[]) )`,
[sourceId, titlesLc, exactSlugs, slugSuffixes],
[sourceIds, titlesLc, exactSlugs, slugSuffixes],
);
} catch {
rows = [];
}
// Hydrate alias-resolved slugs too (their bodies for the synopsis) if not in rows.
const rowBySlug = new Map<string, PageRow>();
for (const r of rows) rowBySlug.set(r.slug, r);
const aliasOnly = resolvedSlugs.filter((s) => !rowBySlug.has(s));
// Hydrate alias-resolved pages too (their bodies for the synopsis) if not in rows.
const rowByKey = new Map<string, PageRow>();
for (const r of rows) rowByKey.set(keyOf(r.source_id, r.slug), r);
const aliasOnly = resolved.filter((p) => !rowByKey.has(keyOf(p.source_id, p.slug)));
if (aliasOnly.length) {
try {
const extra = await engine.executeRaw<PageRow>(
`SELECT slug, title, type, frontmatter, compiled_truth
`SELECT slug, source_id, title, type, frontmatter, compiled_truth
FROM pages
WHERE deleted_at IS NULL AND source_id = $1 AND slug = ANY($2::text[])`,
[sourceId, aliasOnly],
WHERE deleted_at IS NULL AND source_id = ANY($1::text[]) AND slug = ANY($2::text[])`,
[sourceIds, aliasOnly.map((p) => p.slug)],
);
for (const r of extra) rowBySlug.set(r.slug, r);
for (const r of extra) rowByKey.set(keyOf(r.source_id, r.slug), r);
} catch {
/* ignore — alias slug may be stale */
}
}
// Title/slug matches that weren't alias hits, appended after alias hits.
for (const r of rows) pushSlug(r.slug);
// Arm provenance per row is classified in JS (codex D8) — the combined OR
// can't report which predicate matched: an exact lower(title) hit is the
// 'title' arm; anything else got in via slug / slug-suffix.
const titleSet = new Set(titlesLc);
for (const r of rows) {
const titleLc = (r.title ?? '').toLowerCase();
if (titleSet.has(titleLc)) {
push(r.slug, r.source_id, 'title', titleToNorm.get(titleLc));
} else {
// Slug arm: exact slugified-candidate match, else suffix scan.
const tail = r.slug.includes('/') ? r.slug.slice(r.slug.lastIndexOf('/') + 1) : r.slug;
push(r.slug, r.source_id, 'slug-suffix', slugToNorm.get(r.slug) ?? slugToNorm.get(tail));
}
}
// Build pointers in confidence order, applying suppression + cap.
const suppression = opts.suppression ?? 'slug-and-title';
const pointers: ReflexPointer[] = [];
for (const slug of resolvedSlugs) {
const row = rowBySlug.get(slug);
for (const { slug, source_id, arm, matchedNorm } of resolved) {
const row = rowByKey.get(keyOf(source_id, slug));
if (!row) continue;
// Suppression: already present in PRIOR context (slug or title). The current
// turn is deliberately excluded from priorContextText.
// Suppression: already present in PRIOR context. The current turn is
// deliberately excluded from priorContextText. Under windowing
// ('slug-only', codex D7) only the slug counts — a slug appears in prior
// context only when a pointer/page was actually surfaced there, while a
// title appears on any bare mention.
if (priorLc) {
const titleLc = (row.title ?? '').toLowerCase();
if (priorLc.includes(slug.toLowerCase())) continue;
if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue;
if (suppression === 'slug-and-title') {
const titleLc = (row.title ?? '').toLowerCase();
if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue;
}
}
const display = displayForRow(row, displayByNorm);
const synopsis = safeSynopsis(row);
pointers.push({ display, slug, synopsis });
pointers.push({ display, slug, source_id, synopsis, arm, confidence: ARM_CONFIDENCE[arm], matchedNorm });
if (pointers.length >= maxPointers) break;
}
@@ -246,3 +338,30 @@ export function renderPointerBlock(pointers: ReflexPointer[]): string {
}
return lines.join('\n');
}
/**
* v0.43 (#2095, codex D11 + red-team) — ambient-channel feedback logging,
* ACCEPT-SIDE ONLY. Called by the delivery points (the serve IPC server after
* a successful write; the direct-Postgres reflex rung after its per-turn
* timeout admitted the block) — never inside the resolver itself, because a
* pointer block that timed out client-side was NEVER injected into a prompt,
* and logging it would inflate "volunteered" counts and drag the measured
* precision toward zero (corrupting the exact stats users tune
* min_confidence with).
*/
export function logDeliveredReflexPointers(engine: BrainEngine, pointers: ReflexPointer[]): void {
if (!pointers.length) return;
void import('./volunteer-events.ts')
.then(({ logVolunteerEventsFireAndForget, volunteerEventRowsFrom }) => {
logVolunteerEventsFireAndForget(
engine,
volunteerEventRowsFrom(
pointers.map((p) => ({ ...p, rationale: `${p.arm} match "${p.display}"` })),
{ channel: 'reflex' },
),
);
})
.catch(() => {
/* telemetry only */
});
}
+186
View File
@@ -0,0 +1,186 @@
/**
* v0.43 (#2095) — context_volunteer_events: the feedback-loop log behind
* push-based context. One row per page the brain VOLUNTEERED, written
* fire-and-forget by the volunteer_context op, the retrieval-reflex pointer
* path (channel 'reflex'), and `gbrain watch` (channel 'watch').
*
* "Used" is DERIVED, never written: a volunteered page counts as used when
* pages.last_retrieved_at > volunteered_at (the existing bumpLastRetrievedAt
* write-back on get_page/search/query is the open/cite signal). The join is
* approximate by design — last-retrieved is 5-min throttled (false negatives)
* and unrelated reads match too (false positives); stats output carries the
* caveat.
*
* Retention: rows older than VOLUNTEER_EVENTS_TTL_DAYS are pruned by the
* dream cycle's purge phase so conversation-adjacent telemetry never grows
* unbounded. rationale is a deterministic template that may embed the matched
* entity's surface form (which by construction resolved to an existing
* alias/title/slug) — never free conversation text.
*/
import type { BrainEngine } from './../engine.ts';
import { registerBackgroundWorkDrainer } from '../background-work.ts';
export const VOLUNTEER_EVENTS_TTL_DAYS = 90;
export type VolunteerChannel = 'op' | 'reflex' | 'watch';
/**
* Map volunteered pages to event rows for one channel — the ONE place the
* VolunteerEventRow shape is assembled (op / reflex / watch all call this,
* so adding a column is a one-site change).
*/
export function volunteerEventRowsFrom(
pages: Array<{ source_id: string; slug: string; confidence: number; arm: string; rationale: string }>,
opts: { channel: VolunteerChannel; session_id?: string | null; turn?: number | null },
): VolunteerEventRow[] {
return pages.map((p) => ({
source_id: p.source_id,
slug: p.slug,
confidence: p.confidence,
match_arm: p.arm,
rationale: p.rationale,
channel: opts.channel,
session_id: opts.session_id ?? null,
turn: opts.turn ?? null,
}));
}
export interface VolunteerEventRow {
source_id: string;
slug: string;
confidence: number;
match_arm: string;
rationale: string;
channel: VolunteerChannel;
session_id?: string | null;
turn?: number | null;
}
/**
* ONE multi-row parameterized INSERT for a batch of volunteered pages (max 5
* per call by the volunteer cap) — never per-row awaited INSERTs (up to 5
* RTTs ≈ 355ms on a cross-region deployment; eng-review D4). Throws on
* failure; callers run it through the volunteer-events background-work sink
* with try/catch so logging can never fail the op.
*/
export async function insertVolunteerEvents(
engine: BrainEngine,
rows: VolunteerEventRow[],
): Promise<void> {
if (!rows.length) return;
const params: unknown[] = [];
const tuples = rows.map((r) => {
const base = params.length;
params.push(
r.source_id,
r.slug,
r.confidence,
r.match_arm,
r.rationale,
r.channel,
r.session_id ?? null,
r.turn ?? null,
);
const ph = Array.from({ length: 8 }, (_, i) => `$${base + i + 1}`);
return `(${ph.join(', ')})`;
});
await engine.executeRaw(
`INSERT INTO context_volunteer_events
(source_id, slug, confidence, match_arm, rationale, channel, session_id, turn)
VALUES ${tuples.join(', ')}`,
params,
);
}
// ── Fire-and-forget sink (eng-review D4) ─────────────────────────────────
// Mirrors src/core/last-retrieved.ts: track every dangling INSERT promise in
// a module Set, register a drainer so finishCliTeardown settles them
// against a live engine before teardown on EVERY CLI exit path (the commit-1
// drain hoist). Logging failure never fails the caller.
const pendingVolunteerEventWrites = new Set<Promise<unknown>>();
/**
* Log volunteered pages without blocking the hot path. The batched INSERT
* runs as a tracked dangling promise; errors are swallowed (pre-v117 brains,
* transient DB failures — the volunteer result is unaffected).
*/
export function logVolunteerEventsFireAndForget(
engine: BrainEngine,
rows: VolunteerEventRow[],
): void {
if (!rows.length) return;
const p = insertVolunteerEvents(engine, rows).catch(() => {
/* best-effort telemetry — never surfaces */
});
pendingVolunteerEventWrites.add(p);
void p.finally(() => pendingVolunteerEventWrites.delete(p));
}
/** Drain pending event writes (bounded). Same snapshot semantics as last-retrieved. */
export async function awaitPendingVolunteerEventWrites(
timeoutMs = 5_000,
): Promise<{ unfinished: number }> {
if (pendingVolunteerEventWrites.size === 0) return { unfinished: 0 };
const snapshot = Array.from(pendingVolunteerEventWrites);
let timer: ReturnType<typeof setTimeout> | undefined;
const timeout = new Promise<'timeout'>((resolve) => {
timer = setTimeout(() => resolve('timeout'), timeoutMs);
});
const drain = Promise.allSettled(snapshot).then(() => 'drained' as const);
const outcome = await Promise.race([drain, timeout]);
if (timer) clearTimeout(timer);
if (outcome === 'timeout') {
const unfinished = pendingVolunteerEventWrites.size;
// Drop the snapshot so a long-lived process (`gbrain watch`) doesn't
// accumulate references to forever-pending work (last-retrieved C1).
for (const p of snapshot) pendingVolunteerEventWrites.delete(p);
return { unfinished };
}
return { unfinished: 0 };
}
// Registered in the enqueue-owning module (background-work contract): module
// not imported ⇒ nothing enqueued ⇒ nothing to drain. Order 4 — after facts /
// last-retrieved / search-cache / eval-capture; bare INSERTs, no abort.
registerBackgroundWorkDrainer({
name: 'volunteer-events',
order: 4,
drain: (ms) => awaitPendingVolunteerEventWrites(ms),
});
/** Test seam — clears the pending set so each test starts clean. */
export function _resetPendingVolunteerEventWritesForTests(): void {
pendingVolunteerEventWrites.clear();
}
/** Test seam — peek the current pending count. */
export function _peekPendingVolunteerEventWritesForTests(): number {
return pendingVolunteerEventWrites.size;
}
/**
* 90-day GC, called from the dream cycle's purge phase (mirrors
* purgeStaleCheckpoints). Best-effort: returns 0 on any failure (pre-v117
* brains have no table yet).
*/
export async function purgeStaleVolunteerEvents(
engine: BrainEngine,
ttlDays = VOLUNTEER_EVENTS_TTL_DAYS,
): Promise<number> {
try {
const rows = await engine.executeRaw<{ count: string | number }>(
`WITH deleted AS (
DELETE FROM context_volunteer_events
WHERE volunteered_at < now() - ($1 || ' days')::interval
RETURNING 1
)
SELECT count(*)::text AS count FROM deleted`,
[String(ttlDays)],
);
return Number(rows[0]?.count ?? 0);
} catch {
return 0;
}
}
+268
View File
@@ -0,0 +1,268 @@
/**
* v0.43 (#2095) — push-based context: the brain VOLUNTEERS relevant pages
* from a rolling conversation window instead of waiting to be asked.
*
* window text ─ parseWindow() ─→ turns[] ─ extractCandidatesFromWindow()
* │ (recency + frequency +
* │ user-role weights)
* ▼
* resolveEntitiesToPointers(sourceIds[]) alias 0.9 / title 0.8 /
* │ slug-suffix 0.6 (+0.05 boost
* ▼ for ≥2-turn or newest-turn
* gate min_confidence (default 0.7) → mentions)
* suppression (slug-only — windowing) →
* cap (3 default / 5 max)
*
* Zero-LLM, deterministic, precision-biased: push noise is worse than pull
* silence (#2095). At the default gate, slug-suffix matches (0.6+0.05 < 0.7)
* never volunteer — they need an explicit lower min_confidence.
*
* Consumed by three channels: the volunteer_context op, the retrieval-reflex
* window path, and `gbrain watch`. Event logging lives in
* volunteer-events.ts; usage stats here derive "used" from
* pages.last_retrieved_at — APPROXIMATE by design (the 5-min last-retrieved
* throttle causes false negatives; unrelated reads cause false positives).
*/
import type { BrainEngine } from '../engine.ts';
import { normalizeAlias } from '../search/alias-normalize.ts';
import {
extractCandidatesFromWindow,
type WindowTurn,
type WindowEntityCandidate,
} from './entity-salience.ts';
import {
resolveEntitiesToPointers,
ARM_CONFIDENCE,
type ResolveArm,
} from './retrieval-reflex.ts';
export const VOLUNTEER_DEFAULT_MAX_PAGES = 3;
export const VOLUNTEER_MAX_PAGES_CAP = 5;
export const VOLUNTEER_DEFAULT_MIN_CONFIDENCE = 0.7;
/** Deterministic boost for ≥2-turn or newest-turn mentions. */
export const VOLUNTEER_SALIENCE_BOOST = 0.05;
export interface VolunteeredPage {
slug: string;
source_id: string;
display: string;
confidence: number;
arm: ResolveArm;
/** Deterministic template string — never raw conversation text. */
rationale: string;
synopsis: string;
}
export interface VolunteerOpts {
/** Resolved source scope (federated array > scalar — sourceScopeOpts shape). */
sourceIds: string[];
/** Prior context (already-surfaced pointers/pages) for slug-only suppression. */
priorContext?: string;
/**
* Slugs to skip BEFORE the confidence gate and the maxPages cap (O(1)
* membership). `gbrain watch` passes its session-dedupe set here — a
* post-call filter would let a recurring already-pushed entity consume cap
* slots every turn and starve new pages behind it (red-team finding).
*/
excludeSlugs?: ReadonlySet<string>;
maxPages?: number;
minConfidence?: number;
}
/** Shared wire protocol for window turns — watch.ts imports this so the two
* channels can never desynchronize on the prefix grammar. */
export const TURN_PREFIX_RE = /^(user|assistant)\s*:\s?(.*)$/i;
/**
* Lenient window parser: `user:` / `assistant:` line prefixes start a new
* turn (oldest → newest); unprefixed lines continue the current turn; input
* with no prefixes at all is ONE user turn (so `echo "..." | volunteer-context`
* just works). CRLF-tolerant. Empty/whitespace input → [].
*/
export function parseWindow(text: string): WindowTurn[] {
if (!text || !text.trim()) return [];
const turns: WindowTurn[] = [];
let current: WindowTurn | null = null;
for (const rawLine of text.split(/\r?\n/)) {
const m = TURN_PREFIX_RE.exec(rawLine);
if (m) {
if (current) turns.push(current);
current = { role: m[1].toLowerCase() as WindowTurn['role'], text: m[2] ?? '' };
} else if (current) {
current.text += (current.text ? '\n' : '') + rawLine;
} else if (rawLine.trim()) {
// No prefix seen yet — accumulate into an implicit user turn.
current = { role: 'user', text: rawLine };
}
}
if (current) turns.push(current);
// Trim trailing whitespace-only turn bodies.
return turns
.map((t) => ({ role: t.role, text: t.text.trim() }))
.filter((t) => t.text.length > 0);
}
function clampMaxPages(n: number | undefined): number {
if (typeof n !== 'number' || !Number.isFinite(n) || n < 1) return VOLUNTEER_DEFAULT_MAX_PAGES;
return Math.min(Math.floor(n), VOLUNTEER_MAX_PAGES_CAP);
}
function rationaleFor(arm: ResolveArm, display: string, c: WindowEntityCandidate | undefined, windowSize: number): string {
const armText =
arm === 'alias' ? `alias match "${display}"`
: arm === 'title' ? `exact title match "${display}"`
: `slug match "${display}"`;
if (!c) return armText;
const parts = [armText];
if (c.occurrences >= 2) parts.push(`mentioned in ${c.occurrences} of last ${windowSize} turns`);
else if (c.inNewestTurn) parts.push('mentioned in the newest turn');
if (!c.userMention) parts.push('assistant-introduced');
return parts.join('; ');
}
/**
* Volunteer confidence-gated pages for a conversation window. Pure read —
* event logging is the CALLER's job (through the volunteer-events sink).
* Non-relational, zero-LLM; returns [] when nothing clears the gate.
*/
export async function volunteerContext(
engine: BrainEngine,
turns: WindowTurn[],
opts: VolunteerOpts,
): Promise<VolunteeredPage[]> {
if (!turns.length || !opts.sourceIds?.length) return [];
const candidates = extractCandidatesFromWindow(turns);
if (!candidates.length) return [];
const byNorm = new Map<string, WindowEntityCandidate>();
for (const c of candidates) {
const norm = normalizeAlias(c.query);
if (norm && !byNorm.has(norm)) byNorm.set(norm, c);
}
const maxPages = clampMaxPages(opts.maxPages);
const minConfidence =
typeof opts.minConfidence === 'number' && opts.minConfidence >= 0 && opts.minConfidence <= 1
? opts.minConfidence
: VOLUNTEER_DEFAULT_MIN_CONFIDENCE;
// Resolve up to the hard cap so the confidence gate sees the full pool —
// a gated-out alias hit must not shadow a passing title hit behind it.
const block = await resolveEntitiesToPointers(engine, opts.sourceIds[0], candidates, {
sourceIds: opts.sourceIds,
priorContextText: opts.priorContext,
suppression: 'slug-only',
maxPointers: VOLUNTEER_MAX_PAGES_CAP * 2,
});
if (!block) return [];
const out: VolunteeredPage[] = [];
for (const p of block.pointers) {
if (opts.excludeSlugs?.has(p.slug)) continue; // before gate + cap — see VolunteerOpts
// matchedNorm is the resolver's provenance join-key (the candidate that
// resolved the pointer); display-based lookup is the fallback for the
// rare suffix rows where provenance couldn't be recovered.
const cand = (p.matchedNorm ? byNorm.get(p.matchedNorm) : undefined) ?? byNorm.get(normalizeAlias(p.display));
const boost = cand && (cand.occurrences >= 2 || cand.inNewestTurn) ? VOLUNTEER_SALIENCE_BOOST : 0;
const confidence = Math.min(0.99, ARM_CONFIDENCE[p.arm] + boost);
if (confidence < minConfidence) continue;
out.push({
slug: p.slug,
source_id: p.source_id,
display: p.display,
confidence,
arm: p.arm,
rationale: rationaleFor(p.arm, p.display, cand, turns.length),
synopsis: p.synopsis,
});
if (out.length >= maxPages) break;
}
return out;
}
/**
* Canonical human rendering of one volunteered page — shared by
* `gbrain volunteer-context` (cli.ts formatResult) and `gbrain watch` so the
* two surfaces can't drift.
*/
export function formatVolunteeredPage(p: VolunteeredPage): string {
return (
`${p.display}${p.slug} (${p.confidence.toFixed(2)}, ${p.arm}) — ${p.rationale}` +
(p.synopsis ? `\n ${p.synopsis}` : '')
);
}
// ── Usage stats (the feedback loop) ──────────────────────────────────────
export interface VolunteerArmStats {
match_arm: string;
channel: string;
volunteered: number;
used: number;
/** used / volunteered, 0 when nothing volunteered. */
precision: number;
}
export interface VolunteerUsageStats {
days: number;
/** The join is approximate — see the note (false +/- documented in #2095 D9). */
approximate: true;
note: string;
total_volunteered: number;
total_used: number;
by_arm: VolunteerArmStats[];
}
export const VOLUNTEER_STATS_NOTE =
'approximate: "used" = pages.last_retrieved_at > volunteered_at. The 5-min ' +
'last-retrieved throttle causes false negatives; unrelated reads of the same ' +
'page cause false positives.';
/**
* Per-arm/channel precision over the last N days, source-scoped. Read-only;
* returns zeroed stats on pre-v117 brains (no table).
*/
export async function volunteerUsageStats(
engine: BrainEngine,
sourceIds: string[],
days = 30,
): Promise<VolunteerUsageStats> {
const safeDays = Number.isFinite(days) && days > 0 ? Math.floor(days) : 30;
let rows: Array<{ match_arm: string; channel: string; volunteered: string | number; used: string | number }> = [];
try {
rows = await engine.executeRaw(
`SELECT e.match_arm, e.channel,
count(*)::text AS volunteered,
count(*) FILTER (WHERE p.last_retrieved_at > e.volunteered_at)::text AS used
FROM context_volunteer_events e
LEFT JOIN pages p
ON p.source_id = e.source_id AND p.slug = e.slug AND p.deleted_at IS NULL
WHERE e.source_id = ANY($1::text[])
AND e.volunteered_at > now() - ($2 || ' days')::interval
GROUP BY e.match_arm, e.channel
ORDER BY e.match_arm, e.channel`,
[sourceIds, String(safeDays)],
);
} catch {
rows = []; // pre-v117 brain — table doesn't exist yet
}
const by_arm: VolunteerArmStats[] = rows.map((r) => {
const volunteered = Number(r.volunteered);
const used = Number(r.used);
return {
match_arm: r.match_arm,
channel: r.channel,
volunteered,
used,
precision: volunteered > 0 ? Number((used / volunteered).toFixed(3)) : 0,
};
});
return {
days: safeDays,
approximate: true,
note: VOLUNTEER_STATS_NOTE,
total_volunteered: by_arm.reduce((s, a) => s + a.volunteered, 0),
total_used: by_arm.reduce((s, a) => s + a.used, 0),
by_arm,
};
}
+13 -1
View File
@@ -1285,6 +1285,16 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
} catch {
// Non-fatal.
}
// v0.43 (#2095) — 90-day GC of the volunteered-context feedback log.
// Conversation-adjacent telemetry must never grow unbounded. Best-effort:
// purgeStaleVolunteerEvents returns 0 on pre-v117 brains (no table).
let purgedVolunteerEvents = 0;
try {
const { purgeStaleVolunteerEvents } = await import('./context/volunteer-events.ts');
purgedVolunteerEvents = await purgeStaleVolunteerEvents(engine);
} catch {
// Non-fatal.
}
return {
phase: 'purge',
status: 'ok',
@@ -1293,7 +1303,8 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
`purged ${purgedSources.length} source(s), ${purgedPages.count} page(s), ` +
`${purgedClones.count} orphan clone temp dir(s), ${purgedCheckpoints} stale op_checkpoint(s), ` +
`${purgedBrainstormCheckpoints} stale brainstorm checkpoint(s), ` +
`and ${purgedBatchRetryAuditFiles} stale batch-retry audit file(s)`,
`${purgedBatchRetryAuditFiles} stale batch-retry audit file(s), ` +
`and ${purgedVolunteerEvents} stale volunteer event(s)`,
details: {
purged_sources_count: purgedSources.length,
purged_pages_count: purgedPages.count,
@@ -1304,6 +1315,7 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise<Phas
purged_checkpoints_count: purgedCheckpoints,
purged_brainstorm_checkpoints_count: purgedBrainstormCheckpoints,
purged_batch_retry_audit_files_count: purgedBatchRetryAuditFiles,
purged_volunteer_events_count: purgedVolunteerEvents,
},
};
} catch (e) {
+23 -3
View File
@@ -35,6 +35,7 @@
*/
import type { BrainEngine } from './engine.ts';
import { MinionQueue } from './minions/queue.ts';
import { parseUsdLimit, resolveSpendPosture, type SpendPosture } from './spend-posture.ts';
export const COOLDOWN_CONFIG_KEY = 'embed.backfill_cooldown_min';
export const SPEND_CAP_CONFIG_KEY = 'embed.backfill_max_usd_per_source_24h';
@@ -57,6 +58,13 @@ export interface SubmitEmbedBackfillResult {
spend24hUsd?: number;
/** Set when status === 'spend_capped'. Active cap. */
spendCapUsd?: number;
/**
* Set true when `spend.posture=tokenmax` waved the job past the 24h spend
* cap (#2139). The spend is still LEDGERED by the per-job BudgetTracker
* posture removes the ceiling, not the accounting. Cooldown is NOT bypassed
* (it's queue-churn protection, not a spend gate).
*/
spendCapBypassed?: boolean;
}
export interface SubmitEmbedBackfillOpts {
@@ -72,6 +80,8 @@ export interface SubmitEmbedBackfillOpts {
nowMs?: number;
/** Job priority. Default 5 (lower than autopilot's 0; above default jobs). */
priority?: number;
/** Override the resolved spend posture (tests). Default: read from config. */
postureOverride?: SpendPosture;
}
/**
@@ -133,9 +143,12 @@ export async function submitEmbedBackfill(
const cooldownMin =
opts.cooldownMinOverride ??
(await readIntConfig(engine, COOLDOWN_CONFIG_KEY, DEFAULT_COOLDOWN_MIN));
// v0.42.42.0 (#2139): spend cap honors `off`/`unlimited`/`none` → Infinity.
// `0` still falls back to the default (off semantics ≠ 0).
const spendCap =
opts.spendCapUsdOverride ??
(await readIntConfig(engine, SPEND_CAP_CONFIG_KEY, DEFAULT_SPEND_CAP_USD));
(raw => parseUsdLimit(raw, DEFAULT_SPEND_CAP_USD))(await engine.getConfig(SPEND_CAP_CONFIG_KEY));
const posture = opts.postureOverride ?? (await resolveSpendPosture(engine));
// ── Source-level cooldown ─────────────────────────────────────
// Block re-submission if (a) an embed-backfill is currently active for this
@@ -172,9 +185,14 @@ export async function submitEmbedBackfill(
}
// ── 24h rolling spend cap ─────────────────────────────────────
// v0.42.42.0 (#2139): `spend.posture=tokenmax` waves past the cap (the
// operator declared cost isn't the constraint). The per-job BudgetTracker
// still ledgers the spend — posture removes the ceiling, not the accounting.
// An `off`/`unlimited` cap (Infinity) is likewise never tripped.
const spend24hFn = opts.spend24hFn ?? defaultSpend24hForSource;
const spend24h = await spend24hFn(engine, sourceId);
if (spend24h >= spendCap) {
const spendCapBypassed = posture === 'tokenmax' && spend24h >= spendCap;
if (spend24h >= spendCap && !spendCapBypassed) {
return {
status: 'spend_capped',
spend24hUsd: spend24h,
@@ -194,7 +212,9 @@ export async function submitEmbedBackfill(
},
);
return { status: 'submitted', jobId: job.id };
return spendCapBypassed
? { status: 'submitted', jobId: job.id, spendCapBypassed: true, spend24hUsd: spend24h }
: { status: 'submitted', jobId: job.id };
}
/** Round timestamp down to the nearest `bucketMs` boundary. */
+12 -5
View File
@@ -218,16 +218,23 @@ export function willEmbedSynchronously(opts: {
}
/**
* Pure cost-gate decision. The gate BLOCKS (prompt in TTY, exit 2 envelope
* in non-TTY) only when embed runs inline AND the estimated spend exceeds
* the floor. Deferred mode NEVER blocks the backfill cap is the real
* money gate, and blocking the cheap markdown import for cost the import
* doesn't synchronously incur is the bug this fix removes.
* Pure cost-gate decision. The gate BLOCKS (prompt in TTY, auto-defer in
* non-TTY) only when embed runs inline AND the estimated spend exceeds the
* floor. Deferred mode NEVER blocks the backfill cap is the real money gate,
* and blocking the cheap markdown import for cost the import doesn't
* synchronously incur is the bug this fix removes.
*
* v0.42.42.0 (#2139): `spend.posture=tokenmax` makes the gate INFORMATIONAL
* the operator has declared cost isn't the constraint, so it never blocks
* (the caller prints the estimate and proceeds inline). An `off`/`unlimited`
* floor (Infinity) is likewise never exceeded.
*/
export function shouldBlockSync(
costUsd: number,
floorUsd: number,
mode: SyncEmbedMode,
posture: 'gated' | 'tokenmax' = 'gated',
): boolean {
if (posture === 'tokenmax') return false;
return mode === 'inline' && costUsd > floorUsd;
}
+26 -1
View File
@@ -131,7 +131,7 @@ export interface CloneOpts {
export class GitOperationError extends Error {
constructor(
public op: 'clone' | 'pull' | 'remote_get_url',
public op: 'clone' | 'pull' | 'fetch' | 'remote_get_url',
message: string,
public cause?: unknown,
) {
@@ -217,6 +217,31 @@ export function pullRepo(repoPath: string, opts: { timeoutMs?: number } = {}): v
}
}
/**
* Fetch a single remote branch with the SAME SSRF-defensive flags + no-prompt
* env as cloneRepo/pullRepo (GIT_SSRF_FLAGS, --no-recurse-submodules,
* GIT_TERMINAL_PROMPT=0). Used by the sync cost-estimator's fetch-first path
* (#2139) so a cost preview / dry-run never hits a remote through a
* less-protected route than real sync. Throws GitOperationError on failure;
* the estimator catches and falls back to local HEAD.
*/
export function fetchRemote(repoPath: string, branch: string, opts: { timeoutMs?: number } = {}): void {
const args: string[] = ['-C', repoPath, ...GIT_SSRF_FLAGS, 'fetch', ...GIT_SSRF_SUBCOMMAND_FLAGS, 'origin', branch];
try {
execFileSync('git', args, {
stdio: ['ignore', 'pipe', 'pipe'],
timeout: opts.timeoutMs ?? 30_000,
env: { ...process.env, ...GIT_ENV },
});
} catch (e) {
throw new GitOperationError(
'fetch',
`git fetch failed in ${repoPath}: ${(e as Error).message}`,
e,
);
}
}
export type RepoState =
| 'healthy'
| 'missing'
+38
View File
@@ -5232,6 +5232,44 @@ export const MIGRATIONS: Migration[] = [
ON code_edges_chunk (from_symbol_qualified);
`,
},
{
// Renumbered 116 -> 117 at merge: master's v0.42.41.0 triage wave
// claimed v116 (code_edges_source_backfill_and_callee_index) first.
version: 117,
name: 'context_volunteer_events_table',
// #2095 push-based context: feedback-loop log of every page the brain
// VOLUNTEERED (via the volunteer_context op, the retrieval-reflex pointer
// path, or `gbrain watch`). "Used" is derived later by joining
// pages.last_retrieved_at > volunteered_at — no second write path.
// session_id/turn are caller-supplied attribution (nullable). rationale is
// a deterministic template string, NEVER raw conversation text. Rows are
// pruned past 90 days by the dream cycle's purge phase
// (purgeStaleVolunteerEvents). No ::jsonb anywhere. RLS: covered by the
// v35 auto_rls_on_create_table event trigger on Postgres (same as
// v110/v115 tables); pinned by the volunteer-context Postgres e2e.
// Created empty; plain CREATE INDEX is instant — no CONCURRENTLY needed.
// Keep in sync with src/schema.sql, src/core/pglite-schema.ts,
// src/core/schema-embedded.ts.
idempotent: true,
sql: `
CREATE TABLE IF NOT EXISTS context_volunteer_events (
id BIGSERIAL PRIMARY KEY,
source_id TEXT NOT NULL,
slug TEXT NOT NULL,
confidence DOUBLE PRECISION NOT NULL,
match_arm TEXT NOT NULL,
rationale TEXT NOT NULL DEFAULT '',
channel TEXT NOT NULL DEFAULT 'op',
session_id TEXT,
turn INTEGER,
volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx
ON context_volunteer_events (source_id, volunteered_at DESC);
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx
ON context_volunteer_events (source_id, slug);
`,
},
];
export const LATEST_VERSION = MIGRATIONS.length > 0
+15 -5
View File
@@ -40,6 +40,7 @@ import { type DbPacer, createDbPacer, createNoopPacer } from '../../db-pacer.ts'
import { resolvePaceMode, loadPaceModeConfig, readPaceEnv } from '../../pace-mode.ts';
import type { BrainEngine } from '../../engine.ts';
import type { MinionJobContext } from '../types.ts';
import { parseUsdLimit, usdLimitToCap, resolveSpendPosture } from '../../spend-posture.ts';
import { embedBackfillLockId, EMBED_BACKFILL_LOCK_TTL_MIN } from '../../embed-backfill-lock.ts';
@@ -95,12 +96,21 @@ async function resolveBackfillPacer(
}
}
/** Read embed.backfill_max_usd config or default. */
async function readMaxUsd(engine: BrainEngine): Promise<number> {
/**
* Resolve the per-job budget cap (USD) for the BudgetTracker.
*
* v0.42.42.0 (#2139): returns `undefined` = "no cap" (which BudgetTracker
* treats as cap-absent) when the config is `off`/`unlimited`/`none` OR when
* `spend.posture=tokenmax`. NEVER returns Infinity (that would pass through as
* a real ceiling and serialize to `null` in audit rows). Spend is still
* ledgered by the tracker either way posture removes the ceiling, not the
* accounting. `0`/garbage fall back to the $10 default.
*/
async function readMaxUsd(engine: BrainEngine): Promise<number | undefined> {
const posture = await resolveSpendPosture(engine);
if (posture === 'tokenmax') return undefined;
const raw = await engine.getConfig('embed.backfill_max_usd');
if (raw === null || raw === undefined) return DEFAULT_MAX_USD_PER_JOB;
const n = Number(raw);
return Number.isFinite(n) && n > 0 ? n : DEFAULT_MAX_USD_PER_JOB;
return usdLimitToCap(parseUsdLimit(raw, DEFAULT_MAX_USD_PER_JOB));
}
/** Validate + extract typed job params. Throws on malformed input. */
+95
View File
@@ -3145,6 +3145,99 @@ const get_recent_salience: Operation = {
cliHints: { name: 'salience' },
};
// --- v0.43 (#2095): push-based context — the brain volunteers pages ---
const volunteer_context: Operation = {
name: 'volunteer_context',
description:
'Push-based context: volunteer brain pages relevant to a rolling conversation window ' +
'WITHOUT being asked. Zero-LLM, confidence-gated (alias 0.9 / exact-title 0.8 / ' +
'slug-suffix 0.6, +0.05 for multi-turn or newest-turn mentions; default gate 0.7), ' +
'capped at 3 pages (max 5). Returns pointers with one-line rationales + synopses — ' +
'open the page (get_page) before relying on details. Pass stats: true for the ' +
'approximate volunteered-vs-used precision summary (the feedback loop).',
scope: 'read',
params: {
window: {
type: 'string',
description:
"Recent conversation turns, oldest → newest, as 'user:' / 'assistant:' prefixed " +
'lines (unprefixed text = one user turn). Required unless stats: true. ' +
'CLI: piped stdin fills this.',
},
prior_context: {
type: 'string',
description:
'Already-surfaced context (pointer blocks / opened page bodies). Pages whose slug ' +
'appears here are not re-volunteered.',
},
max_pages: { type: 'number', description: 'Max pages to volunteer (default 3, hard cap 5).' },
min_confidence: {
type: 'number',
description:
'Confidence gate 0..1 (default 0.7 — slug-suffix matches need an explicit lower gate).',
},
session_id: { type: 'string', description: 'Optional caller session id, logged for attribution.' },
turn: { type: 'number', description: 'Optional caller turn number, logged for attribution.' },
stats: {
type: 'boolean',
description:
'Return the volunteered-vs-used precision summary instead of volunteering. ' +
'APPROXIMATE: "used" = pages.last_retrieved_at > volunteered_at.',
},
days: { type: 'number', description: 'Stats window in days (default 30; stats mode only).' },
},
handler: async (ctx, p) => {
const { parseWindow, volunteerContext, volunteerUsageStats } = await import('./context/volunteer.ts');
const scope = sourceScopeOpts(ctx);
const sourceIds = scope.sourceIds ?? (scope.sourceId ? [scope.sourceId] : ['default']);
if (p.stats === true) {
return volunteerUsageStats(ctx.engine, sourceIds, typeof p.days === 'number' ? p.days : undefined);
}
if (typeof p.window !== 'string' || !p.window.trim()) {
throw new OperationError(
'invalid_params',
'window is required unless stats: true',
'Pass the recent turns as a string (CLI: pipe them on stdin), or use --stats.',
);
}
const turns = parseWindow(p.window);
const pages = await volunteerContext(ctx.engine, turns, {
sourceIds,
priorContext: typeof p.prior_context === 'string' ? p.prior_context : undefined,
maxPages: typeof p.max_pages === 'number' ? p.max_pages : undefined,
minConfidence: typeof p.min_confidence === 'number' ? p.min_confidence : undefined,
});
// Feedback-loop logging: fire-and-forget batched INSERT through the
// volunteer-events sink (drained at exit). Never fails the op.
if (pages.length) {
try {
const { logVolunteerEventsFireAndForget, volunteerEventRowsFrom } = await import('./context/volunteer-events.ts');
// Trust-boundary clamps (remote MCP callers): cap session_id length so
// a read-scoped token can't bank unbounded TEXT per request, and only
// log integer turns — a non-integer would throw inside the single
// multi-row INSERT and silently drop the whole batch.
const sessionId = typeof p.session_id === 'string' ? p.session_id.slice(0, 256) : null;
const turn =
typeof p.turn === 'number' && Number.isInteger(p.turn) && Math.abs(p.turn) <= 2_147_483_647
? p.turn
: null;
logVolunteerEventsFireAndForget(
ctx.engine,
volunteerEventRowsFrom(pages, { channel: 'op', session_id: sessionId, turn }),
);
} catch {
/* telemetry only */
}
}
return { pages, count: pages.length, window_turns: turns.length };
},
cliHints: { name: 'volunteer-context', stdin: 'window' },
};
const find_anomalies: Operation = {
name: 'find_anomalies',
description: FIND_ANOMALIES_DESCRIPTION,
@@ -4856,6 +4949,8 @@ export const operations: Operation[] = [
whoami, sources_add, sources_list, sources_remove, sources_status,
// v0.29: Salience + anomalies + recent transcripts
get_recent_salience, find_anomalies, get_recent_transcripts,
// v0.43 (#2095): push-based context
volunteer_context,
// v0.31: hot memory (facts table)
extract_facts, recall, forget_fact,
// v0.32.6: contradiction probe MCP surface (M3)
+6
View File
@@ -264,6 +264,12 @@ export class PGLiteEngine implements BrainEngine {
}
}
// NOTE (#2084): PGLite's Emscripten runtime writes the WASM backend's
// proc_exit status into `process.exitCode` (initdb here at create-time,
// the postmaster at close-time), and the writes land asynchronously —
// a snapshot/restore around these awaits does NOT contain them. That is
// why the CLI's exit paths read gbrain's own verdict
// (cli-force-exit.ts currentExitCode), never ambient process.exitCode.
try {
this._db = await preservingProcessExitCode(() =>
PGlite.create({
+21
View File
@@ -945,6 +945,27 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE
);
-- #2095 push-based context: feedback-loop log of volunteered pages.
-- Mirrors migration v117 + src/schema.sql. "Used" derives from
-- pages.last_retrieved_at > volunteered_at; 90-day prune in the dream
-- cycle's purge phase.
CREATE TABLE IF NOT EXISTS context_volunteer_events (
id BIGSERIAL PRIMARY KEY,
source_id TEXT NOT NULL,
slug TEXT NOT NULL,
confidence DOUBLE PRECISION NOT NULL,
match_arm TEXT NOT NULL,
rationale TEXT NOT NULL DEFAULT '',
channel TEXT NOT NULL DEFAULT 'op',
session_id TEXT,
turn INTEGER,
volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx
ON context_volunteer_events (source_id, volunteered_at DESC);
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx
ON context_volunteer_events (source_id, slug);
-- ============================================================
-- migration_impact_log (v0.41.18.0 gbrain onboard wave)
-- ============================================================
+22
View File
@@ -711,6 +711,28 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE
);
-- #2095 push-based context: feedback-loop log of volunteered pages
-- (volunteer_context op / retrieval-reflex pointers / gbrain watch).
-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is
-- a deterministic template string, never raw conversation text. 90-day prune
-- in the dream cycle's purge phase. Mirrors migration v117.
CREATE TABLE IF NOT EXISTS context_volunteer_events (
id BIGSERIAL PRIMARY KEY,
source_id TEXT NOT NULL,
slug TEXT NOT NULL,
confidence DOUBLE PRECISION NOT NULL,
match_arm TEXT NOT NULL,
rationale TEXT NOT NULL DEFAULT '',
channel TEXT NOT NULL DEFAULT 'op',
session_id TEXT,
turn INTEGER,
volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx
ON context_volunteer_events (source_id, volunteered_at DESC);
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx
ON context_volunteer_events (source_id, slug);
-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676)
-- because its \`job_id BIGINT REFERENCES minion_jobs(id)\` FK requires
-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix.
+106
View File
@@ -0,0 +1,106 @@
/**
* Spend posture + USD-limit parsing the single spend-control surface for
* gbrain's cost gates (issue #2139). Two concerns live here:
*
* 1. `spend.posture` (DB-plane config): `'gated'` (default) makes every cost
* gate behave as before; `'tokenmax'` makes them INFORMATIONAL print the
* estimate, proceed, and keep ledgering spend. The operator who sets
* `tokenmax` has declared "cost is not my constraint." Posture removes the
* CEILING, never the ACCOUNTING (the spend ledger still records every
* dollar). It is deliberately SEPARATE from `search.mode=tokenmax` (which
* governs retrieval payload size, not embedding spend); the gate prints a
* hint linking the two when only the search mode is set.
*
* 2. `parseUsdLimit` / `formatUsdLimit`: first-class `off` / `unlimited` /
* `none` on the USD gate knobs (so operators stop setting sentinel values
* like `100000`). The parse layer represents "no limit" as `Infinity` so
* comparisons stay special-case-free (`cost > Infinity` is never true).
* `formatUsdLimit` renders it as the string `'unlimited'` NEVER serialize
* raw `Infinity`, because `JSON.stringify(Infinity)` emits `null`, which is
* ambiguous in audit/ledger rows. At the budget-machinery boundary, callers
* convert `Infinity` `undefined` ("no cap"), which `BudgetTracker` already
* treats as cap-absent.
*
* LEAF module: imports only the engine type so any cost-gate site can pull it
* in without a circular dependency.
*/
import type { BrainEngine } from './engine.ts';
export const SPEND_POSTURE_CONFIG_KEY = 'spend.posture';
export type SpendPosture = 'gated' | 'tokenmax';
/**
* Resolve `spend.posture` from DB-plane config. Fail-open to `'gated'` on a
* missing/unknown value or a config-read error a posture probe must never
* crash a sync, and an unrecognized value must never silently disable gates.
*/
export async function resolveSpendPosture(engine: BrainEngine): Promise<SpendPosture> {
try {
const raw = await engine.getConfig(SPEND_POSTURE_CONFIG_KEY);
return normalizeSpendPosture(raw);
} catch {
return 'gated';
}
}
/** Pure normalizer (testable without an engine). Anything but `tokenmax` → `gated`. */
export function normalizeSpendPosture(raw: unknown): SpendPosture {
if (typeof raw === 'string' && raw.trim().toLowerCase() === 'tokenmax') return 'tokenmax';
return 'gated';
}
/** True iff `raw` is a valid `spend.posture` value (for `config set` validation). */
export function isValidSpendPosture(raw: unknown): boolean {
return typeof raw === 'string' && ['gated', 'tokenmax'].includes(raw.trim().toLowerCase());
}
const OFF_TOKENS = new Set(['off', 'unlimited', 'none']);
/**
* Parse a USD-limit config value.
* - `'off'` / `'unlimited'` / `'none'` (case-insensitive) `Infinity` (no limit)
* - a finite positive number (or `0` when `allowZero`) that number
* - anything else (garbage, negative, empty, NaN) `def`
*
* `allowZero` distinguishes the two knob semantics:
* - `sync.cost_gate_min_usd` uses `allowZero: true` `0` means "block on any
* nonzero spend" (a real operator choice). `off` is the no-limit escape.
* - the backfill caps reject `0` (fall back to the default); only `off`
* disables them. `off` semantics `0` (issue #2139).
*/
export function parseUsdLimit(
raw: unknown,
def: number,
opts: { allowZero?: boolean } = {},
): number {
if (raw === null || raw === undefined) return def;
if (typeof raw === 'string') {
const t = raw.trim().toLowerCase();
if (t === '') return def;
if (OFF_TOKENS.has(t)) return Infinity;
}
const n = Number(raw);
if (!Number.isFinite(n)) return def;
if (n < 0) return def;
if (n === 0) return opts.allowZero ? 0 : def;
return n;
}
/**
* Render a USD limit for human/JSON output. `Infinity` `'unlimited'` (NEVER
* the raw value `JSON.stringify(Infinity)` is `null`). Finite values are
* returned as-is so callers can `$${formatUsdLimit(x)}` or embed in JSON.
*/
export function formatUsdLimit(n: number): string | number {
return Number.isFinite(n) ? n : 'unlimited';
}
/**
* Convert a parsed USD limit to the budget-machinery cap representation:
* `Infinity` `undefined` ("no cap", which `BudgetTracker` treats as
* cap-absent), finite the number. Keeps `null` out of ledger rows.
*/
export function usdLimitToCap(n: number): number | undefined {
return Number.isFinite(n) ? n : undefined;
}
+151
View File
@@ -0,0 +1,151 @@
/**
* Shared sync-delta machinery ONE implementation of "what changed since
* last_commit" consumed by BOTH the sync executor (`performSyncInner` in
* `src/commands/sync.ts`) and the inline-embed cost estimator
* (`src/core/sync-cost-estimate.ts`). Before this module the executor diffed
* `last_commit..pin` while the estimator priced the entire tree, so the
* gate's dollar figure had no relationship to what the sync actually embedded
* (issue #2139: a 400x overestimate that wedged the daily cron). Routing both
* through `computeSyncDelta` makes diff/manifest drift between estimate and
* execution structurally impossible.
*
* Shell-injection safe: `execFileSync` with array args (no `/bin/sh -c`), so a
* `sources.local_path` containing shell metacharacters can never escape same
* posture documented at `git-head.ts:14-19`.
*
* Fail-open ladder (never throws):
*
* computeSyncDelta(repo, from, to)
*
* `git cat-file -t <from>` throws { unavailable, anchor_missing }
* (bookmark object gc'd after a history rewrite nothing to diff;
* caller falls back to a full reconcile / full-tree ceiling)
*
* `git diff --name-status -M from..to` throws { unavailable, diff_failed }
* (oversized post-rewrite diff exceeds the 30s / 100 MiB budget)
*
* ok { ok, manifest } (+ detached working-tree manifest merged in)
*
* NOTE: a present-but-non-ancestor `from` (force-push, squash, mastermain) is
* still diffable `git diff A..B` is an endpoint-tree comparison and does NOT
* require A to be an ancestor of B (unlike a rev-walk or `A...B` merge-base).
* That is the #1970 property this module preserves.
*/
import { execFileSync } from 'node:child_process';
import { buildSyncManifest, type SyncManifest } from './sync.ts';
/** Runs a git subcommand in `repoPath` and returns trimmed stdout (throws on failure). */
export type GitRunner = (repoPath: string, args: string[]) => string;
// Mirrors `git()` + `buildGitInvocation()` in commands/sync.ts: `core.quotepath=false`
// so non-ASCII (CJK) paths arrive as UTF-8; 30s timeout; 100 MiB maxBuffer (a
// 100K-file `--name-status` diff tops out ~10-20 MiB — Node's 1 MiB default
// would ENOBUFS-crash the sync with no log line).
const DEFAULT_GIT_RUNNER: GitRunner = (repoPath, args) =>
execFileSync('git', ['-c', 'core.quotepath=false', '-C', repoPath, ...args], {
encoding: 'utf-8',
timeout: 30_000,
maxBuffer: 100 * 1024 * 1024,
}).trim();
let _gitRunner: GitRunner = DEFAULT_GIT_RUNNER;
/**
* Test seam (probe-seam pattern, matches `git-head.ts:_setGitHeadProbeForTests`)
* so tests drive `computeSyncDelta` without mocking child_process or routing
* through `mock.module` (R2-compliant). Pass `null` to restore the default.
*/
export function _setGitRunnerForTests(fn: GitRunner | null): void {
_gitRunner = fn ?? DEFAULT_GIT_RUNNER;
}
function unique<T>(items: T[]): T[] {
return [...new Set(items)];
}
/**
* Working-tree manifest for a DETACHED HEAD (relocated from sync.ts:557 so the
* estimator can price detached sources identically to how the executor imports
* them). On a detached HEAD, sync syncs from the live working tree: tracked
* changes (`git diff --name-status -M HEAD`) PLUS untracked files (`ls-files
* --others --exclude-standard`). Attached HEADs never call this — their
* incremental path imports ONLY the commit diff (untracked/dirty files are not
* imported), which is why the estimator must not price dirty files on an
* attached repo (issue #2139 phantom-cost class).
*/
export function buildDetachedWorkingTreeManifest(
repoPath: string,
run: GitRunner = _gitRunner,
): SyncManifest {
const manifest = buildSyncManifest(run(repoPath, ['diff', '--name-status', '-M', 'HEAD']));
const untracked = run(repoPath, ['ls-files', '--others', '--exclude-standard'])
.split('\n')
.filter(line => line.length > 0);
return {
added: unique([...manifest.added, ...untracked]),
modified: unique(manifest.modified),
deleted: unique(manifest.deleted),
renamed: manifest.renamed,
};
}
export type SyncDeltaResult =
| { status: 'ok'; manifest: SyncManifest }
| { status: 'unavailable'; reason: 'anchor_missing' | 'diff_failed' };
export interface ComputeSyncDeltaOpts {
/**
* Pre-computed detached working-tree manifest to merge into the commit diff
* (the executor already builds one for its `up_to_date` gate; pass it to
* avoid a redundant `git diff HEAD` + `ls-files`). When omitted and
* `detached` is true, this module builds it.
*/
detachedManifest?: SyncManifest | null;
/** Build the detached manifest internally (estimator path). Ignored if `detachedManifest` is provided. */
detached?: boolean;
}
/**
* The single source of truth for "what changed between two commits in this
* repo." Returns the RAW merged manifest (added/modified/deleted/renamed)
* callers apply their own `isSyncable` filtering + side effects.
*/
export function computeSyncDelta(
repoPath: string,
fromCommit: string,
toCommit: string,
opts: ComputeSyncDeltaOpts = {},
): SyncDeltaResult {
const run = _gitRunner;
// Reachability: a gc'd bookmark object can't be diffed (#1970).
try {
run(repoPath, ['cat-file', '-t', fromCommit]);
} catch {
return { status: 'unavailable', reason: 'anchor_missing' };
}
let diffOutput: string;
try {
diffOutput = run(repoPath, ['diff', '--name-status', '-M', `${fromCommit}..${toCommit}`]);
} catch {
return { status: 'unavailable', reason: 'diff_failed' };
}
const manifest = buildSyncManifest(diffOutput);
const detached =
opts.detachedManifest !== undefined && opts.detachedManifest !== null
? opts.detachedManifest
: opts.detached
? buildDetachedWorkingTreeManifest(repoPath, run)
: null;
if (detached) {
manifest.added = unique([...manifest.added, ...detached.added]);
manifest.modified = unique([...manifest.modified, ...detached.modified]);
manifest.deleted = unique([...manifest.deleted, ...detached.deleted]);
manifest.renamed = [...manifest.renamed, ...detached.renamed];
}
return { status: 'ok', manifest };
}
+19 -8
View File
@@ -13,7 +13,7 @@ import {
startResolveIpcServer,
cleanupStaleSocket,
} from '../core/context/resolve-ipc.ts';
import { resolveEntitiesToPointers } from '../core/context/retrieval-reflex.ts';
import { resolveEntitiesToPointers, logDeliveredReflexPointers } from '../core/context/retrieval-reflex.ts';
export async function startMcpServer(engine: BrainEngine) {
const server = new Server(
@@ -68,13 +68,24 @@ export async function startMcpServer(engine: BrainEngine) {
if (cfg?.engine === 'pglite' && cfg.database_path) {
resolveSocket = resolveSocketPath(cfg.database_path);
const defaultSource = process.env.GBRAIN_SOURCE || 'default';
resolveServer = await startResolveIpcServer(resolveSocket, (req) =>
resolveEntitiesToPointers(
engine,
req.sourceId || defaultSource,
req.candidates ?? [],
{ priorContextText: req.priorContextText, maxPointers: req.maxPointers },
),
resolveServer = await startResolveIpcServer(
resolveSocket,
(req) =>
resolveEntitiesToPointers(
engine,
req.sourceId || defaultSource,
req.candidates ?? [],
{
priorContextText: req.priorContextText,
maxPointers: req.maxPointers,
suppression: req.suppression,
},
),
// The IPC resolve path IS the ambient reflex channel. Logging happens
// at DELIVERY (post-write), not inside the resolver — a block the
// client's 250ms budget abandoned was never injected, and counting it
// would corrupt the volunteered-vs-used precision stats (red-team).
(block) => logDeliveredReflexPointers(engine, block.pointers),
);
}
} catch {
+22
View File
@@ -707,6 +707,28 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths (
FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE
);
-- #2095 push-based context: feedback-loop log of volunteered pages
-- (volunteer_context op / retrieval-reflex pointers / gbrain watch).
-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is
-- a deterministic template string, never raw conversation text. 90-day prune
-- in the dream cycle's purge phase. Mirrors migration v117.
CREATE TABLE IF NOT EXISTS context_volunteer_events (
id BIGSERIAL PRIMARY KEY,
source_id TEXT NOT NULL,
slug TEXT NOT NULL,
confidence DOUBLE PRECISION NOT NULL,
match_arm TEXT NOT NULL,
rationale TEXT NOT NULL DEFAULT '',
channel TEXT NOT NULL DEFAULT 'op',
session_id TEXT,
turn INTEGER,
volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx
ON context_volunteer_events (source_id, volunteered_at DESC);
CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx
ON context_volunteer_events (source_id, slug);
-- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676)
-- because its `job_id BIGINT REFERENCES minion_jobs(id)` FK requires
-- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix.
+34
View File
@@ -0,0 +1,34 @@
/**
* #2084 class pin every exit-code write in src/ routes through
* setCliExitVerdict.
*
* A RAW `process.exitCode = N` is silently ZEROED by the deliberate
* flush-exit: currentExitCode() reads only gbrain's owned verdict (the
* PGLite-Emscripten-pollution defense), so a command that bypasses the
* setter reports success on failure. Caught live twice in one day: doctor's
* FAIL path exited 0 after a merge introduced a raw write. Runtime variants
* are pinned in test/cli-finish-teardown.test.ts; this is the structural
* guard that catches the NEXT raw write at review time.
*/
import { describe, test, expect } from 'bun:test';
import { execSync } from 'child_process';
describe('exit-verdict ownership — no raw process.exitCode assignments', () => {
test('every exit-code write in src/ routes through setCliExitVerdict', () => {
// Two legitimate writers are exempt:
// - cli-force-exit.ts: setCliExitVerdict's own mirror-write.
// - pglite-engine.ts preservingProcessExitCode: #2141's containment
// RESTORE around PGlite.create() — it keeps the GLOBAL tidy for
// external readers and is explicitly not a verdict write (the owned
// channel never reads process.exitCode).
// Whitespace/operator-tolerant: catches `process.exitCode=1`,
// `process.exitCode ??= 1`, `process.exitCode ||= 1`, and the bracket
// form — every shape is equally zeroed by the owned-verdict read.
const hits = execSync(
String.raw`grep -rnE "process(\.|\[')exitCode('\])?[[:space:]]*([?|&]{2})?=[^=]" src --include='*.ts' | grep -v "core/cli-force-exit.ts" | grep -v "core/pglite-engine.ts" || true`,
{ encoding: 'utf-8', cwd: new URL('..', import.meta.url).pathname },
).trim();
expect(hits).toBe('');
});
});
+67
View File
@@ -0,0 +1,67 @@
/**
* v0.43 (#2095) formatResult's volunteer_context human rendering
* (ship coverage G3): pointer lines with confidence/arm/rationale,
* the empty-result message, and the approximate stats summary.
*/
import { describe, test, expect } from 'bun:test';
import { formatResult } from '../src/cli.ts';
describe('formatResult — volunteer_context', () => {
test('renders pointer lines with confidence, arm, rationale, and synopsis', () => {
const out = formatResult('volunteer_context', {
pages: [
{
slug: 'people/alice-example',
source_id: 'default',
display: 'Alice Example',
confidence: 0.85,
arm: 'title',
rationale: 'exact title match "Alice Example"; mentioned in the newest turn',
synopsis: 'Alice is an early founder.',
},
],
count: 1,
window_turns: 3,
});
expect(out).toContain('Alice Example → people/alice-example (0.85, title)');
expect(out).toContain('exact title match');
expect(out).toContain('Alice is an early founder.');
});
test('empty result explains the confidence gate', () => {
const out = formatResult('volunteer_context', { pages: [], count: 0, window_turns: 1 });
expect(out).toContain('Nothing volunteered');
expect(out).toContain('confidence gate');
});
test('stats mode renders totals, per-arm precision, and the approximate note', () => {
const out = formatResult('volunteer_context', {
days: 30,
approximate: true,
note: 'approximate: "used" = pages.last_retrieved_at > volunteered_at.',
total_volunteered: 4,
total_used: 3,
by_arm: [
{ match_arm: 'alias', channel: 'reflex', volunteered: 2, used: 2, precision: 1 },
{ match_arm: 'title', channel: 'op', volunteered: 2, used: 1, precision: 0.5 },
],
});
expect(out).toContain('last 30 day(s)');
expect(out).toContain('approximate');
expect(out).toContain('total: 4 volunteered, 3 used');
expect(out).toContain('alias/reflex: 2/2 used (precision 1)');
expect(out).toContain('title/op: 1/2 used (precision 0.5)');
});
test('stats mode with zero events prints the empty-window line', () => {
const out = formatResult('volunteer_context', {
days: 7,
approximate: true,
note: 'approximate',
total_volunteered: 0,
total_used: 0,
by_arm: [],
});
expect(out).toContain('no volunteer events in the window');
});
});
+47
View File
@@ -0,0 +1,47 @@
/**
* v0.43 (#2084) real-CLI pipe completeness pin (the incident #1959 class).
*
* The synthetic flush-mechanism coverage lives in
* test/flush-then-exit-harness.test.ts (4MB late-reader byte-complete pin).
* This file keeps the IMPLEMENTATION-AGNOSTIC check: the actual CLI, run the
* way agents run it (piped stdout), produces complete, parseable, byte-stable
* output and exits deliberately well under the teardown backstop.
*/
import { describe, test, expect } from 'bun:test';
import { spawnSync } from 'child_process';
import { join, resolve } from 'path';
const REPO = resolve(import.meta.dir, '..');
const CLI = join(REPO, 'src', 'cli.ts');
describe('cli pipe completeness — deliberate exit never truncates piped stdout (#2084)', () => {
test('real CLI: --tools-json over a pipe is complete, parseable, byte-stable, and prompt', () => {
const run = () => {
const t0 = Date.now();
const res = spawnSync('bun', [CLI, '--tools-json'], {
stdio: ['ignore', 'pipe', 'pipe'],
encoding: 'utf-8',
timeout: 60_000,
env: { ...process.env, GBRAIN_SKIP_STARTUP_HOOKS: '1' },
maxBuffer: 64 * 1024 * 1024,
});
return { stdout: res.stdout ?? '', stderr: res.stderr ?? '', status: res.status, ms: Date.now() - t0 };
};
const first = run();
expect(first.status).toBe(0);
expect(Buffer.byteLength(first.stdout, 'utf-8')).toBeGreaterThan(16 * 1024);
// Truncated JSON does not parse — the strongest single-run completeness check.
const parsed = JSON.parse(first.stdout);
expect(Array.isArray(parsed)).toBe(true);
// Deliberate exit, not the teardown backstop. A wall-clock bound is flaky
// on cold CI (bun parse alone runs 10-20s there) — the backstop's banner
// is the truthful signal, same assertion the pgbouncer e2e uses.
expect(first.stderr).not.toContain('force-exiting');
expect(first.stderr).not.toContain('did not return within');
const second = run();
expect(second.status).toBe(0);
expect(second.stdout).toBe(first.stdout);
}, 180_000);
});
+16
View File
@@ -82,4 +82,20 @@ describe('shouldForceExitAfterMain — daemon survival gate', () => {
expect(shouldForceExitAfterMain(['serves'])).toBe(true);
expect(shouldForceExitAfterMain(['serve-cluster'])).toBe(true);
});
test('awaited long-runners exit deliberately when their handler resolves', () => {
// `jobs work`, `jobs watch --follow`, `autopilot`, and `gbrain watch`
// (#2095) all BLOCK inside their awaited handler until done — when
// main() resolves for them, the work is over and the deliberate exit is
// correct (v0.43 #2084 contract). Only commands that RETURN from main()
// while the event loop carries the daemon (`serve`) belong in
// DAEMON_COMMANDS — `watch` blocks in its stdin iteration, so piped EOF
// must flow through the flush-exit instead of hanging on lingering
// sockets.
expect(shouldForceExitAfterMain(['jobs', 'work'])).toBe(true);
expect(shouldForceExitAfterMain(['jobs', 'watch', '--follow'])).toBe(true);
expect(shouldForceExitAfterMain(['autopilot'])).toBe(true);
expect(shouldForceExitAfterMain(['watch'])).toBe(true);
expect(shouldForceExitAfterMain(['watch', '--json'])).toBe(true);
});
});
+9
View File
@@ -29,6 +29,15 @@ describe('KNOWN_CONFIG_KEYS', () => {
expect(KNOWN_CONFIG_KEYS).toContain('models.tier.subagent');
});
test('contains the spend-control keys (v0.42.42.0, #2139) — no --force archaeology', () => {
expect(KNOWN_CONFIG_KEYS).toContain('spend.posture');
expect(KNOWN_CONFIG_KEYS).toContain('sync.cost_gate_min_usd');
expect(KNOWN_CONFIG_KEYS).toContain('sync.federated_v2');
expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_cooldown_min');
expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_max_usd_per_source_24h');
expect(KNOWN_CONFIG_KEYS).toContain('embed.backfill_max_usd');
});
test('no duplicate entries', () => {
const set = new Set(KNOWN_CONFIG_KEYS);
expect(set.size).toBe(KNOWN_CONFIG_KEYS.length);
+4 -1
View File
@@ -26,7 +26,10 @@ describe('resolve IPC', () => {
test('round-trip: client gets the pointer block the server returns', async () => {
const dir = tmpDir();
const sock = resolveSocketPath(dir);
const block: PointerBlock = { pointers: [{ display: 'Alice', slug: 'people/alice', synopsis: 'x' }], text: 'BLOCK' };
const block: PointerBlock = {
pointers: [{ display: 'Alice', slug: 'people/alice', source_id: 'default', synopsis: 'x', arm: 'alias', confidence: 0.9 }],
text: 'BLOCK',
};
const server = await startResolveIpcServer(sock, async (req) => {
expect(req.candidates[0].query).toBe('Alice');
return block;
+13
View File
@@ -8,11 +8,24 @@
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { configureGateway } from '../src/core/ai/gateway.ts';
import { checkFederationHealth } from '../src/commands/doctor.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
// Pin the legacy OpenAI/1536 embedding shape BEFORE initSchema builds the
// content_chunks vector column. beforeAll runs before the legacy-embedding
// preload's restoring beforeEach fires, so the gateway here is whatever the
// PRIOR file in this shard left it — if that file configured a 1280-d model
// and didn't reset, initSchema would build a vector(1280) column and the
// 1536-d coverage fixture below would fail CheckExpectedDim. Establish the
// dimension this file's fixtures assume rather than inheriting it.
configureGateway({
embedding_model: 'openai:text-embedding-3-large',
embedding_dimensions: 1536,
env: { ...process.env },
});
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
+3
View File
@@ -49,6 +49,9 @@ const ALL_TABLES = [
'page_versions',
'ingest_log',
'files',
// v0.43 (#2095): volunteered-context feedback log — no FK to pages (slug
// join), but stale rows poison stats/count assertions across runs.
'context_volunteer_events',
'pages', // last because of foreign keys
'config',
'minion_attachments',
+148
View File
@@ -0,0 +1,148 @@
/**
* v0.43 (#2084 / eng-review TD1) PgBouncer transaction-mode teardown E2E.
*
* Three consecutive waves (#1972 #2015 #2084) fixed pooler-teardown bugs
* that were verified only against one production deployment, because CI had
* no transaction-mode pooler. This file pins the bug CLASS, not exact
* timings: a CLI op against a txn-mode pooled URL must
*
* 1. exit zero with intact stdout, and
* 2. NOT ride the 10s hard-deadline backstop (the
* "[cli] engine.disconnect() did not return within 10000ms" banner is
* the smoking gun pre-#2084 it printed on 100% of query-shaped ops).
*
* Topology: docker-compose.ci.yml runs `pgbouncer` (transaction mode) in
* front of postgres-1. The test uses a DEDICATED database
* (`gbrain_pgbouncer`) created via the direct URL, so it never races the
* TRUNCATE-based fixtures any shard runs against `gbrain_test`.
*
* Gated by GBRAIN_PGBOUNCER_URL + GBRAIN_PGBOUNCER_DIRECT_URL skips
* gracefully outside the docker CI gate. Run manually:
*
* GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@localhost:6543/gbrain_pgbouncer \
* GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test \
* bun test test/e2e/pgbouncer-teardown.test.ts
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs';
import { join, resolve } from 'path';
import { tmpdir } from 'os';
import postgres from 'postgres';
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
const POOLED_URL = process.env.GBRAIN_PGBOUNCER_URL;
const DIRECT_ADMIN_URL = process.env.GBRAIN_PGBOUNCER_DIRECT_URL;
const SKIP = !POOLED_URL || !DIRECT_ADMIN_URL;
const describePooled = SKIP ? describe.skip : describe;
const REPO = resolve(import.meta.dir, '..', '..');
const TEST_DB = 'gbrain_pgbouncer';
const SLUG = 'test/pgbouncer-teardown-fixture';
const MARKER = 'pgbouncer-teardown-marker-content-7c4f';
/** Direct URL pointing at the dedicated test database (same server). */
function directTestDbUrl(): string {
const u = new URL(DIRECT_ADMIN_URL!);
u.pathname = `/${TEST_DB}`;
return u.toString();
}
async function runCli(
args: string[],
env: Record<string, string>,
timeoutMs: number,
): Promise<{ exitCode: number; stdout: string; stderr: string; wallMs: number }> {
const t0 = Date.now();
const proc = Bun.spawn(['bun', 'run', join(REPO, 'src', 'cli.ts'), ...args], {
cwd: REPO,
env: { ...process.env, ...env, GBRAIN_SKIP_STARTUP_HOOKS: '1' },
stdout: 'pipe',
stderr: 'pipe',
});
const killer = setTimeout(() => {
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
}, timeoutMs);
try {
const [stdout, stderr, exitCode] = await Promise.all([
new Response(proc.stdout).text(),
new Response(proc.stderr).text(),
proc.exited,
]);
return { exitCode, stdout, stderr, wallMs: Date.now() - t0 };
} finally {
clearTimeout(killer);
}
}
describePooled('pgbouncer txn-mode teardown (#2084 / TD1)', () => {
let home: string;
beforeAll(async () => {
// Dedicated database on the same server, created via the DIRECT url
// (CREATE DATABASE cannot run through a transaction-mode pooler).
const admin = postgres(DIRECT_ADMIN_URL!, { max: 1 });
try {
await admin.unsafe(`DROP DATABASE IF EXISTS ${TEST_DB} WITH (FORCE)`);
await admin.unsafe(`CREATE DATABASE ${TEST_DB}`);
} finally {
await admin.end({ timeout: 5 });
}
// Schema + fixture via the direct connection (DDL stays off the pooler,
// matching the production split-pool discipline).
const eng = new PostgresEngine();
await eng.connect({ engine: 'postgres', database_url: directTestDbUrl() });
await eng.initSchema();
await eng.putPage(SLUG, {
type: 'note',
title: 'PgBouncer teardown fixture',
compiled_truth: MARKER,
timeline: '',
});
await eng.disconnect();
// Brain config pointing the CLI at the POOLED url.
home = mkdtempSync(join(tmpdir(), 'gbrain-pgbouncer-'));
mkdirSync(join(home, '.gbrain'), { recursive: true });
const pooled = new URL(POOLED_URL!);
pooled.pathname = `/${TEST_DB}`;
writeFileSync(
join(home, '.gbrain', 'config.json'),
JSON.stringify({ engine: 'postgres', database_url: pooled.toString() }) + '\n',
);
}, 240_000);
afterAll(() => {
try { rmSync(home, { recursive: true, force: true }); } catch { /* best effort */ }
});
test('op against the pooled URL exits clean — output intact, no force-exit banner', async () => {
const env = { HOME: home, GBRAIN_HOME: home };
const res = await runCli(['get', SLUG], env, 90_000);
if (res.exitCode !== 0 || /force-exiting/.test(res.stderr)) {
console.error('--- stdout ---\n' + res.stdout);
console.error('--- stderr ---\n' + res.stderr);
}
expect(res.exitCode).toBe(0);
// Output is complete — the #1959 truncation class.
expect(res.stdout).toContain(MARKER);
// The smoking gun: pre-#2084 the hard-deadline banner printed every time
// a query-shaped op ran against a txn-mode pooler.
expect(res.stderr).not.toMatch(/force-exiting/);
expect(res.stderr).not.toMatch(/did not return within/);
// Generous CLASS bound (cold bun parse on CI is 10-20s): the op itself is
// milliseconds; anything that ALSO waited out a 10s teardown backstop
// lands well past this.
expect(res.wallMs).toBeLessThan(60_000);
}, 120_000);
test('second run (warm schema probe) also exits clean through the pooler', async () => {
const env = { HOME: home, GBRAIN_HOME: home };
const res = await runCli(['get', SLUG], env, 90_000);
expect(res.exitCode).toBe(0);
expect(res.stdout).toContain(MARKER);
expect(res.stderr).not.toMatch(/force-exiting/);
}, 120_000);
});
@@ -0,0 +1,97 @@
/**
* v0.43 (#2095) volunteer_context on REAL Postgres (engine parity beyond
* PGLite) + the RLS pin for the v117 table.
*
* The unit suite covers the volunteer core hermetically on PGLite; this file
* proves the same op handler against pgvector Postgres: resolution arms,
* the fire-and-forget event sink landing rows, the stats join, and that
* context_volunteer_events (migration v117) has ROW LEVEL SECURITY enabled (the v35
* auto_rls_on_create_table event trigger covers migration-created tables
* this is the assertion that keeps that mechanism honest for new tables).
*
* Gated by DATABASE_URL skips gracefully without real Postgres.
*/
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts';
import { operationsByName } from '../../src/core/operations.ts';
import {
awaitPendingVolunteerEventWrites,
_resetPendingVolunteerEventWritesForTests,
} from '../../src/core/context/volunteer-events.ts';
const SKIP = !hasDatabase();
const describePg = SKIP ? describe.skip : describe;
function mkCtx(engine: unknown, overrides: Record<string, unknown> = {}) {
return {
engine,
config: {} as never,
logger: { info: () => {}, warn: () => {}, error: () => {} } as never,
dryRun: false,
remote: false,
sourceId: 'default',
...overrides,
} as never;
}
describePg('volunteer_context on real Postgres (#2095)', () => {
beforeAll(async () => {
await setupDB();
const engine = getEngine();
await engine.putPage('people/alice-example', {
type: 'person',
title: 'Alice Example',
compiled_truth: 'Alice is an early founder.',
timeline: '',
});
}, 240_000);
afterAll(async () => {
await teardownDB();
});
test('op volunteers, logs through the sink, and stats join works', async () => {
_resetPendingVolunteerEventWritesForTests();
const engine = getEngine();
const op = operationsByName.volunteer_context;
const result = (await op.handler(mkCtx(engine), {
window: 'user: who knows the seed market?\nassistant: Alice Example does.\nuser: ask her',
session_id: 'e2e-pg',
})) as any;
expect(result.count).toBe(1);
expect(result.pages[0].slug).toBe('people/alice-example');
expect(result.pages[0].arm).toBe('title');
const { unfinished } = await awaitPendingVolunteerEventWrites(10_000);
expect(unfinished).toBe(0);
const rows = await engine.executeRaw<{ slug: string; session_id: string }>(
`SELECT slug, session_id FROM context_volunteer_events WHERE session_id = 'e2e-pg'`,
[],
);
expect(rows.length).toBe(1);
// Mark the page as opened after volunteering → stats counts it used.
await engine.executeRaw(
`UPDATE pages SET last_retrieved_at = now() + interval '1 minute' WHERE slug = 'people/alice-example'`,
[],
);
const stats = (await op.handler(mkCtx(engine), { stats: true })) as any;
expect(stats.approximate).toBe(true);
expect(stats.total_volunteered).toBeGreaterThanOrEqual(1);
expect(stats.total_used).toBeGreaterThanOrEqual(1);
}, 120_000);
test('RLS is enabled on context_volunteer_events (auto-RLS covers v117)', async () => {
const engine = getEngine();
const rows = await engine.executeRaw<{ relrowsecurity: boolean }>(
`SELECT relrowsecurity FROM pg_class
WHERE oid = 'public.context_volunteer_events'::regclass`,
[],
);
expect(rows.length).toBe(1);
expect(rows[0].relrowsecurity).toBe(true);
}, 60_000);
});
+48
View File
@@ -155,6 +155,54 @@ describe('submitEmbedBackfill — 24h spend cap', () => {
expect(result.status).toBe('spend_capped');
expect(result.spendCapUsd).toBe(5);
});
// v0.42.42.0 (#2139): off-switch + tokenmax bypass.
test('cap "off" → submits even at huge spend (Infinity cap never tripped)', async () => {
await engine.setConfig(SPEND_CAP_CONFIG_KEY, 'off');
const result = await submitEmbedBackfill(engine, 'default', {
reason: 'unit',
spend24hFn: async () => 1e9,
});
expect(result.status).toBe('submitted');
});
test('0 falls back to the default cap (off semantics ≠ 0)', async () => {
await engine.setConfig(SPEND_CAP_CONFIG_KEY, '0');
const result = await submitEmbedBackfill(engine, 'default', {
reason: 'unit',
spend24hFn: async () => 25, // == default $25 → capped
});
expect(result.status).toBe('spend_capped');
expect(result.spendCapUsd).toBe(25);
});
test('spend.posture=tokenmax bypasses the cap, marks spendCapBypassed', async () => {
const result = await submitEmbedBackfill(engine, 'default', {
reason: 'unit',
postureOverride: 'tokenmax',
spendCapUsdOverride: 25,
spend24hFn: async () => 100, // way over cap
});
expect(result.status).toBe('submitted');
expect(result.spendCapBypassed).toBe(true);
expect(result.spend24hUsd).toBe(100);
});
test('tokenmax does NOT bypass the cooldown (axis split — churn protection stays)', async () => {
const queue = new MinionQueue(engine);
const job = await queue.add('embed-backfill', { sourceId: 'default' }, {});
await engine.executeRaw(
`UPDATE minion_jobs SET status='completed', finished_at=NOW() - INTERVAL '1 minute' WHERE id=$1`,
[job.id],
);
const result = await submitEmbedBackfill(engine, 'default', {
reason: 'unit',
postureOverride: 'tokenmax',
cooldownMinOverride: 10,
spend24hFn: async () => 1e9,
});
expect(result.status).toBe('cooldown'); // posture lifts the cap, NOT the cooldown
});
});
describe('submitEmbedBackfill — source isolation', () => {
+17
View File
@@ -250,3 +250,20 @@ describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => {
expect(src).toMatch(/buildPgliteInitErrorMessage\(verdict, original\)/);
});
});
describe('v0.42.43.0 #2095 — volunteer-events sink + cycle purge wiring (structural pins)', () => {
test('volunteer-events registers a background-work drainer (order 4)', () => {
// Deleting this registration would silently drop volunteer events on
// every CLI exit with no behavioral test failing — same pin class as the
// other four sinks above.
const src = readFileSync('src/core/context/volunteer-events.ts', 'utf8');
expect(src).toMatch(/registerBackgroundWorkDrainer\(\{[\s\S]*?name:\s*'volunteer-events'/);
expect(src).toMatch(/order:\s*4/);
});
test("the dream cycle's purge phase invokes purgeStaleVolunteerEvents and reports the count", () => {
const src = readFileSync('src/core/cycle.ts', 'utf8');
expect(src).toMatch(/purgeStaleVolunteerEvents\(engine\)/);
expect(src).toMatch(/purged_volunteer_events_count/);
});
});
+64
View File
@@ -2179,3 +2179,67 @@ describe('v112 — pages_links_extracted_at', () => {
});
});
// v0.43 (#2095): context_volunteer_events — push-based-context feedback log.
describe('v117 — context_volunteer_events_table', () => {
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 60_000);
afterAll(async () => { if (engine) await engine.disconnect(); }, 60_000);
test('v117 entry exists, named + idempotent', () => {
const m = MIGRATIONS.find(x => x.version === 117);
expect(m).toBeDefined();
expect(m!.name).toBe('context_volunteer_events_table');
expect(m!.idempotent).toBe(true);
});
test('LATEST_VERSION is at or above 117', () => {
expect(LATEST_VERSION).toBeGreaterThanOrEqual(117);
});
test('table exists after initSchema with the documented columns', async () => {
const rows = await engine.executeRaw<{ column_name: string; is_nullable: string }>(
`SELECT column_name, is_nullable FROM information_schema.columns
WHERE table_name = 'context_volunteer_events' ORDER BY ordinal_position`, [],
);
const byName = new Map(rows.map(r => [r.column_name, r.is_nullable]));
for (const required of ['source_id', 'slug', 'confidence', 'match_arm', 'rationale', 'channel', 'volunteered_at']) {
expect(byName.get(required)).toBe('NO');
}
// Caller-supplied attribution columns are nullable by contract (codex D9).
expect(byName.get('session_id')).toBe('YES');
expect(byName.get('turn')).toBe('YES');
});
test('both source-scoped indexes exist after initSchema', async () => {
const rows = await engine.executeRaw<{ indexname: string }>(
`SELECT indexname FROM pg_indexes WHERE tablename = 'context_volunteer_events'`, [],
);
const names = rows.map(r => r.indexname);
expect(names).toContain('context_volunteer_events_src_time_idx');
expect(names).toContain('context_volunteer_events_src_slug_idx');
});
test('insert + 90-day purge round-trip (purgeStaleVolunteerEvents)', async () => {
const { insertVolunteerEvents, purgeStaleVolunteerEvents } = await import('../src/core/context/volunteer-events.ts');
await insertVolunteerEvents(engine, [
{ source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'alias match', channel: 'op' },
{ source_id: 'default', slug: 'projects/acme-example', confidence: 0.8, match_arm: 'title', rationale: 'exact title', channel: 'reflex', session_id: 's1', turn: 3 },
]);
// Age one row past the TTL, leave the other fresh.
await engine.executeRaw(
`UPDATE context_volunteer_events SET volunteered_at = now() - interval '120 days'
WHERE slug = 'projects/acme-example'`, [],
);
const purged = await purgeStaleVolunteerEvents(engine, 90);
expect(purged).toBe(1);
const left = await engine.executeRaw<{ slug: string }>(
`SELECT slug FROM context_volunteer_events`, [],
);
expect(left.map(r => r.slug)).toEqual(['people/alice-example']);
});
});
+255
View File
@@ -182,3 +182,258 @@ describe('context-engine assemble() — Retrieval Reflex integration', () => {
});
});
});
describe('v0.43 (#2095) — rolling window extraction through assemble()', () => {
const REFLEX_ON = { GBRAIN_RETRIEVAL_REFLEX: 'true' };
test('entity named ONLY in a previous assistant turn yields a pointer now', async () => {
await withEnv(REFLEX_ON, async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
const ce = createGBrainContextEngine({
workspaceDir: '/tmp/rr-test-ws-w1',
resolveEntities: (candidates, opts) =>
resolveEntitiesToPointers(engine, 'default', candidates, opts),
});
// Current turn is a pronoun follow-up; the antecedent was NAMED two
// turns back by the ASSISTANT. Pre-window this never fired.
const res = await ce.assemble({
sessionId: 'w1',
messages: [
{ role: 'user', content: 'who should I talk to about the seed round?' },
{ role: 'assistant', content: 'Alice Example led a similar round last year.' },
{ role: 'user', content: 'what did she invest in?' },
],
});
expect(res.systemPromptAddition).toContain('people/alice-example');
});
});
test('window=1 reproduces the legacy current-turn-only behavior', async () => {
await withEnv({ ...REFLEX_ON, GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '1' }, async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
const ce = createGBrainContextEngine({
workspaceDir: '/tmp/rr-test-ws-w2',
resolveEntities: (candidates, opts) =>
resolveEntitiesToPointers(engine, 'default', candidates, opts),
});
const res = await ce.assemble({
sessionId: 'w2',
messages: [
{ role: 'assistant', content: 'Alice Example led a similar round last year.' },
{ role: 'user', content: 'what did she invest in?' },
],
});
// Current turn has no extractable entity; window=1 must NOT widen.
expect(res.systemPromptAddition).not.toContain('people/alice-example');
});
});
test('windowed suppression is slug-only: a prior-turn MENTION does not suppress (codex D7)', async () => {
await withEnv(REFLEX_ON, async () => {
await seed('people/alice-example', 'Alice Example', 'A founder.');
const ce = createGBrainContextEngine({
workspaceDir: '/tmp/rr-test-ws-w3',
resolveEntities: (candidates, opts) =>
resolveEntitiesToPointers(engine, 'default', candidates, opts),
});
// "Alice Example" appears in a PRIOR turn (a bare mention — prior
// context contains the TITLE). Under the legacy title rule the pointer
// would be suppressed; slug-only windowing must still fire.
const res = await ce.assemble({
sessionId: 'w3',
messages: [
{ role: 'user', content: 'I met Alice Example yesterday' },
{ role: 'assistant', content: 'How did the meeting with Alice Example go?' },
{ role: 'user', content: 'she wants to invest — thoughts?' },
],
});
expect(res.systemPromptAddition).toContain('people/alice-example');
});
});
test('windowed suppression still drops an already-surfaced page (slug in prior context)', async () => {
await withEnv(REFLEX_ON, async () => {
await seed('people/alice-example', 'Alice Example', 'A founder.');
const ce = createGBrainContextEngine({
workspaceDir: '/tmp/rr-test-ws-w4',
resolveEntities: (candidates, opts) =>
resolveEntitiesToPointers(engine, 'default', candidates, opts),
});
const res = await ce.assemble({
sessionId: 'w4',
messages: [
{ role: 'assistant', content: 'Pointer: **Alice Example** → `people/alice-example` (use get_page)' },
{ role: 'user', content: 'tell me more about Alice Example' },
],
});
expect(res.systemPromptAddition).not.toContain('Brain pages mentioned this turn');
});
});
test('fail-open: a throwing resolver under windowing never breaks the turn', async () => {
await withEnv(REFLEX_ON, async () => {
await seed('people/alice-example', 'Alice Example', 'A founder.');
const ce = createGBrainContextEngine({
workspaceDir: '/tmp/rr-test-ws-w5',
resolveEntities: async () => { throw new Error('resolver exploded'); },
});
const res = await ce.assemble({
sessionId: 'w5',
messages: [
{ role: 'assistant', content: 'Alice Example is relevant here.' },
{ role: 'user', content: 'ok tell me about her' },
],
});
expect(res.systemPromptAddition).toContain('Live Context');
expect(res.systemPromptAddition).not.toContain('Brain pages mentioned this turn');
});
});
});
describe('ambient-channel event logging (codex D11 — accept-side logDeliveredReflexPointers)', () => {
test('logDeliveredReflexPointers logs channel=reflex events through the drained sink', async () => {
const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts');
const { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } =
await import('../src/core/context/volunteer-events.ts');
_resetPendingVolunteerEventWritesForTests();
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
await seed('people/alice-example', 'Alice Example', 'A founder.');
const block = await resolveEntitiesToPointers(
engine,
'default',
extractCandidates('what do you think about Alice Example?'),
{},
);
expect(block).not.toBeNull();
logDeliveredReflexPointers(engine, block!.pointers);
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
expect(unfinished).toBe(0);
const rows = await engine.executeRaw<{ channel: string; slug: string; match_arm: string }>(
'SELECT channel, slug, match_arm FROM context_volunteer_events',
[],
);
expect(rows.length).toBe(1);
expect(rows[0].channel).toBe('reflex');
expect(rows[0].slug).toBe('people/alice-example');
expect(rows[0].match_arm).toBe('title');
});
test('the bare resolver logs nothing — delivery is the only logging seam', async () => {
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
await seed('people/alice-example', 'Alice Example', 'A founder.');
const block = await resolveEntitiesToPointers(
engine,
'default',
extractCandidates('about Alice Example'),
{},
);
expect(block).not.toBeNull();
const { awaitPendingVolunteerEventWrites } = await import('../src/core/context/volunteer-events.ts');
await awaitPendingVolunteerEventWrites(5_000);
const rows = await engine.executeRaw<{ channel: string }>('SELECT channel FROM context_volunteer_events', []);
expect(rows.length).toBe(0);
});
test('logDeliveredReflexPointers with an empty pointer list is a no-op', async () => {
const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts');
const { awaitPendingVolunteerEventWrites } = await import('../src/core/context/volunteer-events.ts');
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
logDeliveredReflexPointers(engine, []);
await awaitPendingVolunteerEventWrites(5_000);
const rows = await engine.executeRaw<{ channel: string }>('SELECT channel FROM context_volunteer_events', []);
expect(rows.length).toBe(0);
});
});
describe('serve IPC wiring — suppression passthrough + reflex-channel logging (review hardening)', () => {
test('the IPC round-trip honors slug-only suppression and logs channel=reflex', async () => {
const { startResolveIpcServer, resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } =
await import('../src/core/context/resolve-ipc.ts');
const { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } =
await import('../src/core/context/volunteer-events.ts');
const { mkdtempSync, rmSync } = await import('fs');
const { join } = await import('path');
const { tmpdir } = await import('os');
_resetPendingVolunteerEventWritesForTests();
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
await seed('people/alice-example', 'Alice Example', 'A founder.');
const dir = mkdtempSync(join(tmpdir(), 'rr-ipc-'));
const sock = resolveSocketPath(dir);
// The SAME wiring shape src/mcp/server.ts uses for serve: forwards
// suppression from the request; logging happens at DELIVERY via the
// onDelivered hook (post-write), never inside the resolver.
const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts');
const server = await startResolveIpcServer(
sock,
(req) =>
resolveEntitiesToPointers(engine, req.sourceId || 'default', req.candidates ?? [], {
priorContextText: req.priorContextText,
maxPointers: req.maxPointers,
suppression: req.suppression,
}),
(block) => logDeliveredReflexPointers(engine, block.pointers),
);
expect(server).not.toBeNull();
try {
// slug-only suppression: a TITLE mention in prior context must NOT
// suppress (the windowing contract), and the resolve must log.
const block = await resolveViaIpc(sock, {
candidates: extractCandidates('tell me about Alice Example'),
priorContextText: 'earlier turn merely mentioned Alice Example',
suppression: 'slug-only',
});
expect(block).not.toBe(IPC_UNAVAILABLE);
expect(block).not.toBeNull();
expect((block as { pointers: Array<{ slug: string }> }).pointers[0].slug).toBe('people/alice-example');
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
expect(unfinished).toBe(0);
const rows = await engine.executeRaw<{ channel: string }>(
'SELECT channel FROM context_volunteer_events', [],
);
expect(rows.length).toBe(1);
expect(rows[0].channel).toBe('reflex');
} finally {
server!.close();
rmSync(dir, { recursive: true, force: true });
}
});
});
describe('windowTurnCount — knob edge semantics', () => {
test('0, negative, NaN, and absent all fall back to the default of 4 (1 = legacy off)', async () => {
const { windowTurnCount, DEFAULT_WINDOW_TURNS } = await import('../src/core/context/reflex.ts');
expect(DEFAULT_WINDOW_TURNS).toBe(4);
expect(windowTurnCount(null)).toBe(4);
expect(windowTurnCount({ retrieval_reflex_window_turns: 0 } as never)).toBe(4);
expect(windowTurnCount({ retrieval_reflex_window_turns: -3 } as never)).toBe(4);
expect(windowTurnCount({ retrieval_reflex_window_turns: Number.NaN } as never)).toBe(4);
// The documented "off" switch is 1 (legacy single-turn), not 0.
expect(windowTurnCount({ retrieval_reflex_window_turns: 1 } as never)).toBe(1);
expect(windowTurnCount({ retrieval_reflex_window_turns: 6.9 } as never)).toBe(6);
});
test('the env escape hatch is honored even when config is null (no config file / DB)', async () => {
const { windowTurnCount } = await import('../src/core/context/reflex.ts');
// loadConfig() returns null in a config-less environment (clean CI shard,
// no brain) and drops its env→config mapping — windowTurnCount must still
// read the env var directly, or the documented escape hatch is dead and
// the window silently defaults to 4. withEnv() (not raw process.env
// mutation) keeps the linter + isolation guard happy.
await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '1' }, async () => {
expect(windowTurnCount(null)).toBe(1);
});
await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '7' }, async () => {
expect(windowTurnCount(null)).toBe(7);
// Env wins over a config value too (env is the higher-precedence plane).
expect(windowTurnCount({ retrieval_reflex_window_turns: 3 } as never)).toBe(7);
});
await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: 'not-a-number' }, async () => {
// Garbage env falls through to config / default, not a crash.
expect(windowTurnCount(null)).toBe(4);
});
});
});
+53
View File
@@ -0,0 +1,53 @@
/**
* v0.42.42.0 (#2139) `--max-usd off` / `--max-cost off` uncapped-switch pins
* across enrich / onboard / reindex (the T6 secondary cost gates).
*
* The enrich arg parser carries the real logic (off Infinity sentinel, mapped
* to "no BudgetTracker ceiling" in runEnrichCore), so it gets a direct unit test.
* reindex + onboard detect `off` inline at the CLI dispatch and proceed past the
* confirmation / missing-cap refusal; those are pinned as source-level regression
* guards (the codex pre-landing review found all three half-built these keep
* them from silently regressing without standing up full CLI+gateway harnesses).
*/
import { describe, test, expect } from 'bun:test';
import { parseArgs } from '../src/commands/enrich.ts';
describe('enrich parseArgs — --max-usd off → uncapped (Infinity sentinel)', () => {
test('off / unlimited / none (case-insensitive) → Infinity', () => {
for (const v of ['off', 'OFF', 'unlimited', 'none', 'None']) {
expect(parseArgs(['--max-usd', v]).maxCostUsd).toBe(Infinity);
}
// --max-cost-usd alias too.
expect(parseArgs(['--max-cost-usd', 'off']).maxCostUsd).toBe(Infinity);
});
test('finite positive number passes through; absent → undefined; garbage → undefined', () => {
expect(parseArgs(['--max-usd', '5']).maxCostUsd).toBe(5);
expect(parseArgs([]).maxCostUsd).toBeUndefined();
expect(parseArgs(['--max-usd', 'abc']).maxCostUsd).toBeUndefined();
expect(parseArgs(['--max-usd', '0']).maxCostUsd).toBeUndefined(); // non-positive ignored
});
});
describe('reindex / onboard off-switch dispatch (regression guards)', () => {
test('reindex-code: --max-cost off proceeds past the confirmation gate', async () => {
const src = await Bun.file(new URL('../src/commands/reindex-code.ts', import.meta.url)).text();
// off sets maxCostOff and the gate proceeds when (tokenmax || maxCostOff).
expect(src).toMatch(/maxCostOff\s*=\s*true/);
expect(src).toMatch(/posture === 'tokenmax' \|\| maxCostOff/);
});
test('onboard: --max-usd off lifts the --auto missing-cap refusal', async () => {
const src = await Bun.file(new URL('../src/commands/onboard.ts', import.meta.url)).text();
expect(src).toMatch(/maxUsdOff\s*=/);
// refusal skipped when (maxUsdOff || tokenmax).
expect(src).toMatch(/maxUsdOff \|\| tokenmax/);
});
test('enrich: uncapped path maps the Infinity sentinel to no BudgetTracker ceiling', async () => {
const src = await Bun.file(new URL('../src/commands/enrich.ts', import.meta.url)).text();
// The sentinel must become `undefined` at the tracker (never raw Infinity,
// which serializes to null in audit rows).
expect(src).toMatch(/opts\.maxCostUsd === Infinity \? undefined/);
expect(src).toMatch(/uncapped \? Infinity : parsed\.maxCostUsd/);
});
});
+140
View File
@@ -0,0 +1,140 @@
/**
* v0.42.42.0 (#2139) estimateInlineNewTokens ladder coverage.
*
* The inline cost estimator now MIRRORS EXECUTION (delta, not full-tree
* ceiling) so the gate's dollar figure stops being a ~400x phantom on a busy
* brain. Real temp git repos, no PGLite. estimateInlineNewTokens is exported
* from commands/sync.ts; CHUNKER_VERSION is the live chunker version.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync } from 'fs';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { join } from 'path';
import { estimateInlineNewTokens } from '../src/commands/sync.ts';
import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts';
const CURRENT = String(CHUNKER_VERSION);
let repo: string;
function commitAll(msg: string): string {
execSync('git add -A', { cwd: repo, stdio: 'pipe' });
execSync(`git commit -m "${msg}"`, { cwd: repo, stdio: 'pipe' });
return execSync('git rev-parse HEAD', { cwd: repo, stdio: 'pipe' }).toString().trim();
}
function src(over: Partial<{ last_commit: string | null; chunker_version: string | null; config: Record<string, unknown> }> = {}) {
return {
local_path: repo,
config: over.config ?? {},
last_commit: over.last_commit ?? null,
chunker_version: over.chunker_version ?? null,
};
}
beforeEach(() => {
repo = mkdtempSync(join(tmpdir(), 'gbrain-est-'));
execSync('git init', { cwd: repo, stdio: 'pipe' });
execSync('git config user.email "t@t.com"', { cwd: repo, stdio: 'pipe' });
execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' });
mkdirSync(join(repo, 'topics'), { recursive: true });
});
afterEach(() => {
if (repo) rmSync(repo, { recursive: true, force: true });
});
describe('estimateInlineNewTokens — ladder', () => {
test('chunker drift → full-tree ceiling even with an empty git delta', () => {
writeFileSync(join(repo, 'topics/a.md'), 'some body content here');
const head = commitAll('base');
// git unchanged (last_commit == HEAD) but chunker drifted → must NOT be 0.
const r = estimateInlineNewTokens([src({ last_commit: head, chunker_version: 'STALE-0' })], CURRENT);
expect(r.tokens).toBeGreaterThan(0);
expect(r.estimateKind).toBe('ceiling');
expect(r.ceilingReasons).toContain('chunker_drift');
expect(r.changedSources).toBe(1);
});
test('[D2A headline] HEAD==last_commit + current chunker + DIRTY tree → 0 (unchanged)', () => {
writeFileSync(join(repo, 'topics/a.md'), 'body');
const head = commitAll('base');
// Dirty the tree (untracked scratch + uncommitted edit) — attached sync
// imports nothing, so the estimate must be 0. This is the exact pre-fix
// false-fire shape.
writeFileSync(join(repo, 'scratch.tmp'), 'agent scratch');
writeFileSync(join(repo, 'topics/a.md'), 'uncommitted edit');
const r = estimateInlineNewTokens([src({ last_commit: head, chunker_version: CURRENT })], CURRENT);
expect(r.tokens).toBe(0);
expect(r.estimateKind).toBe('unchanged');
expect(r.unchangedSources).toBe(1);
});
test('first sync (last_commit null) → full-tree ceiling', () => {
writeFileSync(join(repo, 'topics/a.md'), 'body content');
commitAll('base');
const r = estimateInlineNewTokens([src({ last_commit: null, chunker_version: CURRENT })], CURRENT);
expect(r.tokens).toBeGreaterThan(0);
expect(r.estimateKind).toBe('ceiling');
expect(r.ceilingReasons).toContain('first_sync');
});
test('delta rung: only changed committed files priced; deletes cost 0', () => {
writeFileSync(join(repo, 'topics/a.md'), 'a'.repeat(400));
writeFileSync(join(repo, 'topics/b.md'), 'b'.repeat(400));
const base = commitAll('base');
writeFileSync(join(repo, 'topics/a.md'), 'a'.repeat(800)); // modify
rmSync(join(repo, 'topics/b.md')); // delete → 0
commitAll('change');
const r = estimateInlineNewTokens([src({ last_commit: base, chunker_version: CURRENT })], CURRENT);
expect(r.estimateKind).toBe('delta');
expect(r.tokens).toBeGreaterThan(0); // a.md priced
// A pure-delete delta would be 0; here a.md modify keeps it > 0. Sanity:
// the magnitude is delta-scale (one ~800-char file), not full-tree.
});
test('delta rung: non-syncable changed file is filtered out (markdown strategy)', () => {
writeFileSync(join(repo, 'topics/a.md'), 'md');
const base = commitAll('base');
writeFileSync(join(repo, 'notes.txt'), 'x'.repeat(4000)); // .txt under markdown strategy → not syncable
commitAll('add txt');
const r = estimateInlineNewTokens([src({ last_commit: base, chunker_version: CURRENT })], CURRENT);
// No syncable changes → delta priced at 0 tokens (but the source changed).
expect(r.tokens).toBe(0);
expect(r.estimateKind).toBe('delta');
});
test('syncEnabled:false sources are skipped (neither changed nor unchanged)', () => {
writeFileSync(join(repo, 'topics/a.md'), 'body');
commitAll('base');
const r = estimateInlineNewTokens([src({ last_commit: null, config: { syncEnabled: false } })], CURRENT);
expect(r.tokens).toBe(0);
expect(r.changedSources).toBe(0);
expect(r.unchangedSources).toBe(0);
});
test('mixed: one ceiling source + one unchanged source → estimateKind mixed-or-ceiling, reasons captured', () => {
writeFileSync(join(repo, 'topics/a.md'), 'body');
const head = commitAll('base');
const r = estimateInlineNewTokens(
[
src({ last_commit: null, chunker_version: CURRENT }), // first_sync ceiling
src({ last_commit: head, chunker_version: CURRENT }), // unchanged
],
CURRENT,
);
expect(r.ceilingReasons).toContain('first_sync');
expect(r.unchangedSources).toBe(1);
// hadCeiling true, hadDelta false → 'ceiling' aggregate.
expect(r.estimateKind).toBe('ceiling');
});
test('missing local_path source contributes nothing', () => {
const r = estimateInlineNewTokens(
[{ local_path: null, config: {}, last_commit: null, chunker_version: CURRENT }],
CURRENT,
);
expect(r.tokens).toBe(0);
expect(r.changedSources).toBe(0);
});
});
+115 -28
View File
@@ -1,15 +1,17 @@
/**
* v0.41.31 `gbrain sync --all` cost-gate wiring regressions (PGLite).
* `gbrain sync` cost-gate wiring regressions (PGLite).
*
* Pure shouldBlockSync / willEmbedSynchronously logic is pinned in
* test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in
* runSync's --all path:
* Pure shouldBlockSync / willEmbedSynchronously / parseUsdLimit logic is pinned
* in test/sync-cost-preview.test.ts. THIS file pins the end-to-end wiring in
* runSync's --all AND single-source paths:
*
* R-1 (headline): deferred-embed sync --all, non-TTY, with backlog
* emits gate:'deferred_notice' and NEVER exit 2 (the cron-blocking
* bug this release fixes).
* R-2 (protection): inline-embed sync --all (--serial), non-TTY, above
* floor still exit 2 with gate:'confirmation_required'.
* emits gate:'deferred_notice' and NEVER exit 2.
* R-2 (v0.42.42.0, #2139): inline sync --all (--serial), non-TTY, above floor
* AUTO-DEFERS (exit 0, gate:'auto_deferred_embeds') and enqueues an
* embed-backfill job. The exit-2 wedge is gone.
* R-3: chunker drift full-tree CEILING estimate, auto-defers (not exit 2).
* + posture tokenmax, off-switch, format split (#1784/D3A), single-source gate.
*
* Serial-quarantined: stubs process.exit + console.log (process-global).
*/
@@ -21,10 +23,18 @@ import { join } from 'path';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { runSources } from '../src/commands/sources.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { configureGateway, resetGateway } from '../src/core/ai/gateway.ts';
import { configureGateway, resetGateway, __setEmbedTransportForTests } from '../src/core/ai/gateway.ts';
import { CHUNKER_VERSION } from '../src/core/chunkers/code.ts';
import type { ChunkInput } from '../src/core/types.ts';
/** Offline embed stub so inline-proceed paths (posture tokenmax) don't network. */
function stubOfflineEmbed(): void {
__setEmbedTransportForTests(async ({ values }: any) => ({
embeddings: values.map(() => new Array(1536).fill(0)),
usage: { tokens: 0 },
}) as any);
}
let engine: PGLiteEngine;
let repoPath: string;
let headSha: string;
@@ -64,6 +74,7 @@ beforeEach(async () => {
});
afterEach(() => {
__setEmbedTransportForTests(null);
resetGateway();
if (repoPath) rmSync(repoPath, { recursive: true, force: true });
});
@@ -115,39 +126,43 @@ describe('v0.41.31 — sync --all cost gate wiring', () => {
expect(stdout).toContain('"gate":"deferred_notice"');
}, 60_000);
test('R-2: inline sync --all (--serial) above floor still exit 2', async () => {
test('R-2 (#2139): inline sync --all (--serial) above floor AUTO-DEFERS (exit 0, never exit 2) + enqueues backfill', async () => {
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
// Floor 0 → any nonzero inline cost blocks. Source is unsynced
// (last_commit NULL) so estimateInlineNewTokens sees it as changed →
// full-tree tokens > 0 → costUsd > 0 > floor.
// Floor 0 → any nonzero inline cost trips the gate. Source is unsynced
// (last_commit NULL) → first-sync ceiling > 0 > floor.
await engine.setConfig('sync.cost_gate_min_usd', '0');
// --serial forces inline even with v2 on. --json → non-TTY exit-2 path.
// --serial forces inline even with v2 on. --json → non-TTY path → AUTO-DEFER.
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
expect(exitCode).toBe(2);
expect(stdout).toContain('"gate":"confirmation_required"');
expect(exitCode).not.toBe(2);
expect(stdout).toContain('"gate":"auto_deferred_embeds"');
expect(stdout).not.toContain('"gate":"confirmation_required"');
// The run PROCEEDED to import (the wedge is gone) — embeds were deferred,
// not blocked. (The embed-backfill enqueue wiring + its graceful
// missing-table tolerance is pinned in embed-backfill-submit.test.ts; the
// minion_jobs table isn't provisioned in this gate-wiring harness.)
expect(stdout).toContain('"sync_status":"first_sync"');
}, 60_000);
test('R-3: inline, git-unchanged source but STALE chunker_version still estimates (not $0)', async () => {
// The unchanged-source short-circuit requires HEAD==last_commit AND clean
// tree AND chunker_version == current. Here git is unchanged but the
// chunker drifted, so the source must NOT be treated as 0 — sync would
// re-chunk + re-embed everything. floor=0 so any nonzero cost blocks.
test('R-3 (#2139): chunker drift → full-tree CEILING estimate, auto-defers (not exit 2)', async () => {
// git unchanged (HEAD==last_commit) but chunker drifted → the source must
// NOT price $0 (sync would re-chunk + re-embed everything). The estimate is
// the full-tree ceiling; the gate auto-defers rather than wedging.
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, 'STALE-0']);
await engine.setConfig('sync.cost_gate_min_usd', '0');
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
expect(exitCode).toBe(2);
expect(stdout).toContain('"gate":"confirmation_required"');
expect(exitCode).not.toBe(2);
expect(stdout).toContain('"gate":"auto_deferred_embeds"');
expect(stdout).toContain('"estimateKind":"ceiling"');
}, 60_000);
test('R-3 control: inline, git-unchanged + CURRENT chunker_version short-circuits to $0 (no exit 2)', async () => {
// Same setup but chunker_version matches current → the source IS unchanged
// → contributes 0 new-content tokens → below floor → proceeds (no block).
// Proves the short-circuit fires when (and only when) everything matches.
test('R-3 control: git-unchanged + CURRENT chunker → $0 estimate, below floor (no auto-defer)', async () => {
// Mirrors the executor's up_to_date predicate: HEAD==last_commit AND chunker
// matches → 0 new tokens → below floor → proceeds without deferring.
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]);
await engine.setConfig('sync.cost_gate_min_usd', '0');
@@ -155,6 +170,78 @@ describe('v0.41.31 — sync --all cost gate wiring', () => {
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
expect(exitCode).not.toBe(2);
expect(stdout).not.toContain('"gate":"confirmation_required"');
expect(stdout).not.toContain('"gate":"auto_deferred_embeds"');
expect(stdout).toContain('"estimateKind":"unchanged"');
}, 60_000);
test('headline regression: HEAD==last_commit + DIRTY untracked file → $0, no gate (the false-fire)', async () => {
// The exact pre-fix false-fire: a busy brain's working tree is never
// git-clean, but the commits are caught up. The OLD estimator priced the
// whole tree (158M-token phantom); the new one mirrors execution → $0.
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
await engine.executeRaw(`UPDATE sources SET last_commit = $1, chunker_version = $2 WHERE id = 'vault'`, [headSha, String(CHUNKER_VERSION)]);
// Dirty the tree with an untracked non-syncable scratch file (agents/crons
// write constantly) — attached-HEAD sync never imports it.
writeFileSync(join(repoPath, 'scratch.tmp'), 'uncommitted agent scratch');
writeFileSync(join(repoPath, 'topics/foo.md'), 'uncommitted edit, not staged');
await engine.setConfig('sync.cost_gate_min_usd', '0');
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
expect(exitCode).not.toBe(2);
expect(stdout).not.toContain('"gate":"auto_deferred_embeds"');
expect(stdout).toContain('"estimateKind":"unchanged"');
}, 60_000);
test('spend.posture=tokenmax → proceeds inline, gate:posture_tokenmax (informational)', async () => {
stubOfflineEmbed(); // inline embed proceeds — keep it off the network.
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
await engine.setConfig('sync.cost_gate_min_usd', '0');
await engine.setConfig('spend.posture', 'tokenmax');
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
expect(exitCode).not.toBe(2);
expect(stdout).toContain('"gate":"posture_tokenmax"');
expect(stdout).not.toContain('"gate":"auto_deferred_embeds"');
}, 60_000);
test('sync.cost_gate_min_usd=off → floor renders "unlimited", never blocks', async () => {
stubOfflineEmbed();
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
await engine.setConfig('sync.cost_gate_min_usd', 'off');
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--json', '--no-pull']);
expect(exitCode).not.toBe(2);
expect(stdout).toContain('"floorUsd":"unlimited"');
expect(stdout).not.toContain('"gate":"auto_deferred_embeds"');
}, 60_000);
test('format split (#1784/D3A): non-TTY WITHOUT --json emits human text, no JSON envelope', async () => {
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
await engine.setConfig('sync.cost_gate_min_usd', '0');
// No --json: above floor in a non-TTY session → human auto-defer text.
const { exitCode, stdout } = await runSyncCaptured(['--all', '--serial', '--no-pull']);
expect(exitCode).not.toBe(2);
expect(stdout).not.toContain('"gate":'); // no JSON envelope without --json
expect(stdout.toLowerCase()).toContain('deferring embeds');
expect(stdout).toContain('spend.posture'); // self-describing hint present
}, 60_000);
test('single-source sync gets the same gate (auto-defers above floor, exit 0)', async () => {
await runSources(engine, ['add', 'vault', '--path', repoPath, '--no-federated']);
await engine.setConfig('sync.cost_gate_min_usd', '0');
// Single-source (no --all): unsynced → ceiling > 0 → non-TTY auto-defer.
const { exitCode, stdout } = await runSyncCaptured(['--source', 'vault', '--json', '--no-pull']);
expect(exitCode).not.toBe(2);
expect(stdout).toContain('"gate":"auto_deferred_embeds"');
// The gate now exists on the single-source path (was ungated before
// #2139) and proceeds to import rather than blocking.
expect(stdout.toLowerCase()).toContain('imported');
}, 60_000);
});
+86 -1
View File
@@ -13,7 +13,7 @@
* envelope paths don't depend on DB state.
*/
import { describe, test, expect } from 'bun:test';
import { describe, test, expect, beforeEach } from 'bun:test';
import {
EMBEDDING_COST_PER_1K_TOKENS,
estimateEmbeddingCostUsd,
@@ -21,9 +21,27 @@ import {
shouldBlockSync,
} from '../src/core/embedding.ts';
import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts';
import { resetGateway } from '../src/core/ai/gateway.ts';
import { estimateTokens } from '../src/core/chunkers/code.ts';
import {
parseUsdLimit,
formatUsdLimit,
usdLimitToCap,
normalizeSpendPosture,
isValidSpendPosture,
} from '../src/core/spend-posture.ts';
describe('Layer 8 D1 — embedding cost model', () => {
// The estimateEmbeddingCostUsd tests assert the gateway-UNCONFIGURED OpenAI
// fallback ($0.13/Mtok). That fallback only fires when no gateway is
// configured — so guarantee that precondition rather than depending on
// ambient state. Without this, a sibling test file in the same shard that
// configured (and didn't reset) a cheaper model leaks its rate in here and
// these assertions flip (the legacy-embedding preload only restores when the
// gateway slot is EMPTY, so a non-empty foreign config survives).
beforeEach(() => resetGateway());
test('EMBEDDING_COST_PER_1K_TOKENS back-compat constant is the OpenAI 3-large rate', () => {
// Retained only for back-compat imports. Live cost math now resolves the
// CONFIGURED model's rate via embedding-pricing.ts (see model-aware test
@@ -116,6 +134,73 @@ describe('v0.41.31 — shouldBlockSync (cost-gate decision)', () => {
expect(shouldBlockSync(0.0001, 0, 'inline')).toBe(true);
expect(shouldBlockSync(0, 0, 'inline')).toBe(false);
});
// v0.42.42.0 (#2139): posture + Infinity-floor behavior.
test('spend.posture=tokenmax never blocks, even above floor', () => {
expect(shouldBlockSync(999, 0.5, 'inline', 'tokenmax')).toBe(false);
expect(shouldBlockSync(999, 0, 'inline', 'tokenmax')).toBe(false);
});
test('default posture (gated) preserves the legacy decision', () => {
expect(shouldBlockSync(0.51, 0.5, 'inline', 'gated')).toBe(true);
expect(shouldBlockSync(0.03, 0.5, 'inline', 'gated')).toBe(false);
});
test('off/unlimited floor (Infinity) is never exceeded → never blocks', () => {
expect(shouldBlockSync(999, Infinity, 'inline')).toBe(false);
expect(shouldBlockSync(1e9, Infinity, 'inline', 'gated')).toBe(false);
});
});
describe('v0.42.42.0 (#2139) — spend-posture USD-limit parsing', () => {
test('off / unlimited / none (case-insensitive) → Infinity', () => {
for (const v of ['off', 'OFF', 'unlimited', 'Unlimited', 'none', 'NONE', ' off ']) {
expect(parseUsdLimit(v, 25)).toBe(Infinity);
}
});
test('finite positive numbers pass through', () => {
expect(parseUsdLimit('5', 25)).toBe(5);
expect(parseUsdLimit(0.5, 25)).toBe(0.5);
expect(parseUsdLimit('100000', 25)).toBe(100000);
});
test('0 falls back to default unless allowZero', () => {
expect(parseUsdLimit('0', 25)).toBe(25); // backfill cap: off ≠ 0
expect(parseUsdLimit('0', 0.5, { allowZero: true })).toBe(0); // floor: 0 = block-on-any
});
test('garbage / negative / empty / null → default', () => {
expect(parseUsdLimit('abc', 25)).toBe(25);
expect(parseUsdLimit('-3', 25)).toBe(25);
expect(parseUsdLimit('', 25)).toBe(25);
expect(parseUsdLimit(null, 25)).toBe(25);
expect(parseUsdLimit(undefined, 0.5)).toBe(0.5);
});
test('formatUsdLimit: Infinity → "unlimited" (never the JSON.stringify=null trap), finite passthrough', () => {
expect(formatUsdLimit(Infinity)).toBe('unlimited');
expect(formatUsdLimit(5)).toBe(5);
expect(formatUsdLimit(0)).toBe(0);
// The trap this guards: raw Infinity serializes to null.
expect(JSON.stringify({ cap: Infinity })).toBe('{"cap":null}');
expect(JSON.stringify({ cap: formatUsdLimit(Infinity) })).toBe('{"cap":"unlimited"}');
});
test('usdLimitToCap: Infinity → undefined (no cap), finite passthrough', () => {
expect(usdLimitToCap(Infinity)).toBeUndefined();
expect(usdLimitToCap(10)).toBe(10);
});
test('normalizeSpendPosture: only tokenmax is tokenmax; everything else gated', () => {
expect(normalizeSpendPosture('tokenmax')).toBe('tokenmax');
expect(normalizeSpendPosture('TokenMax')).toBe('tokenmax');
expect(normalizeSpendPosture('gated')).toBe('gated');
expect(normalizeSpendPosture('max')).toBe('gated');
expect(normalizeSpendPosture('')).toBe('gated');
expect(normalizeSpendPosture(null)).toBe('gated');
expect(normalizeSpendPosture(42)).toBe('gated');
});
test('isValidSpendPosture accepts gated/tokenmax (case-insensitive), rejects the rest', () => {
expect(isValidSpendPosture('gated')).toBe(true);
expect(isValidSpendPosture('tokenmax')).toBe(true);
expect(isValidSpendPosture('TokenMax')).toBe(true); // normalized lowercase
expect(isValidSpendPosture('max')).toBe(false);
expect(isValidSpendPosture('')).toBe(false);
expect(isValidSpendPosture(7)).toBe(false);
});
});
describe('Layer 8 D1 — estimateTokens (exported from chunkers/code.ts)', () => {
+154
View File
@@ -0,0 +1,154 @@
/**
* v0.42.42.0 (#2139) computeSyncDelta unit coverage.
*
* The shared diff/manifest helper that BOTH the sync executor and the inline
* cost estimator route through (so the gate's dollar figure can't drift from
* what the sync imports). Real temp git repos; no PGLite, no env writes
* (R1/R2-clean). The git-runner seam drives the unavailable branches.
*/
import { describe, test, expect, beforeEach, afterEach } from 'bun:test';
import { mkdtempSync, writeFileSync, rmSync, mkdirSync, renameSync } from 'fs';
import { execSync } from 'child_process';
import { tmpdir } from 'os';
import { join } from 'path';
import {
computeSyncDelta,
buildDetachedWorkingTreeManifest,
_setGitRunnerForTests,
} from '../src/core/sync-delta.ts';
let repo: string;
function git(args: string): string {
return execSync(`git ${args}`, { cwd: repo, stdio: 'pipe' }).toString().trim();
}
function commitAll(msg: string): string {
execSync('git add -A', { cwd: repo, stdio: 'pipe' });
execSync(`git commit -m "${msg}"`, { cwd: repo, stdio: 'pipe' });
return git('rev-parse HEAD');
}
beforeEach(() => {
repo = mkdtempSync(join(tmpdir(), 'gbrain-delta-'));
execSync('git init', { cwd: repo, stdio: 'pipe' });
execSync('git config user.email "t@t.com"', { cwd: repo, stdio: 'pipe' });
execSync('git config user.name "T"', { cwd: repo, stdio: 'pipe' });
mkdirSync(join(repo, 'topics'), { recursive: true });
});
afterEach(() => {
_setGitRunnerForTests(null);
if (repo) rmSync(repo, { recursive: true, force: true });
});
describe('computeSyncDelta — commit diff', () => {
test('A/M/D classified; only committed changes in the manifest', () => {
writeFileSync(join(repo, 'topics/a.md'), 'a');
writeFileSync(join(repo, 'topics/b.md'), 'b');
const base = commitAll('base');
writeFileSync(join(repo, 'topics/a.md'), 'a-edited'); // modify
writeFileSync(join(repo, 'topics/c.md'), 'c'); // add
rmSync(join(repo, 'topics/b.md')); // delete
const head = commitAll('change');
const r = computeSyncDelta(repo, base, head);
expect(r.status).toBe('ok');
if (r.status !== 'ok') return;
expect(r.manifest.modified).toContain('topics/a.md');
expect(r.manifest.added).toContain('topics/c.md');
expect(r.manifest.deleted).toContain('topics/b.md');
});
test('rename → destination path on the renamed list', () => {
writeFileSync(join(repo, 'topics/old.md'), 'x'.repeat(200));
const base = commitAll('base');
renameSync(join(repo, 'topics/old.md'), join(repo, 'topics/new.md'));
const head = commitAll('rename');
const r = computeSyncDelta(repo, base, head);
expect(r.status).toBe('ok');
if (r.status !== 'ok') return;
expect(r.manifest.renamed.map(x => x.to)).toContain('topics/new.md');
});
test('[D2A] attached HEAD: dirty tracked + untracked files are NOT in the manifest', () => {
writeFileSync(join(repo, 'topics/a.md'), 'a');
const base = commitAll('base');
const head = git('rev-parse HEAD'); // HEAD == base, no new commits
// Dirty the tree: an uncommitted edit + an untracked scratch file.
writeFileSync(join(repo, 'topics/a.md'), 'uncommitted edit');
writeFileSync(join(repo, 'scratch.tmp'), 'untracked');
const r = computeSyncDelta(repo, base, head); // not detached → commit diff only
expect(r.status).toBe('ok');
if (r.status !== 'ok') return;
expect(r.manifest.added).toHaveLength(0);
expect(r.manifest.modified).toHaveLength(0);
expect(r.manifest.deleted).toHaveLength(0);
});
});
describe('computeSyncDelta — detached HEAD merges the working-tree manifest', () => {
test('detached + working-tree changes → merged into the manifest', () => {
writeFileSync(join(repo, 'topics/a.md'), 'a');
const base = commitAll('base');
// Detach HEAD and dirty the tree.
execSync(`git checkout --detach ${base}`, { cwd: repo, stdio: 'pipe' });
writeFileSync(join(repo, 'topics/a.md'), 'detached edit'); // tracked modify
writeFileSync(join(repo, 'topics/new.md'), 'new'); // untracked add
const r = computeSyncDelta(repo, base, base, { detached: true });
expect(r.status).toBe('ok');
if (r.status !== 'ok') return;
expect(r.manifest.modified).toContain('topics/a.md');
expect(r.manifest.added).toContain('topics/new.md'); // untracked picked up on detached
});
test('buildDetachedWorkingTreeManifest: clean detached tree → empty manifest', () => {
writeFileSync(join(repo, 'topics/a.md'), 'a');
const base = commitAll('base');
execSync(`git checkout --detach ${base}`, { cwd: repo, stdio: 'pipe' });
const m = buildDetachedWorkingTreeManifest(repo);
expect(m.added).toHaveLength(0);
expect(m.modified).toHaveLength(0);
});
});
describe('computeSyncDelta — fail-open ladder', () => {
test('bogus anchor SHA → unavailable: anchor_missing', () => {
writeFileSync(join(repo, 'topics/a.md'), 'a');
const head = commitAll('base');
const r = computeSyncDelta(repo, 'deadbeefdeadbeefdeadbeefdeadbeefdeadbeef', head);
expect(r.status).toBe('unavailable');
if (r.status === 'unavailable') expect(r.reason).toBe('anchor_missing');
});
test('non-ancestor anchor still diffs (the #1970 property)', () => {
// git diff A..B is endpoint-tree, no ancestry requirement.
writeFileSync(join(repo, 'topics/a.md'), 'a');
const base = commitAll('base');
// Rewrite history: amend creates a new commit not descended from `base`,
// but `base` is still on disk (reflog) → diffable.
writeFileSync(join(repo, 'topics/a.md'), 'rewritten');
execSync('git add -A && git commit --amend -m rewritten', { cwd: repo, stdio: 'pipe' });
const head = git('rev-parse HEAD');
expect(head).not.toBe(base);
const r = computeSyncDelta(repo, base, head);
expect(r.status).toBe('ok'); // orphaned-but-present anchor is still diffable
});
test('injected git failure on the diff → unavailable: diff_failed', () => {
writeFileSync(join(repo, 'topics/a.md'), 'a');
const base = commitAll('base');
const head = git('rev-parse HEAD');
_setGitRunnerForTests((_repo, args) => {
if (args[0] === 'cat-file') return 'commit'; // anchor reachable
if (args[0] === 'diff') throw new Error('simulated oversized diff / timeout');
return '';
});
const r = computeSyncDelta(repo, base, head);
expect(r.status).toBe('unavailable');
if (r.status === 'unavailable') expect(r.reason).toBe('diff_failed');
});
});
+3 -3
View File
@@ -170,10 +170,10 @@ describe('Bug 9 — sync.ts CLI flag wiring', () => {
// performSync's inner ack path only fires when failedFiles.length > 0
// in the current run. This test pins the up-front ack at the top of
// runSync so the flag means "ack whatever is currently flagged".
// v0.42.42.0 (#2139, D13C): the pre-ack is now scoped PER SOURCE — `--all`
// acks every source, single-source acks only its own.
const source = await Bun.file(new URL('../src/commands/sync.ts', import.meta.url)).text();
// Ensure the up-front check exists before the syncAll / performSync
// dispatch, gated on skipFailed.
expect(source).toMatch(/if \(skipFailed\) \{[\s\S]*?unacknowledgedSyncFailures\(\)[\s\S]*?acknowledgeSyncFailures\(\)/);
expect(source).toMatch(/if \(skipFailed\) \{[\s\S]*?syncAll \? acknowledgeFailures\(\) : acknowledgeFailures\(sourceId\)/);
});
test('acknowledgeSyncFailures clears stale failures end-to-end', async () => {
+483
View File
@@ -0,0 +1,483 @@
/**
* v0.43 (#2095) push-based context core: window parsing, multi-turn
* extraction, confidence-gated volunteering, slug-only suppression, privacy,
* and the usage-stats join. Hermetic in-memory PGLite (no file lock),
* modeled on test/retrieval-reflex.test.ts.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { normalizeAlias } from '../src/core/search/alias-normalize.ts';
import {
extractCandidatesFromWindow,
MAX_CANDIDATES,
type WindowTurn,
} from '../src/core/context/entity-salience.ts';
import { resolveEntitiesToPointers } from '../src/core/context/retrieval-reflex.ts';
import {
parseWindow,
volunteerContext,
volunteerUsageStats,
VOLUNTEER_DEFAULT_MIN_CONFIDENCE,
} from '../src/core/context/volunteer.ts';
import { insertVolunteerEvents } from '../src/core/context/volunteer-events.ts';
import { TAKES_FENCE_BEGIN, TAKES_FENCE_END } from '../src/core/takes-fence.ts';
let engine: PGLiteEngine;
async function seed(slug: string, title: string, body: string, source = 'default') {
if (source !== 'default') {
await engine.executeRaw(
`INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2)
ON CONFLICT (id) DO NOTHING`,
[source, `/tmp/${source}`],
);
}
await engine.executeRaw(
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
VALUES ($1, $2, 'person', $3, $4, '')`,
[slug, source, title, body],
);
}
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 60_000);
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await engine.executeRaw('DELETE FROM page_aliases').catch(() => {});
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
await engine.executeRaw('DELETE FROM pages');
});
describe('parseWindow', () => {
test('role prefixes split turns oldest → newest', () => {
const turns = parseWindow('user: ask Alice about the deal\nassistant: noted\nuser: what did she say?');
expect(turns).toEqual([
{ role: 'user', text: 'ask Alice about the deal' },
{ role: 'assistant', text: 'noted' },
{ role: 'user', text: 'what did she say?' },
]);
});
test('CRLF input parses identically', () => {
const turns = parseWindow('user: hello\r\nassistant: hi\r\n');
expect(turns).toEqual([
{ role: 'user', text: 'hello' },
{ role: 'assistant', text: 'hi' },
]);
});
test('unprefixed text is ONE user turn (echo | volunteer-context just works)', () => {
const turns = parseWindow('met with Alice Example today\nshe asked about acme');
expect(turns).toEqual([{ role: 'user', text: 'met with Alice Example today\nshe asked about acme' }]);
});
test('continuation lines append to the open turn', () => {
const turns = parseWindow('user: first line\nsecond line\nassistant: ok');
expect(turns[0]).toEqual({ role: 'user', text: 'first line\nsecond line' });
expect(turns[1]).toEqual({ role: 'assistant', text: 'ok' });
});
test('empty / whitespace input → []', () => {
expect(parseWindow('')).toEqual([]);
expect(parseWindow(' \n \n')).toEqual([]);
});
});
describe('extractCandidatesFromWindow', () => {
test('merges across turns with occurrence + newest-turn metadata', () => {
const turns: WindowTurn[] = [
{ role: 'user', text: 'ask Alice Example about the deal' },
{ role: 'assistant', text: 'Alice Example said she will follow up' },
{ role: 'user', text: 'and ping Bob Sample too' },
];
const cands = extractCandidatesFromWindow(turns);
const alice = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Alice Example'));
const bob = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Bob Sample'));
expect(alice).toBeDefined();
expect(alice!.occurrences).toBe(2);
expect(alice!.inNewestTurn).toBe(false);
expect(alice!.userMention).toBe(true);
expect(bob).toBeDefined();
expect(bob!.inNewestTurn).toBe(true);
});
test('assistant-only mention is flagged (userMention=false)', () => {
const cands = extractCandidatesFromWindow([
{ role: 'assistant', text: 'You should talk to Charlie Demo about this' },
{ role: 'user', text: 'good idea' },
]);
const charlie = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Charlie Demo'));
expect(charlie).toBeDefined();
expect(charlie!.userMention).toBe(false);
});
test('cap holds across a noisy window', () => {
const noisy = Array.from({ length: 30 }, (_, i) => `Entity Number${i} did something.`).join(' ');
const cands = extractCandidatesFromWindow([{ role: 'user', text: noisy }]);
expect(cands.length).toBeLessThanOrEqual(MAX_CANDIDATES);
});
});
describe('volunteerContext', () => {
test('assistant-introduced entity two turns back resolves on a pronoun follow-up', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is an early founder.');
const turns = parseWindow(
'user: who should I talk to about the seed round?\n' +
'assistant: Alice Example led a similar round last year.\n' +
'user: what did she invest in?',
);
const pages = await volunteerContext(engine, turns, { sourceIds: ['default'] });
expect(pages.length).toBe(1);
expect(pages[0].slug).toBe('people/alice-example');
expect(pages[0].arm).toBe('title');
expect(pages[0].rationale).toContain('assistant-introduced');
});
test('excludeSlugs skips BEFORE the cap: an excluded entity never starves a fresh page', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
await seed('people/bob-sample', 'Bob Sample', 'Engineer.');
const turns = parseWindow('user: intro Alice Example to Bob Sample');
const pages = await volunteerContext(engine, turns, {
sourceIds: ['default'],
maxPages: 1,
excludeSlugs: new Set(['people/alice-example']),
});
// A post-call filter would return [] here (Alice burns the single cap
// slot, then gets filtered). The pre-cap exclusion hands Bob the slot.
expect(pages.length).toBe(1);
expect(pages[0].slug).toBe('people/bob-sample');
});
test('confidence gate drops slug-suffix matches at the default threshold', async () => {
// Page resolvable ONLY via slug-suffix: title differs from the mention.
await seed('projects/widget-co', 'The Widget Company Project', 'A project page.');
const turns = parseWindow('user: any updates on Widget-Co this week?');
const gated = await volunteerContext(engine, turns, { sourceIds: ['default'] });
expect(gated).toEqual([]);
// Lowering min_confidence lets it through with honest provenance.
const loose = await volunteerContext(engine, turns, { sourceIds: ['default'], minConfidence: 0.5 });
expect(loose.length).toBe(1);
expect(loose[0].arm).toBe('slug-suffix');
expect(loose[0].confidence).toBeLessThan(VOLUNTEER_DEFAULT_MIN_CONFIDENCE);
});
test('alias arm volunteers with boost when mentioned in the newest turn', async () => {
await seed('people/swami-x', 'Swami X', 'A close friend.');
await engine.setPageAliases('people/swami-x', 'default', [normalizeAlias('Swami')]);
const pages = await volunteerContext(
engine,
parseWindow('user: Spoke with Swami today'),
{ sourceIds: ['default'] },
);
expect(pages.length).toBe(1);
expect(pages[0].arm).toBe('alias');
expect(pages[0].confidence).toBeCloseTo(0.95, 5); // 0.9 + newest-turn boost
expect(pages[0].rationale).toContain('alias match');
});
test('suppression is slug-only under windowing: a prior-turn MENTION does not suppress', async () => {
await seed('people/alice-example', 'Alice Example', 'A founder.');
// Prior context contains the TITLE (a bare mention from an earlier turn)
// but NOT the slug — the page was never actually surfaced.
const pages = await volunteerContext(
engine,
parseWindow('user: ping Alice Example again'),
{ sourceIds: ['default'], priorContext: 'earlier the user said: tell Alice Example the news' },
);
expect(pages.length).toBe(1);
// A slug in prior context (the page WAS surfaced) does suppress.
const suppressed = await volunteerContext(
engine,
parseWindow('user: ping Alice Example again'),
{ sourceIds: ['default'], priorContext: 'pointer: people/alice-example was injected last turn' },
);
expect(suppressed).toEqual([]);
});
test('privacy: takes-fence content never leaks into the synopsis', async () => {
const body = `${TAKES_FENCE_BEGIN}\nSECRET_HUNCH_DO_NOT_LEAK\n${TAKES_FENCE_END}\nAlice is a founder.`;
await seed('people/alice-example', 'Alice Example', body);
const pages = await volunteerContext(engine, parseWindow('user: about Alice Example'), { sourceIds: ['default'] });
expect(pages.length).toBe(1);
expect(JSON.stringify(pages)).not.toContain('SECRET_HUNCH_DO_NOT_LEAK');
});
test('multi-source scope: resolves from both sources, never outside the scope', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.', 'default');
await seed('people/bob-sample', 'Bob Sample', 'Engineer.', 'team-brain');
await seed('people/eve-other', 'Eve Other', 'Out of scope.', 'private-brain');
const turns = parseWindow('user: intro Alice Example to Bob Sample and Eve Other');
const pages = await volunteerContext(engine, turns, { sourceIds: ['default', 'team-brain'] });
const keys = pages.map((p) => `${p.source_id}:${p.slug}`).sort();
expect(keys).toEqual(['default:people/alice-example', 'team-brain:people/bob-sample']);
});
test('zero-candidate fast path: no entities → [] without touching resolution', async () => {
const pages = await volunteerContext(engine, parseWindow('user: ok thanks, sounds good'), {
sourceIds: ['default'],
});
expect(pages).toEqual([]);
});
test('max_pages caps at 5 even when asked for more', async () => {
for (let i = 0; i < 8; i++) {
await seed(`people/person-${i}`, `Person Alpha${i}`, 'A person.');
}
const text = Array.from({ length: 8 }, (_, i) => `Person Alpha${i}`).join(' and ');
const pages = await volunteerContext(engine, parseWindow(`user: intro ${text}`), {
sourceIds: ['default'],
maxPages: 50,
});
expect(pages.length).toBeLessThanOrEqual(5);
});
});
describe('volunteerUsageStats', () => {
test('join math: used = last_retrieved_at > volunteered_at, labeled approximate', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
await seed('people/bob-sample', 'Bob Sample', 'Engineer.');
await insertVolunteerEvents(engine, [
{ source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'op' },
{ source_id: 'default', slug: 'people/bob-sample', confidence: 0.8, match_arm: 'title', rationale: 'r', channel: 'op' },
]);
// Alice was opened AFTER being volunteered; Bob never was.
await engine.executeRaw(
`UPDATE pages SET last_retrieved_at = now() + interval '1 minute' WHERE slug = 'people/alice-example'`, [],
);
const stats = await volunteerUsageStats(engine, ['default'], 30);
expect(stats.approximate).toBe(true);
expect(stats.note).toContain('approximate');
expect(stats.total_volunteered).toBe(2);
expect(stats.total_used).toBe(1);
const alias = stats.by_arm.find((a) => a.match_arm === 'alias')!;
expect(alias.used).toBe(1);
expect(alias.precision).toBe(1);
const title = stats.by_arm.find((a) => a.match_arm === 'title')!;
expect(title.used).toBe(0);
});
test('zero events → zeroed stats, not an error', async () => {
const stats = await volunteerUsageStats(engine, ['default'], 7);
expect(stats.total_volunteered).toBe(0);
expect(stats.by_arm).toEqual([]);
});
});
describe('resolveEntitiesToPointers — new provenance surface (back-compat)', () => {
test('pointers carry arm + confidence + source_id', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
const block = await resolveEntitiesToPointers(
engine,
'default',
[{ display: 'Alice Example', query: 'Alice Example' }],
{},
);
expect(block).not.toBeNull();
expect(block!.pointers[0].arm).toBe('title');
expect(block!.pointers[0].confidence).toBe(0.8);
expect(block!.pointers[0].source_id).toBe('default');
});
test('legacy suppression (slug-and-title) still drops a title mention in prior context', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
const block = await resolveEntitiesToPointers(
engine,
'default',
[{ display: 'Alice Example', query: 'Alice Example' }],
{ priorContextText: 'we discussed Alice Example earlier' },
);
expect(block).toBeNull();
});
});
describe('volunteer_context op (contract surface)', () => {
const { operationsByName } = require('../src/core/operations.ts') as typeof import('../src/core/operations.ts');
const {
awaitPendingVolunteerEventWrites,
_resetPendingVolunteerEventWritesForTests,
} = require('../src/core/context/volunteer-events.ts') as typeof import('../src/core/context/volunteer-events.ts');
function mkCtx(overrides: Record<string, unknown> = {}) {
return {
engine,
config: {} as never,
logger: { info: () => {}, warn: () => {}, error: () => {} } as never,
dryRun: false,
remote: false,
sourceId: 'default',
...overrides,
} as never;
}
test('registered, read-scope, not localOnly, stdin cliHint on window', () => {
const op = operationsByName.volunteer_context;
expect(op).toBeDefined();
expect(op.scope).toBe('read');
expect(op.localOnly).toBeFalsy();
expect(op.cliHints?.name).toBe('volunteer-context');
expect(op.cliHints?.stdin).toBe('window');
});
test('window required unless stats: true', async () => {
const op = operationsByName.volunteer_context;
await expect(op.handler(mkCtx(), {})).rejects.toThrow(/window is required/);
const stats = (await op.handler(mkCtx(), { stats: true })) as any;
expect(stats.approximate).toBe(true);
});
test('volunteers pages and logs events through the drained sink', async () => {
_resetPendingVolunteerEventWritesForTests();
await seed('people/alice-example', 'Alice Example', 'Founder.');
const op = operationsByName.volunteer_context;
const result = (await op.handler(mkCtx(), {
window: 'user: ping Alice Example about the deal',
session_id: 's-42',
turn: 7,
})) as any;
expect(result.count).toBe(1);
expect(result.pages[0].slug).toBe('people/alice-example');
expect(result.window_turns).toBe(1);
// Event row lands via the fire-and-forget sink once drained.
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
expect(unfinished).toBe(0);
const rows = await engine.executeRaw<{ slug: string; channel: string; session_id: string; turn: number }>(
`SELECT slug, channel, session_id, turn FROM context_volunteer_events`, [],
);
expect(rows.length).toBe(1);
expect(rows[0].slug).toBe('people/alice-example');
expect(rows[0].channel).toBe('op');
expect(rows[0].session_id).toBe('s-42');
expect(Number(rows[0].turn)).toBe(7);
});
test('event-log failure never fails the op (failing engine injected for the INSERT)', async () => {
_resetPendingVolunteerEventWritesForTests();
await seed('people/alice-example', 'Alice Example', 'Founder.');
const op = operationsByName.volunteer_context;
// Wrap the engine: reads pass through, INSERTs into the events table throw.
const failingEngine = new Proxy(engine, {
get(target, prop, receiver) {
if (prop === 'executeRaw') {
return (sql: string, params: unknown[]) => {
if (/INSERT INTO context_volunteer_events/.test(sql)) {
return Promise.reject(new Error('telemetry db down'));
}
return target.executeRaw(sql, params);
};
}
return Reflect.get(target, prop, receiver);
},
});
const result = (await op.handler(mkCtx({ engine: failingEngine }), {
window: 'user: ping Alice Example',
})) as any;
expect(result.count).toBe(1); // the volunteer result is unaffected
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
expect(unfinished).toBe(0); // failed write settled (swallowed), not stuck
});
test('federated grant scopes the volunteer (allowedSources)', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.', 'default');
await seed('people/bob-sample', 'Bob Sample', 'Engineer.', 'grant-brain');
await seed('people/eve-other', 'Eve Other', 'Out of grant.', 'secret-brain');
const op = operationsByName.volunteer_context;
const result = (await op.handler(
mkCtx({ remote: true, auth: { allowedSources: ['grant-brain'] } }),
{ window: 'user: intro Alice Example to Bob Sample and Eve Other' },
)) as any;
expect(result.pages.map((p: any) => p.slug)).toEqual(['people/bob-sample']);
});
test('stats mode is source-scoped and returns the approximate note', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
await insertVolunteerEvents(engine, [
{ source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'watch' },
]);
const op = operationsByName.volunteer_context;
const stats = (await op.handler(mkCtx(), { stats: true, days: 7 })) as any;
expect(stats.days).toBe(7);
expect(stats.total_volunteered).toBe(1);
expect(stats.by_arm[0].channel).toBe('watch');
expect(stats.note).toContain('approximate');
});
});
describe('knob clamps — untrusted MCP caller inputs (review hardening)', () => {
test('minConfidence outside [0,1] (or NaN) falls back to the 0.7 default gate', async () => {
await seed('projects/widget-co', 'The Widget Company Project', 'A project.');
const turns = parseWindow('user: updates on Widget-Co?');
for (const bad of [5, -1, Number.NaN]) {
const pages = await volunteerContext(engine, turns, { sourceIds: ['default'], minConfidence: bad });
expect(pages).toEqual([]); // slug-suffix (0.6+boost) stays gated at the default 0.7
}
});
test('maxPages 0 / negative / NaN fall back to the 3-page default', async () => {
for (let i = 0; i < 5; i++) await seed(`people/person-${i}`, `Person Alpha${i}`, 'A person.');
const text = Array.from({ length: 5 }, (_, i) => `Person Alpha${i}`).join(' and ');
for (const bad of [0, -3, Number.NaN]) {
const pages = await volunteerContext(engine, parseWindow(`user: intro ${text}`), {
sourceIds: ['default'],
maxPages: bad,
});
expect(pages.length).toBeLessThanOrEqual(3);
expect(pages.length).toBeGreaterThan(0);
}
});
test('stats days <= 0 / NaN falls back to 30', async () => {
const stats = await volunteerUsageStats(engine, ['default'], -5);
expect(stats.days).toBe(30);
const stats2 = await volunteerUsageStats(engine, ['default'], Number.NaN);
expect(stats2.days).toBe(30);
});
});
describe('window-cap ordering — the newest user mention survives the cap', () => {
test('stale assistant-only chatter is dropped before a newest-turn user entity', async () => {
const { extractCandidatesFromWindow: extract, MAX_CANDIDATES: CAP } = await import('../src/core/context/entity-salience.ts');
// 14 stale assistant-introduced entities in turn 1, then the user names
// ONE entity in the newest turn. The cap (12) must keep the user's.
const stale = Array.from({ length: 14 }, (_, i) => `Stale Chatter${i}`).join(', ');
const cands = extract([
{ role: 'assistant', text: `consider ${stale}.` },
{ role: 'user', text: 'actually ask Alice Example first' },
]);
expect(cands.length).toBeLessThanOrEqual(CAP);
const alice = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Alice Example'));
expect(alice).toBeDefined();
// Recency + user-role weighting puts the newest user mention FIRST.
expect(normalizeAlias(cands[0].query)).toBe(normalizeAlias('Alice Example'));
});
});
describe('volunteer-events sink — timeout branch (long-lived process safety)', () => {
test('a hung write reports unfinished and drops the snapshot (no ghost references)', async () => {
const {
logVolunteerEventsFireAndForget,
awaitPendingVolunteerEventWrites,
_resetPendingVolunteerEventWritesForTests,
_peekPendingVolunteerEventWritesForTests,
} = await import('../src/core/context/volunteer-events.ts');
_resetPendingVolunteerEventWritesForTests();
const hangingEngine = {
executeRaw: () => new Promise(() => { /* never settles */ }),
} as never;
logVolunteerEventsFireAndForget(hangingEngine, [
{ source_id: 'default', slug: 'people/x', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'watch' },
]);
const { unfinished } = await awaitPendingVolunteerEventWrites(20);
expect(unfinished).toBe(1);
// Snapshot dropped so a long-lived `gbrain watch` never accumulates
// references to forever-pending work (the last-retrieved C1 class).
expect(_peekPendingVolunteerEventWritesForTests()).toBe(0);
_resetPendingVolunteerEventWritesForTests();
});
});
+231
View File
@@ -0,0 +1,231 @@
/**
* v0.43 (#2095) `gbrain watch` push transport: streaming loop, rolling
* window, session dedupe, --json shape, event logging on channel 'watch',
* and clean EOF return. Hermetic PGLite + injected line/write deps (no
* subprocess, no real stdin).
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { runWatch, WATCH_HELP } from '../src/commands/watch.ts';
import { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } from '../src/core/context/volunteer-events.ts';
let engine: PGLiteEngine;
async function seed(slug: string, title: string, body: string) {
await engine.executeRaw(
`INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline)
VALUES ($1, 'default', 'person', $2, $3, '')`,
[slug, title, body],
);
}
async function* feed(lines: string[]): AsyncGenerator<string> {
for (const l of lines) yield l;
}
async function watchRun(lines: string[], extraArgs: string[] = []): Promise<string[]> {
const out: string[] = [];
await runWatch(engine, ['--source', 'default', ...extraArgs], {
lines: feed(lines),
write: (s) => out.push(s),
isTTY: false,
});
return out;
}
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
}, 60_000);
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
_resetPendingVolunteerEventWritesForTests();
await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {});
await engine.executeRaw('DELETE FROM pages');
});
describe('gbrain watch (#2095)', () => {
test('--help prints WATCH_HELP and touches nothing', async () => {
const out = await watchRun([], ['--help']);
expect(out.join('')).toBe(WATCH_HELP);
});
test('volunteers per turn and returns cleanly at EOF', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
const out = await watchRun(['ping Alice Example about the deal']);
const text = out.join('');
expect(text).toContain('people/alice-example');
expect(text).toContain('exact title match');
// runWatch RESOLVED — the EOF → clean-return contract (the entrypoint
// flush-exit + finally drain handle the rest in the real CLI).
});
test('rolling window: assistant-introduced entity fires on the pronoun follow-up turn', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
const out = await watchRun([
'user: who should I ask about the round?',
'assistant: Alice Example led one last year.',
'user: what did she invest in?',
]);
expect(out.join('')).toContain('people/alice-example');
});
test('session dedupe: a slug is volunteered at most once per session', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
const out = await watchRun([
'user: ping Alice Example',
'user: ok',
'user: Alice Example again please',
]);
const hits = out.join('').split('people/alice-example').length - 1;
expect(hits).toBe(1);
});
test('--json emits one JSONL row per volunteered page with turn attribution', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
const out = await watchRun(['user: hello there', 'user: ping Alice Example'], ['--json']);
expect(out.length).toBe(1);
const row = JSON.parse(out[0]);
expect(row.slug).toBe('people/alice-example');
expect(row.turn).toBe(2);
expect(row.arm).toBe('title');
expect(typeof row.confidence).toBe('number');
});
test('events land on channel watch with session_id + turn (drained sink)', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
await watchRun(['user: ping Alice Example']);
const { unfinished } = await awaitPendingVolunteerEventWrites(5_000);
expect(unfinished).toBe(0);
const rows = await engine.executeRaw<{ channel: string; session_id: string; turn: number }>(
`SELECT channel, session_id, turn FROM context_volunteer_events`, [],
);
expect(rows.length).toBe(1);
expect(rows[0].channel).toBe('watch');
expect(rows[0].session_id).toMatch(/^watch-/);
expect(Number(rows[0].turn)).toBe(1);
});
test('min-confidence flag gates exactly like the op', async () => {
await seed('projects/widget-co', 'The Widget Company Project', 'A project.');
const gated = await watchRun(['user: updates on Widget-Co?']);
expect(gated.join('')).toBe('');
const loose = await watchRun(['user: updates on Widget-Co?'], ['--min-confidence', '0.5']);
expect(loose.join('')).toContain('projects/widget-co');
});
test('blank lines and CRLF are tolerated; no turn, no volunteer', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
const out = await watchRun(['', ' ', 'user: ping Alice Example\r']);
expect(out.join('')).toContain('people/alice-example');
});
});
describe('gbrain watch — window + cap flags (ship coverage G4)', () => {
test('--window-turns 1: fires on the mention turn itself; the pronoun follow-up adds nothing', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
const out = await watchRun(
[
'assistant: Alice Example led one last year.',
'user: what did she invest in?',
],
['--window-turns', '1', '--json'],
);
// Watch volunteers per turn: the entity fires at turn 1 (its mention
// turn). With window=1 the follow-up turn extracts nothing, so exactly
// one row exists and it carries turn 1 attribution.
expect(out.length).toBe(1);
expect(JSON.parse(out[0]).turn).toBe(1);
});
test('--max-pages 1 caps a multi-entity turn to one volunteered page', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
await seed('people/bob-sample', 'Bob Sample', 'Engineer.');
const out = await watchRun(
['user: intro Alice Example to Bob Sample'],
['--max-pages', '1', '--json'],
);
expect(out.length).toBe(1);
});
});
describe('gbrain watch — red-team hardening', () => {
test('starvation guard: an already-pushed slug never burns the --max-pages cap slot', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
await seed('people/bob-sample', 'Bob Sample', 'Engineer.');
const out = await watchRun(
[
'user: ping Alice Example about the round',
'user: also intro Alice Example to Bob Sample',
],
['--max-pages', '1', '--json'],
);
// Turn 1: Alice takes the single cap slot. Turn 2: Alice is excluded
// BEFORE the cap (not filtered after), so Bob gets the slot instead of
// the turn volunteering nothing forever.
expect(out.length).toBe(2);
expect(JSON.parse(out[0]).slug).toBe('people/alice-example');
expect(JSON.parse(out[1]).slug).toBe('people/bob-sample');
});
test('--window-turns clamps: 0 and negatives behave as window 1', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
const out = await watchRun(
[
'assistant: Alice Example led one last year.',
'user: what did she invest in?',
],
['--window-turns', '0', '--json'],
);
// Clamped to 1: fires on the mention turn only — identical to the
// --window-turns 1 contract above, not a crash or an unbounded window.
expect(out.length).toBe(1);
expect(JSON.parse(out[0]).turn).toBe(1);
});
test('--window-turns absurdly large still runs (hard cap, no unbounded re-scan)', async () => {
await seed('people/alice-example', 'Alice Example', 'Founder.');
const filler = Array.from({ length: 70 }, (_, i) => `user: filler line ${i}`);
const out = await watchRun(
[...filler, 'user: ping Alice Example'],
['--window-turns', '999999', '--json'],
);
expect(out.length).toBe(1);
expect(JSON.parse(out[0]).slug).toBe('people/alice-example');
});
});
describe('gbrain watch — per-turn fail-open (review hardening)', () => {
test('a transient DB error on one turn never kills the stream', async () => {
await seed('people/alice-example', 'Alice Example', 'Alice is a founder.');
// Fail the FIRST resolver query against pages (turn 1's resolution);
// everything else — incl. resolveSourceId's pre-loop check — passes.
let pagesQueries = 0;
const flaky = new Proxy(engine, {
get(target, prop, receiver) {
if (prop === 'executeRaw') {
return (sql: string, params: unknown[]) => {
if (/FROM pages/.test(sql) && ++pagesQueries === 1) {
return Promise.reject(new Error('transient db hiccup'));
}
return target.executeRaw(sql, params);
};
}
return Reflect.get(target, prop, receiver);
},
});
const out: string[] = [];
await runWatch(flaky as never, ['--source', 'default'], {
lines: feed(['user: ping Alice Example', 'user: ping Alice Example please']),
write: (s) => out.push(s),
isTTY: false,
});
// Turn 1 failed open; turn 2 volunteered. The stream survived.
expect(out.join('')).toContain('people/alice-example');
});
});
+75
View File
@@ -0,0 +1,75 @@
/**
* v0.43 (#2095) `gbrain watch` SIGINT lifecycle. SERIAL: spawns a real CLI
* subprocess with a tmpdir brain (the parallel unit shards flake on
* concurrent subprocess spawns same isolation rationale as
* apply-migrations-pglite-spawn.serial.test.ts).
*/
import { describe, test, expect } from 'bun:test';
describe('gbrain watch — SIGINT lifecycle (real subprocess)', () => {
test('SIGINT mid-stream closes the stream and exits cleanly (drain path, exit 0)', async () => {
const { mkdtempSync, mkdirSync, writeFileSync, rmSync } = await import('fs');
const { join, resolve } = await import('path');
const { tmpdir } = await import('os');
const REPO = resolve(import.meta.dir, '..');
const home = mkdtempSync(join(tmpdir(), 'gbrain-watch-sigint-'));
try {
mkdirSync(join(home, '.gbrain'), { recursive: true });
writeFileSync(
join(home, '.gbrain', 'config.json'),
JSON.stringify({
engine: 'pglite',
database_path: join(home, '.gbrain', 'brain.pglite'),
embedding_dimensions: 1536,
}) + '\n',
);
// Piped stdin that NEVER reaches EOF — only SIGINT can end the stream.
const proc = Bun.spawn(['bun', 'run', join(REPO, 'src', 'cli.ts'), 'watch'], {
cwd: REPO,
env: { ...process.env, HOME: home, GBRAIN_HOME: home, GBRAIN_SKIP_STARTUP_HOOKS: '1' },
stdin: 'pipe',
stdout: 'pipe',
stderr: 'pipe',
});
proc.stdin.write('user: nothing relevant here\n');
await proc.stdin.flush();
// Readiness probe: watch prints "[watch] session <id> ready" on stderr
// once engine + source resolution are done and the stdin loop is live.
// A fixed sleep raced cold PGLite init (other tests budget 60s for it)
// — SIGINT before the handler registers means default-disposition kill.
const stderrChunks: string[] = [];
const reader = (proc.stderr as ReadableStream<Uint8Array>).getReader();
const decoder = new TextDecoder();
const deadline = Date.now() + 90_000;
let ready = false;
while (Date.now() < deadline) {
const { value, done } = await reader.read();
if (done) break;
stderrChunks.push(decoder.decode(value, { stream: true }));
if (stderrChunks.join('').includes('ready')) { ready = true; break; }
}
expect(ready).toBe(true);
// Brief settle so the first turn's volunteerContext round-trip is in
// flight or done, then interrupt mid-stream.
await new Promise((r) => setTimeout(r, 500));
proc.kill('SIGINT');
const killer = setTimeout(() => {
try { proc.kill('SIGKILL'); } catch { /* already dead */ }
}, 30_000);
const exitCode = await proc.exited;
clearTimeout(killer);
// Drain the rest of stderr for the banner assertion.
while (true) {
const { value, done } = await reader.read();
if (done) break;
stderrChunks.push(decoder.decode(value, { stream: true }));
}
const stderr = stderrChunks.join('');
// Clean drain-then-exit: no force-exit banner, no SIGKILL (137), exit 0.
expect(stderr).not.toContain('force-exiting');
expect(exitCode).toBe(0);
} finally {
rmSync(home, { recursive: true, force: true });
}
}, 120_000);
});