diff --git a/AGENTS.md b/AGENTS.md index 9efaca672..846a04ce4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -104,8 +104,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont ## Before shipping Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks, -unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a -fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the +guards + typecheck, then 4-shard parallel unit + E2E against four pgvector +containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL` +unset) and tears down. Use `bun run ci:local:diff` for the diff-aware subset during fast iteration on a focused branch. Requires Docker (Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`). diff --git a/CHANGELOG.md b/CHANGELOG.md index a6972e9a6..846c79d67 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,27 @@ All notable changes to GBrain will be documented in this file. +## [0.42.43.0] - 2026-06-12 + +**The brain now volunteers relevant pages instead of waiting to be asked (gbrain#2095).** Retrieval used to be pull-only: a deep session could run for hours with zero brain contributions — not because the brain had nothing, but because nothing prompted the agent to ask, and pages stored under coined names were missed by literal-string queries. Push-based context inverts that, on three channels sharing one zero-LLM, confidence-gated core: the ambient retrieval reflex now reads the last few conversation turns (an entity your assistant introduced two turns ago resolves on the "what did she invest in?" follow-up), a new `volunteer_context` operation gives any agent a per-turn volunteer surface over CLI stdin or MCP, and `gbrain watch` streams volunteered pages as a transcript flows through it. + +Every volunteered page carries an honest confidence (alias match 0.9, exact title 0.8, slug-suffix 0.6, small boosts for repeated or newest-turn mentions; default gate 0.7) and a one-line rationale. A feedback loop closes the tuning circle: volunteered pages are logged, "used" is derived from whether the page actually got retrieved afterwards, and `gbrain volunteer-context --stats` reports per-arm precision (labeled approximate, because the retrieval signal is throttled). Suppression learned the difference between a page that was actually surfaced and one merely mentioned — under windowing only a surfaced page is held back, so prior-turn mentions can't silence themselves. + +This release also lands on top of v0.42.42.0's exit-contract work as a strict superset: the transaction-mode pooler topology behind three consecutive teardown waves is now reproduced in the local CI gate (a real pooler service + an end-to-end teardown test), the exit-verdict sweep is completed across every command surface (notably `gbrain doctor`, whose FAIL verdict could still report exit 0), and a structural guard makes the next raw exit-code write fail in CI instead of silently reporting success on failure. + +### Added +- **`volunteer_context` operation** (CLI: `gbrain volunteer-context`, MCP tool) — pipe recent turns in (`user:` / `assistant:` prefixed lines, or plain text), get confidence-gated page pointers with rationales and synopses out. `--stats` returns the volunteered-vs-used precision summary. Per-call knobs: `max_pages`, `min_confidence`, `session_id`/`turn` attribution. +- **`gbrain watch`** — the streaming push transport: feed a transcript on stdin, volunteered pages stream out (`--json` for JSONL), each slug at most once per session. Piped input exits cleanly at end-of-input; interactive sessions run until Ctrl-C. +- **Rolling-window retrieval reflex** — the ambient channel extracts entities from the last 4 turns (configurable via `retrieval_reflex_window_turns` / `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`; 1 restores the previous single-turn behavior). Assistant-introduced entities and named-antecedent follow-ups now surface pointers with zero agent-initiated queries. +- **Volunteered-context feedback log** — volunteered pages are recorded best-effort (channel, arm, confidence, optional session/turn) with 90-day retention handled by the nightly cycle; rationales are deterministic templates, never raw conversation text. Synopses always strip the takes/facts privacy fences before reaching a prompt. +- **Transaction-mode pooler in the local CI gate** — `bun run ci:local` now runs a real transaction-pooling service in front of Postgres with an end-to-end teardown test, so the bug class behind gbrain#1972/#2015/#2084 is reproducible before it ships, not after. + +### Fixed +- **`gbrain doctor` exits 1 on FAIL again on every engine.** Its verdict write predated the v0.42.42.0 exit-verdict channel and was being silently zeroed; swept, plus a structural test that fails CI on the next raw exit-code write anywhere in the CLI. +- **Reflex pointer suppression under multi-turn windows** distinguishes "page already surfaced" from "entity merely mentioned earlier" — without this, window extraction would have suppressed every prior-turn entity by construction. + +### To take advantage of v0.42.43.0 +`gbrain upgrade` (applies the new feedback-log migration automatically). The wider reflex window is on by default for context-engine hosts — set `retrieval_reflex_window_turns: 1` in `~/.gbrain/config.json` to restore single-turn behavior. Agents without the context engine: call `volunteer_context` per turn (window in, pointers out), or pipe a transcript through `gbrain watch --json`. After a few days, `gbrain volunteer-context --stats` shows which resolution arms are earning their keep; raise or lower `min_confidence` accordingly. ## [0.42.42.0] - 2026-06-12 **`gbrain query` no longer pays a flat 10-second exit tax on managed Postgres behind a transaction-mode pooler — and CLI exit codes finally tell the truth on PGLite.** On deployments where the pooler holds sockets open past the bounded pool drain (gbrain#2084, a residual of gbrain#1972), every query printed its results and then sat for 10 seconds until the force-exit banner fired. The cause was two-layered: the hard-deadline timer was armed *before* the operation handler, so a multi-second search on a large brain burned the teardown budget (and any operation slower than 10 seconds was silently killed mid-run with exit 0 and truncated output); and the CLI never exited explicitly on success — it waited for Bun's event loop to drain, which a stuck pooler socket can hold open forever. diff --git a/CLAUDE.md b/CLAUDE.md index 56d63874d..592b5e1dc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP +Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -97,6 +97,7 @@ detail on demand.) | any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry | | search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` | | search modes / cost knobs | `docs/guides/search-modes.md` | +| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` | | schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` | | thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` | | the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry | diff --git a/TODOS.md b/TODOS.md index 92c22f146..230a59860 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,65 @@ # TODOS +## gbrain#2095 push-based context follow-ups (v0.43+) + +Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`). +Deliberately scoped OUT of v1 per the eng-review scope decision (success criteria +are the bar). Plan + GSTACK REVIEW REPORT at +`~/.claude/plans/system-instruction-you-are-working-cheerful-elephant.md`. + +- [ ] **P3 — SSE/HTTP push channel via serve-http.** The op + `gbrain watch` cover + pull-per-turn and stdin streaming; a serve-http SSE feed would push volunteered + pages to remote agents without a local CLI. **Why:** thin-client/remote-MCP + deployments get push too. **Cons:** async plumbing + auth scoping; no consumer + wired today. **Where:** `src/commands/serve-http.ts` + `src/core/context/volunteer.ts`. + **Blocked by:** a real consumer (revisit when one exists). +- [ ] **P3 — policy skill + doctor check for push-context.** The ambient reflex + needed doctor visibility because silent failure was invisible; volunteer is + invoked-on-demand so v1 skipped it. If `volunteer-context --stats` adoption shows + agents not discovering the surface, ship a `push-context` recipe (mirror + `recipes/retrieval-reflex/`) + a doctor check reading the events table. + **Where:** `recipes/`, `src/commands/doctor.ts`. +- [ ] **P3 — structured `messages[]` param for volunteer_context.** v1 takes a + string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract. + If MCP callers accumulate parsing bugs, add a structured array param beside it. + **Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`. +- [ ] **P3 — index shapes for the per-turn resolver query.** The arm-2 resolver + (`retrieval-reflex.ts`: `lower(title) = ANY() OR slug = ANY() OR slug LIKE + ANY('%/...')`) predates #2095 but now runs per turn on three channels + (reflex window, volunteer_context, watch) federated across sources. Neither + the leading-wildcard suffix arm nor `lower(title)` is index-served. If + per-turn latency telemetry on large brains comes back hot: add + `(source_id, lower(title))` btree + a reverse(slug) text_pattern_ops (or + gin_trgm) index, or split the OR into three index-friendly queries. + **Where:** `src/core/context/retrieval-reflex.ts`, migration. +- [ ] **P3 — batch the volunteer-events pruner's first run after a long gap.** + `purgeStaleVolunteerEvents` is one unbatched DELETE with a bare + `volunteered_at` predicate (full scan; fine for a TTL-bounded table). Edge: + a brain whose dream cycle was off for months could hit the pooler's ~2min + statement_timeout on the first prune, get swallowed by the catch, and never + make progress. If observed: id-batched chunks (`DELETE ... WHERE id IN + (SELECT ... LIMIT 10000)` looped). **Where:** + `src/core/context/volunteer-events.ts:purgeStaleVolunteerEvents`. +- [ ] **P3 — route `gbrain watch` through the serve resolve-IPC on PGLite.** + `watch` connects directly, so on a PGLite brain it monopolizes the single + connection for its whole (potentially hours-long) session — a concurrent + `gbrain serve` or any write path blocks on the lock until watch exits. + WATCH_HELP documents the monopoly; the fix is an IPC rung in watch's + resolver (reuse `resolveViaIpc` like the ambient reflex's ladder) so a + running serve answers and watch never takes the lock. **Why:** watch + + serve concurrently is the natural agent topology. **Where:** + `src/commands/watch.ts`, `src/core/context/resolve-ipc.ts` (red-team RT2). +- [ ] **P3 — capability/version gate for host-injected reflex resolvers.** + Windowing switched the orchestrator's suppression request to 'slug-only'; + a host resolver built against the pre-window contract that still applies + title-whole-word suppression silently self-suppresses every windowed + entity. The contract is documented at `ResolveEntitiesFn` (reflex.ts), but + nothing detects a stale host. Add a capability handshake (e.g. resolver + advertises `supportsSuppressionModes`) and fall back to + `window_turns: 1` semantics when absent. **Where:** + `src/core/context/reflex.ts:ResolveEntitiesFn` + the OpenClaw plugin + contract (red-team RT4). + ## gbrain triage wave follow-ups (filed v0.42.41.0) Deferred from the v0.42.41.0 fix wave (eng-reviewed as separate scope, not hotfixes). @@ -32,11 +92,13 @@ Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extra is deterministic + precision-biased. See plan + GSTACK REVIEW REPORT at `~/.claude/plans/system-instruction-you-are-working-wild-yeti.md`. -- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The v1 extractor - (`src/core/context/entity-salience.ts`) misses lowercase names, many non-Latin - scripts, pronoun follow-ups ("what about her?"), and assistant-introduced entities. - These need conversation state or an LLM pass. **Why:** higher recall on the read - side. **Where:** `entity-salience.ts` + the orchestrator's `priorContextText`. +- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The extractor + (`src/core/context/entity-salience.ts`) misses lowercase names and many non-Latin + scripts; these need an LLM pass or script-aware heuristics. **Why:** higher recall + on the read side. **Where:** `entity-salience.ts`. *(Partially done by the #2095 + wave: `extractCandidatesFromWindow` now covers assistant-introduced entities and + pronoun follow-ups whose antecedent was NAMED in the rolling window; true pronoun + coreference for never-named antecedents remains with the LLM-pass idea.)* - [ ] **P3 — recall knob: optional fuzzy/prefix-expansion resolution.** The resolver (`src/core/context/retrieval-reflex.ts`) is exact-only (alias + title + slug-suffix) for precision. Revisit adding `resolveEntitySlug`'s trgm-fuzzy / prefix-expansion diff --git a/VERSION b/VERSION index b7fef61ef..a8d19f9fa 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.42.0 \ No newline at end of file +0.42.43.0 diff --git a/docker-compose.ci.yml b/docker-compose.ci.yml index fc42df13a..787899d3a 100644 --- a/docker-compose.ci.yml +++ b/docker-compose.ci.yml @@ -85,6 +85,40 @@ services: volumes: - gbrain-ci-pg-data-4:/var/lib/postgresql/data + # v0.43 (#2084 / eng-review TD1): PgBouncer in TRANSACTION pooling mode + # fronting postgres-1 — the production topology (Supabase direct :5432 + + # pooled :6543) behind three consecutive pooler-teardown waves + # (#1972 → #2015 → #2084) that CI could never reproduce. + # test/e2e/pgbouncer-teardown.test.ts uses a DEDICATED database + # (gbrain_pgbouncer) on postgres-1 so it never races shard 1's + # TRUNCATE-based fixtures; pgbouncer's wildcard [databases] section + # forwards any dbname to DB_HOST. + pgbouncer: + image: edoburu/pgbouncer:latest + environment: + DB_HOST: postgres-1 + DB_PORT: "5432" + DB_USER: postgres + DB_PASSWORD: postgres + POOL_MODE: transaction + # plain (CI-only): pg16 stores SCRAM verifiers, and pgbouncer can only + # answer the server's SCRAM challenge when its userlist holds the + # PLAINTEXT password — an md5-hashed userlist fails with + # "server login failed: wrong password type". + AUTH_TYPE: plain + MAX_CLIENT_CONN: "200" + DEFAULT_POOL_SIZE: "10" + # gbrain's client sets statement_timeout + idle_in_transaction_session_timeout + # as startup parameters (db.ts buildConnectionParams); the Supabase pooler + # whitelists them, so this pooler must too or every connection is refused + # before the teardown path is even reached. + IGNORE_STARTUP_PARAMETERS: extra_float_digits,statement_timeout,idle_in_transaction_session_timeout,search_path + ports: + - "${GBRAIN_CI_PGBOUNCER_PORT:-6543}:5432" + depends_on: + postgres-1: + condition: service_healthy + runner: image: oven/bun:1 working_dir: /app @@ -97,6 +131,8 @@ services: condition: service_healthy postgres-4: condition: service_healthy + pgbouncer: + condition: service_started # No global DATABASE_URL — scripts/ci-local.sh sets per-shard URL via -e. # Unit phase explicitly unsets DATABASE_URL so test/e2e/* gracefully skip. volumes: diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 14ba6765f..3cff19ddc 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -12,15 +12,17 @@ Before shipping (/ship) or reviewing (/review), always run the full test suite. Two equivalent paths: **Path A — local CI gate (recommended, v0.23.1+):** -- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), unit - tests with `DATABASE_URL` unset, and all 29 E2E files sequentially against a - fresh pgvector container. Stronger than PR CI's 2-file Tier 1 set; closer to - what nightly Tier 1 catches. Spins up + tears down postgres automatically via - `docker-compose.ci.yml`. Override the host port with - `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides. +- `bun run ci:local` runs the entire stack inside Docker: gitleaks (host), + guards + typecheck, then 4-shard parallel unit + E2E against four pgvector + containers plus a transaction-mode PgBouncer service (unit phase keeps + `DATABASE_URL` unset; `--no-shard` for the legacy sequential flow). Stronger + than PR CI's 2-file Tier 1 set; closer to what nightly Tier 1 catches. Spins + up + tears down postgres automatically via `docker-compose.ci.yml`. Override + the host port with `GBRAIN_CI_PG_PORT=5435 bun run ci:local` if 5434 collides. - `bun run ci:local:diff` runs only the E2E files matched by the diff selector - (`scripts/select-e2e.ts`), falling back to all 29 on unmapped src/ paths or - schema/skills/package.json changes. Fast iteration during a focused branch. + (`scripts/select-e2e.ts`), falling back to ALL E2E files on unmapped src/ + paths or schema/skills/package.json changes. Fast iteration during a focused + branch. **Path B — manual lifecycle (still supported):** - `bun test` — unit tests (no database required) diff --git a/docs/TESTING.md b/docs/TESTING.md index d39384b98..93b33252c 100644 --- a/docs/TESTING.md +++ b/docs/TESTING.md @@ -15,7 +15,7 @@ Seven test command tiers, each with a clear scope: | `bun run test:slow` | Just the `*.slow.test.ts` set (intentional cold-path correctness checks). | seconds-to-minutes | When touching slow-path code. | | `bun run test:serial` | Just the `*.serial.test.ts` set (cross-file-contention quarantine; one bun process per file for true module-registry isolation). | ~1s per quarantined file | Debugging a specific quarantined file. | | `bun run test:e2e` | Real Postgres E2E. Requires Docker + `DATABASE_URL`. Sequential. | ~5-10min | Pre-ship; nightly. | -| `bun run check:all` | All 7 historical pre-checks (privacy + jsonb + progress + no-legacy-getconnection + trailing-newline + wasm + exports-count). Superset of `verify`. | ~10s | Local-only sweep. The 4 not in `verify` are nice-to-haves. | +| `bun run check:all` | The historical pre-check scripts (22, chained sequentially in package.json). Overlaps `verify` heavily but is NOT a superset — `verify`'s `CHECKS` array in `scripts/run-verify-parallel.sh` (~30 entries incl. typecheck) is the authoritative gate; `check:all` keeps a few local-only extras (trailing-newline, exports-count, no-legacy-getconnection). | ~10s | Local-only sweep for the extras. | ### CI vs local: intentionally divergent file sets @@ -126,6 +126,12 @@ Unit tests and what they cover: - `test/cli-finish-teardown.test.ts` — the #2084 teardown contract: `computeTeardownDeadlineMs` formula/floor/live-registry scaling + `GBRAIN_TEARDOWN_DEADLINE_MS` override (garbage/zero/negative values fall back to the formula); `finishCliTeardown` clean path (drain BEFORE disconnect, no exit, no warn), backstop on hung drain or disconnect (honors an errored op's exit code), throwing drain/disconnect warned + swallowed; the gbrain-owned verdict channel is immune to PGLite WASM `process.exitCode` writes; `flushThenExit` unit coverage with mocked streams (exits once after both stream callbacks, non-TTY aliveness grace, blocked-pipe guard, EPIPE-safe, `GBRAIN_FLUSH_GRACE_MS` override). - `test/flush-then-exit-harness.test.ts` — real spawned-Bun pipe semantics for `flushThenExit` (fixture: `test/fixtures/flush-then-exit-harness.ts`): a 4MB piped stdout payload arrives byte-complete with the exit code even with a late reader, small output survives exit with a concurrent reader, and the fence resolves promptly (wall time well under the guard + grace ceiling). - `test/cli-should-force-exit.test.ts` — `shouldForceExitAfterMain` daemon-survival gate: `serve` (stdio and `--http`) never force-exits, including with preceding global flags; op commands / empty / flag-only argv do; the #2084 case that space-separated global-flag VALUES can't fake a command (`--timeout 30s serve` resolves to the `serve` daemon, not a `30s` command). +- `test/cli-exit-verdict-pin.test.ts` — #2084 structural class pin: greps `src/` so the NEXT raw `process.exitCode =` write fails CI (a raw write bypasses the gbrain-owned verdict channel and gets silently zeroed by the deliberate flush-exit — the bug that made doctor's FAIL path exit 0). Runtime variants live in `test/cli-finish-teardown.test.ts`; this is the review-time guard. +- `test/cli-pipe-truncation.test.ts` — real-CLI pipe completeness (the #1959 incident class), implementation-agnostic: the actual CLI run the way agents run it (piped stdout) produces complete, parseable, byte-stable `--tools-json` output and exits deliberately, well under the teardown backstop. Synthetic flush-mechanism coverage stays in `test/flush-then-exit-harness.test.ts`. +- `test/volunteer-context.test.ts` — push-based context core (#2095), hermetic in-memory PGLite: `parseWindow` lenient `user:`/`assistant:` parsing, multi-turn window extraction, confidence-gated volunteering (arm confidences, multi-turn/newest-turn boosts, `min_confidence` gate, max-pages cap), slug-only suppression, privacy (rationales are deterministic templates; synopses pass the takes/facts fence), and the approximate usage-stats join. +- `test/watch-command.test.ts` — `gbrain watch` push transport (#2095): streaming loop, rolling window, session dedupe, `--json` JSONL shape, `channel: 'watch'` event logging, clean EOF return. Hermetic PGLite + injected line/write deps (no subprocess, no real stdin). +- `test/watch-sigint.serial.test.ts` — `gbrain watch` SIGINT lifecycle against a real spawned CLI subprocess with a tmpdir brain. SERIAL: parallel unit shards flake on concurrent subprocess spawns (same rationale as `apply-migrations-pglite-spawn.serial.test.ts`). +- `test/cli-format-volunteer.test.ts` — `formatResult`'s `volunteer_context` human rendering: pointer lines with confidence/arm/rationale, the empty-result message, the approximate stats summary. - `test/config.test.ts` — config redaction. - `test/files.test.ts` — MIME/hash. - `test/import-file.test.ts` — import pipeline. @@ -133,7 +139,7 @@ Unit tests and what they cover: - `test/file-migration.test.ts` — file migration. - `test/file-resolver.test.ts` — file resolution. - `test/import-resume.test.ts` — import checkpoints. -- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion. +- `test/migrate.test.ts` — migration: v8/v9 helper-btree-index SQL structural assertions; 1000-row wall-clock fixtures guarding the O(n²)→O(n log n) fix; v12/v13 SQL shape; `sqlFor` + `transaction:false` runner semantics; the `max_stalled DEFAULT 1` regression guard; v24 `sqlFor.pglite: ''` no-op assertion; v117 `context_volunteer_events` (named + idempotent entry, documented columns + both source-scoped indexes after `initSchema`, insert + 90-day `purgeStaleVolunteerEvents` round-trip). - `test/bootstrap.test.ts` — bootstrap contract: no-op on fresh install, idempotent across two `initSchema()` calls, no-op on modern brain that already has every probed column, full bootstrap path on a simulated legacy brain, fresh-install regression guard, legacy `links` shape coverage. - `test/schema-bootstrap-coverage.test.ts` — CI guard. `REQUIRED_BOOTSTRAP_COVERAGE` lists every forward reference in `PGLITE_SCHEMA_SQL`; the test fails loudly if `applyForwardReferenceBootstrap` skips one (extend both arrays when adding a column-with-index to the embedded schema blob). Also parses `src/core/migrate.ts` source text for every `ALTER TABLE ... ADD COLUMN` (top-level `sql:`, `sqlFor.{postgres,pglite}` overrides, AND handler-body `engine.runMigration(N, \`ALTER TABLE ...\`)`) and asserts each (table, column) pair is covered by the bootstrap OR by the schema blob's CREATE TABLE bodies — catching the column-only forward-reference class (e.g. `sources.archived`, `oauth_clients.source_id`) that a CREATE INDEX parser alone can't see. `parseBaseTableColumns` strips SQL line + block comments before identifying column names so commented-out lines don't hide adjacent columns. - `test/helpers/schema-diff.ts` + `test/helpers/schema-diff.test.ts` + `test/e2e/schema-drift.test.ts` — cross-engine schema parity gate. Helper exports pure `snapshotSchema(query)` / `diffSnapshots(pg, pglite, opts)` / `formatDiffForFailure(diff)` / `isCleanDiff(diff)` over a four-tuple per column (`data_type`, `udt_name`, `is_nullable`, `column_default`). E2E test spins up fresh PGLite + Postgres, runs `engine.initSchema()` on each, snapshots `information_schema.columns`, then diffs. 2-table allowlist (`files`, `file_migration_ledger`) — every other Postgres table must reach PGLite via `PGLITE_SCHEMA_SQL` or a migration's `sqlFor.pglite` branch. Sentinels for `oauth_clients`, `mcp_request_log`, `access_tokens`, `eval_candidates` give tighter blame messages. Skips without `DATABASE_URL`. Wired into `scripts/e2e-test-map.ts` so changes to `src/schema.sql`, `src/core/pglite-schema.ts`, or `src/core/migrate.ts` trigger it. The failure message names every drift with a paste-ready hint pointing at `src/core/pglite-schema.ts`. @@ -223,6 +229,8 @@ E2E tests live in `test/e2e/` and run against real Postgres+pgvector (require `D - `test/e2e/upgrade.test.ts` — check-update against real GitHub API (network required). - `test/e2e/minions-shell-pglite.test.ts` — PGLite `--follow` inline shell-job path (in-memory, no `DATABASE_URL` required) — the path the minion-orchestrator skill documents for dev use. - `test/e2e/pglite-cli-exit.serial.test.ts` — real spawned-CLI exit behavior on PGLite (in-memory, no `DATABASE_URL`): read commands (`search`/`get`/`query`) exit 0 promptly; CLI_ONLY `capture` exits clean and frees the single-writer lock; the `#2084` describes pin every swept disconnect site — a failed op exits 1 with the error on stderr, and the dashboard, read-only-timeout, doctor, and `dream --dry-run` paths all exit with no force-exit banner. +- `test/e2e/pgbouncer-teardown.test.ts` — PgBouncer TRANSACTION-mode teardown (#2084 / the #1972→#2015→#2084 class). Pins the bug CLASS, not timings: a CLI op against a txn-mode pooled URL exits 0 with intact stdout and does NOT ride the 10s hard-deadline backstop (the `engine.disconnect() did not return` banner is the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). Gated by `GBRAIN_PGBOUNCER_URL` + `GBRAIN_PGBOUNCER_DIRECT_URL` (NOT `DATABASE_URL`) — set automatically by `bun run ci:local`'s `pgbouncer` compose service; skips gracefully elsewhere. Uses a DEDICATED `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures. +- `test/e2e/volunteer-context-postgres.test.ts` — `volunteer_context` on REAL Postgres (#2095; engine parity beyond the hermetic PGLite unit suite): resolution arms through the actual op handler, the fire-and-forget volunteer-event sink landing rows, the stats join, and the RLS pin that `context_volunteer_events` has ROW LEVEL SECURITY enabled (keeps the v35 auto-RLS event trigger honest for migration-created tables). `DATABASE_URL`-gated. - `test/e2e/openclaw-reference-compat.test.ts` — `check-resolvable` + `skillpack install` against a minimal AGENTS.md workspace fixture (`test/fixtures/openclaw-reference-minimal/`), regression guard for the OpenClaw deployment shape. - `test/e2e/search-swamp.test.ts` — reproduces the source-swamp case. Seeds a curated `originals/talks/article-outline-fat-code` page against two `/chat/` pages stuffed with the same multi-word phrase. Asserts the article wins keyword AND vector ranking, that `detail=high` lets the chat swamp re-surface, and that `source_id` passes through the two-stage CTE intact. PGLite in-memory. - `test/e2e/search-exclude.test.ts` — `test/` + `archive/` pages hidden by default, `include_slug_prefixes` opts back in, caller-supplied `exclude_slug_prefixes` adds to defaults. Both keyword and vector search paths. diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index 667217659..005fd9e0c 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -11,16 +11,17 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/operations.ts` — Contract-first operation definitions (the foundation). Exports upload validators `validateUploadPath`, `validatePageSlug`, `validateFilename`, plus `matchesSlugAllowList(slug, prefixes)` (glob matcher: `/*` matches recursive children; bare `` matches exact only). `OperationContext.remote` is a REQUIRED field flagging untrusted callers; `OperationContext.allowedSlugPrefixes` is the trusted-workspace allow-list set by the dream cycle; `OperationContext.auth?: AuthInfo` is threaded through HTTP dispatch for scope enforcement in `serve-http.ts` before the op runs. `put_page` enforces: when `viaSubagent` and `allowedSlugPrefixes` is set, slug must match the allow-list; else the legacy `wiki/agents//...` namespace check applies. Auto-link skipped only when `remote=true && !trustedWorkspace`. Every `Operation` carries `scope?: 'read' | 'write' | 'admin'` + `localOnly?: boolean`; `sync_brain`, `file_upload`, `file_list`, `file_url` are `admin + localOnly` (rejected over HTTP). Four trust-boundary call sites (`put_page` allowlist, file_upload trust-narrowing, submit_job protected-name guard, auto-link skip) use FAIL-CLOSED semantics: `ctx.remote === false` for trusted-only sites, `ctx.remote !== false` for "untrust unless explicit-false" — anything not strictly `false` is treated as remote (closes the HTTP MCP shell-job RCE where a read+write OAuth token could submit `shell` jobs). `sourceScopeOpts(ctx)` encodes the source-scoped read precedence ladder — federated array (`ctx.auth.allowedSources`) wins over scalar (`ctx.sourceId`/`ctx.auth.sourceId`) over nothing; every read-side op handler routes through it so a source-bound OAuth client can't see neighboring sources via `search`/`query`/`list_pages`/`get_page`/`find_experts`/query's image path. `put_page`'s inline disk write-through is the shared `writePageThrough` helper (`src/core/write-through.ts`), ATOMIC via temp-sibling + rename so a crash or concurrent `gbrain sync` can't read a half-written `.md`; same helper backs `gbrain brainstorm/lsd --save`. Link provenance surface (#1941): `add_link` (`gbrain link`/`link-add`) + `remove_link` (`gbrain unlink`/`link-rm`) expose `link_source`/`link_type`; `add_link` rejects the reconciliation-managed built-ins via `MANAGED_LINK_SOURCES` (`markdown`/`frontmatter`/`mentions`/`wikilink-resolved`) and defaults omitted provenance to `'manual'` (the engine's own default stays `'markdown'` for internal callers); `list_link_sources` (`gbrain link-sources`, read) lists provenances via `sourceScopeOpts`. CLI aliases register through `cliHints.aliases` (collision-guarded in `src/cli.ts`). - `src/core/engine.ts` — Pluggable engine interface (BrainEngine). `clampSearchLimit(limit, default, cap)` takes an explicit cap so per-operation caps can be tighter than `MAX_SEARCH_LIMIT`. Exports `LinkBatchInput`/`TimelineBatchInput` for the bulk-insert API (`addLinksBatch`/`addTimelineEntriesBatch`). `readonly kind: 'postgres' | 'pglite'` discriminator lets `src/core/migrate.ts` and others branch without `instanceof` + dynamic imports. Methods: `batchLoadEmotionalInputs(slugs?)` (CTE-shaped read with per-table aggregates so page × N tags × M takes never produces N×M rows), `setEmotionalWeightBatch(rows)` (`UPDATE FROM unnest($1::text[],$2::text[],$3::real[])` composite-keyed on `(slug, source_id)`), `getRecentSalience(opts)`, `findAnomalies(opts)`. `PageFilters` has `sort?: 'updated_desc'|'updated_asc'|'created_desc'|'slug'` + `PAGE_SORT_SQL` whitelist consumed by both engines. `listAllPageRefs(): Promise>` ordered by `(source_id, slug)` — cheap cross-source enumeration replacing the `getAllSlugs()→getPage(slug)` N+1 (which silently defaulted to `source_id='default'`); parity across postgres-engine.ts + pglite-engine.ts; Pinned by `test/e2e/multi-source-bug-class.test.ts`. `SearchOpts`+`PageFilters` add `sourceIds?: string[]` (federated read axis; both engines apply `WHERE source_id = ANY($N::text[])` when set, preserve scalar `sourceId` fast path when unset); `traverseGraph(slug, depth, opts?)` and `traversePaths(slug, opts?)` accept `opts.sourceId`/`opts.sourceIds`. `traverseGraph` opts has `frontierCap?: number` (per-iteration recursive-CTE cap, approx per-BFS-layer); return type `Promise` for MCP wire stability; export `TraverseGraphOpts`; Postgres uses parenthesized `LIMIT N ORDER BY (slug, id)` inside the recursive term, PGLite mirrors with positional params; Pinned by `test/regressions/v0_36_frontier_cap.test.ts`. Phantom-redirect methods: `refreshPageBody(slug, sourceId, compiled_truth, timeline, content_hash)` narrow-UPDATEs three columns + updated_at, skipping soft-deleted rows (content_hash refresh required so `gbrain sync` sees the canonical as unchanged after fence merge); `migrateFactsToCanonical(phantomSlug, canonicalSlug, sourceId)` UPDATEs `entity_slug`+`source_markdown_slug` on every active fact row keyed on the phantom, preserving embedding/validUntil/kind/status/source_session/confidence; parity at `test/phantom-redirect-engine-parity.test.ts`. `getAdjacencyBoosts(pageIds): Promise>` powers the per-query graph-signals stage — one SQL query returning inbound-link counts among top-K plus a cross-source count (links from differing `source_id`); `COALESCE(p.source_id,'default')` null safety, `HAVING >= 1`, cross-source CASE-WHEN excludes the target's own source; parity SQL across both engines; `SearchResult` gains optional `base_score`, `backlink_boost`, `salience_boost`, `recency_boost`, `exact_match_boost`, `graph_adjacency_boost`, `graph_cross_source_boost`, `session_demote_factor`, `reranker_delta` + internal staging fields; Pinned by `test/e2e/graph-signals-engine.test.ts`. Two REQUIRED methods: `deletePages(slugs, {sourceId}): Promise` (single-batch primitive returning slugs actually deleted) and `resolveSlugsByPaths(paths, {sourceId}): Promise>` (batch path→slug lookup); `sourceId` REQUIRED on both at the type level (asymmetric with single-row `deletePage` which keeps optional/'default'); both short-circuit on empty input and throw when `> DELETE_BATCH_SIZE`. Embedding-signature stale-detection quartet: `countStaleChunks(opts?)` gains optional `signature?: string` widening the stale predicate from `embedding IS NULL` to ALSO include chunks whose JOINed page `embedding_signature IS NOT NULL AND <> $signature` (NULL signature is GRANDFATHERED, never counted; omit `signature` for the legacy NULL-only count); `sumStaleChunkChars(opts?: {sourceId?, signature?}): Promise` = `SUM(LENGTH(chunk_text))` over stale chunks (same predicate + embed_skip filter + optional sourceId scope), used by `gbrain sync --all` cost preview via `estimateCostFromChars`; `setPageEmbeddingSignature(slug, {sourceId?, signature})` stamps `pages.embedding_signature` after a page's chunks (re)embed, idempotent no-op when page absent; `invalidateStaleSignatureEmbeddings({signature, sourceId?}): Promise` NULLs `embedding`+`embedded_at` on every chunk whose page signature is set AND differs, returning the count, called BEFORE `listStaleChunks` so signature-drift pages flow through the NULL-embedding keyset cursor unchanged (NULL never invalidated). Widens `findOrphanPages(opts?: {sourceId?, sourceIds?})` (candidate-side scoping only; inbound links counted from any source). Pinned by `test/sum-stale-chunk-chars.test.ts`, `test/embedding-signature-stale.test.ts`, `test/e2e/engine-parity.test.ts`. Free-text alias layer: `resolveAliases(aliasNorms, opts?): Promise>>` (READ; maps each normalized alias to declaring `(slug, source_id)` pairs, source-scoped) and `setPageAliases(slug, sourceId, aliasNorms)` (WRITE; replaces the full alias set, delete-then-insert, empty clears, idempotent on the unique triple), called by the `importFromContent` ingest projection and the `reindex --aliases` backfill; parity across both engines, Pinned by `test/search/page-aliases-engine.test.ts`. `searchVector` in both engines injects the shared `buildBestPerPagePoolCte` per-page max-pool so a page surfaces on its strongest chunk. `executeRawDirect(sql, params?, opts?)` is the lock-hot-path sibling of `executeRaw`: same single-statement contract, but routes to the direct session-mode pool when dual-pool is active (Postgres/Supabase port 5432) so a long-held lock heartbeat survives the transaction pooler's per-transaction connection recycling; PGLite delegates straight to `executeRaw` (no pooler). Both engines implement it; the Minion lock path (`claim`/`renewLock`) is the consumer. `reconnect(ctx?: {error?})` is a REQUIRED lifecycle method on BOTH engines: it recovers a dropped connection using the config captured at the last `connect()`, so callers (autopilot health probe, `batchRetry`) never `disconnect()` + bare `connect()` (which loses the config and throws `database_url undefined` forever, and opens a null-connection window). PostgresEngine rebuilds its pool with a `_reconnecting` reentrancy guard and emits pool-recovery audit; PGLite is single-writer in-process so it just restores the saved data dir for parity. - `src/core/engine-constants.ts` — single source of truth for engine batch-sizing constants. Exports `DELETE_BATCH_SIZE = 500` consumed by both engines' `deletePages` + `resolveSlugsByPaths` and by the sync delete + rename loops. Lives outside `engine.ts` (the interface module) to avoid circular-import worry — bounded per-statement work for predictable lock hold time + write amplification. -- `src/core/background-work.ts` (#1762/#1745/#1775) — process background-work registry: the single owner of "drain every fire-and-forget DB-write sink before the CLI disconnects." `registerBackgroundWorkDrainer({name, order, drain(timeoutMs), abort?})` + `drainAllBackgroundWorkForCliExit({timeoutMs})` over a `Map` (idempotent registration by name; `__registerDrainerForTest` returns an unregister handle). Drains in explicit `(order, name)` order — facts FIRST (order 0) so its abort-path DB `logIngest` runs against the freshest live engine — and AWAITS `abort()` only when `drain()` reports `unfinished>0`. Best-effort per drainer: one sink's failure never blocks the others or the disconnect. FOUR sinks register at module import (rule-of-four): `facts/queue.ts` (order 0; `abort`=`shutdown()` cancels a hung facts:absorb Haiku via internalAbort), `last-retrieved.ts` (order 1), `search/hybrid.ts` (order 2; `awaitPendingSearchCacheWrites` bounded via `Promise.race`), `eval-capture.ts` (order 3; `captureEvalCandidate` self-tracks its promise via `awaitPendingEvalCaptures`). Every cli.ts teardown site reaches it through `finishCliTeardown` (`src/core/cli-force-exit.ts`), which drains the registry before `engine.disconnect()` — closing the PGLite busy-loop where `db.close()` raced an in-flight job and pinned the single-writer lock (#1762). Exports `backgroundWorkSinkCount()` so the teardown helper computes its backstop deadline from the registered sink count. CLI-EXIT-ONLY: the facts `shutdown()` abort is permanent process state, never call in a long-lived `gbrain serve`. Companion changes: `src/core/ai/gateway.ts` `withDefaultTimeout(caller, ms)` bounds every outbound AI call (chat 300s, embed+multimodal 60s; env `GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS`; composed with caller signals via `AbortSignal.any`) and the teardown backstop honors an errored op's exit code so a hung disconnect can't mask failure as success (see `cli-force-exit.ts`); `src/core/postgres-engine.ts` `reconnect()` module-mode branch re-establishes via idempotent `db.connect()` + `connectionManager.setReadPool` refresh instead of `db.disconnect()` (no null window for concurrent ops; fail-loud on real connect failure — #1745); `src/core/search/hybrid.ts` `embedQueryBounded` + a shared `QueryEmbedDeadline` (6s, floored 2s per embed via `MIN_QUERY_EMBED_BUDGET_MS`; env `GBRAIN_QUERY_EMBED_TIMEOUT_MS`) bounds the cache-lookup AND inner query embeds so a stalled provider falls back to keyword instead of stalling the whole op (#1775). Incorporates + hardens PR #1763 (@ElliotDrel). Pinned by `test/core/background-work.test.ts`, `test/search/query-embed-deadline.test.ts`, `test/eval-capture-drain.test.ts`, `test/e2e/postgres-reconnect-singleton.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts`, `test/fix-wave-structural.test.ts`. +- `src/core/background-work.ts` (#1762/#1745/#1775) — process background-work registry: the single owner of "drain every fire-and-forget DB-write sink before the CLI disconnects." `registerBackgroundWorkDrainer({name, order, drain(timeoutMs), abort?})` + `drainAllBackgroundWorkForCliExit({timeoutMs})` over a `Map` (idempotent registration by name; `__registerDrainerForTest` returns an unregister handle). Drains in explicit `(order, name)` order — facts FIRST (order 0) so its abort-path DB `logIngest` runs against the freshest live engine — and AWAITS `abort()` only when `drain()` reports `unfinished>0`. Best-effort per drainer: one sink's failure never blocks the others or the disconnect. FIVE sinks register at module import: `facts/queue.ts` (order 0; `abort`=`shutdown()` cancels a hung facts:absorb Haiku via internalAbort), `last-retrieved.ts` (order 1), `search/hybrid.ts` (order 2; `awaitPendingSearchCacheWrites` bounded via `Promise.race`), `eval-capture.ts` (order 3; `captureEvalCandidate` self-tracks its promise via `awaitPendingEvalCaptures`), `context/volunteer-events.ts` (order 4, #2095 — batched volunteer-event INSERTs, drained like the rest). Every cli.ts teardown site reaches it through `finishCliTeardown` (`src/core/cli-force-exit.ts`), which drains the registry before `engine.disconnect()` — closing the PGLite busy-loop where `db.close()` raced an in-flight job and pinned the single-writer lock (#1762). Exports `backgroundWorkSinkCount()` so the teardown helper computes its backstop deadline from the registered sink count. CLI-EXIT-ONLY: the facts `shutdown()` abort is permanent process state, never call in a long-lived `gbrain serve`. Companion changes: `src/core/ai/gateway.ts` `withDefaultTimeout(caller, ms)` bounds every outbound AI call (chat 300s, embed+multimodal 60s; env `GBRAIN_AI_{CHAT,EMBED,MULTIMODAL}_TIMEOUT_MS`; composed with caller signals via `AbortSignal.any`) and the teardown backstop honors an errored op's exit code so a hung disconnect can't mask failure as success (see `cli-force-exit.ts`); `src/core/postgres-engine.ts` `reconnect()` module-mode branch re-establishes via idempotent `db.connect()` + `connectionManager.setReadPool` refresh instead of `db.disconnect()` (no null window for concurrent ops; fail-loud on real connect failure — #1745); `src/core/search/hybrid.ts` `embedQueryBounded` + a shared `QueryEmbedDeadline` (6s, floored 2s per embed via `MIN_QUERY_EMBED_BUDGET_MS`; env `GBRAIN_QUERY_EMBED_TIMEOUT_MS`) bounds the cache-lookup AND inner query embeds so a stalled provider falls back to keyword instead of stalling the whole op (#1775). Incorporates + hardens PR #1763 (@ElliotDrel). Pinned by `test/core/background-work.test.ts`, `test/search/query-embed-deadline.test.ts`, `test/eval-capture-drain.test.ts`, `test/e2e/postgres-reconnect-singleton.test.ts`, `test/e2e/pglite-cli-exit.serial.test.ts`, `test/fix-wave-structural.test.ts`. - `src/core/search/graph-signals.ts` — per-query graph-signals helper. `applyGraphSignals(results, engine, opts)` runs as the 4th post-fusion stage (after backlink/salience/recency). Three boosts: `ADJACENCY_BOOST=1.05` (page linked from 2+ OTHER top-K results — local hub for THIS query), `CROSS_SOURCE_BOOST=1.10` (page linked from 2+ DIFFERENT sources — corroborated across team brains, dormant in single-source brains), `SESSION_DEMOTE=0.95` (3+ results from same chat session — keep the highest-scoring at full score, demote the rest). All three inherit the floor-ratio gate preventing weak pages from being boosted past strong ones via popularity. `computeScoreDistribution(results)` emits min/p25/p50/p75/p95/max + `reorder_band_width`. `sessionPrefix(slug)` extracts the chat-session anchor (`chat/2026-05-15-...`). Pure `pairedBootstrapPValue(deltas, resamples, rng)` exported for eval gates. Test seam via `adjacencyFn` DI. Fail-open: any error logs via `logGraphSignalsFailure` (JSONL audit via `audit-writer`) and returns the input array unchanged. Pinned by `test/search/graph-signals.test.ts` (incl. the IRON-RULE floor-gate regression). - `src/core/search/hybrid.ts` extension — `runPostFusionStages` has a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Each post-fusion stage stamps its multiplier: `applyBacklinkBoost`→`backlink_boost`, `applySalienceBoost`→`salience_boost`, `applyRecencyBoost`→`recency_boost`. `applyReranker` (earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution powers `gbrain search --explain` — every boost surface carries its own field so `formatResultsExplain` reads them all without coupling to internal stage ordering. - `src/core/search/explain-formatter.ts` — renders `SearchResult[]` as a multi-line per-result breakdown for `gbrain search --explain`. Reads every boost-stamping field. Handles the "no boosts applied" empty path. 4-decimal precision with trailing-zero strip. Pinned by `test/search/explain-formatter.test.ts`. - `src/core/search/mode.ts` extension — `graph_signals: boolean` knob in `ModeBundle` (defaults: `conservative=false`, `balanced=true`, `tokenmax=true`). `KNOBS_HASH_VERSION` appends a `gs=` parts entry per the cache-key contamination convention so a graph-on cache write can't be served to a graph-off lookup. `SearchKeyOverrides` + `SearchPerCallOpts` + `loadOverridesFromConfig` + `SEARCH_MODE_CONFIG_KEYS` + `resolveSearchMode` + `attributeKnob` all carry the field. Opt-out: `gbrain config set search.graph_signals false`. Mid-deploy `query_cache` rows from before the upgrade hash differently — natural row segregation, clears within `cache.ttl_seconds` (3600s default). -- `src/core/context-engine.ts` + `src/openclaw-context-engine.ts` — the deterministic context engine OpenClaw loads on every turn (`assemble()` injects the Live Context block, zero-LLM). `createGBrainContextEngine({workspaceDir, resolveEntities?})` accepts an OPTIONAL host-injected resolver (`ENGINE_API_VERSION` 0.2.0, additive — older hosts work unchanged; the plugin entry maps `ctx.resolveEntities`/`ctx.brainQuery` onto it). `assemble()` runs the Retrieval Reflex after the Live Context block: extracts the current turn's user text, builds prior-context text (every message EXCEPT the current turn — suppression must not see the triggering mention), and appends the pointer block. `warmReflex()` fires at construction. -- `src/core/context/` — Retrieval Reflex (Layer 1, issue #1981). `entity-salience.ts`: pure, zero-LLM, precision-biased `extractCandidates(text)` (capitalized runs + `@handles`, STOPWORDS + soft COMMON_WORDS + sentence-start guard, deterministic, capped). `retrieval-reflex.ts`: `resolveEntitiesToPointers(engine, sourceId, candidates, opts)` — alias arm (`resolveAliases`, caught per-arm for pre-v110 brains) + exact title/slug-suffix arm (the recall fix: real slugs are namespaced `people/x` but `slugify` drops the prefix); synopsis runs through `stripTakesFence`/`stripFactsFence` (the same privacy boundary `get_page` applies) so private facts never reach the prompt; suppression scans `priorContextText` only; capped at `MAX_POINTERS`. `reflex.ts`: the orchestrator + engine-aware resolver ladder (host `resolveEntities` → PGLite serve IPC → Postgres cached process-singleton → disabled), zero-candidate fast path, fail-open + timeout, heartbeat write for the doctor check, `reflexEnabled(cfg)` (file/env gate, default ON; DB-plane does NOT gate — `assemble()` is sync). `resolve-ipc.ts`: local unix-socket resolve protocol (client + server) so PGLite resolves through the single connection `gbrain serve` holds (a second opener would hit the exclusive lock; a subprocess would force-steal it past the 5-min staleness window and crash). Wired into `src/mcp/server.ts` (serve binds `/.gbrain-resolve.sock` on PGLite, cleaned up on shutdown). Doctor surface: `retrieval_reflex_health` in `src/commands/doctor.ts` (reads the heartbeat for truthful runtime status; categorized in `doctor-categories.ts`). Config: `retrieval_reflex` + `retrieval_reflex_max_pointers` in `src/core/config.ts`. Policy layer ships as the `retrieval-reflex` recipe (`recipes/retrieval-reflex/`). Pinned by `test/context/entity-salience.test.ts`, `test/retrieval-reflex.test.ts`, `test/context/resolve-ipc.test.ts`, `test/doctor-retrieval-reflex.test.ts`. +- `src/core/context-engine.ts` + `src/openclaw-context-engine.ts` — the deterministic context engine OpenClaw loads on every turn (`assemble()` injects the Live Context block, zero-LLM). `createGBrainContextEngine({workspaceDir, resolveEntities?})` accepts an OPTIONAL host-injected resolver (`ENGINE_API_VERSION` 0.2.0, additive — older hosts work unchanged; the plugin entry maps `ctx.resolveEntities`/`ctx.brainQuery` onto it). `assemble()` runs the Retrieval Reflex after the Live Context block: extracts the current turn's user text, builds prior-context text (every message EXCEPT the current turn — suppression must not see the triggering mention), passes the rolling window (`getWindowTurns`, last 12 user/assistant turns; the reflex slices to its configured `retrieval_reflex_window_turns`), and appends the pointer block. `warmReflex()` fires at construction. +- `src/core/context/` — Retrieval Reflex (Layer 1, issue #1981). `entity-salience.ts`: pure, zero-LLM, precision-biased `extractCandidates(text)` (capitalized runs + `@handles`, STOPWORDS + soft COMMON_WORDS + sentence-start guard, deterministic, capped) + `extractCandidatesFromWindow(turns)` (#2095: merges per-turn extraction across the last N turns by normalizeAlias form with occurrence/newest-turn/user-mention metadata; salience-ordered — recency > frequency > user-role — so the cap drops stale assistant chatter first). `retrieval-reflex.ts`: `resolveEntitiesToPointers(engine, sourceId, candidates, opts)` — alias arm (`resolveAliases`, caught per-arm for pre-v110 brains) + exact title/slug-suffix arm (the recall fix: real slugs are namespaced `people/x` but `slugify` drops the prefix); pointers carry `source_id`/`arm`/`confidence`/`matchedNorm` (#2095 — `ARM_CONFIDENCE` alias 0.9 / title 0.8 / slug-suffix 0.6 lives next to the arm definitions; arm-2 provenance classified in JS since the combined OR can't report which predicate matched); opts: `sourceIds?` federated scope (alias arm loops per source, arm 2 uses `source_id = ANY`), `suppression?` ('slug-and-title' legacy default; 'slug-only' REQUIRED under windowing — the title rule would suppress every entity merely mentioned in a prior window turn), ambient-channel event logging is DELIVERY-side, not in-resolver — `logDeliveredReflexPointers(engine, pointers)` fires only once a block is actually handed to the consumer (serve's resolve-IPC `onDelivered` hook post-write; `buildReflexAddition` post-timeout on the direct rung), so abandoned/timed-out blocks never pollute the volunteered-vs-used stats; synopsis runs through `stripTakesFence`/`stripFactsFence` (the same privacy boundary `get_page` applies) so private facts never reach the prompt; capped at `MAX_POINTERS`. `reflex.ts`: the orchestrator + engine-aware resolver ladder (host `resolveEntities` → PGLite serve IPC → Postgres cached process-singleton → disabled), zero-candidate fast path, fail-open + timeout, heartbeat write for the doctor check, `reflexEnabled(cfg)` (file/env gate, default ON; DB-plane does NOT gate — `assemble()` is sync); windowed extraction when `windowTurns` present and `retrieval_reflex_window_turns` (default 4; 1 = exact legacy behavior) > 1 — switches suppression to slug-only; accept-side reflex-channel logging fires after the per-turn timeout admits the block (direct-Postgres rung only — IPC logs server-side at delivery; host-injected resolvers are a documented gap). `resolve-ipc.ts`: local unix-socket resolve protocol (client + server) so PGLite resolves through the single connection `gbrain serve` holds (a second opener would hit the exclusive lock; a subprocess would force-steal it past the 5-min staleness window and crash). Wired into `src/mcp/server.ts` (serve binds `/.gbrain-resolve.sock` on PGLite, cleaned up on shutdown). Doctor surface: `retrieval_reflex_health` in `src/commands/doctor.ts` (reads the heartbeat for truthful runtime status; categorized in `doctor-categories.ts`). Config: `retrieval_reflex` + `retrieval_reflex_max_pointers` + `retrieval_reflex_window_turns` in `src/core/config.ts` (env `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`). `volunteer.ts` (#2095): `parseWindow` (lenient `user:`/`assistant:` prefixes, unprefixed → one user turn), `volunteerContext` (extract → resolve → +0.05 multi-turn/newest-turn boost → `min_confidence` 0.7 gate → cap 3/5; deterministic rationale strings, never raw conversation text; slug-only suppression), `volunteerUsageStats` (per-arm/channel precision from the `pages.last_retrieved_at > volunteered_at` join — APPROXIMATE: the 5-min last-retrieved throttle causes false negatives, unrelated reads false positives). `volunteer-events.ts` (#2095): `insertVolunteerEvents` (ONE multi-row parameterized INSERT), `logVolunteerEventsFireAndForget` + bounded drain registered as the `volunteer-events` background-work sink (order 4), `purgeStaleVolunteerEvents` (90-day GC, called from the dream cycle's purge phase). Policy layer ships as the `retrieval-reflex` recipe (`recipes/retrieval-reflex/`). Pinned by `test/context/entity-salience.test.ts`, `test/retrieval-reflex.test.ts`, `test/context/resolve-ipc.test.ts`, `test/doctor-retrieval-reflex.test.ts`, `test/volunteer-context.test.ts`, `test/e2e/volunteer-context-postgres.test.ts`. +- `src/commands/watch.ts` — `gbrain watch` (#2095): the push transport. Reads turns from stdin as they arrive (`user:`/`assistant:` prefixes; unprefixed = user turn), keeps a rolling in-process window (`--window-turns`, default 4), calls `volunteerContext` per turn, streams pointers to stdout (`--json` for JSONL with turn attribution), logs `channel: 'watch'` events with a per-session id. Session dedupe feeds already-pushed slugs back as priorContext so the core's slug-only suppression dedupes. Blocks in the stdin iteration (interactive alive until Ctrl-C/Ctrl-D; piped ends at EOF) — deliberately NOT in DAEMON_COMMANDS; SIGINT closes the stream so teardown flows through finishCliTeardown. Per-turn resolution failures are fail-open. Registered in CLI_ONLY + CLI_ONLY_SELF_HELP + THIN_CLIENT_REFUSED_COMMANDS (thin clients use the `volunteer_context` MCP op). Pinned by `test/watch-command.test.ts`. - `src/commands/integrations.ts` — recipe install. The resolver-row install fence is keyed by `manifest.recipe` (`gbrain::resolver-rows`), so a second `copy-into-host-repo` recipe no longer writes a block mislabeled with the first recipe's name. Pinned by `test/integrations-install.test.ts`. - `src/core/audit/audit-writer.ts` — shared JSONL audit primitive consolidating the hand-rolled audit modules. Exports `createAuditWriter({kind, recordSchema})` returning `{log, readRecent}` plus shared helpers `computeIsoWeekFilename(kind, now?)` and `resolveAuditDir()` (honors `GBRAIN_AUDIT_DIR`). ISO-week file rotation; best-effort writes (stderr warn on failure, never throws); read-path scans current-week + previous-week files for boundary spans. Refactored onto it for parity: `src/core/rerank-audit.ts`, `src/core/audit-slug-fallback.ts`, `src/core/minions/handlers/shell-audit.ts`, `src/core/minions/handlers/supervisor-audit.ts`, `src/core/facts/phantom-audit.ts` (each module's public API preserved bit-for-bit). The `graph-signals-failures` audit (`logGraphSignalsFailure`) uses the same primitive. One hand-rolled audit remains at `src/core/skillpack/audit.ts`. Pinned by `test/audit/audit-writer.test.ts`. -- `src/core/cli-force-exit.ts` (#2084) — single owner of one-shot CLI exit + teardown, designed as a PAIR with the `import.meta.main` seam at the bottom of `src/cli.ts`. `finishCliTeardown({engine, drainTimeoutMs?})` is teardown-ONLY (never exits on the clean path): arms a REF'D backstop (unref'd would let a hung teardown exit naturally, skipping the flush and exiting with whatever PGLite scribbled into `process.exitCode`) whose deadline is COMPUTED from the bounds it guards (`computeTeardownDeadlineMs` = sinks × drainTimeoutMs + facts-abort grace + 2 × pool-end bound + slack, floor 10s; `GBRAIN_TEARDOWN_DEADLINE_MS` env override is the incident escape hatch), drains every background-work sink, disconnects the engine (a throw is warned + swallowed — the exit code reports the OPERATION, not the cleanup), then returns. The exit VERDICT lives in a gbrain-owned channel (`setCliExitVerdict`/`currentExitCode`; mirror-writes `process.exitCode` but NEVER reads it back) because PGLite's Emscripten runtime scribbles its own status into `process.exitCode` at arbitrary points mid-run — every writer that means to set the CLI exit code (op-dispatch catch, reindex, frontmatter, transcripts, brainstorm, autopilot) calls `setCliExitVerdict`. The deadline arms at TEARDOWN start, never before the op handler (the pre-#2084 placement measured handler + teardown combined, so PgBouncer deployments paid a flat 10s force-exit tax on every query and any >10s op was killed mid-run with exit 0). All nine cli.ts disconnect sites route through it; the ONE process exit happens in cli.ts's `main().then/catch` via `flushThenExit(currentExitCode())`, gated by `shouldForceExitAfterMain()` (daemon list: `serve`) — the CLI never waits for Bun's event loop to drain, because `endPoolBounded` deliberately races past stuck PgBouncer sockets that would keep it alive. `flushThenExit(code)` fences stdout+stderr (`write('', cb)` raced with an unref'd guard, EPIPE-safe both sync and async) then holds a REF'D aliveness grace for non-TTY stdio before `process.exit` — Bun delivers queued pipe writes only while the process is alive (no flush API reaches `process.stdout`'s native queue; write callbacks fire on accept, not delivery), so the grace IS the flush (#1959 truncation class). Scope claim is deliberately cli.ts-only: command modules' mid-run engine lifecycles stay local (process-exit semantics inside them would be wrong) and are absorbed by the final explicit exit. Pinned by `test/cli-finish-teardown.test.ts`, `test/flush-then-exit-harness.test.ts` (real spawned-Bun pipe semantics), `test/cli-should-force-exit.test.ts`, the `#2084` describes in `test/fix-wave-structural.test.ts` + `test/e2e/pglite-cli-exit.serial.test.ts`. +- `src/core/cli-force-exit.ts` (#2084) — single owner of one-shot CLI exit + teardown, designed as a PAIR with the `import.meta.main` seam at the bottom of `src/cli.ts`. `finishCliTeardown({engine, drainTimeoutMs?})` is teardown-ONLY (never exits on the clean path): arms a REF'D backstop (unref'd would let a hung teardown exit naturally, skipping the flush and exiting with whatever PGLite scribbled into `process.exitCode`) whose deadline is COMPUTED from the bounds it guards (`computeTeardownDeadlineMs` = sinks × drainTimeoutMs + facts-abort grace + 2 × pool-end bound + slack, floor 10s; `GBRAIN_TEARDOWN_DEADLINE_MS` env override is the incident escape hatch), drains every background-work sink, disconnects the engine (a throw is warned + swallowed — the exit code reports the OPERATION, not the cleanup), then returns. The exit VERDICT lives in a gbrain-owned channel (`setCliExitVerdict`/`currentExitCode`; mirror-writes `process.exitCode` but NEVER reads it back) because PGLite's Emscripten runtime scribbles its own status into `process.exitCode` at arbitrary points mid-run — every writer that means to set the CLI exit code (op-dispatch catch, reindex, frontmatter, transcripts, brainstorm, autopilot, doctor's FAIL verdict, extract, and cli.ts's swept inner exits — friction, claw-test, smoke-test, the no-DB eval runners, status/status-thin, whoknows-thin) calls `setCliExitVerdict`; `test/cli-exit-verdict-pin.test.ts` greps src/ so the next raw `process.exitCode =` write fails CI instead of silently reporting success on failure. The deadline arms at TEARDOWN start, never before the op handler (the pre-#2084 placement measured handler + teardown combined, so PgBouncer deployments paid a flat 10s force-exit tax on every query and any >10s op was killed mid-run with exit 0). All nine cli.ts disconnect sites route through it; the ONE process exit happens in cli.ts's `main().then/catch` via `flushThenExit(currentExitCode())`, gated by `shouldForceExitAfterMain()` (daemon list: `serve`) — the CLI never waits for Bun's event loop to drain, because `endPoolBounded` deliberately races past stuck PgBouncer sockets that would keep it alive. `flushThenExit(code)` fences stdout+stderr (`write('', cb)` raced with an unref'd guard, EPIPE-safe both sync and async) then holds a REF'D aliveness grace for non-TTY stdio before `process.exit` — Bun delivers queued pipe writes only while the process is alive (no flush API reaches `process.stdout`'s native queue; write callbacks fire on accept, not delivery), so the grace IS the flush (#1959 truncation class). Scope claim is deliberately cli.ts-only: command modules' mid-run engine lifecycles stay local (process-exit semantics inside them would be wrong) and are absorbed by the final explicit exit. Pinned by `test/cli-finish-teardown.test.ts`, `test/flush-then-exit-harness.test.ts` (real spawned-Bun pipe semantics), `test/cli-should-force-exit.test.ts`, `test/cli-pipe-truncation.test.ts` (real-CLI piped --tools-json byte-stable), `test/cli-exit-verdict-pin.test.ts`, the `#2084` describes in `test/fix-wave-structural.test.ts` + `test/e2e/pglite-cli-exit.serial.test.ts`, and `test/e2e/pgbouncer-teardown.test.ts` (CI transaction-mode pooler — the #1972/#2015/#2084 class, finally reproducible in CI). - `src/core/cli-options.ts` extension — `CliOptions` gains `explain: boolean`. `parseGlobalFlags` recognizes `--explain` anywhere in argv (stripped before command dispatch). `src/cli.ts` `formatResult` for `search` + `query` cases routes to `formatResultsExplain` from `src/core/search/explain-formatter.ts` when `CliOptions.explain` is set; falls through to the existing JSON / human formatters otherwise. - `src/commands/search.ts:gbrain search stats` extension — `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property; `_meta.metric_glossary` adds `graph_signals.enabled` + `graph_signals.failures_by_reason`. Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts`. @@ -322,8 +323,8 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/markdown.ts` — Frontmatter parsing + body splitter. `coerceFrontmatterString(v)` coerces a non-string `title`/`slug`/`type` to a deterministic string at parse time so a YAML-typed value never reaches `.toLowerCase()` and throws (the #1939 wedge: `title: 2024-06-01` parsed as a `Date`, `title: 1458` as a number, and the throw blocked the sync bookmark from advancing); a `Date` becomes its UTC ISO date (`2024-06-01`, machine-independent and matching the on-disk token, unlike `String(date)`), `null`/`undefined` become `''`, everything else uses `String()`. `splitBody` requires an explicit timeline sentinel (``, `--- timeline ---`, or `---` immediately before `## Timeline`/`## History`). Plain `---` in body text is a markdown horizontal rule, not a separator. `inferType` auto-types `/wiki/analysis/` → analysis, `/wiki/guides/` → guide, `/wiki/hardware/` → hardware, `/wiki/architecture/` → architecture, `/writing/` → writing (plus existing people/companies/deals/etc heuristics). - `scripts/check-jsonb-pattern.sh` — CI grep guard. Fails the build if anyone reintroduces (a) the `${JSON.stringify(x)}::jsonb` interpolation pattern (postgres.js v3 double-encodes it), or (b) `max_stalled INTEGER NOT NULL DEFAULT 1` in any schema source file (must be DEFAULT 5 to preserve SIGKILL-rescue). Wired into `bun test`. - `scripts/check-source-id-projection.sh` — CI grep guard for the multi-source bug class. Greps `src/core/postgres-engine.ts` + `src/core/pglite-engine.ts` for `SELECT.*FROM pages` projections matching the `rowToPage` feeder shape (id + slug + type + title) and fails if `source_id` is missing. `Page.source_id` is required at the type level; a projection dropping the column produces `Page` rows with `source_id: undefined` while TypeScript's `: string` lies about it. Wired into `bun run verify` + `bun run check:all`. -- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up `pgvector/pgvector:pg16` + `oven/bun:1` with named volumes (`gbrain-ci-pg-data`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs unit tests with `DATABASE_URL` unset, then runs all 29 E2E files sequentially. `--diff` swaps in the diff-aware selector; `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set. -- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all 29 files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases. +- `docker-compose.ci.yml` + `scripts/ci-local.sh` — Local CI gate. `bun run ci:local` spins up four `pgvector/pgvector:pg16` services (postgres-1..4) + `oven/bun:1` with named volumes (`gbrain-ci-pg-data-{1..4}`, `gbrain-ci-node-modules`, `gbrain-ci-bun-cache`), runs gitleaks on host, smoke-tests `scripts/run-e2e.sh` argv handling, runs guards + typecheck, then the Tier 1 default: 4-shard parallel unit + E2E (`xargs -P4`, one Postgres per shard; unit phase keeps `DATABASE_URL` unset). `--no-shard` falls back to the legacy unsharded sequential flow (debug aid); `--diff` runs the diff-aware selector unsharded. Also runs a `pgbouncer` service (`edoburu/pgbouncer`, `POOL_MODE: transaction`, `AUTH_TYPE: plain` — pg16 stores SCRAM verifiers, so the userlist must hold the plaintext password; `IGNORE_STARTUP_PARAMETERS` whitelists gbrain's `statement_timeout`/`idle_in_transaction_session_timeout` startup params the way the Supabase pooler does) fronting postgres-1 on host port `GBRAIN_CI_PGBOUNCER_PORT` (default 6543); every E2E invocation exports `GBRAIN_PGBOUNCER_URL` (pooled; dedicated `gbrain_pgbouncer` database so it never races the `gbrain_test` TRUNCATE fixtures) + `GBRAIN_PGBOUNCER_DIRECT_URL`, consumed by `test/e2e/pgbouncer-teardown.test.ts` — the transaction-mode teardown bug class (#1972/#2015/#2084) reproduced in the local gate instead of only in production. `--no-pull` skips upstream pulls; `--clean` nukes named volumes. Postgres host port defaults to 5434; override with `GBRAIN_CI_PG_PORT=NNNN`. Stronger gate than PR CI's 2-file Tier 1 set. +- `scripts/select-e2e.ts` + `scripts/e2e-test-map.ts` — Diff-aware E2E test selector. Reads three git sources (committed `origin/master...HEAD`, working-tree `HEAD`, and `git ls-files --others --exclude-standard` for untracked NOT-gitignored files), classifies as EMPTY / DOC_ONLY / SRC. Fail-closed: EMPTY → all files; DOC_ONLY (every path matches the README/CLAUDE/AGENTS/CHANGELOG/TODOS allowlist) → empty stdout; SRC → escape-hatch paths (schema, package.json, skills/) trigger all, else the hand-tuned `E2E_TEST_MAP` glob narrows, and an unmapped src/ change still emits ALL files (never silently nothing). Pure-function exports `selectTests`, `classify`, `matchGlob`. `bun run ci:select-e2e` prints the current selection on stdout. `test/select-e2e.test.ts` covers all 4 branches plus 3 regression guards (skills/, untracked files, unmapped src/) — 24 cases. - `scripts/run-e2e.sh` — Sequential E2E runner. Accepts an optional argv-driven file list (used by `ci:local:diff`) and a `--dry-run-list` flag that prints the resolved file list and exits (used by `ci-local.sh`'s startup smoke-test). Falls back to `test/e2e/*.test.ts` when invoked with no args. - `scripts/llms-config.ts` + `scripts/build-llms.ts` — Generator for `llms.txt` (llmstxt.org-spec web index) + `llms-full.txt` (inlined single-fetch bundle). Curated config drives both. Run `bun run build:llms` after adding a new doc. `LLMS_REPO_BASE` env lets forks regenerate with their own URL base. `FULL_SIZE_BUDGET` (600KB) caps the inline bundle; generator WARNs if exceeded. Committed output has no runtime consumer; committed for GitHub browsing and fork-safe fetching. - `AGENTS.md` — Local-clone entry point for non-Claude agents (Codex, Cursor, OpenClaw, Aider). Mirrors `CLAUDE.md` intent via relative links. Claude Code keeps using `CLAUDE.md`. diff --git a/docs/guides/push-context.md b/docs/guides/push-context.md new file mode 100644 index 000000000..e4e104756 --- /dev/null +++ b/docs/guides/push-context.md @@ -0,0 +1,79 @@ +# Push-based context (#2095, v0.42.43.0) + +Retrieval used to be pull-only: the agent had to *know to ask* before the brain +contributed anything. Push-based context inverts that — the brain volunteers +relevant pages from the recent conversation, confidence-gated so push noise +never becomes worse than pull silence. + +Three channels share one zero-LLM core (`src/core/context/volunteer.ts`): + +| Channel | Surface | When to use | +|---|---|---| +| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call | +| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn | +| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out | + +## How it decides + +1. **Extract** entities across the last N turns (capitalized runs, `@handles`), + merged with recency / frequency / user-role salience. Assistant-introduced + entities and "what did she invest in?" follow-ups whose antecedent was named + in the window now resolve. +2. **Resolve** through the alias table, exact titles, and slug suffixes — each + arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6, + +0.05 when mentioned in ≥2 turns or the newest turn. +3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an + explicit lower gate), suppress pages already surfaced (slug-presence only), + cap at 3 pages (hard cap 5). + +## CLI + +```bash +# one-shot: pipe recent turns (oldest → newest) +printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \ + | gbrain volunteer-context + +# streaming: volunteered pages print as the transcript flows +some-transcript-feed | gbrain watch --json + +# the feedback loop: how often were volunteered pages actually opened? +gbrain volunteer-context --stats +``` + +Stats are **approximate** by design: "used" means `pages.last_retrieved_at > +volunteered_at` — the 5-minute last-retrieved throttle causes false negatives +and unrelated reads of the same page cause false positives. Use the per-arm +precision to tune `min_confidence`, not as an exact metric. + +**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its +connection for the whole session — a concurrent `gbrain serve` or any write +path blocks until watch exits. On a PGLite brain, run watch in bursts (piped +input exits at EOF) or use the ambient reflex channel instead, which routes +through a running serve's resolve socket rather than taking the lock. Routing +watch through that same socket is a filed follow-up (TODOS.md). Postgres +brains are unaffected. + +## Config + +| Key | Default | What it does | +|---|---|---| +| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) | +| `retrieval_reflex` | true | the ambient channel's master switch | +| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn | + +Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch` +(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch); +on the op only: `prior_context` (text whose already-surfaced slugs are suppressed), +`session_id` / `turn` attribution params (watch stamps its own per-session id and +turn numbers in the feedback log), and `days` to size the `--stats` window. + +## Storage + privacy + +Volunteered pages log to `context_volunteer_events` (migration v117): slug, +arm, confidence, channel, optional session/turn — the rationale is a +deterministic template string, never raw conversation text. Event writes are +best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal, +not an audit trail. Rows are pruned after 90 days by the dream cycle's purge +phase. Synopses always strip the takes/facts fences — the same strip `get_page` +applies to untrusted callers, applied unconditionally here so private fence rows +never reach a prompt regardless of caller trust. diff --git a/llms-full.txt b/llms-full.txt index 558367394..1ef630b27 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -117,8 +117,9 @@ writing or reviewing an operation, consult `src/core/operations.ts` for the cont ## Before shipping Easiest path: `bun run ci:local` runs the full CI gate inside Docker (gitleaks, -unit tests with `DATABASE_URL` unset, then all 29 E2E files sequentially against a -fresh pgvector container) and tears down. Use `bun run ci:local:diff` for the +guards + typecheck, then 4-shard parallel unit + E2E against four pgvector +containers plus a transaction-mode PgBouncer; unit phase keeps `DATABASE_URL` +unset) and tears down. Use `bun run ci:local:diff` for the diff-aware subset during fast iteration on a focused branch. Requires Docker (Docker Desktop / OrbStack / Colima) and `gitleaks` (`brew install gitleaks`). @@ -186,7 +187,7 @@ mount, CEO-class with multiple team brains) and ## Architecture -Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP +Contract-first: `src/core/operations.ts` defines ~90 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.42.43.0 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`) dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat markdown files (tool-agnostic, work with both CLI and plugin contexts). @@ -245,6 +246,7 @@ detail on demand.) | any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry | | search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` | | search modes / cost knobs | `docs/guides/search-modes.md` | +| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` | | schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` | | thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` | | the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry | @@ -3372,6 +3374,92 @@ the bundled resolver lives at [`skills/RESOLVER.md`](../../skills/RESOLVER.md). --- +## docs/guides/push-context.md + +Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md + +# Push-based context (#2095, v0.42.43.0) + +Retrieval used to be pull-only: the agent had to *know to ask* before the brain +contributed anything. Push-based context inverts that — the brain volunteers +relevant pages from the recent conversation, confidence-gated so push noise +never becomes worse than pull silence. + +Three channels share one zero-LLM core (`src/core/context/volunteer.ts`): + +| Channel | Surface | When to use | +|---|---|---| +| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call | +| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn | +| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out | + +## How it decides + +1. **Extract** entities across the last N turns (capitalized runs, `@handles`), + merged with recency / frequency / user-role salience. Assistant-introduced + entities and "what did she invest in?" follow-ups whose antecedent was named + in the window now resolve. +2. **Resolve** through the alias table, exact titles, and slug suffixes — each + arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6, + +0.05 when mentioned in ≥2 turns or the newest turn. +3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an + explicit lower gate), suppress pages already surfaced (slug-presence only), + cap at 3 pages (hard cap 5). + +## CLI + +```bash +# one-shot: pipe recent turns (oldest → newest) +printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \ + | gbrain volunteer-context + +# streaming: volunteered pages print as the transcript flows +some-transcript-feed | gbrain watch --json + +# the feedback loop: how often were volunteered pages actually opened? +gbrain volunteer-context --stats +``` + +Stats are **approximate** by design: "used" means `pages.last_retrieved_at > +volunteered_at` — the 5-minute last-retrieved throttle causes false negatives +and unrelated reads of the same page cause false positives. Use the per-arm +precision to tune `min_confidence`, not as an exact metric. + +**PGLite + `gbrain watch`:** PGLite is single-connection, and watch holds its +connection for the whole session — a concurrent `gbrain serve` or any write +path blocks until watch exits. On a PGLite brain, run watch in bursts (piped +input exits at EOF) or use the ambient reflex channel instead, which routes +through a running serve's resolve socket rather than taking the lock. Routing +watch through that same socket is a filed follow-up (TODOS.md). Postgres +brains are unaffected. + +## Config + +| Key | Default | What it does | +|---|---|---| +| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) | +| `retrieval_reflex` | true | the ambient channel's master switch | +| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn | + +Per-call knobs: `max_pages` + `min_confidence` on both the op and `gbrain watch` +(`--max-pages` / `--min-confidence`, plus `--window-turns` / `--source` on watch); +on the op only: `prior_context` (text whose already-surfaced slugs are suppressed), +`session_id` / `turn` attribution params (watch stamps its own per-session id and +turn numbers in the feedback log), and `days` to size the `--stats` window. + +## Storage + privacy + +Volunteered pages log to `context_volunteer_events` (migration v117): slug, +arm, confidence, channel, optional session/turn — the rationale is a +deterministic template string, never raw conversation text. Event writes are +best-effort (fire-and-forget, drained at CLI exit) — the log is a tuning signal, +not an audit trail. Rows are pruned after 90 days by the dream cycle's purge +phase. Synopses always strip the takes/facts fences — the same strip `get_page` +applies to untrusted callers, applied unconditionally here so private fence rows +never reach a prompt regardless of caller trust. + +--- + ## docs/mcp/DEPLOY.md Source: https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md diff --git a/llms.txt b/llms.txt index 8ca28d88a..34d94b6d4 100644 --- a/llms.txt +++ b/llms.txt @@ -25,6 +25,7 @@ Repo: https://github.com/garrytan/gbrain - [docs/guides/minions-deployment.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/minions-deployment.md): Deploying the gbrain jobs worker: crontab + watchdog, inline --follow, systemd/Procfile/fly.toml, upgrade checklist. - [docs/guides/quiet-hours.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/quiet-hours.md): Notification hold + timezone-aware delivery. - [docs/guides/scaling-skills.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/scaling-skills.md): Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss. +- [docs/guides/push-context.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/guides/push-context.md): Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop. - [docs/mcp/DEPLOY.md](https://raw.githubusercontent.com/garrytan/gbrain/master/docs/mcp/DEPLOY.md): MCP server deployment. ## AI providers diff --git a/package.json b/package.json index a345afb96..23c21a7d0 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.42.0" + "version": "0.42.43.0" } diff --git a/scripts/ci-local.sh b/scripts/ci-local.sh index b7407fa6b..06ec621d6 100755 --- a/scripts/ci-local.sh +++ b/scripts/ci-local.sh @@ -196,7 +196,10 @@ SELECTED=$(bun run scripts/select-e2e.ts) if [ -z "$SELECTED" ]; then echo "[runner] selector emitted nothing (doc-only diff); skipping E2E." else - DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test echo "$SELECTED" | xargs bash scripts/run-e2e.sh + DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \ + GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \ + GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \ + echo "$SELECTED" | xargs bash scripts/run-e2e.sh fi' else RUN_PHASES_CMD='echo "[runner] guards + typecheck" @@ -208,7 +211,10 @@ bun run typecheck echo "[runner] unit (unsharded, DATABASE_URL unset)" env -u DATABASE_URL bash scripts/run-unit-shard.sh echo "[runner] e2e (unsharded)" -DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test bash scripts/run-e2e.sh' +DATABASE_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \ +GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \ +GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \ +bash scripts/run-e2e.sh' fi else # Tier 1 sharded path. Each shard runs unit+E2E sequentially against its @@ -257,10 +263,14 @@ printf '%s\\n' 1 2 3 4 | xargs -P4 -I{} sh -c ' if [ -s /tmp/e2e-selected.txt ]; then SHARD=\${shard}/4 \\ DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\ + GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\ + GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\ xargs -a /tmp/e2e-selected.txt bash scripts/run-e2e.sh >> \$log 2>&1 else SHARD=\${shard}/4 \\ DATABASE_URL=postgresql://postgres:postgres@postgres-\${shard}:5432/gbrain_test \\ + GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@pgbouncer:5432/gbrain_pgbouncer \\ + GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@postgres-1:5432/gbrain_test \\ bash scripts/run-e2e.sh >> \$log 2>&1 fi e2e_exit=\$? diff --git a/scripts/llms-config.ts b/scripts/llms-config.ts index e81154c6a..b2b338a86 100644 --- a/scripts/llms-config.ts +++ b/scripts/llms-config.ts @@ -151,6 +151,12 @@ export const SECTIONS: DocSection[] = [ "Three-tier architecture for agents with 300+ skills: always-loaded, resolver-routed, and dormant. Per-turn token math, the v0.41.7.0 compact list-format resolver, and the `gbrain doctor` safety net. 306 skills, ~21K tokens freed per turn, zero capability loss.", path: "docs/guides/scaling-skills.md", }, + { + title: "docs/guides/push-context.md", + description: + "Push-based context: the brain volunteers confidence-gated pages from the rolling conversation window. Three channels (ambient reflex, volunteer_context op, gbrain watch), config knobs, and the volunteered-vs-used feedback loop.", + path: "docs/guides/push-context.md", + }, { title: "docs/mcp/DEPLOY.md", description: "MCP server deployment.", diff --git a/src/cli.ts b/src/cli.ts index 82b7e22a6..4cab59fbd 100755 --- a/src/cli.ts +++ b/src/cli.ts @@ -24,6 +24,7 @@ import type { GBrainConfig } from './core/config.ts'; import type { AIGatewayConfig } from './core/ai/types.ts'; import type { BrainEngine } from './core/engine.ts'; import { operations, OperationError } from './core/operations.ts'; +import { formatVolunteeredPage } from './core/context/volunteer.ts'; import type { Operation, OperationContext } from './core/operations.ts'; import { shouldForceExitAfterMain, finishCliTeardown, flushThenExit, currentExitCode, setCliExitVerdict } from './core/cli-force-exit.ts'; import { serializeMarkdown } from './core/markdown.ts'; @@ -43,7 +44,7 @@ for (const op of operations) { } // CLI-only commands that bypass the operation layer -const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade']); +const CLI_ONLY = new Set(['init', 'reinit-pglite', 'upgrade', 'post-upgrade', 'check-update', 'integrations', 'publish', 'check-backlinks', 'lint', 'report', 'import', 'export', 'files', 'embed', 'serve', 'call', 'config', 'doctor', 'migrate', 'eval', 'sync', 'extract', 'extract-conversation-facts', 'enrich', 'features', 'autopilot', 'graph-query', 'jobs', 'agent', 'apply-migrations', 'skillpack-check', 'skillpack', 'resolvers', 'integrity', 'repair-jsonb', 'orphans', 'sources', 'mounts', 'dream', 'check-resolvable', 'routing-eval', 'skillify', 'smoke-test', 'providers', 'storage', 'repos', 'code-def', 'code-refs', 'reindex', 'reindex-code', 'reindex-frontmatter', 'code-callers', 'code-callees', 'frontmatter', 'auth', 'friction', 'claw-test', 'book-mirror', 'takes', 'think', 'salience', 'anomalies', 'transcripts', 'models', 'remote', 'recall', 'forget', 'edges-backfill', 'cache', 'ze-switch', 'founder', 'brainstorm', 'lsd', 'schema', 'capture', 'onboard', 'conversation-parser', 'status', 'connect', 'skillopt', 'quarantine', 'self-upgrade', 'watch']); // CLI-only commands whose handlers print their own --help text. These are // excluded from the generic short-circuit so detailed per-command and // per-subcommand usage stays reachable. @@ -67,6 +68,8 @@ const CLI_ONLY_SELF_HELP = new Set([ 'capture', // v0.42 self-upgrade ships its own usage (flags + the agent-skill story). 'self-upgrade', + // v0.43 (#2095): watch ships WATCH_HELP (flags + the stdin-turn protocol). + 'watch', // v0.37 fix wave (Lane D.4 + CDX2-12): sync's --no-embed flag was // unreachable via help because the dispatcher's generic CLI-only // short-circuit fired before runSync could print its own usage block. @@ -802,8 +805,27 @@ async function makeContext(engine: BrainEngine, params: Record) }; } -function formatResult(opName: string, result: unknown): string { +// Exported for tests (same import-safety contract as cliAliases/printOpHelp). +export function formatResult(opName: string, result: unknown): string { switch (opName) { + case 'volunteer_context': { + const r = result as any; + // Stats mode (the feedback loop). + if (r && r.approximate === true && Array.isArray(r.by_arm)) { + const lines = [ + `volunteered-context precision — last ${r.days} day(s) (${r.note})`, + `total: ${r.total_volunteered} volunteered, ${r.total_used} used`, + ]; + for (const a of r.by_arm) { + lines.push(` ${a.match_arm}/${a.channel}: ${a.used}/${a.volunteered} used (precision ${a.precision})`); + } + if (!r.by_arm.length) lines.push(' (no volunteer events in the window)'); + return lines.join('\n') + '\n'; + } + const pages = (r?.pages ?? []) as any[]; + if (!pages.length) return 'Nothing volunteered (no entity cleared the confidence gate).\n'; + return pages.map((p) => formatVolunteeredPage(p)).join('\n') + '\n'; + } case 'get_page': { const r = result as any; if (r.error === 'ambiguous_slug') { @@ -925,6 +947,9 @@ function formatResult(opName: string, result: unknown): string { const THIN_CLIENT_REFUSED_COMMANDS = new Set([ 'sync', 'embed', 'extract', 'extract-conversation-facts', 'enrich', 'migrate', 'apply-migrations', 'repair-jsonb', 'orphans', 'integrity', 'serve', + // v0.43 (#2095): watch streams against a LOCAL engine; thin clients get + // the volunteer_context MCP op instead. + 'watch', // v0.31.1 (CDX-2 op coverage matrix): more local-only commands 'dream', 'transcripts', 'storage', // v0.31.1 CDX-2 audit: takes/sources have multiple subcommands; some @@ -1156,11 +1181,14 @@ async function handleCliOnly(command: string, args: string[]) { } if (command === 'friction') { const { runFriction } = await import('./commands/friction.ts'); - process.exit(runFriction(args)); + // #2084 inner-exit sweep: verdict + return so teardown + the flush seam run. + setCliExitVerdict(runFriction(args)); + return; } if (command === 'claw-test') { const { runClawTest } = await import('./commands/claw-test.ts'); - process.exit(await runClawTest(args)); + setCliExitVerdict(await runClawTest(args)); + return; } if (command === 'report') { const { runReport } = await import('./commands/report.ts'); @@ -1269,7 +1297,7 @@ async function handleCliOnly(command: string, args: string[]) { execSync(`bash "${scriptPath}"`, { stdio: 'inherit', env: { ...process.env } }); } catch (e: any) { // Non-zero exit = some tests failed (exit code = failure count) - process.exit(e.status ?? 1); + setCliExitVerdict(e.status ?? 1); } return; } @@ -1318,7 +1346,8 @@ async function handleCliOnly(command: string, args: string[]) { // The handler self-configures the AI gateway from loadConfig() + process.env. if (command === 'eval' && args[0] === 'cross-modal') { const { runEvalCrossModal } = await import('./commands/eval-cross-modal.ts'); - process.exit(await runEvalCrossModal(args.slice(1))); + setCliExitVerdict(await runEvalCrossModal(args.slice(1))); + return; } // v0.32 EXP-5 (codex review #10): `eval takes-quality replay ` @@ -1329,7 +1358,8 @@ async function handleCliOnly(command: string, args: string[]) { // engine-required path below. if (command === 'eval' && args[0] === 'takes-quality' && args[1] === 'replay') { const { runReplayNoBrain } = await import('./commands/eval-takes-quality.ts'); - process.exit(await runReplayNoBrain(args.slice(2))); + setCliExitVerdict(await runReplayNoBrain(args.slice(2))); + return; } // v0.28.8: longmemeval brings its own in-memory PGLite. Bypassing @@ -1361,7 +1391,8 @@ async function handleCliOnly(command: string, args: string[]) { // gate runs on machines with no `~/.gbrain/config.json`. if (command === 'eval' && args[0] === 'conversation-parser') { const { runEvalConversationParser } = await import('./commands/eval-conversation-parser.ts'); - process.exit(await runEvalConversationParser(args.slice(1))); + setCliExitVerdict(await runEvalConversationParser(args.slice(1))); + return; } // v0.41.13.0: `gbrain conversation-parser list-builtins | validate @@ -1389,7 +1420,8 @@ async function handleCliOnly(command: string, args: string[]) { const cfgPre = loadConfig(); if (isThinClient(cfgPre)) { const { runEvalWhoknows } = await import('./commands/eval-whoknows.ts'); - process.exit(await runEvalWhoknows(null, args.slice(1))); + setCliExitVerdict(await runEvalWhoknows(null, args.slice(1))); + return; } } @@ -1403,7 +1435,8 @@ async function handleCliOnly(command: string, args: string[]) { if (cfgPre && isThinClient(cfgPre)) { const { runStatus } = await import('./commands/status.ts'); const result = await runStatus(null, args); - process.exit(result.exitCode); + setCliExitVerdict(result.exitCode); + return; } } @@ -1716,8 +1749,8 @@ async function handleCliOnly(command: string, args: string[]) { case 'status': { const { runStatus } = await import('./commands/status.ts'); const result = await runStatus(engine, args); - process.exit(result.exitCode); - // eslint-disable-next-line no-unreachable + // #2084 inner-exit sweep: a mid-switch exit skips the finally teardown. + setCliExitVerdict(result.exitCode); break; } // v0.38 — Capture: single human-facing entrypoint for ingestion. @@ -1887,6 +1920,15 @@ async function handleCliOnly(command: string, args: string[]) { await runQuarantine(engine, args); break; } + case 'watch': { + // v0.43 (#2095): push-based context transport. Blocks in the stdin + // iteration (interactive stays alive; piped exits at EOF), then the + // finally below runs finishCliTeardown (volunteer events drain with + // every other sink) and the import.meta.main seam flush-exits. + const { runWatch } = await import('./commands/watch.ts'); + await runWatch(engine, args); + break; + } case 'storage': { const { runStorage } = await import('./commands/storage.ts'); await runStorage(engine, args); @@ -2267,6 +2309,8 @@ ADMIN --public-url URL Public issuer URL (required behind proxy/tunnel) connect --token Wire Claude Code to a remote gbrain (bearer token) [--install] [--json] Print the paste-ready command, or --install to run it + watch [--json] Push-based context: pipe conversation turns in, + volunteered brain pages stream out (#2095) call '' Raw tool invocation version Version info --tools-json Tool discovery (JSON) diff --git a/src/commands/watch.ts b/src/commands/watch.ts new file mode 100644 index 000000000..9c990e7d2 --- /dev/null +++ b/src/commands/watch.ts @@ -0,0 +1,190 @@ +/** + * v0.43 (#2095) — `gbrain watch`: the push transport for push-based context. + * + * Reads conversation turns from stdin AS THEY ARRIVE (plain text = a user + * turn; `user:` / `assistant:` prefixed lines set the role), maintains a + * rolling window in-process, and volunteers confidence-gated brain pages to + * stdout after every turn. The consumer pipes its transcript in and reads + * volunteered pointers out — no per-entity CLI round-trips. + * + * Lifecycle: BLOCKS in the stdin iteration (like `gbrain jobs work`), so an + * interactive TTY session stays alive until Ctrl-C / Ctrl-D and piped input + * exits at EOF. Either way the handler RETURNS, the CLI_ONLY finally runs + * finishCliTeardown (volunteer events bank before teardown), and the + * entrypoint flush-exit ends the process deliberately — which is exactly why + * `watch` is NOT in DAEMON_COMMANDS: it never returns from main() while work + * is still running. SIGINT closes the stream and flows through the same + * drain path instead of killing mid-write. + * + * Session dedupe: a slug is volunteered at most once per watch session — + * already-pushed slugs ride VolunteerOpts.excludeSlugs, skipped inside the + * core's pointer loop BEFORE the confidence gate and maxPages cap (O(1) + * membership; a post-call filter would let a recurring entity starve new + * pages out of the cap — red-team finding). + * + * PGLite note: watch holds the single-writer engine for the whole session, + * so on the default engine it cannot run concurrently with `gbrain serve` + * (or any other gbrain process) against the same brain — run it against + * Postgres, or stop serve first. Routing watch through the serve resolve-IPC + * socket (like the ambient reflex) is a filed follow-up. + */ + +import { createInterface } from 'node:readline'; +import type { BrainEngine } from '../core/engine.ts'; +import { + volunteerContext, + formatVolunteeredPage, + TURN_PREFIX_RE, + VOLUNTEER_DEFAULT_MAX_PAGES, + VOLUNTEER_DEFAULT_MIN_CONFIDENCE, +} from '../core/context/volunteer.ts'; +import type { WindowTurn } from '../core/context/entity-salience.ts'; +import { DEFAULT_WINDOW_TURNS, windowTurnCount } from '../core/context/reflex.ts'; +import { loadConfig } from '../core/config.ts'; +import { logVolunteerEventsFireAndForget, volunteerEventRowsFrom } from '../core/context/volunteer-events.ts'; + +export const WATCH_HELP = `gbrain watch — push-based context: volunteer brain pages per conversation turn (#2095) + +Reads turns from stdin (one per line; 'user:' / 'assistant:' prefixes set the +role, unprefixed lines are user turns) and prints confidence-gated page +pointers with rationales after each turn. A slug is volunteered at most once +per session. Piped input exits at EOF; interactive sessions exit on Ctrl-C. + +Usage: + some-transcript-feed | gbrain watch [--json] + gbrain watch # interactive: type turns, Ctrl-C to end + +PGLite brains: watch holds the single-writer engine for the whole session — +it cannot run alongside \`gbrain serve\` (or other gbrain processes) against +the same brain. Use a Postgres brain for concurrent access. + +Flags: + --json JSONL output (one volunteered page per line) + --window-turns N rolling extraction window (default ${DEFAULT_WINDOW_TURNS}) + --max-pages N max pages volunteered per turn (default ${VOLUNTEER_DEFAULT_MAX_PAGES}, cap 5) + --min-confidence X confidence gate 0..1 (default ${VOLUNTEER_DEFAULT_MIN_CONFIDENCE}) + --source source scope (defaults to the canonical 6-tier resolution) + --help this text +`; + +function numFlag(args: string[], flag: string): number | undefined { + const i = args.indexOf(flag); + if (i < 0 || i + 1 >= args.length) return undefined; + const n = Number(args[i + 1]); + return Number.isFinite(n) ? n : undefined; +} + +function strFlag(args: string[], flag: string): string | undefined { + const i = args.indexOf(flag); + return i >= 0 && i + 1 < args.length ? args[i + 1] : undefined; +} + +export interface WatchIoDeps { + /** Injected line source for tests (defaults to readline over stdin). */ + lines?: AsyncIterable; + write?: (s: string) => void; + isTTY?: boolean; +} + +export async function runWatch(engine: BrainEngine, args: string[], deps: WatchIoDeps = {}): Promise { + if (args.includes('--help') || args.includes('-h')) { + (deps.write ?? ((s: string) => process.stdout.write(s)))(WATCH_HELP); + return; + } + + const json = args.includes('--json'); + // --window-turns wins; otherwise the same config knob the ambient reflex + // honors (retrieval_reflex_window_turns, default 4) applies here too. + // Hard cap 64: an unbounded window re-scans every retained turn on every + // turn over an hours-long session — the cost class the priorContext fix + // removed, reintroduced via a config typo (red-team finding). + const windowTurns = Math.min( + 64, + Math.max(1, Math.floor(numFlag(args, '--window-turns') ?? windowTurnCount(loadConfig()))), + ); + const maxPages = numFlag(args, '--max-pages'); + const minConfidence = numFlag(args, '--min-confidence'); + const write = deps.write ?? ((s: string) => process.stdout.write(s)); + const isTTY = deps.isTTY ?? Boolean(process.stdin.isTTY); + + const { resolveSourceId } = await import('../core/source-resolver.ts'); + const sourceId = await resolveSourceId(engine, strFlag(args, '--source') ?? null, process.cwd()); + const sourceIds = [sourceId]; + const sessionId = `watch-${process.pid}-${Date.now().toString(36)}`; + + if (!deps.lines) { + // The ready line doubles as a machine-readable readiness signal for + // scripted consumers (and the SIGINT lifecycle test): engine + source + // resolution are done, the stdin loop starts next. + process.stderr.write( + isTTY + ? `[watch] interactive session ${sessionId} ready — type turns ('assistant: ...' to set role), Ctrl-C to end\n` + : `[watch] session ${sessionId} ready\n`, + ); + } + + const rl = deps.lines + ? null + : createInterface({ input: process.stdin, crlfDelay: Infinity }); + const lines: AsyncIterable = deps.lines ?? (rl as AsyncIterable); + + // SIGINT closes the stream so the for-await ends and the normal + // drain-then-exit path runs (never a mid-write kill). + const onSigint = () => { + rl?.close(); + }; + process.on('SIGINT', onSigint); + + const window: WindowTurn[] = []; + const pushedSlugs = new Set(); // session dedupe (slug-only suppression input) + let turnNo = 0; + + try { + for await (const rawLine of lines) { + const line = rawLine.replace(/\r$/, ''); + if (!line.trim()) continue; + const m = TURN_PREFIX_RE.exec(line); + const turn: WindowTurn = m + ? { role: m[1].toLowerCase() as WindowTurn['role'], text: (m[2] ?? '').trim() } + : { role: 'user', text: line.trim() }; + if (!turn.text) continue; + turnNo++; + window.push(turn); + if (window.length > windowTurns) window.splice(0, window.length - windowTurns); + + let pages; + try { + pages = await volunteerContext(engine, [...window], { + sourceIds, + maxPages, + minConfidence, + // Session dedupe: skipped inside the core BEFORE the gate + cap + // (O(1) per pointer) so a recurring slug can't starve new pages. + excludeSlugs: pushedSlugs, + }); + } catch { + continue; // fail-open per turn: a transient DB error never kills the stream + } + if (!pages.length) continue; + + for (const p of pages) pushedSlugs.add(p.slug); + logVolunteerEventsFireAndForget( + engine, + volunteerEventRowsFrom(pages, { channel: 'watch', session_id: sessionId, turn: turnNo }), + ); + + if (json) { + for (const p of pages) { + write(JSON.stringify({ turn: turnNo, ...p }) + '\n'); + } + } else { + for (const p of pages) { + write(formatVolunteeredPage(p) + '\n'); + } + } + } + } finally { + process.off('SIGINT', onSigint); + rl?.close(); + } +} diff --git a/src/core/config.ts b/src/core/config.ts index 70ddca5b7..333e4d11a 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -169,6 +169,14 @@ export interface GBrainConfig { retrieval_reflex?: boolean; /** Max pointers injected per turn (default 3). File-plane only. */ retrieval_reflex_max_pointers?: number; + /** + * v0.43 (#2095) — how many recent turns the reflex extracts entities from + * (default 4). 1 reproduces the legacy current-turn-only behavior (and the + * legacy slug+title suppression). File-plane / env + * (GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS) only — same plane as the other + * reflex knobs. + */ + retrieval_reflex_window_turns?: number; embedding_image_ocr?: boolean; embedding_image_ocr_model?: string; @@ -533,6 +541,10 @@ export function loadConfig(): GBrainConfig | null { ...(process.env.GBRAIN_RETRIEVAL_REFLEX ? { retrieval_reflex: !(process.env.GBRAIN_RETRIEVAL_REFLEX === 'false' || process.env.GBRAIN_RETRIEVAL_REFLEX === '0') } : {}), + ...(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS && + Number.isFinite(Number(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS)) + ? { retrieval_reflex_window_turns: Number(process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS) } + : {}), ...(process.env.GBRAIN_REMOTE_CLIENT_SECRET && fileConfig?.remote_mcp ? { remote_mcp: { ...fileConfig.remote_mcp, oauth_client_secret: process.env.GBRAIN_REMOTE_CLIENT_SECRET } } : {}), diff --git a/src/core/context-engine.ts b/src/core/context-engine.ts index 080e517da..e38114b98 100644 --- a/src/core/context-engine.ts +++ b/src/core/context-engine.ts @@ -179,6 +179,29 @@ function getLastUserText(messages: AgentMessage[]): string { return ''; } +/** + * v0.43 (#2095): the rolling extraction window — the most recent + * user/assistant turns (oldest → newest, current turn last), capped here at + * a generous max; the reflex slices to its configured + * retrieval_reflex_window_turns (default 4). + */ +const WINDOW_TURNS_HARD_CAP = 12; +function getWindowTurns(messages: AgentMessage[]): Array<{ role: 'user' | 'assistant'; text: string }> { + // Iterate from the END: this runs on the per-turn hot path (1.5s reflex + // budget) and only the last 12 turns matter — flattening every content + // block of a multi-hundred-turn session just to slice the tail would make + // the cost grow with session length. + const out: Array<{ role: 'user' | 'assistant'; text: string }> = []; + for (let i = messages.length - 1; i >= 0 && out.length < WINDOW_TURNS_HARD_CAP; i--) { + const m = messages[i]; + if (m?.role !== 'user' && m?.role !== 'assistant') continue; + const text = messageText(m.content); + if (!text) continue; + out.push({ role: m.role, text }); + } + return out.reverse(); +} + /** * Joined text of everything the agent has ALREADY seen — every message EXCEPT * the current turn (the last user message). Used for "already in context" @@ -649,6 +672,9 @@ export function createGBrainContextEngine(ctx: { workspaceDir, currentUserText: getLastUserText(messages), priorContextText: getPriorContextText(messages), + // v0.43 (#2095): rolling window — assistant-introduced entities and + // named-antecedent follow-ups from recent turns now resolve too. + windowTurns: getWindowTurns(messages), resolveEntities: ctx.resolveEntities, }); diff --git a/src/core/context/entity-salience.ts b/src/core/context/entity-salience.ts index a51d20ef2..91082a0dc 100644 --- a/src/core/context/entity-salience.ts +++ b/src/core/context/entity-salience.ts @@ -7,12 +7,15 @@ * touching the brain, so it must be fast (one regex pass) and precision-biased: * a false candidate costs a wasted resolve and, worse, a misleading pointer. * - * DELIBERATE v1 limits (documented, not bugs — see issue #1981 / eng-review): + * DELIBERATE limits (documented, not bugs — see issue #1981 / eng-review): * - Proper-case + ASCII biased. Misses lowercase names ("garry") and many * non-Latin scripts. - * - Current-user-message only. No pronoun follow-ups ("what about her?"), no - * entities the assistant introduced. - * These are TODOs, not v1 scope. Do NOT market this as full "human-like recall". + * - extractCandidates is single-turn. The v0.43 (#2095) window layer + * (extractCandidatesFromWindow) widens extraction across the last N turns + * — assistant-introduced entities and "what about her?" follow-ups whose + * antecedent was NAMED in the window now resolve; true pronoun + * coreference (never-named antecedents) remains out of scope. + * Do NOT market this as full "human-like recall". * * Resolution (alias/slug lookup) lives in retrieval-reflex.ts; this module only * decides WHAT to look up. @@ -172,3 +175,83 @@ export function extractCandidates(text: string): EntityCandidate[] { } return out; } + +// ── Rolling-window extraction (v0.43, #2095 push-based context) ────────── + +export interface WindowTurn { + role: 'user' | 'assistant'; + text: string; +} + +/** + * A window candidate with the salience metadata the volunteer layer's + * confidence boost reads: how many turns mentioned it, whether the NEWEST + * turn did, and whether the USER (vs only the assistant) ever said it. + */ +export interface WindowEntityCandidate extends EntityCandidate { + /** Number of distinct turns that mentioned this candidate. */ + occurrences: number; + /** Mentioned in the newest (last) turn of the window. */ + inNewestTurn: boolean; + /** Mentioned in at least one USER turn (assistant-only mentions rank lower). */ + userMention: boolean; +} + +/** + * Extract candidates across the last N turns (oldest → newest). Pure, + * zero-LLM: runs the per-turn extractor on each turn and merges by the + * normalizeAlias form. Ordering is salience-aware — recency of last mention, + * cross-turn frequency, and a user-role boost — so when the merged set + * exceeds MAX_CANDIDATES, the dropped tail is the stalest assistant-only + * chatter, not the entity the user just named. + */ +export function extractCandidatesFromWindow(turns: WindowTurn[]): WindowEntityCandidate[] { + if (!turns?.length) return []; + interface WAcc extends WindowEntityCandidate { + lastTurnIdx: number; + order: number; + } + const acc = new Map(); + let order = 0; + const lastIdx = turns.length - 1; + + for (let i = 0; i < turns.length; i++) { + const turn = turns[i]; + if (!turn?.text) continue; + for (const c of extractCandidates(turn.text)) { + const norm = normalizeAlias(c.query); + if (!norm) continue; + const existing = acc.get(norm); + if (existing) { + existing.occurrences += 1; + existing.lastTurnIdx = i; + existing.inNewestTurn = existing.inNewestTurn || i === lastIdx; + if (turn.role === 'user' && !existing.userMention) { + // First USER-said surface form beats an assistant-introduced one + // for the display label. + existing.display = c.display; + existing.userMention = true; + } + } else { + acc.set(norm, { + display: c.display, + query: c.query, + occurrences: 1, + lastTurnIdx: i, + inNewestTurn: i === lastIdx, + userMention: turn.role === 'user', + order: order++, + }); + } + } + } + + // Salience weight: recency dominates, then frequency, then user-role. + // Deterministic tie-break on first-seen order. + const weight = (c: WAcc) => + (c.lastTurnIdx + 1) / turns.length + Math.min(c.occurrences, 4) * 0.1 + (c.userMention ? 0.15 : 0); + return Array.from(acc.values()) + .sort((a, b) => weight(b) - weight(a) || a.order - b.order) + .slice(0, MAX_CANDIDATES) + .map(({ lastTurnIdx: _l, order: _o, ...rest }) => rest); +} diff --git a/src/core/context/reflex.ts b/src/core/context/reflex.ts index afec4054a..188dbea21 100644 --- a/src/core/context/reflex.ts +++ b/src/core/context/reflex.ts @@ -20,7 +20,12 @@ import { join } from 'node:path'; import { mkdirSync, appendFileSync } from 'node:fs'; import { loadConfig, type GBrainConfig } from '../config.ts'; import type { BrainEngine } from '../engine.ts'; -import { extractCandidates, type EntityCandidate } from './entity-salience.ts'; +import { + extractCandidates, + extractCandidatesFromWindow, + type EntityCandidate, + type WindowTurn, +} from './entity-salience.ts'; import { resolveEntitiesToPointers, DEFAULT_MAX_POINTERS, @@ -28,22 +33,69 @@ import { } from './retrieval-reflex.ts'; import { resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } from './resolve-ipc.ts'; -/** Host capability shape (D1=A): candidates in, pointers out. Narrow by design. */ +/** Per-turn resolver options shared by every rung of the ladder. */ +export interface ResolveEntitiesOpts { + priorContextText?: string; + maxPointers?: number; + /** v0.43 (#2095): 'slug-only' under windowing — see ResolvePointersOpts. */ + suppression?: 'slug-and-title' | 'slug-only'; +} + +/** + * Host capability shape (D1=A): candidates in, pointers out. Narrow by design. + * + * CONTRACT (red-team): a host resolver MUST honor `opts.suppression`. Under + * windowing the orchestrator passes 'slug-only' — a resolver that keeps + * applying the legacy title-whole-word rule will suppress every entity merely + * mentioned in a prior window turn and silently disable the feature. Hosts + * built against the pre-window contract should be upgraded or pinned to + * `retrieval_reflex_window_turns: 1`. (A capability/version gate so the + * orchestrator can detect a stale host is a filed TODO.) + */ export type ResolveEntitiesFn = ( candidates: EntityCandidate[], - opts: { priorContextText?: string; maxPointers?: number }, + opts: ResolveEntitiesOpts, ) => Promise; export interface ReflexParams { workspaceDir: string; - /** The current turn's user text (drives extraction). */ + /** The current turn's user text (drives extraction when no window is given). */ currentUserText: string; /** Joined PRIOR turns + loaded page bodies — EXCLUDES the current turn (suppression). */ priorContextText: string; + /** + * v0.43 (#2095): recent turns (oldest → newest, current turn last). When + * present and the configured window is > 1, extraction widens to the last + * N turns (assistant-introduced entities + named-antecedent follow-ups now + * resolve) and suppression switches to slug-only (codex D7 — the title rule + * would suppress every entity merely mentioned in a prior window turn). + */ + windowTurns?: WindowTurn[]; /** Host-provided resolver, if the OpenClaw plugin contract supplied one. */ resolveEntities?: ResolveEntitiesFn; } +/** Default extraction window (turns). 1 = legacy current-turn-only. */ +export const DEFAULT_WINDOW_TURNS = 4; + +export function windowTurnCount(cfg: GBrainConfig | null): number { + // Env plane is read DIRECTLY here (mirroring reflexEnabled's direct + // process.env read), not just via loadConfig's env→config mapping. When + // there's no config file AND no DATABASE_URL, loadConfig() returns null and + // drops that mapping entirely — so without this, the documented + // GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS escape hatch would be silently + // ignored and the window would fall back to the default of 4 (a real + // config-less-environment bug, e.g. a clean CI shard with no brain). + const env = process.env.GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS; + if (env != null && env !== '') { + const e = Number(env); + if (Number.isFinite(e) && e >= 1) return Math.floor(e); + } + const n = cfg?.retrieval_reflex_window_turns; + if (typeof n === 'number' && Number.isFinite(n) && n >= 1) return Math.floor(n); + return DEFAULT_WINDOW_TURNS; +} + const TIMEOUT_MS = 1500; // generous per-turn ceiling; the work is usually <100ms const HEARTBEAT_PATH = join(homedir(), '.gbrain', 'integrations', 'retrieval-reflex', 'heartbeat.jsonl'); @@ -68,14 +120,37 @@ export async function buildReflexAddition(params: ReflexParams): Promise 1. Window=1 reproduces the legacy + // current-turn-only behavior exactly (including suppression mode). + const windowN = windowTurnCount(cfg); + const windowed = windowN > 1 && (params.windowTurns?.length ?? 0) > 0; + const candidates: EntityCandidate[] = windowed + ? extractCandidatesFromWindow(params.windowTurns!.slice(-windowN)) + : extractCandidates(params.currentUserText); + // Zero-candidate fast path: regex passes only, no brain touch. if (!candidates.length) return null; - const opts = { priorContextText: params.priorContextText, maxPointers: maxPointers(cfg) }; + const opts: ResolveEntitiesOpts = { + priorContextText: params.priorContextText, + maxPointers: maxPointers(cfg), + suppression: windowed ? 'slug-only' : 'slug-and-title', + }; const block = await withTimeout(resolve(params, cfg, candidates, opts), TIMEOUT_MS); if (!block || !block.pointers.length) return null; + // Accept-side reflex-channel logging (red-team): the block survived the + // per-turn timeout, so these pointers ARE being injected. Only the + // direct-Postgres rung has an engine here; the IPC rung logs server-side + // at delivery; host-injected resolvers can't log (documented gap). + if (!params.resolveEntities && isPostgres(cfg)) { + const engine = await getPostgresEngine(cfg); + if (engine) { + const { logDeliveredReflexPointers } = await import('./retrieval-reflex.ts'); + logDeliveredReflexPointers(engine, block.pointers); + } + } + writeHeartbeat(cfg, block.pointers.length); return block.text; } catch { @@ -87,7 +162,7 @@ async function resolve( params: ReflexParams, cfg: GBrainConfig | null, candidates: EntityCandidate[], - opts: { priorContextText?: string; maxPointers?: number }, + opts: ResolveEntitiesOpts, ): Promise { // 1. Host capability (any engine). if (params.resolveEntities) { diff --git a/src/core/context/resolve-ipc.ts b/src/core/context/resolve-ipc.ts index 3955ea3e1..55e705e93 100644 --- a/src/core/context/resolve-ipc.ts +++ b/src/core/context/resolve-ipc.ts @@ -35,6 +35,8 @@ export interface ResolveRequest { priorContextText?: string; maxPointers?: number; sourceId?: string; + /** v0.43 (#2095, codex D7): suppression mode — 'slug-only' under windowing. */ + suppression?: 'slug-and-title' | 'slug-only'; } export type ResolveHandler = (req: ResolveRequest) => Promise; @@ -98,6 +100,13 @@ export async function resolveViaIpc( export async function startResolveIpcServer( socketPath: string, handler: ResolveHandler, + /** + * v0.43 (#2095, red-team): fired ONLY after the response was successfully + * written to the client — the accept-side seam for reflex-channel feedback + * logging. A block the client never received (timeout, dead socket) was + * never injected into a prompt and must not count as "volunteered". + */ + onDelivered?: (block: PointerBlock, req: ResolveRequest) => void, ): Promise { // Remove a stale socket file if present (a previous serve that didn't clean up). cleanupStaleSocket(socketPath); @@ -113,14 +122,23 @@ export async function startResolveIpcServer( if (nl < 0) return; const line = buf.slice(0, nl); let resp: string; + let delivered: { block: PointerBlock; req: ResolveRequest } | null = null; try { const req = JSON.parse(line) as ResolveRequest; const block = await handler(req); resp = JSON.stringify({ ok: true, block }); + if (block) delivered = { block, req }; } catch (e) { resp = JSON.stringify({ ok: false, error: (e as Error).message }); } - try { conn.write(resp + '\n'); } catch { /* client gone */ } + try { + conn.write(resp + '\n'); + // Write accepted — the client (250ms budget) may still have hung + // up, but this is the closest observable delivery point. + if (delivered && onDelivered) { + try { onDelivered(delivered.block, delivered.req); } catch { /* telemetry only */ } + } + } catch { /* client gone — do NOT log undelivered pointers */ } conn.end(); }); conn.on('error', () => { try { conn.destroy(); } catch { /* noop */ } }); diff --git a/src/core/context/retrieval-reflex.ts b/src/core/context/retrieval-reflex.ts index d1cb8e3af..1f9b3d70f 100644 --- a/src/core/context/retrieval-reflex.ts +++ b/src/core/context/retrieval-reflex.ts @@ -34,10 +34,40 @@ import type { EntityCandidate } from './entity-salience.ts'; export const DEFAULT_MAX_POINTERS = 3; const SYNOPSIS_MAX = 160; +/** Which resolution arm produced a pointer (provenance → honest confidence). */ +export type ResolveArm = 'alias' | 'title' | 'slug-suffix'; + +/** + * v0.43 (#2095) — arm → confidence. Lives HERE, next to the arm definitions, + * so arm identity and its score can't drift apart (eng-review note). The + * volunteer layer imports these; small deterministic boosts (multi-turn / + * newest-turn mention) are added on top there. + */ +export const ARM_CONFIDENCE: Record = { + alias: 0.9, + title: 0.8, + 'slug-suffix': 0.6, +}; + export interface ReflexPointer { display: string; slug: string; + /** Which brain source the page lives in (federated callers need it for dedup). */ + source_id: string; synopsis: string; + /** Resolution provenance (v0.43 #2095). */ + arm: ResolveArm; + /** Base arm confidence (ARM_CONFIDENCE[arm]); callers may boost. */ + confidence: number; + /** + * normalizeAlias form of the CANDIDATE that resolved this pointer (v0.43 + * #2095) — lets the volunteer layer join pointers back to window-salience + * metadata without guessing from the display label (which falls back to the + * page title when the candidate surface differs, e.g. alias "Swami" → + * title "Swami X"). Absent only when suffix classification couldn't + * recover the source candidate. + */ + matchedNorm?: string; } export interface PointerBlock { @@ -55,10 +85,30 @@ export interface ResolvePointersOpts { * message's own mention would suppress every pointer (eng-review/Codex fix). */ priorContextText?: string; + /** + * v0.43 (#2095, codex D7) — suppression mode. + * 'slug-and-title' (default, window=1 legacy): suppress when the slug OR + * the page title appears whole-word in prior context. + * 'slug-only' (REQUIRED under multi-turn windowing): suppress on slug + * presence only. Slugs only enter context when a pointer block or page + * body was actually injected; a bare mention of "Alice Example" in a + * prior turn never contains `people/alice-example`. The title rule + * would suppress every entity merely MENTIONED in a prior window turn + * — breaking window extraction by construction. + */ + suppression?: 'slug-and-title' | 'slug-only'; + /** + * v0.43 (#2095) — federated scope: resolve across these sources instead of + * the single positional sourceId. Precedence mirrors sourceScopeOpts + * (federated array > scalar). Alias arm loops per source; the title/slug + * arm uses source_id = ANY(...) in one query. + */ + sourceIds?: string[]; } interface PageRow { slug: string; + source_id: string; title: string; type: string | null; frontmatter: Record | null; @@ -88,40 +138,63 @@ export async function resolveEntitiesToPointers( const titlesLc: string[] = []; const exactSlugs: string[] = []; const slugSuffixes: string[] = []; + // Reverse maps for arm-2 provenance (which candidate produced a row) — + // populated in this same pass so the derivations happen exactly once. + const titleToNorm = new Map(); + const slugToNorm = new Map(); for (const c of candidates) { const norm = normalizeAlias(c.query); if (!norm) continue; if (!displayByNorm.has(norm)) displayByNorm.set(norm, c.display); aliasNorms.push(norm); - titlesLc.push(c.query.toLowerCase()); + const tl = c.query.toLowerCase(); + titlesLc.push(tl); + if (!titleToNorm.has(tl)) titleToNorm.set(tl, norm); const s = slugify(c.query); if (s) { exactSlugs.push(s); slugSuffixes.push(`%/${s}`); + if (!slugToNorm.has(s)) slugToNorm.set(s, norm); } } if (!aliasNorms.length) return null; - // Ordered set of resolved slugs (alias hits first → higher confidence). - const resolvedSlugs: string[] = []; + // Federated scope (v0.43 #2095): explicit sourceIds win over the scalar. + const sourceIds = opts.sourceIds?.length ? opts.sourceIds : [sourceId]; + + // Ordered set of resolved (source, slug) pairs with arm provenance — + // alias hits pushed first → higher confidence. + const resolved: Array<{ slug: string; source_id: string; arm: ResolveArm; matchedNorm?: string }> = []; const seen = new Set(); - const pushSlug = (slug: string) => { - if (slug && !seen.has(slug)) { - seen.add(slug); - resolvedSlugs.push(slug); + // Neither source ids nor slugs contain spaces, so a space separator is safe. + const keyOf = (src: string, slug: string) => `${src} ${slug}`; + const push = (slug: string, src: string, arm: ResolveArm, matchedNorm?: string) => { + if (!slug) return; + const k = keyOf(src, slug); + if (!seen.has(k)) { + seen.add(k); + resolved.push({ slug, source_id: src, arm, matchedNorm }); } }; - - // Arm 1 — alias-first. Unambiguous single-slug hits only. Guarded: pre-v110 - // brains throw "relation page_aliases does not exist" — swallow and continue. - try { - const aliasMap = await engine.resolveAliases(aliasNorms, { sourceId }); + // Arm 1 — alias-first. Unambiguous single-slug hits only, per source (no + // engine-interface change for federation). Guarded: pre-v110 brains throw + // "relation page_aliases does not exist" — swallow and continue. + // Per-source lookups are independent — run them concurrently so a + // federated caller (M granted sources) pays one RTT, not M sequential + // ones (~71ms each cross-region; the reflex runs under a 1.5s budget). + // Results are folded back in sourceIds order so pointer ordering stays + // deterministic. Per-source failures degrade independently (pre-v110 + // brains have no page_aliases table). + const aliasResults = await Promise.allSettled( + sourceIds.map((src) => engine.resolveAliases(aliasNorms, { sourceId: src })), + ); + for (let i = 0; i < sourceIds.length; i++) { + const r = aliasResults[i]; + if (r.status !== 'fulfilled') continue; for (const norm of aliasNorms) { - const hits = aliasMap.get(norm); - if (hits && hits.length === 1) pushSlug(hits[0].slug); + const hits = r.value.get(norm); + if (hits && hits.length === 1) push(hits[0].slug, sourceIds[i], 'alias', norm); } - } catch { - /* no page_aliases table (pre-v110) — degrade to the title/slug arm */ } // Arm 2 — exact title OR slug-suffix. This is the recall fix: a bare "Alice @@ -130,53 +203,72 @@ export async function resolveEntitiesToPointers( let rows: PageRow[] = []; try { rows = await engine.executeRaw( - `SELECT slug, title, type, frontmatter, compiled_truth + `SELECT slug, source_id, title, type, frontmatter, compiled_truth FROM pages WHERE deleted_at IS NULL - AND source_id = $1 + AND source_id = ANY($1::text[]) AND ( lower(title) = ANY($2::text[]) OR slug = ANY($3::text[]) OR slug LIKE ANY($4::text[]) )`, - [sourceId, titlesLc, exactSlugs, slugSuffixes], + [sourceIds, titlesLc, exactSlugs, slugSuffixes], ); } catch { rows = []; } - // Hydrate alias-resolved slugs too (their bodies for the synopsis) if not in rows. - const rowBySlug = new Map(); - for (const r of rows) rowBySlug.set(r.slug, r); - const aliasOnly = resolvedSlugs.filter((s) => !rowBySlug.has(s)); + // Hydrate alias-resolved pages too (their bodies for the synopsis) if not in rows. + const rowByKey = new Map(); + for (const r of rows) rowByKey.set(keyOf(r.source_id, r.slug), r); + const aliasOnly = resolved.filter((p) => !rowByKey.has(keyOf(p.source_id, p.slug))); if (aliasOnly.length) { try { const extra = await engine.executeRaw( - `SELECT slug, title, type, frontmatter, compiled_truth + `SELECT slug, source_id, title, type, frontmatter, compiled_truth FROM pages - WHERE deleted_at IS NULL AND source_id = $1 AND slug = ANY($2::text[])`, - [sourceId, aliasOnly], + WHERE deleted_at IS NULL AND source_id = ANY($1::text[]) AND slug = ANY($2::text[])`, + [sourceIds, aliasOnly.map((p) => p.slug)], ); - for (const r of extra) rowBySlug.set(r.slug, r); + for (const r of extra) rowByKey.set(keyOf(r.source_id, r.slug), r); } catch { /* ignore — alias slug may be stale */ } } // Title/slug matches that weren't alias hits, appended after alias hits. - for (const r of rows) pushSlug(r.slug); + // Arm provenance per row is classified in JS (codex D8) — the combined OR + // can't report which predicate matched: an exact lower(title) hit is the + // 'title' arm; anything else got in via slug / slug-suffix. + const titleSet = new Set(titlesLc); + for (const r of rows) { + const titleLc = (r.title ?? '').toLowerCase(); + if (titleSet.has(titleLc)) { + push(r.slug, r.source_id, 'title', titleToNorm.get(titleLc)); + } else { + // Slug arm: exact slugified-candidate match, else suffix scan. + const tail = r.slug.includes('/') ? r.slug.slice(r.slug.lastIndexOf('/') + 1) : r.slug; + push(r.slug, r.source_id, 'slug-suffix', slugToNorm.get(r.slug) ?? slugToNorm.get(tail)); + } + } // Build pointers in confidence order, applying suppression + cap. + const suppression = opts.suppression ?? 'slug-and-title'; const pointers: ReflexPointer[] = []; - for (const slug of resolvedSlugs) { - const row = rowBySlug.get(slug); + for (const { slug, source_id, arm, matchedNorm } of resolved) { + const row = rowByKey.get(keyOf(source_id, slug)); if (!row) continue; - // Suppression: already present in PRIOR context (slug or title). The current - // turn is deliberately excluded from priorContextText. + // Suppression: already present in PRIOR context. The current turn is + // deliberately excluded from priorContextText. Under windowing + // ('slug-only', codex D7) only the slug counts — a slug appears in prior + // context only when a pointer/page was actually surfaced there, while a + // title appears on any bare mention. if (priorLc) { - const titleLc = (row.title ?? '').toLowerCase(); if (priorLc.includes(slug.toLowerCase())) continue; - if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue; + if (suppression === 'slug-and-title') { + const titleLc = (row.title ?? '').toLowerCase(); + if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue; + } } const display = displayForRow(row, displayByNorm); const synopsis = safeSynopsis(row); - pointers.push({ display, slug, synopsis }); + pointers.push({ display, slug, source_id, synopsis, arm, confidence: ARM_CONFIDENCE[arm], matchedNorm }); if (pointers.length >= maxPointers) break; } @@ -246,3 +338,30 @@ export function renderPointerBlock(pointers: ReflexPointer[]): string { } return lines.join('\n'); } + +/** + * v0.43 (#2095, codex D11 + red-team) — ambient-channel feedback logging, + * ACCEPT-SIDE ONLY. Called by the delivery points (the serve IPC server after + * a successful write; the direct-Postgres reflex rung after its per-turn + * timeout admitted the block) — never inside the resolver itself, because a + * pointer block that timed out client-side was NEVER injected into a prompt, + * and logging it would inflate "volunteered" counts and drag the measured + * precision toward zero (corrupting the exact stats users tune + * min_confidence with). + */ +export function logDeliveredReflexPointers(engine: BrainEngine, pointers: ReflexPointer[]): void { + if (!pointers.length) return; + void import('./volunteer-events.ts') + .then(({ logVolunteerEventsFireAndForget, volunteerEventRowsFrom }) => { + logVolunteerEventsFireAndForget( + engine, + volunteerEventRowsFrom( + pointers.map((p) => ({ ...p, rationale: `${p.arm} match "${p.display}"` })), + { channel: 'reflex' }, + ), + ); + }) + .catch(() => { + /* telemetry only */ + }); +} diff --git a/src/core/context/volunteer-events.ts b/src/core/context/volunteer-events.ts new file mode 100644 index 000000000..2ec14a7b6 --- /dev/null +++ b/src/core/context/volunteer-events.ts @@ -0,0 +1,186 @@ +/** + * v0.43 (#2095) — context_volunteer_events: the feedback-loop log behind + * push-based context. One row per page the brain VOLUNTEERED, written + * fire-and-forget by the volunteer_context op, the retrieval-reflex pointer + * path (channel 'reflex'), and `gbrain watch` (channel 'watch'). + * + * "Used" is DERIVED, never written: a volunteered page counts as used when + * pages.last_retrieved_at > volunteered_at (the existing bumpLastRetrievedAt + * write-back on get_page/search/query is the open/cite signal). The join is + * approximate by design — last-retrieved is 5-min throttled (false negatives) + * and unrelated reads match too (false positives); stats output carries the + * caveat. + * + * Retention: rows older than VOLUNTEER_EVENTS_TTL_DAYS are pruned by the + * dream cycle's purge phase so conversation-adjacent telemetry never grows + * unbounded. rationale is a deterministic template that may embed the matched + * entity's surface form (which by construction resolved to an existing + * alias/title/slug) — never free conversation text. + */ + +import type { BrainEngine } from './../engine.ts'; +import { registerBackgroundWorkDrainer } from '../background-work.ts'; + +export const VOLUNTEER_EVENTS_TTL_DAYS = 90; + +export type VolunteerChannel = 'op' | 'reflex' | 'watch'; + +/** + * Map volunteered pages to event rows for one channel — the ONE place the + * VolunteerEventRow shape is assembled (op / reflex / watch all call this, + * so adding a column is a one-site change). + */ +export function volunteerEventRowsFrom( + pages: Array<{ source_id: string; slug: string; confidence: number; arm: string; rationale: string }>, + opts: { channel: VolunteerChannel; session_id?: string | null; turn?: number | null }, +): VolunteerEventRow[] { + return pages.map((p) => ({ + source_id: p.source_id, + slug: p.slug, + confidence: p.confidence, + match_arm: p.arm, + rationale: p.rationale, + channel: opts.channel, + session_id: opts.session_id ?? null, + turn: opts.turn ?? null, + })); +} + +export interface VolunteerEventRow { + source_id: string; + slug: string; + confidence: number; + match_arm: string; + rationale: string; + channel: VolunteerChannel; + session_id?: string | null; + turn?: number | null; +} + +/** + * ONE multi-row parameterized INSERT for a batch of volunteered pages (max 5 + * per call by the volunteer cap) — never per-row awaited INSERTs (up to 5 + * RTTs ≈ 355ms on a cross-region deployment; eng-review D4). Throws on + * failure; callers run it through the volunteer-events background-work sink + * with try/catch so logging can never fail the op. + */ +export async function insertVolunteerEvents( + engine: BrainEngine, + rows: VolunteerEventRow[], +): Promise { + if (!rows.length) return; + const params: unknown[] = []; + const tuples = rows.map((r) => { + const base = params.length; + params.push( + r.source_id, + r.slug, + r.confidence, + r.match_arm, + r.rationale, + r.channel, + r.session_id ?? null, + r.turn ?? null, + ); + const ph = Array.from({ length: 8 }, (_, i) => `$${base + i + 1}`); + return `(${ph.join(', ')})`; + }); + await engine.executeRaw( + `INSERT INTO context_volunteer_events + (source_id, slug, confidence, match_arm, rationale, channel, session_id, turn) + VALUES ${tuples.join(', ')}`, + params, + ); +} + +// ── Fire-and-forget sink (eng-review D4) ───────────────────────────────── +// Mirrors src/core/last-retrieved.ts: track every dangling INSERT promise in +// a module Set, register a drainer so finishCliTeardown settles them +// against a live engine before teardown on EVERY CLI exit path (the commit-1 +// drain hoist). Logging failure never fails the caller. + +const pendingVolunteerEventWrites = new Set>(); + +/** + * Log volunteered pages without blocking the hot path. The batched INSERT + * runs as a tracked dangling promise; errors are swallowed (pre-v117 brains, + * transient DB failures — the volunteer result is unaffected). + */ +export function logVolunteerEventsFireAndForget( + engine: BrainEngine, + rows: VolunteerEventRow[], +): void { + if (!rows.length) return; + const p = insertVolunteerEvents(engine, rows).catch(() => { + /* best-effort telemetry — never surfaces */ + }); + pendingVolunteerEventWrites.add(p); + void p.finally(() => pendingVolunteerEventWrites.delete(p)); +} + +/** Drain pending event writes (bounded). Same snapshot semantics as last-retrieved. */ +export async function awaitPendingVolunteerEventWrites( + timeoutMs = 5_000, +): Promise<{ unfinished: number }> { + if (pendingVolunteerEventWrites.size === 0) return { unfinished: 0 }; + const snapshot = Array.from(pendingVolunteerEventWrites); + let timer: ReturnType | undefined; + const timeout = new Promise<'timeout'>((resolve) => { + timer = setTimeout(() => resolve('timeout'), timeoutMs); + }); + const drain = Promise.allSettled(snapshot).then(() => 'drained' as const); + const outcome = await Promise.race([drain, timeout]); + if (timer) clearTimeout(timer); + if (outcome === 'timeout') { + const unfinished = pendingVolunteerEventWrites.size; + // Drop the snapshot so a long-lived process (`gbrain watch`) doesn't + // accumulate references to forever-pending work (last-retrieved C1). + for (const p of snapshot) pendingVolunteerEventWrites.delete(p); + return { unfinished }; + } + return { unfinished: 0 }; +} + +// Registered in the enqueue-owning module (background-work contract): module +// not imported ⇒ nothing enqueued ⇒ nothing to drain. Order 4 — after facts / +// last-retrieved / search-cache / eval-capture; bare INSERTs, no abort. +registerBackgroundWorkDrainer({ + name: 'volunteer-events', + order: 4, + drain: (ms) => awaitPendingVolunteerEventWrites(ms), +}); + +/** Test seam — clears the pending set so each test starts clean. */ +export function _resetPendingVolunteerEventWritesForTests(): void { + pendingVolunteerEventWrites.clear(); +} + +/** Test seam — peek the current pending count. */ +export function _peekPendingVolunteerEventWritesForTests(): number { + return pendingVolunteerEventWrites.size; +} + +/** + * 90-day GC, called from the dream cycle's purge phase (mirrors + * purgeStaleCheckpoints). Best-effort: returns 0 on any failure (pre-v117 + * brains have no table yet). + */ +export async function purgeStaleVolunteerEvents( + engine: BrainEngine, + ttlDays = VOLUNTEER_EVENTS_TTL_DAYS, +): Promise { + try { + const rows = await engine.executeRaw<{ count: string | number }>( + `WITH deleted AS ( + DELETE FROM context_volunteer_events + WHERE volunteered_at < now() - ($1 || ' days')::interval + RETURNING 1 + ) + SELECT count(*)::text AS count FROM deleted`, + [String(ttlDays)], + ); + return Number(rows[0]?.count ?? 0); + } catch { + return 0; + } +} diff --git a/src/core/context/volunteer.ts b/src/core/context/volunteer.ts new file mode 100644 index 000000000..dcdd721ad --- /dev/null +++ b/src/core/context/volunteer.ts @@ -0,0 +1,268 @@ +/** + * v0.43 (#2095) — push-based context: the brain VOLUNTEERS relevant pages + * from a rolling conversation window instead of waiting to be asked. + * + * window text ─ parseWindow() ─→ turns[] ─ extractCandidatesFromWindow() + * │ (recency + frequency + + * │ user-role weights) + * ▼ + * resolveEntitiesToPointers(sourceIds[]) alias 0.9 / title 0.8 / + * │ slug-suffix 0.6 (+0.05 boost + * ▼ for ≥2-turn or newest-turn + * gate min_confidence (default 0.7) → mentions) + * suppression (slug-only — windowing) → + * cap (3 default / 5 max) + * + * Zero-LLM, deterministic, precision-biased: push noise is worse than pull + * silence (#2095). At the default gate, slug-suffix matches (0.6+0.05 < 0.7) + * never volunteer — they need an explicit lower min_confidence. + * + * Consumed by three channels: the volunteer_context op, the retrieval-reflex + * window path, and `gbrain watch`. Event logging lives in + * volunteer-events.ts; usage stats here derive "used" from + * pages.last_retrieved_at — APPROXIMATE by design (the 5-min last-retrieved + * throttle causes false negatives; unrelated reads cause false positives). + */ + +import type { BrainEngine } from '../engine.ts'; +import { normalizeAlias } from '../search/alias-normalize.ts'; +import { + extractCandidatesFromWindow, + type WindowTurn, + type WindowEntityCandidate, +} from './entity-salience.ts'; +import { + resolveEntitiesToPointers, + ARM_CONFIDENCE, + type ResolveArm, +} from './retrieval-reflex.ts'; + +export const VOLUNTEER_DEFAULT_MAX_PAGES = 3; +export const VOLUNTEER_MAX_PAGES_CAP = 5; +export const VOLUNTEER_DEFAULT_MIN_CONFIDENCE = 0.7; +/** Deterministic boost for ≥2-turn or newest-turn mentions. */ +export const VOLUNTEER_SALIENCE_BOOST = 0.05; + +export interface VolunteeredPage { + slug: string; + source_id: string; + display: string; + confidence: number; + arm: ResolveArm; + /** Deterministic template string — never raw conversation text. */ + rationale: string; + synopsis: string; +} + +export interface VolunteerOpts { + /** Resolved source scope (federated array > scalar — sourceScopeOpts shape). */ + sourceIds: string[]; + /** Prior context (already-surfaced pointers/pages) for slug-only suppression. */ + priorContext?: string; + /** + * Slugs to skip BEFORE the confidence gate and the maxPages cap (O(1) + * membership). `gbrain watch` passes its session-dedupe set here — a + * post-call filter would let a recurring already-pushed entity consume cap + * slots every turn and starve new pages behind it (red-team finding). + */ + excludeSlugs?: ReadonlySet; + maxPages?: number; + minConfidence?: number; +} + +/** Shared wire protocol for window turns — watch.ts imports this so the two + * channels can never desynchronize on the prefix grammar. */ +export const TURN_PREFIX_RE = /^(user|assistant)\s*:\s?(.*)$/i; + +/** + * Lenient window parser: `user:` / `assistant:` line prefixes start a new + * turn (oldest → newest); unprefixed lines continue the current turn; input + * with no prefixes at all is ONE user turn (so `echo "..." | volunteer-context` + * just works). CRLF-tolerant. Empty/whitespace input → []. + */ +export function parseWindow(text: string): WindowTurn[] { + if (!text || !text.trim()) return []; + const turns: WindowTurn[] = []; + let current: WindowTurn | null = null; + for (const rawLine of text.split(/\r?\n/)) { + const m = TURN_PREFIX_RE.exec(rawLine); + if (m) { + if (current) turns.push(current); + current = { role: m[1].toLowerCase() as WindowTurn['role'], text: m[2] ?? '' }; + } else if (current) { + current.text += (current.text ? '\n' : '') + rawLine; + } else if (rawLine.trim()) { + // No prefix seen yet — accumulate into an implicit user turn. + current = { role: 'user', text: rawLine }; + } + } + if (current) turns.push(current); + // Trim trailing whitespace-only turn bodies. + return turns + .map((t) => ({ role: t.role, text: t.text.trim() })) + .filter((t) => t.text.length > 0); +} + +function clampMaxPages(n: number | undefined): number { + if (typeof n !== 'number' || !Number.isFinite(n) || n < 1) return VOLUNTEER_DEFAULT_MAX_PAGES; + return Math.min(Math.floor(n), VOLUNTEER_MAX_PAGES_CAP); +} + +function rationaleFor(arm: ResolveArm, display: string, c: WindowEntityCandidate | undefined, windowSize: number): string { + const armText = + arm === 'alias' ? `alias match "${display}"` + : arm === 'title' ? `exact title match "${display}"` + : `slug match "${display}"`; + if (!c) return armText; + const parts = [armText]; + if (c.occurrences >= 2) parts.push(`mentioned in ${c.occurrences} of last ${windowSize} turns`); + else if (c.inNewestTurn) parts.push('mentioned in the newest turn'); + if (!c.userMention) parts.push('assistant-introduced'); + return parts.join('; '); +} + +/** + * Volunteer confidence-gated pages for a conversation window. Pure read — + * event logging is the CALLER's job (through the volunteer-events sink). + * Non-relational, zero-LLM; returns [] when nothing clears the gate. + */ +export async function volunteerContext( + engine: BrainEngine, + turns: WindowTurn[], + opts: VolunteerOpts, +): Promise { + if (!turns.length || !opts.sourceIds?.length) return []; + const candidates = extractCandidatesFromWindow(turns); + if (!candidates.length) return []; + const byNorm = new Map(); + for (const c of candidates) { + const norm = normalizeAlias(c.query); + if (norm && !byNorm.has(norm)) byNorm.set(norm, c); + } + + const maxPages = clampMaxPages(opts.maxPages); + const minConfidence = + typeof opts.minConfidence === 'number' && opts.minConfidence >= 0 && opts.minConfidence <= 1 + ? opts.minConfidence + : VOLUNTEER_DEFAULT_MIN_CONFIDENCE; + + // Resolve up to the hard cap so the confidence gate sees the full pool — + // a gated-out alias hit must not shadow a passing title hit behind it. + const block = await resolveEntitiesToPointers(engine, opts.sourceIds[0], candidates, { + sourceIds: opts.sourceIds, + priorContextText: opts.priorContext, + suppression: 'slug-only', + maxPointers: VOLUNTEER_MAX_PAGES_CAP * 2, + }); + if (!block) return []; + + const out: VolunteeredPage[] = []; + for (const p of block.pointers) { + if (opts.excludeSlugs?.has(p.slug)) continue; // before gate + cap — see VolunteerOpts + // matchedNorm is the resolver's provenance join-key (the candidate that + // resolved the pointer); display-based lookup is the fallback for the + // rare suffix rows where provenance couldn't be recovered. + const cand = (p.matchedNorm ? byNorm.get(p.matchedNorm) : undefined) ?? byNorm.get(normalizeAlias(p.display)); + const boost = cand && (cand.occurrences >= 2 || cand.inNewestTurn) ? VOLUNTEER_SALIENCE_BOOST : 0; + const confidence = Math.min(0.99, ARM_CONFIDENCE[p.arm] + boost); + if (confidence < minConfidence) continue; + out.push({ + slug: p.slug, + source_id: p.source_id, + display: p.display, + confidence, + arm: p.arm, + rationale: rationaleFor(p.arm, p.display, cand, turns.length), + synopsis: p.synopsis, + }); + if (out.length >= maxPages) break; + } + return out; +} + +/** + * Canonical human rendering of one volunteered page — shared by + * `gbrain volunteer-context` (cli.ts formatResult) and `gbrain watch` so the + * two surfaces can't drift. + */ +export function formatVolunteeredPage(p: VolunteeredPage): string { + return ( + `${p.display} → ${p.slug} (${p.confidence.toFixed(2)}, ${p.arm}) — ${p.rationale}` + + (p.synopsis ? `\n ${p.synopsis}` : '') + ); +} + +// ── Usage stats (the feedback loop) ────────────────────────────────────── + +export interface VolunteerArmStats { + match_arm: string; + channel: string; + volunteered: number; + used: number; + /** used / volunteered, 0 when nothing volunteered. */ + precision: number; +} + +export interface VolunteerUsageStats { + days: number; + /** The join is approximate — see the note (false +/- documented in #2095 D9). */ + approximate: true; + note: string; + total_volunteered: number; + total_used: number; + by_arm: VolunteerArmStats[]; +} + +export const VOLUNTEER_STATS_NOTE = + 'approximate: "used" = pages.last_retrieved_at > volunteered_at. The 5-min ' + + 'last-retrieved throttle causes false negatives; unrelated reads of the same ' + + 'page cause false positives.'; + +/** + * Per-arm/channel precision over the last N days, source-scoped. Read-only; + * returns zeroed stats on pre-v117 brains (no table). + */ +export async function volunteerUsageStats( + engine: BrainEngine, + sourceIds: string[], + days = 30, +): Promise { + const safeDays = Number.isFinite(days) && days > 0 ? Math.floor(days) : 30; + let rows: Array<{ match_arm: string; channel: string; volunteered: string | number; used: string | number }> = []; + try { + rows = await engine.executeRaw( + `SELECT e.match_arm, e.channel, + count(*)::text AS volunteered, + count(*) FILTER (WHERE p.last_retrieved_at > e.volunteered_at)::text AS used + FROM context_volunteer_events e + LEFT JOIN pages p + ON p.source_id = e.source_id AND p.slug = e.slug AND p.deleted_at IS NULL + WHERE e.source_id = ANY($1::text[]) + AND e.volunteered_at > now() - ($2 || ' days')::interval + GROUP BY e.match_arm, e.channel + ORDER BY e.match_arm, e.channel`, + [sourceIds, String(safeDays)], + ); + } catch { + rows = []; // pre-v117 brain — table doesn't exist yet + } + const by_arm: VolunteerArmStats[] = rows.map((r) => { + const volunteered = Number(r.volunteered); + const used = Number(r.used); + return { + match_arm: r.match_arm, + channel: r.channel, + volunteered, + used, + precision: volunteered > 0 ? Number((used / volunteered).toFixed(3)) : 0, + }; + }); + return { + days: safeDays, + approximate: true, + note: VOLUNTEER_STATS_NOTE, + total_volunteered: by_arm.reduce((s, a) => s + a.volunteered, 0), + total_used: by_arm.reduce((s, a) => s + a.used, 0), + by_arm, + }; +} diff --git a/src/core/cycle.ts b/src/core/cycle.ts index 116e5420f..17a31bccb 100644 --- a/src/core/cycle.ts +++ b/src/core/cycle.ts @@ -1285,6 +1285,16 @@ async function runPhasePurge(engine: BrainEngine, dryRun: boolean): Promise 117 at merge: master's v0.42.41.0 triage wave + // claimed v116 (code_edges_source_backfill_and_callee_index) first. + version: 117, + name: 'context_volunteer_events_table', + // #2095 push-based context: feedback-loop log of every page the brain + // VOLUNTEERED (via the volunteer_context op, the retrieval-reflex pointer + // path, or `gbrain watch`). "Used" is derived later by joining + // pages.last_retrieved_at > volunteered_at — no second write path. + // session_id/turn are caller-supplied attribution (nullable). rationale is + // a deterministic template string, NEVER raw conversation text. Rows are + // pruned past 90 days by the dream cycle's purge phase + // (purgeStaleVolunteerEvents). No ::jsonb anywhere. RLS: covered by the + // v35 auto_rls_on_create_table event trigger on Postgres (same as + // v110/v115 tables); pinned by the volunteer-context Postgres e2e. + // Created empty; plain CREATE INDEX is instant — no CONCURRENTLY needed. + // Keep in sync with src/schema.sql, src/core/pglite-schema.ts, + // src/core/schema-embedded.ts. + idempotent: true, + sql: ` + CREATE TABLE IF NOT EXISTS context_volunteer_events ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + slug TEXT NOT NULL, + confidence DOUBLE PRECISION NOT NULL, + match_arm TEXT NOT NULL, + rationale TEXT NOT NULL DEFAULT '', + channel TEXT NOT NULL DEFAULT 'op', + session_id TEXT, + turn INTEGER, + volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now() + ); + CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx + ON context_volunteer_events (source_id, volunteered_at DESC); + CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx + ON context_volunteer_events (source_id, slug); + `, + }, ]; export const LATEST_VERSION = MIGRATIONS.length > 0 diff --git a/src/core/operations.ts b/src/core/operations.ts index aa6f41e41..eb6adc70f 100644 --- a/src/core/operations.ts +++ b/src/core/operations.ts @@ -3145,6 +3145,99 @@ const get_recent_salience: Operation = { cliHints: { name: 'salience' }, }; +// --- v0.43 (#2095): push-based context — the brain volunteers pages --- + +const volunteer_context: Operation = { + name: 'volunteer_context', + description: + 'Push-based context: volunteer brain pages relevant to a rolling conversation window ' + + 'WITHOUT being asked. Zero-LLM, confidence-gated (alias 0.9 / exact-title 0.8 / ' + + 'slug-suffix 0.6, +0.05 for multi-turn or newest-turn mentions; default gate 0.7), ' + + 'capped at 3 pages (max 5). Returns pointers with one-line rationales + synopses — ' + + 'open the page (get_page) before relying on details. Pass stats: true for the ' + + 'approximate volunteered-vs-used precision summary (the feedback loop).', + scope: 'read', + params: { + window: { + type: 'string', + description: + "Recent conversation turns, oldest → newest, as 'user:' / 'assistant:' prefixed " + + 'lines (unprefixed text = one user turn). Required unless stats: true. ' + + 'CLI: piped stdin fills this.', + }, + prior_context: { + type: 'string', + description: + 'Already-surfaced context (pointer blocks / opened page bodies). Pages whose slug ' + + 'appears here are not re-volunteered.', + }, + max_pages: { type: 'number', description: 'Max pages to volunteer (default 3, hard cap 5).' }, + min_confidence: { + type: 'number', + description: + 'Confidence gate 0..1 (default 0.7 — slug-suffix matches need an explicit lower gate).', + }, + session_id: { type: 'string', description: 'Optional caller session id, logged for attribution.' }, + turn: { type: 'number', description: 'Optional caller turn number, logged for attribution.' }, + stats: { + type: 'boolean', + description: + 'Return the volunteered-vs-used precision summary instead of volunteering. ' + + 'APPROXIMATE: "used" = pages.last_retrieved_at > volunteered_at.', + }, + days: { type: 'number', description: 'Stats window in days (default 30; stats mode only).' }, + }, + handler: async (ctx, p) => { + const { parseWindow, volunteerContext, volunteerUsageStats } = await import('./context/volunteer.ts'); + const scope = sourceScopeOpts(ctx); + const sourceIds = scope.sourceIds ?? (scope.sourceId ? [scope.sourceId] : ['default']); + + if (p.stats === true) { + return volunteerUsageStats(ctx.engine, sourceIds, typeof p.days === 'number' ? p.days : undefined); + } + + if (typeof p.window !== 'string' || !p.window.trim()) { + throw new OperationError( + 'invalid_params', + 'window is required unless stats: true', + 'Pass the recent turns as a string (CLI: pipe them on stdin), or use --stats.', + ); + } + const turns = parseWindow(p.window); + const pages = await volunteerContext(ctx.engine, turns, { + sourceIds, + priorContext: typeof p.prior_context === 'string' ? p.prior_context : undefined, + maxPages: typeof p.max_pages === 'number' ? p.max_pages : undefined, + minConfidence: typeof p.min_confidence === 'number' ? p.min_confidence : undefined, + }); + + // Feedback-loop logging: fire-and-forget batched INSERT through the + // volunteer-events sink (drained at exit). Never fails the op. + if (pages.length) { + try { + const { logVolunteerEventsFireAndForget, volunteerEventRowsFrom } = await import('./context/volunteer-events.ts'); + // Trust-boundary clamps (remote MCP callers): cap session_id length so + // a read-scoped token can't bank unbounded TEXT per request, and only + // log integer turns — a non-integer would throw inside the single + // multi-row INSERT and silently drop the whole batch. + const sessionId = typeof p.session_id === 'string' ? p.session_id.slice(0, 256) : null; + const turn = + typeof p.turn === 'number' && Number.isInteger(p.turn) && Math.abs(p.turn) <= 2_147_483_647 + ? p.turn + : null; + logVolunteerEventsFireAndForget( + ctx.engine, + volunteerEventRowsFrom(pages, { channel: 'op', session_id: sessionId, turn }), + ); + } catch { + /* telemetry only */ + } + } + return { pages, count: pages.length, window_turns: turns.length }; + }, + cliHints: { name: 'volunteer-context', stdin: 'window' }, +}; + const find_anomalies: Operation = { name: 'find_anomalies', description: FIND_ANOMALIES_DESCRIPTION, @@ -4856,6 +4949,8 @@ export const operations: Operation[] = [ whoami, sources_add, sources_list, sources_remove, sources_status, // v0.29: Salience + anomalies + recent transcripts get_recent_salience, find_anomalies, get_recent_transcripts, + // v0.43 (#2095): push-based context + volunteer_context, // v0.31: hot memory (facts table) extract_facts, recall, forget_fact, // v0.32.6: contradiction probe MCP surface (M3) diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 8808efec5..6fdb57d07 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -264,6 +264,12 @@ export class PGLiteEngine implements BrainEngine { } } + // NOTE (#2084): PGLite's Emscripten runtime writes the WASM backend's + // proc_exit status into `process.exitCode` (initdb here at create-time, + // the postmaster at close-time), and the writes land asynchronously — + // a snapshot/restore around these awaits does NOT contain them. That is + // why the CLI's exit paths read gbrain's own verdict + // (cli-force-exit.ts currentExitCode), never ambient process.exitCode. try { this._db = await preservingProcessExitCode(() => PGlite.create({ diff --git a/src/core/pglite-schema.ts b/src/core/pglite-schema.ts index 44c3cc418..b0d7da947 100644 --- a/src/core/pglite-schema.ts +++ b/src/core/pglite-schema.ts @@ -945,6 +945,27 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths ( FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE ); +-- #2095 push-based context: feedback-loop log of volunteered pages. +-- Mirrors migration v117 + src/schema.sql. "Used" derives from +-- pages.last_retrieved_at > volunteered_at; 90-day prune in the dream +-- cycle's purge phase. +CREATE TABLE IF NOT EXISTS context_volunteer_events ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + slug TEXT NOT NULL, + confidence DOUBLE PRECISION NOT NULL, + match_arm TEXT NOT NULL, + rationale TEXT NOT NULL DEFAULT '', + channel TEXT NOT NULL DEFAULT 'op', + session_id TEXT, + turn INTEGER, + volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx + ON context_volunteer_events (source_id, volunteered_at DESC); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx + ON context_volunteer_events (source_id, slug); + -- ============================================================ -- migration_impact_log (v0.41.18.0 — gbrain onboard wave) -- ============================================================ diff --git a/src/core/schema-embedded.ts b/src/core/schema-embedded.ts index 2e30b49f1..e0f313dc8 100644 --- a/src/core/schema-embedded.ts +++ b/src/core/schema-embedded.ts @@ -711,6 +711,28 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths ( FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE ); +-- #2095 push-based context: feedback-loop log of volunteered pages +-- (volunteer_context op / retrieval-reflex pointers / gbrain watch). +-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is +-- a deterministic template string, never raw conversation text. 90-day prune +-- in the dream cycle's purge phase. Mirrors migration v117. +CREATE TABLE IF NOT EXISTS context_volunteer_events ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + slug TEXT NOT NULL, + confidence DOUBLE PRECISION NOT NULL, + match_arm TEXT NOT NULL, + rationale TEXT NOT NULL DEFAULT '', + channel TEXT NOT NULL DEFAULT 'op', + session_id TEXT, + turn INTEGER, + volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx + ON context_volunteer_events (source_id, volunteered_at DESC); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx + ON context_volunteer_events (source_id, slug); + -- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676) -- because its \`job_id BIGINT REFERENCES minion_jobs(id)\` FK requires -- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix. diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 9ea0a7470..ae5ab4c3f 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -13,7 +13,7 @@ import { startResolveIpcServer, cleanupStaleSocket, } from '../core/context/resolve-ipc.ts'; -import { resolveEntitiesToPointers } from '../core/context/retrieval-reflex.ts'; +import { resolveEntitiesToPointers, logDeliveredReflexPointers } from '../core/context/retrieval-reflex.ts'; export async function startMcpServer(engine: BrainEngine) { const server = new Server( @@ -68,13 +68,24 @@ export async function startMcpServer(engine: BrainEngine) { if (cfg?.engine === 'pglite' && cfg.database_path) { resolveSocket = resolveSocketPath(cfg.database_path); const defaultSource = process.env.GBRAIN_SOURCE || 'default'; - resolveServer = await startResolveIpcServer(resolveSocket, (req) => - resolveEntitiesToPointers( - engine, - req.sourceId || defaultSource, - req.candidates ?? [], - { priorContextText: req.priorContextText, maxPointers: req.maxPointers }, - ), + resolveServer = await startResolveIpcServer( + resolveSocket, + (req) => + resolveEntitiesToPointers( + engine, + req.sourceId || defaultSource, + req.candidates ?? [], + { + priorContextText: req.priorContextText, + maxPointers: req.maxPointers, + suppression: req.suppression, + }, + ), + // The IPC resolve path IS the ambient reflex channel. Logging happens + // at DELIVERY (post-write), not inside the resolver — a block the + // client's 250ms budget abandoned was never injected, and counting it + // would corrupt the volunteered-vs-used precision stats (red-team). + (block) => logDeliveredReflexPointers(engine, block.pointers), ); } } catch { diff --git a/src/schema.sql b/src/schema.sql index 3a5f32ef7..d05b45117 100644 --- a/src/schema.sql +++ b/src/schema.sql @@ -707,6 +707,28 @@ CREATE TABLE IF NOT EXISTS op_checkpoint_paths ( FOREIGN KEY (op, fingerprint) REFERENCES op_checkpoints (op, fingerprint) ON DELETE CASCADE ); +-- #2095 push-based context: feedback-loop log of volunteered pages +-- (volunteer_context op / retrieval-reflex pointers / gbrain watch). +-- "Used" derives from pages.last_retrieved_at > volunteered_at; rationale is +-- a deterministic template string, never raw conversation text. 90-day prune +-- in the dream cycle's purge phase. Mirrors migration v117. +CREATE TABLE IF NOT EXISTS context_volunteer_events ( + id BIGSERIAL PRIMARY KEY, + source_id TEXT NOT NULL, + slug TEXT NOT NULL, + confidence DOUBLE PRECISION NOT NULL, + match_arm TEXT NOT NULL, + rationale TEXT NOT NULL DEFAULT '', + channel TEXT NOT NULL DEFAULT 'op', + session_id TEXT, + turn INTEGER, + volunteered_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_time_idx + ON context_volunteer_events (source_id, volunteered_at DESC); +CREATE INDEX IF NOT EXISTS context_volunteer_events_src_slug_idx + ON context_volunteer_events (source_id, slug); + -- migration_impact_log moved BELOW minion_jobs (was here, lines 645-676) -- because its `job_id BIGINT REFERENCES minion_jobs(id)` FK requires -- minion_jobs to exist FIRST during SCHEMA_SQL replay. v0.41.25.0 fix. diff --git a/test/cli-exit-verdict-pin.test.ts b/test/cli-exit-verdict-pin.test.ts new file mode 100644 index 000000000..75322ca47 --- /dev/null +++ b/test/cli-exit-verdict-pin.test.ts @@ -0,0 +1,34 @@ +/** + * #2084 class pin — every exit-code write in src/ routes through + * setCliExitVerdict. + * + * A RAW `process.exitCode = N` is silently ZEROED by the deliberate + * flush-exit: currentExitCode() reads only gbrain's owned verdict (the + * PGLite-Emscripten-pollution defense), so a command that bypasses the + * setter reports success on failure. Caught live twice in one day: doctor's + * FAIL path exited 0 after a merge introduced a raw write. Runtime variants + * are pinned in test/cli-finish-teardown.test.ts; this is the structural + * guard that catches the NEXT raw write at review time. + */ + +import { describe, test, expect } from 'bun:test'; +import { execSync } from 'child_process'; + +describe('exit-verdict ownership — no raw process.exitCode assignments', () => { + test('every exit-code write in src/ routes through setCliExitVerdict', () => { + // Two legitimate writers are exempt: + // - cli-force-exit.ts: setCliExitVerdict's own mirror-write. + // - pglite-engine.ts preservingProcessExitCode: #2141's containment + // RESTORE around PGlite.create() — it keeps the GLOBAL tidy for + // external readers and is explicitly not a verdict write (the owned + // channel never reads process.exitCode). + // Whitespace/operator-tolerant: catches `process.exitCode=1`, + // `process.exitCode ??= 1`, `process.exitCode ||= 1`, and the bracket + // form — every shape is equally zeroed by the owned-verdict read. + const hits = execSync( + String.raw`grep -rnE "process(\.|\[')exitCode('\])?[[:space:]]*([?|&]{2})?=[^=]" src --include='*.ts' | grep -v "core/cli-force-exit.ts" | grep -v "core/pglite-engine.ts" || true`, + { encoding: 'utf-8', cwd: new URL('..', import.meta.url).pathname }, + ).trim(); + expect(hits).toBe(''); + }); +}); diff --git a/test/cli-format-volunteer.test.ts b/test/cli-format-volunteer.test.ts new file mode 100644 index 000000000..3e7f24226 --- /dev/null +++ b/test/cli-format-volunteer.test.ts @@ -0,0 +1,67 @@ +/** + * v0.43 (#2095) — formatResult's volunteer_context human rendering + * (ship coverage G3): pointer lines with confidence/arm/rationale, + * the empty-result message, and the approximate stats summary. + */ +import { describe, test, expect } from 'bun:test'; +import { formatResult } from '../src/cli.ts'; + +describe('formatResult — volunteer_context', () => { + test('renders pointer lines with confidence, arm, rationale, and synopsis', () => { + const out = formatResult('volunteer_context', { + pages: [ + { + slug: 'people/alice-example', + source_id: 'default', + display: 'Alice Example', + confidence: 0.85, + arm: 'title', + rationale: 'exact title match "Alice Example"; mentioned in the newest turn', + synopsis: 'Alice is an early founder.', + }, + ], + count: 1, + window_turns: 3, + }); + expect(out).toContain('Alice Example → people/alice-example (0.85, title)'); + expect(out).toContain('exact title match'); + expect(out).toContain('Alice is an early founder.'); + }); + + test('empty result explains the confidence gate', () => { + const out = formatResult('volunteer_context', { pages: [], count: 0, window_turns: 1 }); + expect(out).toContain('Nothing volunteered'); + expect(out).toContain('confidence gate'); + }); + + test('stats mode renders totals, per-arm precision, and the approximate note', () => { + const out = formatResult('volunteer_context', { + days: 30, + approximate: true, + note: 'approximate: "used" = pages.last_retrieved_at > volunteered_at.', + total_volunteered: 4, + total_used: 3, + by_arm: [ + { match_arm: 'alias', channel: 'reflex', volunteered: 2, used: 2, precision: 1 }, + { match_arm: 'title', channel: 'op', volunteered: 2, used: 1, precision: 0.5 }, + ], + }); + expect(out).toContain('last 30 day(s)'); + expect(out).toContain('approximate'); + expect(out).toContain('total: 4 volunteered, 3 used'); + expect(out).toContain('alias/reflex: 2/2 used (precision 1)'); + expect(out).toContain('title/op: 1/2 used (precision 0.5)'); + }); + + test('stats mode with zero events prints the empty-window line', () => { + const out = formatResult('volunteer_context', { + days: 7, + approximate: true, + note: 'approximate', + total_volunteered: 0, + total_used: 0, + by_arm: [], + }); + expect(out).toContain('no volunteer events in the window'); + }); +}); diff --git a/test/cli-pipe-truncation.test.ts b/test/cli-pipe-truncation.test.ts new file mode 100644 index 000000000..bea683ef7 --- /dev/null +++ b/test/cli-pipe-truncation.test.ts @@ -0,0 +1,47 @@ +/** + * v0.43 (#2084) — real-CLI pipe completeness pin (the incident #1959 class). + * + * The synthetic flush-mechanism coverage lives in + * test/flush-then-exit-harness.test.ts (4MB late-reader byte-complete pin). + * This file keeps the IMPLEMENTATION-AGNOSTIC check: the actual CLI, run the + * way agents run it (piped stdout), produces complete, parseable, byte-stable + * output and exits deliberately — well under the teardown backstop. + */ + +import { describe, test, expect } from 'bun:test'; +import { spawnSync } from 'child_process'; +import { join, resolve } from 'path'; + +const REPO = resolve(import.meta.dir, '..'); +const CLI = join(REPO, 'src', 'cli.ts'); + +describe('cli pipe completeness — deliberate exit never truncates piped stdout (#2084)', () => { + test('real CLI: --tools-json over a pipe is complete, parseable, byte-stable, and prompt', () => { + const run = () => { + const t0 = Date.now(); + const res = spawnSync('bun', [CLI, '--tools-json'], { + stdio: ['ignore', 'pipe', 'pipe'], + encoding: 'utf-8', + timeout: 60_000, + env: { ...process.env, GBRAIN_SKIP_STARTUP_HOOKS: '1' }, + maxBuffer: 64 * 1024 * 1024, + }); + return { stdout: res.stdout ?? '', stderr: res.stderr ?? '', status: res.status, ms: Date.now() - t0 }; + }; + const first = run(); + expect(first.status).toBe(0); + expect(Buffer.byteLength(first.stdout, 'utf-8')).toBeGreaterThan(16 * 1024); + // Truncated JSON does not parse — the strongest single-run completeness check. + const parsed = JSON.parse(first.stdout); + expect(Array.isArray(parsed)).toBe(true); + // Deliberate exit, not the teardown backstop. A wall-clock bound is flaky + // on cold CI (bun parse alone runs 10-20s there) — the backstop's banner + // is the truthful signal, same assertion the pgbouncer e2e uses. + expect(first.stderr).not.toContain('force-exiting'); + expect(first.stderr).not.toContain('did not return within'); + + const second = run(); + expect(second.status).toBe(0); + expect(second.stdout).toBe(first.stdout); + }, 180_000); +}); diff --git a/test/cli-should-force-exit.test.ts b/test/cli-should-force-exit.test.ts index 9831c7325..1cc5f8a48 100644 --- a/test/cli-should-force-exit.test.ts +++ b/test/cli-should-force-exit.test.ts @@ -82,4 +82,20 @@ describe('shouldForceExitAfterMain — daemon survival gate', () => { expect(shouldForceExitAfterMain(['serves'])).toBe(true); expect(shouldForceExitAfterMain(['serve-cluster'])).toBe(true); }); + + test('awaited long-runners exit deliberately when their handler resolves', () => { + // `jobs work`, `jobs watch --follow`, `autopilot`, and `gbrain watch` + // (#2095) all BLOCK inside their awaited handler until done — when + // main() resolves for them, the work is over and the deliberate exit is + // correct (v0.43 #2084 contract). Only commands that RETURN from main() + // while the event loop carries the daemon (`serve`) belong in + // DAEMON_COMMANDS — `watch` blocks in its stdin iteration, so piped EOF + // must flow through the flush-exit instead of hanging on lingering + // sockets. + expect(shouldForceExitAfterMain(['jobs', 'work'])).toBe(true); + expect(shouldForceExitAfterMain(['jobs', 'watch', '--follow'])).toBe(true); + expect(shouldForceExitAfterMain(['autopilot'])).toBe(true); + expect(shouldForceExitAfterMain(['watch'])).toBe(true); + expect(shouldForceExitAfterMain(['watch', '--json'])).toBe(true); + }); }); diff --git a/test/context/resolve-ipc.test.ts b/test/context/resolve-ipc.test.ts index c9e83f8f0..a9687b88d 100644 --- a/test/context/resolve-ipc.test.ts +++ b/test/context/resolve-ipc.test.ts @@ -26,7 +26,10 @@ describe('resolve IPC', () => { test('round-trip: client gets the pointer block the server returns', async () => { const dir = tmpDir(); const sock = resolveSocketPath(dir); - const block: PointerBlock = { pointers: [{ display: 'Alice', slug: 'people/alice', synopsis: 'x' }], text: 'BLOCK' }; + const block: PointerBlock = { + pointers: [{ display: 'Alice', slug: 'people/alice', source_id: 'default', synopsis: 'x', arm: 'alias', confidence: 0.9 }], + text: 'BLOCK', + }; const server = await startResolveIpcServer(sock, async (req) => { expect(req.candidates[0].query).toBe('Alice'); return block; diff --git a/test/doctor-federation-health.test.ts b/test/doctor-federation-health.test.ts index 091ef92f9..71a808fd8 100644 --- a/test/doctor-federation-health.test.ts +++ b/test/doctor-federation-health.test.ts @@ -8,11 +8,24 @@ */ import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { configureGateway } from '../src/core/ai/gateway.ts'; import { checkFederationHealth } from '../src/commands/doctor.ts'; let engine: PGLiteEngine; beforeAll(async () => { + // Pin the legacy OpenAI/1536 embedding shape BEFORE initSchema builds the + // content_chunks vector column. beforeAll runs before the legacy-embedding + // preload's restoring beforeEach fires, so the gateway here is whatever the + // PRIOR file in this shard left it — if that file configured a 1280-d model + // and didn't reset, initSchema would build a vector(1280) column and the + // 1536-d coverage fixture below would fail CheckExpectedDim. Establish the + // dimension this file's fixtures assume rather than inheriting it. + configureGateway({ + embedding_model: 'openai:text-embedding-3-large', + embedding_dimensions: 1536, + env: { ...process.env }, + }); engine = new PGLiteEngine(); await engine.connect({}); await engine.initSchema(); diff --git a/test/e2e/helpers.ts b/test/e2e/helpers.ts index 683a83dd3..56cec2ff5 100644 --- a/test/e2e/helpers.ts +++ b/test/e2e/helpers.ts @@ -49,6 +49,9 @@ const ALL_TABLES = [ 'page_versions', 'ingest_log', 'files', + // v0.43 (#2095): volunteered-context feedback log — no FK to pages (slug + // join), but stale rows poison stats/count assertions across runs. + 'context_volunteer_events', 'pages', // last because of foreign keys 'config', 'minion_attachments', diff --git a/test/e2e/pgbouncer-teardown.test.ts b/test/e2e/pgbouncer-teardown.test.ts new file mode 100644 index 000000000..29b3feb9e --- /dev/null +++ b/test/e2e/pgbouncer-teardown.test.ts @@ -0,0 +1,148 @@ +/** + * v0.43 (#2084 / eng-review TD1) — PgBouncer transaction-mode teardown E2E. + * + * Three consecutive waves (#1972 → #2015 → #2084) fixed pooler-teardown bugs + * that were verified only against one production deployment, because CI had + * no transaction-mode pooler. This file pins the bug CLASS, not exact + * timings: a CLI op against a txn-mode pooled URL must + * + * 1. exit zero with intact stdout, and + * 2. NOT ride the 10s hard-deadline backstop (the + * "[cli] engine.disconnect() did not return within 10000ms" banner is + * the smoking gun — pre-#2084 it printed on 100% of query-shaped ops). + * + * Topology: docker-compose.ci.yml runs `pgbouncer` (transaction mode) in + * front of postgres-1. The test uses a DEDICATED database + * (`gbrain_pgbouncer`) created via the direct URL, so it never races the + * TRUNCATE-based fixtures any shard runs against `gbrain_test`. + * + * Gated by GBRAIN_PGBOUNCER_URL + GBRAIN_PGBOUNCER_DIRECT_URL — skips + * gracefully outside the docker CI gate. Run manually: + * + * GBRAIN_PGBOUNCER_URL=postgresql://postgres:postgres@localhost:6543/gbrain_pgbouncer \ + * GBRAIN_PGBOUNCER_DIRECT_URL=postgresql://postgres:postgres@localhost:5434/gbrain_test \ + * bun test test/e2e/pgbouncer-teardown.test.ts + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'fs'; +import { join, resolve } from 'path'; +import { tmpdir } from 'os'; +import postgres from 'postgres'; +import { PostgresEngine } from '../../src/core/postgres-engine.ts'; + +const POOLED_URL = process.env.GBRAIN_PGBOUNCER_URL; +const DIRECT_ADMIN_URL = process.env.GBRAIN_PGBOUNCER_DIRECT_URL; +const SKIP = !POOLED_URL || !DIRECT_ADMIN_URL; +const describePooled = SKIP ? describe.skip : describe; + +const REPO = resolve(import.meta.dir, '..', '..'); +const TEST_DB = 'gbrain_pgbouncer'; +const SLUG = 'test/pgbouncer-teardown-fixture'; +const MARKER = 'pgbouncer-teardown-marker-content-7c4f'; + +/** Direct URL pointing at the dedicated test database (same server). */ +function directTestDbUrl(): string { + const u = new URL(DIRECT_ADMIN_URL!); + u.pathname = `/${TEST_DB}`; + return u.toString(); +} + +async function runCli( + args: string[], + env: Record, + timeoutMs: number, +): Promise<{ exitCode: number; stdout: string; stderr: string; wallMs: number }> { + const t0 = Date.now(); + const proc = Bun.spawn(['bun', 'run', join(REPO, 'src', 'cli.ts'), ...args], { + cwd: REPO, + env: { ...process.env, ...env, GBRAIN_SKIP_STARTUP_HOOKS: '1' }, + stdout: 'pipe', + stderr: 'pipe', + }); + const killer = setTimeout(() => { + try { proc.kill('SIGKILL'); } catch { /* already dead */ } + }, timeoutMs); + try { + const [stdout, stderr, exitCode] = await Promise.all([ + new Response(proc.stdout).text(), + new Response(proc.stderr).text(), + proc.exited, + ]); + return { exitCode, stdout, stderr, wallMs: Date.now() - t0 }; + } finally { + clearTimeout(killer); + } +} + +describePooled('pgbouncer txn-mode teardown (#2084 / TD1)', () => { + let home: string; + + beforeAll(async () => { + // Dedicated database on the same server, created via the DIRECT url + // (CREATE DATABASE cannot run through a transaction-mode pooler). + const admin = postgres(DIRECT_ADMIN_URL!, { max: 1 }); + try { + await admin.unsafe(`DROP DATABASE IF EXISTS ${TEST_DB} WITH (FORCE)`); + await admin.unsafe(`CREATE DATABASE ${TEST_DB}`); + } finally { + await admin.end({ timeout: 5 }); + } + + // Schema + fixture via the direct connection (DDL stays off the pooler, + // matching the production split-pool discipline). + const eng = new PostgresEngine(); + await eng.connect({ engine: 'postgres', database_url: directTestDbUrl() }); + await eng.initSchema(); + await eng.putPage(SLUG, { + type: 'note', + title: 'PgBouncer teardown fixture', + compiled_truth: MARKER, + timeline: '', + }); + await eng.disconnect(); + + // Brain config pointing the CLI at the POOLED url. + home = mkdtempSync(join(tmpdir(), 'gbrain-pgbouncer-')); + mkdirSync(join(home, '.gbrain'), { recursive: true }); + const pooled = new URL(POOLED_URL!); + pooled.pathname = `/${TEST_DB}`; + writeFileSync( + join(home, '.gbrain', 'config.json'), + JSON.stringify({ engine: 'postgres', database_url: pooled.toString() }) + '\n', + ); + }, 240_000); + + afterAll(() => { + try { rmSync(home, { recursive: true, force: true }); } catch { /* best effort */ } + }); + + test('op against the pooled URL exits clean — output intact, no force-exit banner', async () => { + const env = { HOME: home, GBRAIN_HOME: home }; + const res = await runCli(['get', SLUG], env, 90_000); + + if (res.exitCode !== 0 || /force-exiting/.test(res.stderr)) { + console.error('--- stdout ---\n' + res.stdout); + console.error('--- stderr ---\n' + res.stderr); + } + expect(res.exitCode).toBe(0); + // Output is complete — the #1959 truncation class. + expect(res.stdout).toContain(MARKER); + // The smoking gun: pre-#2084 the hard-deadline banner printed every time + // a query-shaped op ran against a txn-mode pooler. + expect(res.stderr).not.toMatch(/force-exiting/); + expect(res.stderr).not.toMatch(/did not return within/); + // Generous CLASS bound (cold bun parse on CI is 10-20s): the op itself is + // milliseconds; anything that ALSO waited out a 10s teardown backstop + // lands well past this. + expect(res.wallMs).toBeLessThan(60_000); + }, 120_000); + + test('second run (warm schema probe) also exits clean through the pooler', async () => { + const env = { HOME: home, GBRAIN_HOME: home }; + const res = await runCli(['get', SLUG], env, 90_000); + expect(res.exitCode).toBe(0); + expect(res.stdout).toContain(MARKER); + expect(res.stderr).not.toMatch(/force-exiting/); + }, 120_000); +}); diff --git a/test/e2e/volunteer-context-postgres.test.ts b/test/e2e/volunteer-context-postgres.test.ts new file mode 100644 index 000000000..a516fdd53 --- /dev/null +++ b/test/e2e/volunteer-context-postgres.test.ts @@ -0,0 +1,97 @@ +/** + * v0.43 (#2095) — volunteer_context on REAL Postgres (engine parity beyond + * PGLite) + the RLS pin for the v117 table. + * + * The unit suite covers the volunteer core hermetically on PGLite; this file + * proves the same op handler against pgvector Postgres: resolution arms, + * the fire-and-forget event sink landing rows, the stats join, and that + * context_volunteer_events (migration v117) has ROW LEVEL SECURITY enabled (the v35 + * auto_rls_on_create_table event trigger covers migration-created tables — + * this is the assertion that keeps that mechanism honest for new tables). + * + * Gated by DATABASE_URL — skips gracefully without real Postgres. + */ + +import { describe, test, expect, beforeAll, afterAll } from 'bun:test'; +import { hasDatabase, setupDB, teardownDB, getEngine } from './helpers.ts'; +import { operationsByName } from '../../src/core/operations.ts'; +import { + awaitPendingVolunteerEventWrites, + _resetPendingVolunteerEventWritesForTests, +} from '../../src/core/context/volunteer-events.ts'; + +const SKIP = !hasDatabase(); +const describePg = SKIP ? describe.skip : describe; + +function mkCtx(engine: unknown, overrides: Record = {}) { + return { + engine, + config: {} as never, + logger: { info: () => {}, warn: () => {}, error: () => {} } as never, + dryRun: false, + remote: false, + sourceId: 'default', + ...overrides, + } as never; +} + +describePg('volunteer_context on real Postgres (#2095)', () => { + beforeAll(async () => { + await setupDB(); + const engine = getEngine(); + await engine.putPage('people/alice-example', { + type: 'person', + title: 'Alice Example', + compiled_truth: 'Alice is an early founder.', + timeline: '', + }); + }, 240_000); + + afterAll(async () => { + await teardownDB(); + }); + + test('op volunteers, logs through the sink, and stats join works', async () => { + _resetPendingVolunteerEventWritesForTests(); + const engine = getEngine(); + const op = operationsByName.volunteer_context; + + const result = (await op.handler(mkCtx(engine), { + window: 'user: who knows the seed market?\nassistant: Alice Example does.\nuser: ask her', + session_id: 'e2e-pg', + })) as any; + expect(result.count).toBe(1); + expect(result.pages[0].slug).toBe('people/alice-example'); + expect(result.pages[0].arm).toBe('title'); + + const { unfinished } = await awaitPendingVolunteerEventWrites(10_000); + expect(unfinished).toBe(0); + + const rows = await engine.executeRaw<{ slug: string; session_id: string }>( + `SELECT slug, session_id FROM context_volunteer_events WHERE session_id = 'e2e-pg'`, + [], + ); + expect(rows.length).toBe(1); + + // Mark the page as opened after volunteering → stats counts it used. + await engine.executeRaw( + `UPDATE pages SET last_retrieved_at = now() + interval '1 minute' WHERE slug = 'people/alice-example'`, + [], + ); + const stats = (await op.handler(mkCtx(engine), { stats: true })) as any; + expect(stats.approximate).toBe(true); + expect(stats.total_volunteered).toBeGreaterThanOrEqual(1); + expect(stats.total_used).toBeGreaterThanOrEqual(1); + }, 120_000); + + test('RLS is enabled on context_volunteer_events (auto-RLS covers v117)', async () => { + const engine = getEngine(); + const rows = await engine.executeRaw<{ relrowsecurity: boolean }>( + `SELECT relrowsecurity FROM pg_class + WHERE oid = 'public.context_volunteer_events'::regclass`, + [], + ); + expect(rows.length).toBe(1); + expect(rows[0].relrowsecurity).toBe(true); + }, 60_000); +}); diff --git a/test/fix-wave-structural.test.ts b/test/fix-wave-structural.test.ts index e2b668e70..1b734617c 100644 --- a/test/fix-wave-structural.test.ts +++ b/test/fix-wave-structural.test.ts @@ -250,3 +250,20 @@ describe('v0.41.8.0 #1340 — PGLite WASM init classifier', () => { expect(src).toMatch(/buildPgliteInitErrorMessage\(verdict, original\)/); }); }); + +describe('v0.42.43.0 #2095 — volunteer-events sink + cycle purge wiring (structural pins)', () => { + test('volunteer-events registers a background-work drainer (order 4)', () => { + // Deleting this registration would silently drop volunteer events on + // every CLI exit with no behavioral test failing — same pin class as the + // other four sinks above. + const src = readFileSync('src/core/context/volunteer-events.ts', 'utf8'); + expect(src).toMatch(/registerBackgroundWorkDrainer\(\{[\s\S]*?name:\s*'volunteer-events'/); + expect(src).toMatch(/order:\s*4/); + }); + + test("the dream cycle's purge phase invokes purgeStaleVolunteerEvents and reports the count", () => { + const src = readFileSync('src/core/cycle.ts', 'utf8'); + expect(src).toMatch(/purgeStaleVolunteerEvents\(engine\)/); + expect(src).toMatch(/purged_volunteer_events_count/); + }); +}); diff --git a/test/migrate.test.ts b/test/migrate.test.ts index 67fd83248..13067b2c0 100644 --- a/test/migrate.test.ts +++ b/test/migrate.test.ts @@ -2179,3 +2179,67 @@ describe('v112 — pages_links_extracted_at', () => { }); }); + +// v0.43 (#2095): context_volunteer_events — push-based-context feedback log. +describe('v117 — context_volunteer_events_table', () => { + let engine: PGLiteEngine; + beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); + }, 60_000); + afterAll(async () => { if (engine) await engine.disconnect(); }, 60_000); + + test('v117 entry exists, named + idempotent', () => { + const m = MIGRATIONS.find(x => x.version === 117); + expect(m).toBeDefined(); + expect(m!.name).toBe('context_volunteer_events_table'); + expect(m!.idempotent).toBe(true); + }); + + test('LATEST_VERSION is at or above 117', () => { + expect(LATEST_VERSION).toBeGreaterThanOrEqual(117); + }); + + test('table exists after initSchema with the documented columns', async () => { + const rows = await engine.executeRaw<{ column_name: string; is_nullable: string }>( + `SELECT column_name, is_nullable FROM information_schema.columns + WHERE table_name = 'context_volunteer_events' ORDER BY ordinal_position`, [], + ); + const byName = new Map(rows.map(r => [r.column_name, r.is_nullable])); + for (const required of ['source_id', 'slug', 'confidence', 'match_arm', 'rationale', 'channel', 'volunteered_at']) { + expect(byName.get(required)).toBe('NO'); + } + // Caller-supplied attribution columns are nullable by contract (codex D9). + expect(byName.get('session_id')).toBe('YES'); + expect(byName.get('turn')).toBe('YES'); + }); + + test('both source-scoped indexes exist after initSchema', async () => { + const rows = await engine.executeRaw<{ indexname: string }>( + `SELECT indexname FROM pg_indexes WHERE tablename = 'context_volunteer_events'`, [], + ); + const names = rows.map(r => r.indexname); + expect(names).toContain('context_volunteer_events_src_time_idx'); + expect(names).toContain('context_volunteer_events_src_slug_idx'); + }); + + test('insert + 90-day purge round-trip (purgeStaleVolunteerEvents)', async () => { + const { insertVolunteerEvents, purgeStaleVolunteerEvents } = await import('../src/core/context/volunteer-events.ts'); + await insertVolunteerEvents(engine, [ + { source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'alias match', channel: 'op' }, + { source_id: 'default', slug: 'projects/acme-example', confidence: 0.8, match_arm: 'title', rationale: 'exact title', channel: 'reflex', session_id: 's1', turn: 3 }, + ]); + // Age one row past the TTL, leave the other fresh. + await engine.executeRaw( + `UPDATE context_volunteer_events SET volunteered_at = now() - interval '120 days' + WHERE slug = 'projects/acme-example'`, [], + ); + const purged = await purgeStaleVolunteerEvents(engine, 90); + expect(purged).toBe(1); + const left = await engine.executeRaw<{ slug: string }>( + `SELECT slug FROM context_volunteer_events`, [], + ); + expect(left.map(r => r.slug)).toEqual(['people/alice-example']); + }); +}); diff --git a/test/retrieval-reflex.test.ts b/test/retrieval-reflex.test.ts index 385af152b..f79e0c04c 100644 --- a/test/retrieval-reflex.test.ts +++ b/test/retrieval-reflex.test.ts @@ -182,3 +182,258 @@ describe('context-engine assemble() — Retrieval Reflex integration', () => { }); }); }); + +describe('v0.43 (#2095) — rolling window extraction through assemble()', () => { + const REFLEX_ON = { GBRAIN_RETRIEVAL_REFLEX: 'true' }; + + test('entity named ONLY in a previous assistant turn yields a pointer now', async () => { + await withEnv(REFLEX_ON, async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w1', + resolveEntities: (candidates, opts) => + resolveEntitiesToPointers(engine, 'default', candidates, opts), + }); + // Current turn is a pronoun follow-up; the antecedent was NAMED two + // turns back by the ASSISTANT. Pre-window this never fired. + const res = await ce.assemble({ + sessionId: 'w1', + messages: [ + { role: 'user', content: 'who should I talk to about the seed round?' }, + { role: 'assistant', content: 'Alice Example led a similar round last year.' }, + { role: 'user', content: 'what did she invest in?' }, + ], + }); + expect(res.systemPromptAddition).toContain('people/alice-example'); + }); + }); + + test('window=1 reproduces the legacy current-turn-only behavior', async () => { + await withEnv({ ...REFLEX_ON, GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '1' }, async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w2', + resolveEntities: (candidates, opts) => + resolveEntitiesToPointers(engine, 'default', candidates, opts), + }); + const res = await ce.assemble({ + sessionId: 'w2', + messages: [ + { role: 'assistant', content: 'Alice Example led a similar round last year.' }, + { role: 'user', content: 'what did she invest in?' }, + ], + }); + // Current turn has no extractable entity; window=1 must NOT widen. + expect(res.systemPromptAddition).not.toContain('people/alice-example'); + }); + }); + + test('windowed suppression is slug-only: a prior-turn MENTION does not suppress (codex D7)', async () => { + await withEnv(REFLEX_ON, async () => { + await seed('people/alice-example', 'Alice Example', 'A founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w3', + resolveEntities: (candidates, opts) => + resolveEntitiesToPointers(engine, 'default', candidates, opts), + }); + // "Alice Example" appears in a PRIOR turn (a bare mention — prior + // context contains the TITLE). Under the legacy title rule the pointer + // would be suppressed; slug-only windowing must still fire. + const res = await ce.assemble({ + sessionId: 'w3', + messages: [ + { role: 'user', content: 'I met Alice Example yesterday' }, + { role: 'assistant', content: 'How did the meeting with Alice Example go?' }, + { role: 'user', content: 'she wants to invest — thoughts?' }, + ], + }); + expect(res.systemPromptAddition).toContain('people/alice-example'); + }); + }); + + test('windowed suppression still drops an already-surfaced page (slug in prior context)', async () => { + await withEnv(REFLEX_ON, async () => { + await seed('people/alice-example', 'Alice Example', 'A founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w4', + resolveEntities: (candidates, opts) => + resolveEntitiesToPointers(engine, 'default', candidates, opts), + }); + const res = await ce.assemble({ + sessionId: 'w4', + messages: [ + { role: 'assistant', content: 'Pointer: **Alice Example** → `people/alice-example` (use get_page)' }, + { role: 'user', content: 'tell me more about Alice Example' }, + ], + }); + expect(res.systemPromptAddition).not.toContain('Brain pages mentioned this turn'); + }); + }); + + test('fail-open: a throwing resolver under windowing never breaks the turn', async () => { + await withEnv(REFLEX_ON, async () => { + await seed('people/alice-example', 'Alice Example', 'A founder.'); + const ce = createGBrainContextEngine({ + workspaceDir: '/tmp/rr-test-ws-w5', + resolveEntities: async () => { throw new Error('resolver exploded'); }, + }); + const res = await ce.assemble({ + sessionId: 'w5', + messages: [ + { role: 'assistant', content: 'Alice Example is relevant here.' }, + { role: 'user', content: 'ok tell me about her' }, + ], + }); + expect(res.systemPromptAddition).toContain('Live Context'); + expect(res.systemPromptAddition).not.toContain('Brain pages mentioned this turn'); + }); + }); +}); + +describe('ambient-channel event logging (codex D11 — accept-side logDeliveredReflexPointers)', () => { + test('logDeliveredReflexPointers logs channel=reflex events through the drained sink', async () => { + const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts'); + const { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } = + await import('../src/core/context/volunteer-events.ts'); + _resetPendingVolunteerEventWritesForTests(); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await seed('people/alice-example', 'Alice Example', 'A founder.'); + + const block = await resolveEntitiesToPointers( + engine, + 'default', + extractCandidates('what do you think about Alice Example?'), + {}, + ); + expect(block).not.toBeNull(); + logDeliveredReflexPointers(engine, block!.pointers); + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); + const rows = await engine.executeRaw<{ channel: string; slug: string; match_arm: string }>( + 'SELECT channel, slug, match_arm FROM context_volunteer_events', + [], + ); + expect(rows.length).toBe(1); + expect(rows[0].channel).toBe('reflex'); + expect(rows[0].slug).toBe('people/alice-example'); + expect(rows[0].match_arm).toBe('title'); + }); + + test('the bare resolver logs nothing — delivery is the only logging seam', async () => { + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await seed('people/alice-example', 'Alice Example', 'A founder.'); + const block = await resolveEntitiesToPointers( + engine, + 'default', + extractCandidates('about Alice Example'), + {}, + ); + expect(block).not.toBeNull(); + const { awaitPendingVolunteerEventWrites } = await import('../src/core/context/volunteer-events.ts'); + await awaitPendingVolunteerEventWrites(5_000); + const rows = await engine.executeRaw<{ channel: string }>('SELECT channel FROM context_volunteer_events', []); + expect(rows.length).toBe(0); + }); + + test('logDeliveredReflexPointers with an empty pointer list is a no-op', async () => { + const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts'); + const { awaitPendingVolunteerEventWrites } = await import('../src/core/context/volunteer-events.ts'); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + logDeliveredReflexPointers(engine, []); + await awaitPendingVolunteerEventWrites(5_000); + const rows = await engine.executeRaw<{ channel: string }>('SELECT channel FROM context_volunteer_events', []); + expect(rows.length).toBe(0); + }); +}); + +describe('serve IPC wiring — suppression passthrough + reflex-channel logging (review hardening)', () => { + test('the IPC round-trip honors slug-only suppression and logs channel=reflex', async () => { + const { startResolveIpcServer, resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } = + await import('../src/core/context/resolve-ipc.ts'); + const { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } = + await import('../src/core/context/volunteer-events.ts'); + const { mkdtempSync, rmSync } = await import('fs'); + const { join } = await import('path'); + const { tmpdir } = await import('os'); + + _resetPendingVolunteerEventWritesForTests(); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await seed('people/alice-example', 'Alice Example', 'A founder.'); + + const dir = mkdtempSync(join(tmpdir(), 'rr-ipc-')); + const sock = resolveSocketPath(dir); + // The SAME wiring shape src/mcp/server.ts uses for serve: forwards + // suppression from the request; logging happens at DELIVERY via the + // onDelivered hook (post-write), never inside the resolver. + const { logDeliveredReflexPointers } = await import('../src/core/context/retrieval-reflex.ts'); + const server = await startResolveIpcServer( + sock, + (req) => + resolveEntitiesToPointers(engine, req.sourceId || 'default', req.candidates ?? [], { + priorContextText: req.priorContextText, + maxPointers: req.maxPointers, + suppression: req.suppression, + }), + (block) => logDeliveredReflexPointers(engine, block.pointers), + ); + expect(server).not.toBeNull(); + try { + // slug-only suppression: a TITLE mention in prior context must NOT + // suppress (the windowing contract), and the resolve must log. + const block = await resolveViaIpc(sock, { + candidates: extractCandidates('tell me about Alice Example'), + priorContextText: 'earlier turn merely mentioned Alice Example', + suppression: 'slug-only', + }); + expect(block).not.toBe(IPC_UNAVAILABLE); + expect(block).not.toBeNull(); + expect((block as { pointers: Array<{ slug: string }> }).pointers[0].slug).toBe('people/alice-example'); + + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); + const rows = await engine.executeRaw<{ channel: string }>( + 'SELECT channel FROM context_volunteer_events', [], + ); + expect(rows.length).toBe(1); + expect(rows[0].channel).toBe('reflex'); + } finally { + server!.close(); + rmSync(dir, { recursive: true, force: true }); + } + }); +}); + +describe('windowTurnCount — knob edge semantics', () => { + test('0, negative, NaN, and absent all fall back to the default of 4 (1 = legacy off)', async () => { + const { windowTurnCount, DEFAULT_WINDOW_TURNS } = await import('../src/core/context/reflex.ts'); + expect(DEFAULT_WINDOW_TURNS).toBe(4); + expect(windowTurnCount(null)).toBe(4); + expect(windowTurnCount({ retrieval_reflex_window_turns: 0 } as never)).toBe(4); + expect(windowTurnCount({ retrieval_reflex_window_turns: -3 } as never)).toBe(4); + expect(windowTurnCount({ retrieval_reflex_window_turns: Number.NaN } as never)).toBe(4); + // The documented "off" switch is 1 (legacy single-turn), not 0. + expect(windowTurnCount({ retrieval_reflex_window_turns: 1 } as never)).toBe(1); + expect(windowTurnCount({ retrieval_reflex_window_turns: 6.9 } as never)).toBe(6); + }); + + test('the env escape hatch is honored even when config is null (no config file / DB)', async () => { + const { windowTurnCount } = await import('../src/core/context/reflex.ts'); + // loadConfig() returns null in a config-less environment (clean CI shard, + // no brain) and drops its env→config mapping — windowTurnCount must still + // read the env var directly, or the documented escape hatch is dead and + // the window silently defaults to 4. withEnv() (not raw process.env + // mutation) keeps the linter + isolation guard happy. + await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '1' }, async () => { + expect(windowTurnCount(null)).toBe(1); + }); + await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: '7' }, async () => { + expect(windowTurnCount(null)).toBe(7); + // Env wins over a config value too (env is the higher-precedence plane). + expect(windowTurnCount({ retrieval_reflex_window_turns: 3 } as never)).toBe(7); + }); + await withEnv({ GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS: 'not-a-number' }, async () => { + // Garbage env falls through to config / default, not a crash. + expect(windowTurnCount(null)).toBe(4); + }); + }); +}); diff --git a/test/sync-cost-preview.test.ts b/test/sync-cost-preview.test.ts index 7f492c032..52d4dba6b 100644 --- a/test/sync-cost-preview.test.ts +++ b/test/sync-cost-preview.test.ts @@ -13,7 +13,7 @@ * envelope paths don't depend on DB state. */ -import { describe, test, expect } from 'bun:test'; +import { describe, test, expect, beforeEach } from 'bun:test'; import { EMBEDDING_COST_PER_1K_TOKENS, estimateEmbeddingCostUsd, @@ -21,9 +21,20 @@ import { shouldBlockSync, } from '../src/core/embedding.ts'; import { lookupEmbeddingPrice } from '../src/core/embedding-pricing.ts'; +import { resetGateway } from '../src/core/ai/gateway.ts'; import { estimateTokens } from '../src/core/chunkers/code.ts'; describe('Layer 8 D1 — embedding cost model', () => { + // The estimateEmbeddingCostUsd tests assert the gateway-UNCONFIGURED OpenAI + // fallback ($0.13/Mtok). That fallback only fires when no gateway is + // configured — so guarantee that precondition rather than depending on + // ambient state. Without this, a sibling test file in the same shard that + // configured (and didn't reset) a cheaper model leaks its rate in here and + // these assertions flip (the legacy-embedding preload only restores when the + // gateway slot is EMPTY, so a non-empty foreign config survives). + beforeEach(() => resetGateway()); + + test('EMBEDDING_COST_PER_1K_TOKENS back-compat constant is the OpenAI 3-large rate', () => { // Retained only for back-compat imports. Live cost math now resolves the // CONFIGURED model's rate via embedding-pricing.ts (see model-aware test diff --git a/test/volunteer-context.test.ts b/test/volunteer-context.test.ts new file mode 100644 index 000000000..a972ef4f9 --- /dev/null +++ b/test/volunteer-context.test.ts @@ -0,0 +1,483 @@ +/** + * v0.43 (#2095) — push-based context core: window parsing, multi-turn + * extraction, confidence-gated volunteering, slug-only suppression, privacy, + * and the usage-stats join. Hermetic in-memory PGLite (no file lock), + * modeled on test/retrieval-reflex.test.ts. + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { normalizeAlias } from '../src/core/search/alias-normalize.ts'; +import { + extractCandidatesFromWindow, + MAX_CANDIDATES, + type WindowTurn, +} from '../src/core/context/entity-salience.ts'; +import { resolveEntitiesToPointers } from '../src/core/context/retrieval-reflex.ts'; +import { + parseWindow, + volunteerContext, + volunteerUsageStats, + VOLUNTEER_DEFAULT_MIN_CONFIDENCE, +} from '../src/core/context/volunteer.ts'; +import { insertVolunteerEvents } from '../src/core/context/volunteer-events.ts'; +import { TAKES_FENCE_BEGIN, TAKES_FENCE_END } from '../src/core/takes-fence.ts'; + +let engine: PGLiteEngine; + +async function seed(slug: string, title: string, body: string, source = 'default') { + if (source !== 'default') { + await engine.executeRaw( + `INSERT INTO sources (id, name, local_path) VALUES ($1, $1, $2) + ON CONFLICT (id) DO NOTHING`, + [source, `/tmp/${source}`], + ); + } + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES ($1, $2, 'person', $3, $4, '')`, + [slug, source, title, body], + ); +} + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 60_000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await engine.executeRaw('DELETE FROM page_aliases').catch(() => {}); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await engine.executeRaw('DELETE FROM pages'); +}); + +describe('parseWindow', () => { + test('role prefixes split turns oldest → newest', () => { + const turns = parseWindow('user: ask Alice about the deal\nassistant: noted\nuser: what did she say?'); + expect(turns).toEqual([ + { role: 'user', text: 'ask Alice about the deal' }, + { role: 'assistant', text: 'noted' }, + { role: 'user', text: 'what did she say?' }, + ]); + }); + + test('CRLF input parses identically', () => { + const turns = parseWindow('user: hello\r\nassistant: hi\r\n'); + expect(turns).toEqual([ + { role: 'user', text: 'hello' }, + { role: 'assistant', text: 'hi' }, + ]); + }); + + test('unprefixed text is ONE user turn (echo | volunteer-context just works)', () => { + const turns = parseWindow('met with Alice Example today\nshe asked about acme'); + expect(turns).toEqual([{ role: 'user', text: 'met with Alice Example today\nshe asked about acme' }]); + }); + + test('continuation lines append to the open turn', () => { + const turns = parseWindow('user: first line\nsecond line\nassistant: ok'); + expect(turns[0]).toEqual({ role: 'user', text: 'first line\nsecond line' }); + expect(turns[1]).toEqual({ role: 'assistant', text: 'ok' }); + }); + + test('empty / whitespace input → []', () => { + expect(parseWindow('')).toEqual([]); + expect(parseWindow(' \n \n')).toEqual([]); + }); +}); + +describe('extractCandidatesFromWindow', () => { + test('merges across turns with occurrence + newest-turn metadata', () => { + const turns: WindowTurn[] = [ + { role: 'user', text: 'ask Alice Example about the deal' }, + { role: 'assistant', text: 'Alice Example said she will follow up' }, + { role: 'user', text: 'and ping Bob Sample too' }, + ]; + const cands = extractCandidatesFromWindow(turns); + const alice = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Alice Example')); + const bob = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Bob Sample')); + expect(alice).toBeDefined(); + expect(alice!.occurrences).toBe(2); + expect(alice!.inNewestTurn).toBe(false); + expect(alice!.userMention).toBe(true); + expect(bob).toBeDefined(); + expect(bob!.inNewestTurn).toBe(true); + }); + + test('assistant-only mention is flagged (userMention=false)', () => { + const cands = extractCandidatesFromWindow([ + { role: 'assistant', text: 'You should talk to Charlie Demo about this' }, + { role: 'user', text: 'good idea' }, + ]); + const charlie = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Charlie Demo')); + expect(charlie).toBeDefined(); + expect(charlie!.userMention).toBe(false); + }); + + test('cap holds across a noisy window', () => { + const noisy = Array.from({ length: 30 }, (_, i) => `Entity Number${i} did something.`).join(' '); + const cands = extractCandidatesFromWindow([{ role: 'user', text: noisy }]); + expect(cands.length).toBeLessThanOrEqual(MAX_CANDIDATES); + }); +}); + +describe('volunteerContext', () => { + test('assistant-introduced entity two turns back resolves on a pronoun follow-up', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is an early founder.'); + const turns = parseWindow( + 'user: who should I talk to about the seed round?\n' + + 'assistant: Alice Example led a similar round last year.\n' + + 'user: what did she invest in?', + ); + const pages = await volunteerContext(engine, turns, { sourceIds: ['default'] }); + expect(pages.length).toBe(1); + expect(pages[0].slug).toBe('people/alice-example'); + expect(pages[0].arm).toBe('title'); + expect(pages[0].rationale).toContain('assistant-introduced'); + }); + + test('excludeSlugs skips BEFORE the cap: an excluded entity never starves a fresh page', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.'); + const turns = parseWindow('user: intro Alice Example to Bob Sample'); + const pages = await volunteerContext(engine, turns, { + sourceIds: ['default'], + maxPages: 1, + excludeSlugs: new Set(['people/alice-example']), + }); + // A post-call filter would return [] here (Alice burns the single cap + // slot, then gets filtered). The pre-cap exclusion hands Bob the slot. + expect(pages.length).toBe(1); + expect(pages[0].slug).toBe('people/bob-sample'); + }); + + test('confidence gate drops slug-suffix matches at the default threshold', async () => { + // Page resolvable ONLY via slug-suffix: title differs from the mention. + await seed('projects/widget-co', 'The Widget Company Project', 'A project page.'); + const turns = parseWindow('user: any updates on Widget-Co this week?'); + const gated = await volunteerContext(engine, turns, { sourceIds: ['default'] }); + expect(gated).toEqual([]); + // Lowering min_confidence lets it through with honest provenance. + const loose = await volunteerContext(engine, turns, { sourceIds: ['default'], minConfidence: 0.5 }); + expect(loose.length).toBe(1); + expect(loose[0].arm).toBe('slug-suffix'); + expect(loose[0].confidence).toBeLessThan(VOLUNTEER_DEFAULT_MIN_CONFIDENCE); + }); + + test('alias arm volunteers with boost when mentioned in the newest turn', async () => { + await seed('people/swami-x', 'Swami X', 'A close friend.'); + await engine.setPageAliases('people/swami-x', 'default', [normalizeAlias('Swami')]); + const pages = await volunteerContext( + engine, + parseWindow('user: Spoke with Swami today'), + { sourceIds: ['default'] }, + ); + expect(pages.length).toBe(1); + expect(pages[0].arm).toBe('alias'); + expect(pages[0].confidence).toBeCloseTo(0.95, 5); // 0.9 + newest-turn boost + expect(pages[0].rationale).toContain('alias match'); + }); + + test('suppression is slug-only under windowing: a prior-turn MENTION does not suppress', async () => { + await seed('people/alice-example', 'Alice Example', 'A founder.'); + // Prior context contains the TITLE (a bare mention from an earlier turn) + // but NOT the slug — the page was never actually surfaced. + const pages = await volunteerContext( + engine, + parseWindow('user: ping Alice Example again'), + { sourceIds: ['default'], priorContext: 'earlier the user said: tell Alice Example the news' }, + ); + expect(pages.length).toBe(1); + // A slug in prior context (the page WAS surfaced) does suppress. + const suppressed = await volunteerContext( + engine, + parseWindow('user: ping Alice Example again'), + { sourceIds: ['default'], priorContext: 'pointer: people/alice-example was injected last turn' }, + ); + expect(suppressed).toEqual([]); + }); + + test('privacy: takes-fence content never leaks into the synopsis', async () => { + const body = `${TAKES_FENCE_BEGIN}\nSECRET_HUNCH_DO_NOT_LEAK\n${TAKES_FENCE_END}\nAlice is a founder.`; + await seed('people/alice-example', 'Alice Example', body); + const pages = await volunteerContext(engine, parseWindow('user: about Alice Example'), { sourceIds: ['default'] }); + expect(pages.length).toBe(1); + expect(JSON.stringify(pages)).not.toContain('SECRET_HUNCH_DO_NOT_LEAK'); + }); + + test('multi-source scope: resolves from both sources, never outside the scope', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.', 'default'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.', 'team-brain'); + await seed('people/eve-other', 'Eve Other', 'Out of scope.', 'private-brain'); + const turns = parseWindow('user: intro Alice Example to Bob Sample and Eve Other'); + const pages = await volunteerContext(engine, turns, { sourceIds: ['default', 'team-brain'] }); + const keys = pages.map((p) => `${p.source_id}:${p.slug}`).sort(); + expect(keys).toEqual(['default:people/alice-example', 'team-brain:people/bob-sample']); + }); + + test('zero-candidate fast path: no entities → [] without touching resolution', async () => { + const pages = await volunteerContext(engine, parseWindow('user: ok thanks, sounds good'), { + sourceIds: ['default'], + }); + expect(pages).toEqual([]); + }); + + test('max_pages caps at 5 even when asked for more', async () => { + for (let i = 0; i < 8; i++) { + await seed(`people/person-${i}`, `Person Alpha${i}`, 'A person.'); + } + const text = Array.from({ length: 8 }, (_, i) => `Person Alpha${i}`).join(' and '); + const pages = await volunteerContext(engine, parseWindow(`user: intro ${text}`), { + sourceIds: ['default'], + maxPages: 50, + }); + expect(pages.length).toBeLessThanOrEqual(5); + }); +}); + +describe('volunteerUsageStats', () => { + test('join math: used = last_retrieved_at > volunteered_at, labeled approximate', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.'); + await insertVolunteerEvents(engine, [ + { source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'op' }, + { source_id: 'default', slug: 'people/bob-sample', confidence: 0.8, match_arm: 'title', rationale: 'r', channel: 'op' }, + ]); + // Alice was opened AFTER being volunteered; Bob never was. + await engine.executeRaw( + `UPDATE pages SET last_retrieved_at = now() + interval '1 minute' WHERE slug = 'people/alice-example'`, [], + ); + const stats = await volunteerUsageStats(engine, ['default'], 30); + expect(stats.approximate).toBe(true); + expect(stats.note).toContain('approximate'); + expect(stats.total_volunteered).toBe(2); + expect(stats.total_used).toBe(1); + const alias = stats.by_arm.find((a) => a.match_arm === 'alias')!; + expect(alias.used).toBe(1); + expect(alias.precision).toBe(1); + const title = stats.by_arm.find((a) => a.match_arm === 'title')!; + expect(title.used).toBe(0); + }); + + test('zero events → zeroed stats, not an error', async () => { + const stats = await volunteerUsageStats(engine, ['default'], 7); + expect(stats.total_volunteered).toBe(0); + expect(stats.by_arm).toEqual([]); + }); +}); + +describe('resolveEntitiesToPointers — new provenance surface (back-compat)', () => { + test('pointers carry arm + confidence + source_id', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const block = await resolveEntitiesToPointers( + engine, + 'default', + [{ display: 'Alice Example', query: 'Alice Example' }], + {}, + ); + expect(block).not.toBeNull(); + expect(block!.pointers[0].arm).toBe('title'); + expect(block!.pointers[0].confidence).toBe(0.8); + expect(block!.pointers[0].source_id).toBe('default'); + }); + + test('legacy suppression (slug-and-title) still drops a title mention in prior context', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const block = await resolveEntitiesToPointers( + engine, + 'default', + [{ display: 'Alice Example', query: 'Alice Example' }], + { priorContextText: 'we discussed Alice Example earlier' }, + ); + expect(block).toBeNull(); + }); +}); + +describe('volunteer_context op (contract surface)', () => { + const { operationsByName } = require('../src/core/operations.ts') as typeof import('../src/core/operations.ts'); + const { + awaitPendingVolunteerEventWrites, + _resetPendingVolunteerEventWritesForTests, + } = require('../src/core/context/volunteer-events.ts') as typeof import('../src/core/context/volunteer-events.ts'); + + function mkCtx(overrides: Record = {}) { + return { + engine, + config: {} as never, + logger: { info: () => {}, warn: () => {}, error: () => {} } as never, + dryRun: false, + remote: false, + sourceId: 'default', + ...overrides, + } as never; + } + + test('registered, read-scope, not localOnly, stdin cliHint on window', () => { + const op = operationsByName.volunteer_context; + expect(op).toBeDefined(); + expect(op.scope).toBe('read'); + expect(op.localOnly).toBeFalsy(); + expect(op.cliHints?.name).toBe('volunteer-context'); + expect(op.cliHints?.stdin).toBe('window'); + }); + + test('window required unless stats: true', async () => { + const op = operationsByName.volunteer_context; + await expect(op.handler(mkCtx(), {})).rejects.toThrow(/window is required/); + const stats = (await op.handler(mkCtx(), { stats: true })) as any; + expect(stats.approximate).toBe(true); + }); + + test('volunteers pages and logs events through the drained sink', async () => { + _resetPendingVolunteerEventWritesForTests(); + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const op = operationsByName.volunteer_context; + const result = (await op.handler(mkCtx(), { + window: 'user: ping Alice Example about the deal', + session_id: 's-42', + turn: 7, + })) as any; + expect(result.count).toBe(1); + expect(result.pages[0].slug).toBe('people/alice-example'); + expect(result.window_turns).toBe(1); + // Event row lands via the fire-and-forget sink once drained. + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); + const rows = await engine.executeRaw<{ slug: string; channel: string; session_id: string; turn: number }>( + `SELECT slug, channel, session_id, turn FROM context_volunteer_events`, [], + ); + expect(rows.length).toBe(1); + expect(rows[0].slug).toBe('people/alice-example'); + expect(rows[0].channel).toBe('op'); + expect(rows[0].session_id).toBe('s-42'); + expect(Number(rows[0].turn)).toBe(7); + }); + + test('event-log failure never fails the op (failing engine injected for the INSERT)', async () => { + _resetPendingVolunteerEventWritesForTests(); + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const op = operationsByName.volunteer_context; + // Wrap the engine: reads pass through, INSERTs into the events table throw. + const failingEngine = new Proxy(engine, { + get(target, prop, receiver) { + if (prop === 'executeRaw') { + return (sql: string, params: unknown[]) => { + if (/INSERT INTO context_volunteer_events/.test(sql)) { + return Promise.reject(new Error('telemetry db down')); + } + return target.executeRaw(sql, params); + }; + } + return Reflect.get(target, prop, receiver); + }, + }); + const result = (await op.handler(mkCtx({ engine: failingEngine }), { + window: 'user: ping Alice Example', + })) as any; + expect(result.count).toBe(1); // the volunteer result is unaffected + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); // failed write settled (swallowed), not stuck + }); + + test('federated grant scopes the volunteer (allowedSources)', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.', 'default'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.', 'grant-brain'); + await seed('people/eve-other', 'Eve Other', 'Out of grant.', 'secret-brain'); + const op = operationsByName.volunteer_context; + const result = (await op.handler( + mkCtx({ remote: true, auth: { allowedSources: ['grant-brain'] } }), + { window: 'user: intro Alice Example to Bob Sample and Eve Other' }, + )) as any; + expect(result.pages.map((p: any) => p.slug)).toEqual(['people/bob-sample']); + }); + + test('stats mode is source-scoped and returns the approximate note', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await insertVolunteerEvents(engine, [ + { source_id: 'default', slug: 'people/alice-example', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'watch' }, + ]); + const op = operationsByName.volunteer_context; + const stats = (await op.handler(mkCtx(), { stats: true, days: 7 })) as any; + expect(stats.days).toBe(7); + expect(stats.total_volunteered).toBe(1); + expect(stats.by_arm[0].channel).toBe('watch'); + expect(stats.note).toContain('approximate'); + }); +}); + +describe('knob clamps — untrusted MCP caller inputs (review hardening)', () => { + test('minConfidence outside [0,1] (or NaN) falls back to the 0.7 default gate', async () => { + await seed('projects/widget-co', 'The Widget Company Project', 'A project.'); + const turns = parseWindow('user: updates on Widget-Co?'); + for (const bad of [5, -1, Number.NaN]) { + const pages = await volunteerContext(engine, turns, { sourceIds: ['default'], minConfidence: bad }); + expect(pages).toEqual([]); // slug-suffix (0.6+boost) stays gated at the default 0.7 + } + }); + + test('maxPages 0 / negative / NaN fall back to the 3-page default', async () => { + for (let i = 0; i < 5; i++) await seed(`people/person-${i}`, `Person Alpha${i}`, 'A person.'); + const text = Array.from({ length: 5 }, (_, i) => `Person Alpha${i}`).join(' and '); + for (const bad of [0, -3, Number.NaN]) { + const pages = await volunteerContext(engine, parseWindow(`user: intro ${text}`), { + sourceIds: ['default'], + maxPages: bad, + }); + expect(pages.length).toBeLessThanOrEqual(3); + expect(pages.length).toBeGreaterThan(0); + } + }); + + test('stats days <= 0 / NaN falls back to 30', async () => { + const stats = await volunteerUsageStats(engine, ['default'], -5); + expect(stats.days).toBe(30); + const stats2 = await volunteerUsageStats(engine, ['default'], Number.NaN); + expect(stats2.days).toBe(30); + }); +}); + +describe('window-cap ordering — the newest user mention survives the cap', () => { + test('stale assistant-only chatter is dropped before a newest-turn user entity', async () => { + const { extractCandidatesFromWindow: extract, MAX_CANDIDATES: CAP } = await import('../src/core/context/entity-salience.ts'); + // 14 stale assistant-introduced entities in turn 1, then the user names + // ONE entity in the newest turn. The cap (12) must keep the user's. + const stale = Array.from({ length: 14 }, (_, i) => `Stale Chatter${i}`).join(', '); + const cands = extract([ + { role: 'assistant', text: `consider ${stale}.` }, + { role: 'user', text: 'actually ask Alice Example first' }, + ]); + expect(cands.length).toBeLessThanOrEqual(CAP); + const alice = cands.find((c) => normalizeAlias(c.query) === normalizeAlias('Alice Example')); + expect(alice).toBeDefined(); + // Recency + user-role weighting puts the newest user mention FIRST. + expect(normalizeAlias(cands[0].query)).toBe(normalizeAlias('Alice Example')); + }); +}); + +describe('volunteer-events sink — timeout branch (long-lived process safety)', () => { + test('a hung write reports unfinished and drops the snapshot (no ghost references)', async () => { + const { + logVolunteerEventsFireAndForget, + awaitPendingVolunteerEventWrites, + _resetPendingVolunteerEventWritesForTests, + _peekPendingVolunteerEventWritesForTests, + } = await import('../src/core/context/volunteer-events.ts'); + _resetPendingVolunteerEventWritesForTests(); + const hangingEngine = { + executeRaw: () => new Promise(() => { /* never settles */ }), + } as never; + logVolunteerEventsFireAndForget(hangingEngine, [ + { source_id: 'default', slug: 'people/x', confidence: 0.9, match_arm: 'alias', rationale: 'r', channel: 'watch' }, + ]); + const { unfinished } = await awaitPendingVolunteerEventWrites(20); + expect(unfinished).toBe(1); + // Snapshot dropped so a long-lived `gbrain watch` never accumulates + // references to forever-pending work (the last-retrieved C1 class). + expect(_peekPendingVolunteerEventWritesForTests()).toBe(0); + _resetPendingVolunteerEventWritesForTests(); + }); +}); diff --git a/test/watch-command.test.ts b/test/watch-command.test.ts new file mode 100644 index 000000000..b29801b0b --- /dev/null +++ b/test/watch-command.test.ts @@ -0,0 +1,231 @@ +/** + * v0.43 (#2095) — `gbrain watch` push transport: streaming loop, rolling + * window, session dedupe, --json shape, event logging on channel 'watch', + * and clean EOF return. Hermetic PGLite + injected line/write deps (no + * subprocess, no real stdin). + */ +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { runWatch, WATCH_HELP } from '../src/commands/watch.ts'; +import { awaitPendingVolunteerEventWrites, _resetPendingVolunteerEventWritesForTests } from '../src/core/context/volunteer-events.ts'; + +let engine: PGLiteEngine; + +async function seed(slug: string, title: string, body: string) { + await engine.executeRaw( + `INSERT INTO pages (slug, source_id, type, title, compiled_truth, timeline) + VALUES ($1, 'default', 'person', $2, $3, '')`, + [slug, title, body], + ); +} + +async function* feed(lines: string[]): AsyncGenerator { + for (const l of lines) yield l; +} + +async function watchRun(lines: string[], extraArgs: string[] = []): Promise { + const out: string[] = []; + await runWatch(engine, ['--source', 'default', ...extraArgs], { + lines: feed(lines), + write: (s) => out.push(s), + isTTY: false, + }); + return out; +} + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}, 60_000); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + _resetPendingVolunteerEventWritesForTests(); + await engine.executeRaw('DELETE FROM context_volunteer_events').catch(() => {}); + await engine.executeRaw('DELETE FROM pages'); +}); + +describe('gbrain watch (#2095)', () => { + test('--help prints WATCH_HELP and touches nothing', async () => { + const out = await watchRun([], ['--help']); + expect(out.join('')).toBe(WATCH_HELP); + }); + + test('volunteers per turn and returns cleanly at EOF', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun(['ping Alice Example about the deal']); + const text = out.join(''); + expect(text).toContain('people/alice-example'); + expect(text).toContain('exact title match'); + // runWatch RESOLVED — the EOF → clean-return contract (the entrypoint + // flush-exit + finally drain handle the rest in the real CLI). + }); + + test('rolling window: assistant-introduced entity fires on the pronoun follow-up turn', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun([ + 'user: who should I ask about the round?', + 'assistant: Alice Example led one last year.', + 'user: what did she invest in?', + ]); + expect(out.join('')).toContain('people/alice-example'); + }); + + test('session dedupe: a slug is volunteered at most once per session', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun([ + 'user: ping Alice Example', + 'user: ok', + 'user: Alice Example again please', + ]); + const hits = out.join('').split('people/alice-example').length - 1; + expect(hits).toBe(1); + }); + + test('--json emits one JSONL row per volunteered page with turn attribution', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun(['user: hello there', 'user: ping Alice Example'], ['--json']); + expect(out.length).toBe(1); + const row = JSON.parse(out[0]); + expect(row.slug).toBe('people/alice-example'); + expect(row.turn).toBe(2); + expect(row.arm).toBe('title'); + expect(typeof row.confidence).toBe('number'); + }); + + test('events land on channel watch with session_id + turn (drained sink)', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + await watchRun(['user: ping Alice Example']); + const { unfinished } = await awaitPendingVolunteerEventWrites(5_000); + expect(unfinished).toBe(0); + const rows = await engine.executeRaw<{ channel: string; session_id: string; turn: number }>( + `SELECT channel, session_id, turn FROM context_volunteer_events`, [], + ); + expect(rows.length).toBe(1); + expect(rows[0].channel).toBe('watch'); + expect(rows[0].session_id).toMatch(/^watch-/); + expect(Number(rows[0].turn)).toBe(1); + }); + + test('min-confidence flag gates exactly like the op', async () => { + await seed('projects/widget-co', 'The Widget Company Project', 'A project.'); + const gated = await watchRun(['user: updates on Widget-Co?']); + expect(gated.join('')).toBe(''); + const loose = await watchRun(['user: updates on Widget-Co?'], ['--min-confidence', '0.5']); + expect(loose.join('')).toContain('projects/widget-co'); + }); + + test('blank lines and CRLF are tolerated; no turn, no volunteer', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun(['', ' ', 'user: ping Alice Example\r']); + expect(out.join('')).toContain('people/alice-example'); + }); +}); + +describe('gbrain watch — window + cap flags (ship coverage G4)', () => { + test('--window-turns 1: fires on the mention turn itself; the pronoun follow-up adds nothing', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + const out = await watchRun( + [ + 'assistant: Alice Example led one last year.', + 'user: what did she invest in?', + ], + ['--window-turns', '1', '--json'], + ); + // Watch volunteers per turn: the entity fires at turn 1 (its mention + // turn). With window=1 the follow-up turn extracts nothing, so exactly + // one row exists and it carries turn 1 attribution. + expect(out.length).toBe(1); + expect(JSON.parse(out[0]).turn).toBe(1); + }); + + test('--max-pages 1 caps a multi-entity turn to one volunteered page', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.'); + const out = await watchRun( + ['user: intro Alice Example to Bob Sample'], + ['--max-pages', '1', '--json'], + ); + expect(out.length).toBe(1); + }); +}); + +describe('gbrain watch — red-team hardening', () => { + test('starvation guard: an already-pushed slug never burns the --max-pages cap slot', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + await seed('people/bob-sample', 'Bob Sample', 'Engineer.'); + const out = await watchRun( + [ + 'user: ping Alice Example about the round', + 'user: also intro Alice Example to Bob Sample', + ], + ['--max-pages', '1', '--json'], + ); + // Turn 1: Alice takes the single cap slot. Turn 2: Alice is excluded + // BEFORE the cap (not filtered after), so Bob gets the slot instead of + // the turn volunteering nothing forever. + expect(out.length).toBe(2); + expect(JSON.parse(out[0]).slug).toBe('people/alice-example'); + expect(JSON.parse(out[1]).slug).toBe('people/bob-sample'); + }); + + test('--window-turns clamps: 0 and negatives behave as window 1', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const out = await watchRun( + [ + 'assistant: Alice Example led one last year.', + 'user: what did she invest in?', + ], + ['--window-turns', '0', '--json'], + ); + // Clamped to 1: fires on the mention turn only — identical to the + // --window-turns 1 contract above, not a crash or an unbounded window. + expect(out.length).toBe(1); + expect(JSON.parse(out[0]).turn).toBe(1); + }); + + test('--window-turns absurdly large still runs (hard cap, no unbounded re-scan)', async () => { + await seed('people/alice-example', 'Alice Example', 'Founder.'); + const filler = Array.from({ length: 70 }, (_, i) => `user: filler line ${i}`); + const out = await watchRun( + [...filler, 'user: ping Alice Example'], + ['--window-turns', '999999', '--json'], + ); + expect(out.length).toBe(1); + expect(JSON.parse(out[0]).slug).toBe('people/alice-example'); + }); +}); + +describe('gbrain watch — per-turn fail-open (review hardening)', () => { + test('a transient DB error on one turn never kills the stream', async () => { + await seed('people/alice-example', 'Alice Example', 'Alice is a founder.'); + // Fail the FIRST resolver query against pages (turn 1's resolution); + // everything else — incl. resolveSourceId's pre-loop check — passes. + let pagesQueries = 0; + const flaky = new Proxy(engine, { + get(target, prop, receiver) { + if (prop === 'executeRaw') { + return (sql: string, params: unknown[]) => { + if (/FROM pages/.test(sql) && ++pagesQueries === 1) { + return Promise.reject(new Error('transient db hiccup')); + } + return target.executeRaw(sql, params); + }; + } + return Reflect.get(target, prop, receiver); + }, + }); + const out: string[] = []; + await runWatch(flaky as never, ['--source', 'default'], { + lines: feed(['user: ping Alice Example', 'user: ping Alice Example please']), + write: (s) => out.push(s), + isTTY: false, + }); + // Turn 1 failed open; turn 2 volunteered. The stream survived. + expect(out.join('')).toContain('people/alice-example'); + }); +}); diff --git a/test/watch-sigint.serial.test.ts b/test/watch-sigint.serial.test.ts new file mode 100644 index 000000000..62edfe713 --- /dev/null +++ b/test/watch-sigint.serial.test.ts @@ -0,0 +1,75 @@ +/** + * v0.43 (#2095) — `gbrain watch` SIGINT lifecycle. SERIAL: spawns a real CLI + * subprocess with a tmpdir brain (the parallel unit shards flake on + * concurrent subprocess spawns — same isolation rationale as + * apply-migrations-pglite-spawn.serial.test.ts). + */ +import { describe, test, expect } from 'bun:test'; + +describe('gbrain watch — SIGINT lifecycle (real subprocess)', () => { + test('SIGINT mid-stream closes the stream and exits cleanly (drain path, exit 0)', async () => { + const { mkdtempSync, mkdirSync, writeFileSync, rmSync } = await import('fs'); + const { join, resolve } = await import('path'); + const { tmpdir } = await import('os'); + const REPO = resolve(import.meta.dir, '..'); + const home = mkdtempSync(join(tmpdir(), 'gbrain-watch-sigint-')); + try { + mkdirSync(join(home, '.gbrain'), { recursive: true }); + writeFileSync( + join(home, '.gbrain', 'config.json'), + JSON.stringify({ + engine: 'pglite', + database_path: join(home, '.gbrain', 'brain.pglite'), + embedding_dimensions: 1536, + }) + '\n', + ); + // Piped stdin that NEVER reaches EOF — only SIGINT can end the stream. + const proc = Bun.spawn(['bun', 'run', join(REPO, 'src', 'cli.ts'), 'watch'], { + cwd: REPO, + env: { ...process.env, HOME: home, GBRAIN_HOME: home, GBRAIN_SKIP_STARTUP_HOOKS: '1' }, + stdin: 'pipe', + stdout: 'pipe', + stderr: 'pipe', + }); + proc.stdin.write('user: nothing relevant here\n'); + await proc.stdin.flush(); + // Readiness probe: watch prints "[watch] session ready" on stderr + // once engine + source resolution are done and the stdin loop is live. + // A fixed sleep raced cold PGLite init (other tests budget 60s for it) + // — SIGINT before the handler registers means default-disposition kill. + const stderrChunks: string[] = []; + const reader = (proc.stderr as ReadableStream).getReader(); + const decoder = new TextDecoder(); + const deadline = Date.now() + 90_000; + let ready = false; + while (Date.now() < deadline) { + const { value, done } = await reader.read(); + if (done) break; + stderrChunks.push(decoder.decode(value, { stream: true })); + if (stderrChunks.join('').includes('ready')) { ready = true; break; } + } + expect(ready).toBe(true); + // Brief settle so the first turn's volunteerContext round-trip is in + // flight or done, then interrupt mid-stream. + await new Promise((r) => setTimeout(r, 500)); + proc.kill('SIGINT'); + const killer = setTimeout(() => { + try { proc.kill('SIGKILL'); } catch { /* already dead */ } + }, 30_000); + const exitCode = await proc.exited; + clearTimeout(killer); + // Drain the rest of stderr for the banner assertion. + while (true) { + const { value, done } = await reader.read(); + if (done) break; + stderrChunks.push(decoder.decode(value, { stream: true })); + } + const stderr = stderrChunks.join(''); + // Clean drain-then-exit: no force-exit banner, no SIGKILL (137), exit 0. + expect(stderr).not.toContain('force-exiting'); + expect(exitCode).toBe(0); + } finally { + rmSync(home, { recursive: true, force: true }); + } + }, 120_000); +});