docs: KEY_FILES + push-context guide + TODOS for the #2084/#2095 wave

- docs/architecture/KEY_FILES.md (current-state): context entries gain the
  window extractor, arm provenance/confidence, suppression modes, volunteer
  + volunteer-events modules; background-work entry now lists FIVE sinks and
  the drainThenDisconnect owner-disconnect contract; new entries for
  src/core/cli-force-exit.ts (the exit contract) and src/commands/watch.ts.
- docs/guides/push-context.md (new): the three channels (reflex/op/watch),
  the confidence model, CLI usage, config keys, and the approximate-stats
  caveat. Linked from CLAUDE.md's reference map.
- CLAUDE.md: ops line mentions volunteer_context + the guide link;
  bun run build:llms regenerated in the same commit (freshness test green).
- TODOS.md: #2095 deferrals filed (SSE push channel, policy skill + doctor
  check, structured messages[] param); the #1981 entity-detection TODO
  narrowed (window extraction covered assistant-introduced entities +
  named-antecedent follow-ups); the drain-hoist P3 marked DONE by the
  #2084 wave.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-06-11 22:27:38 -07:00
co-authored by Claude Fable 5
parent f77740599e
commit 6f9d8d4241
5 changed files with 111 additions and 21 deletions
+2 -1
View File
@@ -38,7 +38,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.43 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -97,6 +97,7 @@ detail on demand.)
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
| search modes / cost knobs | `docs/guides/search-modes.md` |
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |
+37 -16
View File
@@ -1,16 +1,42 @@
# TODOS
## gbrain#2095 push-based context follow-ups (v0.43+)
Filed from the #2095 wave (volunteer_context op + reflex window + `gbrain watch`).
Deliberately scoped OUT of v1 per the eng-review scope decision (success criteria
are the bar). Plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-cheerful-elephant.md`.
- [ ] **P3 — SSE/HTTP push channel via serve-http.** The op + `gbrain watch` cover
pull-per-turn and stdin streaming; a serve-http SSE feed would push volunteered
pages to remote agents without a local CLI. **Why:** thin-client/remote-MCP
deployments get push too. **Cons:** async plumbing + auth scoping; no consumer
wired today. **Where:** `src/commands/serve-http.ts` + `src/core/context/volunteer.ts`.
**Blocked by:** a real consumer (revisit when one exists).
- [ ] **P3 — policy skill + doctor check for push-context.** The ambient reflex
needed doctor visibility because silent failure was invisible; volunteer is
invoked-on-demand so v1 skipped it. If `volunteer-context --stats` adoption shows
agents not discovering the surface, ship a `push-context` recipe (mirror
`recipes/retrieval-reflex/`) + a doctor check reading the events table.
**Where:** `recipes/`, `src/commands/doctor.ts`.
- [ ] **P3 — structured `messages[]` param for volunteer_context.** v1 takes a
string window (`user:`/`assistant:` prefixes) to avoid a dual-shape contract.
If MCP callers accumulate parsing bugs, add a structured array param beside it.
**Where:** `src/core/operations.ts:volunteer_context` + `src/core/context/volunteer.ts:parseWindow`.
## gbrain#1981 Retrieval Reflex follow-ups (v0.43+)
Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extractor
is deterministic + precision-biased. See plan + GSTACK REVIEW REPORT at
`~/.claude/plans/system-instruction-you-are-working-wild-yeti.md`.
- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The v1 extractor
(`src/core/context/entity-salience.ts`) misses lowercase names, many non-Latin
scripts, pronoun follow-ups ("what about her?"), and assistant-introduced entities.
These need conversation state or an LLM pass. **Why:** higher recall on the read
side. **Where:** `entity-salience.ts` + the orchestrator's `priorContextText`.
- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The extractor
(`src/core/context/entity-salience.ts`) misses lowercase names and many non-Latin
scripts; these need an LLM pass or script-aware heuristics. **Why:** higher recall
on the read side. **Where:** `entity-salience.ts`. *(Partially done by the #2095
wave: `extractCandidatesFromWindow` now covers assistant-introduced entities and
pronoun follow-ups whose antecedent was NAMED in the rolling window; true pronoun
coreference for never-named antecedents remains with the LLM-pass idea.)*
- [ ] **P3 — recall knob: optional fuzzy/prefix-expansion resolution.** The resolver
(`src/core/context/retrieval-reflex.ts`) is exact-only (alias + title + slug-suffix)
for precision. Revisit adding `resolveEntitySlug`'s trgm-fuzzy / prefix-expansion
@@ -213,17 +239,12 @@ GSTACK REVIEW REPORT at
can't desync per-engine, bounded against CLI-hang by a top-level forced
cleanup. Do this BEFORE introducing any concurrent module-engine connect path.
- [ ] **P3 — `dream` + CLI_ONLY fall-through paths don't drain the facts /
last-retrieved queues before the owner disconnect.** The op-dispatch path
(`cli.ts:~282-314`) drains `getFactsQueue().drainPending()` +
`awaitPendingLastRetrievedWrites()` before `engine.disconnect()`; the `dream`
owner-disconnect (`cli.ts:~1164`) and the fall-through owner-disconnect
(`cli.ts:~1785`) do not. If the dream cycle ever enqueues a facts:absorb /
last-retrieved write that's still in flight at disconnect, the owner nulls the
singleton and the write throws "No database connection". Pre-existing (not
introduced by the #1471 ownership fix), surfaced by the Claude adversarial
review (F5). Fix: hoist the same drain-before-disconnect block the op-dispatch
path uses into a shared helper and call it on all three owner-disconnect sites.
- [x] **P3 — `dream` + CLI_ONLY fall-through paths don't drain the facts /
last-retrieved queues before the owner disconnect.** DONE in the #2084 wave:
`drainThenDisconnect(engine)` in `src/cli.ts` is the shared helper, applied at
ALL eight owner-disconnect sites (op-dispatch, CLI_ONLY fall-through, search
dashboard, doctor remediation x3, ze-switch, dream, read-only-timeout path),
with its own bounded hard-deadline. Pinned by `test/cli-drain-then-disconnect.test.ts`.
## v0.42.x AI SDK v6 tool-schema fix follow-ups (#1782/#1764)
Surfaced by the codex outside-voice pass during `/plan-eng-review` and
File diff suppressed because one or more lines are too long
+65
View File
@@ -0,0 +1,65 @@
# Push-based context (#2095, v0.43)
Retrieval used to be pull-only: the agent had to *know to ask* before the brain
contributed anything. Push-based context inverts that — the brain volunteers
relevant pages from the recent conversation, confidence-gated so push noise
never becomes worse than pull silence.
Three channels share one zero-LLM core (`src/core/context/volunteer.ts`):
| Channel | Surface | When to use |
|---|---|---|
| `reflex` | automatic, inside the context engine | default-on for plugin hosts; nothing to call |
| `op` | `gbrain volunteer-context` / MCP `volunteer_context` | agents without the plugin; one call per turn |
| `watch` | `gbrain watch` | stream a transcript in, volunteered pages stream out |
## How it decides
1. **Extract** entities across the last N turns (capitalized runs, `@handles`),
merged with recency / frequency / user-role salience. Assistant-introduced
entities and "what did she invest in?" follow-ups whose antecedent was named
in the window now resolve.
2. **Resolve** through the alias table, exact titles, and slug suffixes — each
arm carries an honest confidence: alias 0.9, exact title 0.8, slug-suffix 0.6,
+0.05 when mentioned in ≥2 turns or the newest turn.
3. **Gate** at `min_confidence` (default 0.7 — slug-suffix matches need an
explicit lower gate), suppress pages already surfaced (slug-presence only),
cap at 3 pages (hard cap 5).
## CLI
```bash
# one-shot: pipe recent turns (oldest → newest)
printf 'user: ask alice-example about the deal\nassistant: noted\nuser: what did she say?\n' \
| gbrain volunteer-context
# streaming: volunteered pages print as the transcript flows
some-transcript-feed | gbrain watch --json
# the feedback loop: how often were volunteered pages actually opened?
gbrain volunteer-context --stats
```
Stats are **approximate** by design: "used" means `pages.last_retrieved_at >
volunteered_at` — the 5-minute last-retrieved throttle causes false negatives
and unrelated reads of the same page cause false positives. Use the per-arm
precision to tune `min_confidence`, not as an exact metric.
## Config
| Key | Default | What it does |
|---|---|---|
| `retrieval_reflex_window_turns` | 4 | turns the ambient reflex extracts from; 1 = legacy current-turn-only (file/env plane: `GBRAIN_RETRIEVAL_REFLEX_WINDOW_TURNS`) |
| `retrieval_reflex` | true | the ambient channel's master switch |
| `retrieval_reflex_max_pointers` | 3 | pointer cap per turn |
Per-call knobs on the op/watch: `max_pages`, `min_confidence`, `session_id`,
`turn` (attribution columns in the feedback log).
## Storage + privacy
Volunteered pages log to `context_volunteer_events` (migration v116): slug,
arm, confidence, channel, optional session/turn — the rationale is a
deterministic template string, never raw conversation text. Rows are pruned
after 90 days by the dream cycle's purge phase. Synopses pass through the same
takes/facts-fence privacy boundary as `get_page`.
+2 -1
View File
@@ -186,7 +186,7 @@ mount, CEO-class with multiple team brains) and
## Architecture
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`). CLI and MCP
Contract-first: `src/core/operations.ts` defines ~47 shared operations (v0.29 adds `get_recent_salience`, `find_anomalies`, `get_recent_transcripts`; v0.43 adds `volunteer_context` — push-based context, see `docs/guides/push-context.md`). CLI and MCP
server are both generated from this single source. Engine factory (`src/core/engine-factory.ts`)
dynamically imports the configured engine (`'pglite'` or `'postgres'`). Skills are fat
markdown files (tool-agnostic, work with both CLI and plugin contexts).
@@ -245,6 +245,7 @@ detail on demand.)
| any file in `src/` (what it does + its invariants) | `docs/architecture/KEY_FILES.md` — find the file's entry |
| search / ranking / hybrid / retrieval | `docs/architecture/RETRIEVAL.md` + the `search/*` entries in `KEY_FILES.md` |
| search modes / cost knobs | `docs/guides/search-modes.md` |
| push-based context (volunteer/watch/reflex window) | `docs/guides/push-context.md` |
| schema packs / page types / extraction | `docs/architecture/schema-packs.md`, `type-taxonomy.md`, `lens-packs.md` |
| thin-client / remote MCP / cross-modal | `docs/architecture/thin-client.md` |
| the CLI surface (commands + flags) | `gbrain --help` / `gbrain --tools-json`, plus the relevant `KEY_FILES.md` entry |