diff --git a/CHANGELOG.md b/CHANGELOG.md index 29c12a8d6..bf7c55fbb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,26 @@ All notable changes to GBrain will be documented in this file. +## [0.42.39.0] - 2026-06-09 + +**Your agent now learns a brain page exists the moment you name someone — instead of talking about a close contact for four messages without ever opening their page.** gbrain was great at *storing* knowledge and at injecting deterministic per-turn context, but it never taught the agent the *policy* of retrieval: when to look something up, and what to pull. That lived in each user's hand-rolled instructions and failed silently. The Retrieval Reflex makes it a property of having a brain. + +Two layers, on by default. A **deterministic pointer layer** in the context engine scans each turn's message for salient, resolvable entities (capitalized names, `@handles`) and injects a compact pointer — name → slug → one-line summary → "open the page before relying on details." Zero-LLM, fail-open, capped, and judgment-gated: it points, it never auto-dumps the page body, and it stays silent on trivial mentions or entities already in context. A **policy skill** (installed into your agent's resolver via `gbrain integrations install retrieval-reflex`) encodes the trigger policy and the pointer → full-page → graph-neighbors escalation ladder so the agent knows what to do with the pointer. + +It works on every engine without leaking a second database connection. PGLite holds a single connection (your `gbrain serve` owns it), so the context engine resolves *through* the live holder over a local socket rather than opening its own — and on Postgres it uses a cached direct connection. Synopses run through the same privacy boundary `get_page` applies, so private facts never reach the prompt. `gbrain doctor` reports whether the reflex is actually firing. + +### Added +- **Retrieval Reflex deterministic pointer layer (gbrain#1981).** The context engine injects compact, privacy-safe entity pointers per turn — on by default, zero-LLM, fail-open, capped. Disable with `GBRAIN_RETRIEVAL_REFLEX=false` or `retrieval_reflex: false` in `~/.gbrain/config.json` (file/env plane). +- **`retrieval-reflex` recipe + policy skill.** Installs the when/what-to-retrieve policy into your agent's resolver: `gbrain integrations install retrieval-reflex --target `. +- **`retrieval_reflex_health` doctor check.** Reports the deterministic layer's real runtime status (observed firing, resolve path, policy-skill install state). + +### Fixed +- **Resolver-row install fence is now keyed by recipe id.** Installing a second `copy-into-host-repo` recipe previously wrote a block mislabeled with the first recipe's name; `--refresh`/uninstall now find their own rows. + +### To take advantage of v0.42.39.0 + +`gbrain upgrade`. The deterministic pointer layer is on automatically — no config needed. To give the agent the matching policy skill, run `gbrain integrations install retrieval-reflex --target `, then `gbrain doctor` to confirm `retrieval_reflex_health` is green. + ## [0.42.38.0] - 2026-06-09 **Three independent job-layer bugs that left autopilot wedged or swallowed a command's output are fixed, each traced to source.** A triage of the job/lock/teardown layer (gbrain#1972) pulled them into one wave. diff --git a/TODOS.md b/TODOS.md index 1bd0164d1..fc74cfaf6 100644 --- a/TODOS.md +++ b/TODOS.md @@ -1,5 +1,21 @@ # TODOS +## gbrain#1981 Retrieval Reflex follow-ups (v0.43+) + +Filed from the #1981 ship (v0.42.39.0). Deliberately scoped OUT — the v1 extractor +is deterministic + precision-biased. See plan + GSTACK REVIEW REPORT at +`~/.claude/plans/system-instruction-you-are-working-wild-yeti.md`. + +- [ ] **P3 — broaden entity detection beyond proper-case ASCII.** The v1 extractor + (`src/core/context/entity-salience.ts`) misses lowercase names, many non-Latin + scripts, pronoun follow-ups ("what about her?"), and assistant-introduced entities. + These need conversation state or an LLM pass. **Why:** higher recall on the read + side. **Where:** `entity-salience.ts` + the orchestrator's `priorContextText`. +- [ ] **P3 — recall knob: optional fuzzy/prefix-expansion resolution.** The resolver + (`src/core/context/retrieval-reflex.ts`) is exact-only (alias + title + slug-suffix) + for precision. Revisit adding `resolveEntitySlug`'s trgm-fuzzy / prefix-expansion + arm, gated on an unambiguous single hit, if recall telemetry comes back weak. + ## gbrain#1972 job-layer follow-up (v0.43+) Filed from the #1972 fix (stale-lock reaper + bounded disconnect + complete diff --git a/VERSION b/VERSION index 79e10702e..2a5cc6444 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.42.38.0 +0.42.39.0 diff --git a/docs/architecture/KEY_FILES.md b/docs/architecture/KEY_FILES.md index dd25c0c2e..198b2759a 100644 --- a/docs/architecture/KEY_FILES.md +++ b/docs/architecture/KEY_FILES.md @@ -16,6 +16,9 @@ per-release `**vX.Y.Z:**` narration — CI enforces this - `src/core/search/hybrid.ts` extension — `runPostFusionStages` has a 4th stage (`graphSignalsEnabled`, `onGraphMeta`, `onScoreDistribution`). `base_score` stamped at function entry idempotently (captured ONCE before any boost stage mutates `score`). Each post-fusion stage stamps its multiplier: `applyBacklinkBoost`→`backlink_boost`, `applySalienceBoost`→`salience_boost`, `applyRecencyBoost`→`recency_boost`. `applyReranker` (earlier in the pipeline) stamps `reranker_delta` as a rank delta (positive = improved). `applyExactMatchBoost` in `src/core/search/intent-weights.ts` stamps `exact_match_boost` when fired. Per-stage attribution powers `gbrain search --explain` — every boost surface carries its own field so `formatResultsExplain` reads them all without coupling to internal stage ordering. - `src/core/search/explain-formatter.ts` — renders `SearchResult[]` as a multi-line per-result breakdown for `gbrain search --explain`. Reads every boost-stamping field. Handles the "no boosts applied" empty path. 4-decimal precision with trailing-zero strip. Pinned by `test/search/explain-formatter.test.ts`. - `src/core/search/mode.ts` extension — `graph_signals: boolean` knob in `ModeBundle` (defaults: `conservative=false`, `balanced=true`, `tokenmax=true`). `KNOBS_HASH_VERSION` appends a `gs=` parts entry per the cache-key contamination convention so a graph-on cache write can't be served to a graph-off lookup. `SearchKeyOverrides` + `SearchPerCallOpts` + `loadOverridesFromConfig` + `SEARCH_MODE_CONFIG_KEYS` + `resolveSearchMode` + `attributeKnob` all carry the field. Opt-out: `gbrain config set search.graph_signals false`. Mid-deploy `query_cache` rows from before the upgrade hash differently — natural row segregation, clears within `cache.ttl_seconds` (3600s default). +- `src/core/context-engine.ts` + `src/openclaw-context-engine.ts` — the deterministic context engine OpenClaw loads on every turn (`assemble()` injects the Live Context block, zero-LLM). `createGBrainContextEngine({workspaceDir, resolveEntities?})` accepts an OPTIONAL host-injected resolver (`ENGINE_API_VERSION` 0.2.0, additive — older hosts work unchanged; the plugin entry maps `ctx.resolveEntities`/`ctx.brainQuery` onto it). `assemble()` runs the Retrieval Reflex after the Live Context block: extracts the current turn's user text, builds prior-context text (every message EXCEPT the current turn — suppression must not see the triggering mention), and appends the pointer block. `warmReflex()` fires at construction. +- `src/core/context/` — Retrieval Reflex (Layer 1, issue #1981). `entity-salience.ts`: pure, zero-LLM, precision-biased `extractCandidates(text)` (capitalized runs + `@handles`, STOPWORDS + soft COMMON_WORDS + sentence-start guard, deterministic, capped). `retrieval-reflex.ts`: `resolveEntitiesToPointers(engine, sourceId, candidates, opts)` — alias arm (`resolveAliases`, caught per-arm for pre-v110 brains) + exact title/slug-suffix arm (the recall fix: real slugs are namespaced `people/x` but `slugify` drops the prefix); synopsis runs through `stripTakesFence`/`stripFactsFence` (the same privacy boundary `get_page` applies) so private facts never reach the prompt; suppression scans `priorContextText` only; capped at `MAX_POINTERS`. `reflex.ts`: the orchestrator + engine-aware resolver ladder (host `resolveEntities` → PGLite serve IPC → Postgres cached process-singleton → disabled), zero-candidate fast path, fail-open + timeout, heartbeat write for the doctor check, `reflexEnabled(cfg)` (file/env gate, default ON; DB-plane does NOT gate — `assemble()` is sync). `resolve-ipc.ts`: local unix-socket resolve protocol (client + server) so PGLite resolves through the single connection `gbrain serve` holds (a second opener would hit the exclusive lock; a subprocess would force-steal it past the 5-min staleness window and crash). Wired into `src/mcp/server.ts` (serve binds `/.gbrain-resolve.sock` on PGLite, cleaned up on shutdown). Doctor surface: `retrieval_reflex_health` in `src/commands/doctor.ts` (reads the heartbeat for truthful runtime status; categorized in `doctor-categories.ts`). Config: `retrieval_reflex` + `retrieval_reflex_max_pointers` in `src/core/config.ts`. Policy layer ships as the `retrieval-reflex` recipe (`recipes/retrieval-reflex/`). Pinned by `test/context/entity-salience.test.ts`, `test/retrieval-reflex.test.ts`, `test/context/resolve-ipc.test.ts`, `test/doctor-retrieval-reflex.test.ts`. +- `src/commands/integrations.ts` — recipe install. The resolver-row install fence is keyed by `manifest.recipe` (`gbrain::resolver-rows`), so a second `copy-into-host-repo` recipe no longer writes a block mislabeled with the first recipe's name. Pinned by `test/integrations-install.test.ts`. - `src/core/audit/audit-writer.ts` — shared JSONL audit primitive consolidating the hand-rolled audit modules. Exports `createAuditWriter({kind, recordSchema})` returning `{log, readRecent}` plus shared helpers `computeIsoWeekFilename(kind, now?)` and `resolveAuditDir()` (honors `GBRAIN_AUDIT_DIR`). ISO-week file rotation; best-effort writes (stderr warn on failure, never throws); read-path scans current-week + previous-week files for boundary spans. Refactored onto it for parity: `src/core/rerank-audit.ts`, `src/core/audit-slug-fallback.ts`, `src/core/minions/handlers/shell-audit.ts`, `src/core/minions/handlers/supervisor-audit.ts`, `src/core/facts/phantom-audit.ts` (each module's public API preserved bit-for-bit). The `graph-signals-failures` audit (`logGraphSignalsFailure`) uses the same primitive. One hand-rolled audit remains at `src/core/skillpack/audit.ts`. Pinned by `test/audit/audit-writer.test.ts`. - `src/core/cli-options.ts` extension — `CliOptions` gains `explain: boolean`. `parseGlobalFlags` recognizes `--explain` anywhere in argv (stripped before command dispatch). `src/cli.ts` `formatResult` for `search` + `query` cases routes to `formatResultsExplain` from `src/core/search/explain-formatter.ts` when `CliOptions.explain` is set; falls through to the existing JSON / human formatters otherwise. - `src/commands/search.ts:gbrain search stats` extension — `graph_signals` section (enabled/source/failures_count/failures_by_reason). JSON envelope adds a `graph_signals` sibling property; `_meta.metric_glossary` adds `graph_signals.enabled` + `graph_signals.failures_by_reason`. Human output prints the section after the existing block. Reads `search.graph_signals` config first, falls back to the mode default. Pinned by `test/search/search-stats-graph-signals.test.ts`. diff --git a/package.json b/package.json index 9d30c7d7d..e986cad10 100644 --- a/package.json +++ b/package.json @@ -143,5 +143,5 @@ "bun": ">=1.3.10" }, "license": "MIT", - "version": "0.42.38.0" + "version": "0.42.39.0" } diff --git a/recipes/retrieval-reflex.md b/recipes/retrieval-reflex.md new file mode 100644 index 000000000..8c0e25ced --- /dev/null +++ b/recipes/retrieval-reflex.md @@ -0,0 +1,54 @@ +--- +id: retrieval-reflex +name: Retrieval Reflex +version: 0.1.0 +description: Teaches the host agent WHEN to look something up and WHAT to pull. Ships a policy skill (trigger + retrieval spec) into the host resolver; pairs with the deterministic pointer layer in the context engine. +category: reflex +install_kind: copy-into-host-repo +requires: [] +secrets: [] +health_checks: + - type: command + argv: [gbrain, doctor, --json] + label: Retrieval reflex wiring (see retrieval_reflex_health) +setup_time: 2 min +cost_estimate: "$0 — zero-LLM deterministic layer + a prose policy skill" +--- + +# Retrieval Reflex: teach the agent *when* and *what* to retrieve + +gbrain is great at **storing** knowledge and at **injecting deterministic +context** every turn. It does not, by itself, teach the host agent the *policy* +of retrieval: **when** to look something up and **what** to pull. Without it, +the agent can discuss a person who has a rich brain page for several messages +without ever opening it — then answer generically about facts the brain already +knew. + +This reflex has two halves: + +1. **Deterministic pointer layer (automatic, on by default).** The + `gbrain-context` engine scans each turn's user message for salient, + resolvable entities and injects a compact pointer (name → slug → one-line + summary) so the agent *knows the page exists*. Zero-LLM, fail-open. Nothing + to install — it's on unless `retrieval_reflex` is disabled in + `~/.gbrain/config.json` or `GBRAIN_RETRIEVAL_REFLEX=false`. + +2. **Policy skill (this recipe installs it).** A SKILL fragment in the host + resolver that encodes the trigger policy and retrieval spec the agent + follows when a pointer appears or an entity becomes the subject. + +## IMPORTANT: Instructions for the Agent + +**You are the installer.** Run these steps on behalf of the user. + +1. Confirm the deterministic layer isn't disabled: + `gbrain doctor --json | jq '.checks[] | select(.name=="retrieval_reflex_health")'` +2. Install the policy skill into the host repo (the OpenClaw/agent repo that + holds `skills/RESOLVER.md` or `AGENTS.md`): + `gbrain integrations install retrieval-reflex --target ` +3. Verify: re-run `gbrain doctor` and confirm `retrieval_reflex_health` is `ok`. + +The deterministic layer needs no install. On a PGLite brain it resolves through +the running `gbrain serve` (or a host-provided capability); if neither is +available it stays disabled and this policy skill carries the behavior — the +doctor check reports which. diff --git a/recipes/retrieval-reflex/install/manifest.json b/recipes/retrieval-reflex/install/manifest.json new file mode 100644 index 000000000..ec89d6416 --- /dev/null +++ b/recipes/retrieval-reflex/install/manifest.json @@ -0,0 +1,16 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "recipe": "retrieval-reflex", + "version": "0.1.0", + "install_kind": "copy-into-host-repo", + "description": "src → target mapping consumed by `gbrain integrations install retrieval-reflex`. Policy-only recipe: ships one SKILL.md into the host resolver and appends a resolver row. The deterministic pointer layer lives in the gbrain context engine and needs no install.", + "target_root_relative_to_host_repo": "skills/retrieval-reflex", + "skills_target_root_relative_to_host_repo": "skills", + "files": [], + "skills": [ + { "src": "skills/retrieval-reflex/SKILL.md", "target": "skills/retrieval-reflex/SKILL.md", "mode": "0644" } + ], + "resolver_rows_to_append": [ + "retrieval-reflex | a named person/company/project/place becomes the subject; a brain-page pointer appears in context; \"who is\", \"what do we know about\", \"tell me about\"; about to assert a non-trivial detail about a named entity" + ] +} diff --git a/recipes/retrieval-reflex/skills/retrieval-reflex/SKILL.md b/recipes/retrieval-reflex/skills/retrieval-reflex/SKILL.md new file mode 100644 index 000000000..3fec8b376 --- /dev/null +++ b/recipes/retrieval-reflex/skills/retrieval-reflex/SKILL.md @@ -0,0 +1,59 @@ +--- +name: retrieval-reflex +version: 0.1.0 +description: When/what to retrieve — open the brain page for a salient entity before answering from memory. +triggers: + - "who is" + - "what do we know about" + - "tell me about" +mutating: false +writes_pages: false +writes_to: [] +tools: [get_page, query, graph, backlinks] +--- + +# Retrieval Reflex — retrieve on demand, when an entity is salient + +A person doesn't bulk-load their whole address book into working memory. They +retrieve **on demand**, when an entity becomes **salient**, use it, and drop it. +Encode that reflex. The brain probably has the data — if a name is salient and +you haven't opened its page, open it before you answer. + +## Trigger policy — WHEN to retrieve + +Retrieve when ANY of these holds AND the page isn't already loaded in context: + +- An entity (person / company / project / deal / place) is the **subject** of + the message, or a decision/judgment about it is being made, or the exchange is + substantive / relational / emotional about it. +- A **brain-page pointer** appeared in context this turn (the deterministic + layer told you the page exists) — open it before relying on details. +- A name or term appears that you **don't recognize** and that looks notable → + do a quick resolve (the human reflex). +- You're about to **assert a non-trivial detail** about an entity (attribution, + status, history) → verify against the brain first. Say "let me check", not a guess. + +**Skip** trivial passing mentions, logistics pings, and anything already loaded. +Judgment first — retrieve when it changes the quality of the reply, not reflexively. + +## Retrieval spec — WHAT to pull, and when to stop + +Escalate only as far as the task needs: + +1. **Pointer / metadata.** If a pointer is already in context (slug + one-line + summary), and the task only needs identity, stop there. +2. **Full page.** When the entity is the subject or details matter, open it: + `get_page ` (MCP) — read the page before relying on specifics. +3. **Linked neighbors.** Only when relationship context is needed, pull + `graph` / `backlinks` for the slug. + +**Resolve only the name(s) the current task needs, use them, drop them.** No +bulk-loading the inner circle. + +## The failure this prevents + +If you've discussed a named person for more than a message without opening their +page, open it now. The write side captures everything; the read side only helps +if you actually look. + +See also: `skills/query/SKILL.md` (search the brain), `skills/brain-ops/SKILL.md`. diff --git a/src/commands/doctor.ts b/src/commands/doctor.ts index 08651157e..99fcdda8b 100644 --- a/src/commands/doctor.ts +++ b/src/commands/doctor.ts @@ -24,7 +24,10 @@ import { categorizeCheck, type CheckCategory } from '../core/doctor-categories.t import { rankIssues, type RankedIssue } from '../core/doctor-cause-rank.ts'; import { getCliOptions, cliOptsToProgressOptions } from '../core/cli-options.ts'; import type { DbUrlSource } from '../core/config.ts'; -import { gbrainPath } from '../core/config.ts'; +import { gbrainPath, loadConfig } from '../core/config.ts'; +import { reflexEnabled } from '../core/context/reflex.ts'; +import { resolveSocketPath } from '../core/context/resolve-ipc.ts'; +import { homedir } from 'os'; import { dirname, isAbsolute, join, resolve as resolvePath } from 'path'; import { fileURLToPath } from 'url'; import { existsSync, readFileSync, readdirSync, statSync } from 'fs'; @@ -3912,6 +3915,93 @@ export async function computePoolReapHealthCheck( return null; } +/** + * Retrieval Reflex health (#1981). Read-only, fail-open. The deterministic + * pointer layer is on by default; this reports the TRUTH, not an aspiration: + * - config/env disabled → warn (pointer layer off) + * - heartbeat fired recently → ok, "active" (it's demonstrably working, + * whatever path — Postgres/IPC/host) + * - enabled, no recent heartbeat → ok if a viable path looks present + * (postgres, or pglite serve socket), + * else warn (likely inactive — policy + * skill carries). Never claims a host + * capability it can't observe. + * Policy-skill install state is reported in details (it ships into the HOST + * repo, so absence in gbrain's own skills dir is expected, not a failure). + */ +export function buildRetrievalReflexCheck(skillsDir: string | null): Check { + const name = 'retrieval_reflex_health'; + try { + const cfg = loadConfig(); + const enabled = reflexEnabled(cfg); + const engineKind = cfg?.engine ?? 'unknown'; + const skillInstalled = !!skillsDir && existsSync(join(skillsDir, 'retrieval-reflex', 'SKILL.md')); + + if (!enabled) { + return { + name, + status: 'warn', + message: 'retrieval reflex disabled (config/env) — entity pointer layer off', + details: { enabled: false, engine: engineKind, policy_skill_installed: skillInstalled }, + }; + } + + // Heartbeat is the authority for "is it firing". + const hbPath = join(homedir(), '.gbrain', 'integrations', 'retrieval-reflex', 'heartbeat.jsonl'); + let lastFired: string | null = null; + try { + if (existsSync(hbPath)) { + const lines = readFileSync(hbPath, 'utf8').trim().split('\n').filter(Boolean); + const last = lines.length ? JSON.parse(lines[lines.length - 1]) : null; + if (last && typeof last.ts === 'string') lastFired = last.ts; + } + } catch { /* heartbeat unreadable — treat as never fired */ } + const firedRecently = + !!lastFired && Date.now() - new Date(lastFired).getTime() < 7 * 24 * 60 * 60 * 1000; + + // Detect a viable resolve path the doctor CAN see (host ctx.brainQuery is invisible). + let pathDesc: string; + let viablePathVisible: boolean; + if (engineKind === 'postgres') { + pathDesc = 'postgres direct'; + viablePathVisible = true; + } else if (engineKind === 'pglite' && cfg?.database_path) { + const socket = resolveSocketPath(cfg.database_path); + viablePathVisible = existsSync(socket); + pathDesc = viablePathVisible ? 'pglite via serve IPC' : 'pglite — serve IPC socket not present'; + } else { + pathDesc = `engine ${engineKind}`; + viablePathVisible = false; + } + + const runtimeMsg = firedRecently + ? `active (last fired ${lastFired})` + : viablePathVisible + ? 'enabled; not observed firing yet' + : 'enabled but no observed activity and no visible resolve path (host capability may still supply it; policy skill carries otherwise)'; + + const status: Check['status'] = firedRecently || viablePathVisible ? 'ok' : 'warn'; + const skillHint = skillInstalled + ? '' + : ' — policy skill not installed; run `gbrain integrations install retrieval-reflex --target `'; + return { + name, + status, + message: `${pathDesc}; ${runtimeMsg}${skillHint}`, + details: { + enabled: true, + engine: engineKind, + path: pathDesc, + fired_recently: firedRecently, + last_fired: lastFired, + policy_skill_installed: skillInstalled, + }, + }; + } catch (e) { + return { name, status: 'warn', message: `could not check: ${(e as Error).message}` }; + } +} + export async function buildChecks( engine: BrainEngine | null, args: string[], @@ -4024,6 +4114,15 @@ export async function buildChecks( checks.push({ name: 'resolver_health', status: 'warn', message: 'Could not find skills directory' }); } + // 1b. Retrieval Reflex health (#1981, SKILL group — gated). Truthful runtime + // status: the deterministic pointer layer is on by default; the heartbeat file + // (written by the context engine when it actually injects) is the authority for + // "is it firing". The doctor cannot see the OpenClaw host capability directly, + // so it never claims "enabled via host"; it reports observed activity instead. + if (scope === 'all') { + checks.push(buildRetrievalReflexCheck(skillsDir)); + } + // 2. Skill conformance (SKILL group — gated) if (scope === 'all' && skillsDir) { const conformanceResult = checkSkillConformance(skillsDir); diff --git a/src/commands/init.ts b/src/commands/init.ts index 3f7dbd636..08cf8432c 100644 --- a/src/commands/init.ts +++ b/src/commands/init.ts @@ -1452,6 +1452,12 @@ export function reportModStatus(): void { } console.log('Resolver: skills/RESOLVER.md'); console.log('Soul audit: run `gbrain soul-audit` to customize agent identity'); + // Retrieval Reflex (#1981): the deterministic pointer layer is ON by default + // (no action needed). The policy skill is installed into the HOST repo on + // request — we PRINT the command rather than silently mutating the host repo. + console.log('Retrieval reflex: on by default (entity pointers injected per turn)'); + console.log(' Install the policy skill into your agent repo:'); + console.log(' gbrain integrations install retrieval-reflex --target '); console.log(''); } diff --git a/src/commands/integrations.ts b/src/commands/integrations.ts index 575ff0a9e..92cda4e20 100644 --- a/src/commands/integrations.ts +++ b/src/commands/integrations.ts @@ -1438,9 +1438,15 @@ export async function installRecipeIntoHostRepo( } catch { /* not present */ } } if (resolverPath) { - const rowsBlock = `\n\n\n` + + // Fence the appended rows by the RECIPE id, not a hardcoded recipe name. + // The pre-v0.42 fence was literally `gbrain:agent-voice:resolver-rows`, + // so any second copy-into-host-repo recipe (e.g. retrieval-reflex) wrote + // an agent-voice-labeled block — and `--refresh`/uninstall keyed on the + // recipe id would never find it. Derive the fence from manifest.recipe. + const fenceId = manifest.recipe || 'gbrain'; + const rowsBlock = `\n\n\n` + manifest.resolver_rows_to_append.map((r) => `- ${r}`).join('\n') + - '\n\n'; + `\n\n`; fsAppendFileSync(resolverPath, rowsBlock); } else { console.warn( diff --git a/src/core/config.ts b/src/core/config.ts index 0aa815211..ff1ee2273 100644 --- a/src/core/config.ts +++ b/src/core/config.ts @@ -155,6 +155,20 @@ export interface GBrainConfig { embedding_multimodal?: boolean; /** Model override for multimodal embeddings (e.g. "voyage:voyage-multimodal-3"). */ embedding_multimodal_model?: string; + + /** + * v0.42 (#1981) — Retrieval Reflex. The context engine's deterministic + * per-turn entity-pointer injection. Default ON (absent key = enabled). + * + * IMPORTANT: this is a FILE-PLANE / env gate only. The context engine reads + * it via the synchronous `loadConfig()` during `assemble()`, which never + * touches the DB. So `gbrain config set retrieval_reflex false` (DB plane) + * does NOT disable the reflex — set it in `~/.gbrain/config.json` or via + * `GBRAIN_RETRIEVAL_REFLEX=false`. + */ + retrieval_reflex?: boolean; + /** Max pointers injected per turn (default 3). File-plane only. */ + retrieval_reflex_max_pointers?: number; embedding_image_ocr?: boolean; embedding_image_ocr_model?: string; @@ -426,6 +440,9 @@ export function loadConfig(): GBrainConfig | null { ...(process.env.GBRAIN_EMBEDDING_IMAGE_OCR_MODEL ? { embedding_image_ocr_model: process.env.GBRAIN_EMBEDDING_IMAGE_OCR_MODEL } : {}), + ...(process.env.GBRAIN_RETRIEVAL_REFLEX + ? { retrieval_reflex: !(process.env.GBRAIN_RETRIEVAL_REFLEX === 'false' || process.env.GBRAIN_RETRIEVAL_REFLEX === '0') } + : {}), ...(process.env.GBRAIN_REMOTE_CLIENT_SECRET && fileConfig?.remote_mcp ? { remote_mcp: { ...fileConfig.remote_mcp, oauth_client_secret: process.env.GBRAIN_REMOTE_CLIENT_SECRET } } : {}), diff --git a/src/core/context-engine.ts b/src/core/context-engine.ts index 9c369b109..080e517da 100644 --- a/src/core/context-engine.ts +++ b/src/core/context-engine.ts @@ -15,6 +15,7 @@ import { readFileSync, existsSync, statSync } from 'fs'; import { join } from 'path'; +import { buildReflexAddition, warmReflex, type ResolveEntitiesFn as ReflexResolveEntitiesFn } from './context/reflex.ts'; // Types inlined from openclaw/plugin-sdk to avoid hard dependency during development. // At runtime inside OpenClaw, the real SDK is available; these types ensure build compat. @@ -111,7 +112,10 @@ export const ENGINE_NAME = 'GBrain Context Engine'; * semantic: this is an interface-stability marker for OpenClaw's loader, * not a release tag. */ -export const ENGINE_API_VERSION = '0.1.0'; +// 0.2.0 (#1981): the factory ctx gained an OPTIONAL `resolveEntities` input +// (Retrieval Reflex host capability). Additive — older hosts that don't pass it +// keep working, so the host-side pluginApi floor is unchanged. +export const ENGINE_API_VERSION = '0.2.0'; /** @deprecated Use ENGINE_API_VERSION. Kept for back-compat with v0.32.5 callers. */ export const ENGINE_VERSION = ENGINE_API_VERSION; @@ -151,6 +155,50 @@ function sanitizeForPrompt(s: string, maxLen: number = 100): string { return s.replace(/[\n\r\t\x00-\x1F\x7F]/g, ' ').slice(0, maxLen).trim(); } +/** + * Coerce a message's `content` (string or structured block array) to plain text + * for the Retrieval Reflex extractor / suppression scan. Best-effort: pulls + * `.text` out of content blocks, ignores non-text parts. + */ +function messageText(content: unknown): string { + if (typeof content === 'string') return content; + if (Array.isArray(content)) { + return content + .map((b) => (b && typeof b === 'object' && typeof (b as any).text === 'string' ? (b as any).text : '')) + .filter(Boolean) + .join(' '); + } + return ''; +} + +/** Text of the current turn = the LAST user-role message. '' if none. */ +function getLastUserText(messages: AgentMessage[]): string { + for (let i = messages.length - 1; i >= 0; i--) { + if (messages[i]?.role === 'user') return messageText(messages[i].content); + } + return ''; +} + +/** + * Joined text of everything the agent has ALREADY seen — every message EXCEPT + * the current turn (the last user message). Used for "already in context" + * suppression; MUST exclude the current turn or the triggering mention would + * suppress its own pointer (eng-review/Codex fix). Capped for scan cost. + */ +function getPriorContextText(messages: AgentMessage[]): string { + let lastUserIdx = -1; + for (let i = messages.length - 1; i >= 0; i--) { + if (messages[i]?.role === 'user') { lastUserIdx = i; break; } + } + const parts: string[] = []; + for (let i = 0; i < messages.length; i++) { + if (i === lastUserIdx) continue; + const t = messageText(messages[i]?.content); + if (t) parts.push(t); + } + return parts.join('\n').slice(-20_000); +} + /** Common airport → timezone mapping */ const AIRPORT_TZ: Record = { SFO: 'US/Pacific', LAX: 'US/Pacific', SJC: 'US/Pacific', SEA: 'US/Pacific', PDX: 'US/Pacific', @@ -552,8 +600,19 @@ function formatContextBlock(ctx: LiveContext): string { export function createGBrainContextEngine(ctx: { workspaceDir?: string; + /** + * Retrieval Reflex (#1981, D1=A): optional host-provided resolver. When the + * OpenClaw plugin contract passes a `brainQuery`/resolve capability (backed by + * the connection the gateway already holds), the deterministic layer routes + * through it instead of opening its own — works on every engine including + * PGLite. Absent → the engine falls to the serve IPC / Postgres-direct ladder. + */ + resolveEntities?: ReflexResolveEntitiesFn; }): ContextEngine { const workspaceDir = ctx.workspaceDir ?? process.cwd(); + // Warm the Postgres connection ahead of the first salient turn (no-op for + // PGLite/host paths). Fire-and-forget; never blocks engine construction. + warmReflex(); const engine: ContextEngine = { info: { @@ -582,9 +641,21 @@ export function createGBrainContextEngine(ctx: { citationsMode, }); - // 3. Combine: live context + memory prompt + // 2b. Retrieval Reflex (#1981): detect salient entities in THIS turn that + // resolve to existing brain pages and inject compact pointers. Zero-LLM, + // fail-open, time-bounded — returns null (no addition) on any error or when + // nothing salient resolves. Detect + point, never auto-dump bodies. + const reflexAddition = await buildReflexAddition({ + workspaceDir, + currentUserText: getLastUserText(messages), + priorContextText: getPriorContextText(messages), + resolveEntities: ctx.resolveEntities, + }); + + // 3. Combine: live context + memory prompt + reflex pointers const parts = [contextBlock]; if (memoryAddition) parts.push(memoryAddition); + if (reflexAddition) parts.push(reflexAddition); // 4. Pass through messages unchanged (legacy assembly) return { diff --git a/src/core/context/entity-salience.ts b/src/core/context/entity-salience.ts new file mode 100644 index 000000000..a51d20ef2 --- /dev/null +++ b/src/core/context/entity-salience.ts @@ -0,0 +1,174 @@ +/** + * Retrieval Reflex — pure entity-salience extractor (issue #1981, Layer 1). + * + * Zero-LLM, zero-DB, SDK-free. Scans ONE turn's user text for candidate entity + * surface-forms (capitalized token runs, @handles) that are worth resolving + * against the brain index. The context engine runs this on every turn before + * touching the brain, so it must be fast (one regex pass) and precision-biased: + * a false candidate costs a wasted resolve and, worse, a misleading pointer. + * + * DELIBERATE v1 limits (documented, not bugs — see issue #1981 / eng-review): + * - Proper-case + ASCII biased. Misses lowercase names ("garry") and many + * non-Latin scripts. + * - Current-user-message only. No pronoun follow-ups ("what about her?"), no + * entities the assistant introduced. + * These are TODOs, not v1 scope. Do NOT market this as full "human-like recall". + * + * Resolution (alias/slug lookup) lives in retrieval-reflex.ts; this module only + * decides WHAT to look up. + */ + +import { normalizeAlias } from '../search/alias-normalize.ts'; + +export interface EntityCandidate { + /** Surface form for the pointer label, e.g. "Garry Tan" or "@garry". */ + display: string; + /** Text fed to alias-normalize / slugify for resolution (no leading @, no possessive). */ + query: string; +} + +/** Max candidates returned per turn — bounds downstream DB work regardless of pointer cap. */ +export const MAX_CANDIDATES = 12; + +/** + * HARD stopwords — function words that are never an entity, even capitalized + * mid-sentence. Pronouns, articles/determiners, auxiliaries, conjunctions, + * and the most common sentence openers. Compared in lowercase. + */ +const STOPWORDS = new Set([ + // pronouns + 'i', "i'm", "i've", "i'll", 'you', "you're", 'he', 'she', 'it', "it's", 'we', "we're", + 'they', "they're", 'me', 'him', 'her', 'us', 'them', 'my', 'your', 'his', 'their', 'our', + 'mine', 'yours', 'hers', 'theirs', 'ours', 'this', 'that', 'these', 'those', 'who', 'whom', + // articles / determiners / conjunctions / prepositions (common openers) + 'the', 'a', 'an', 'and', 'or', 'but', 'so', 'if', 'as', 'at', 'by', 'for', 'in', 'of', + 'on', 'to', 'up', 'with', 'from', 'into', 'over', 'than', 'then', 'also', 'just', + // question words / auxiliaries + 'what', 'when', 'where', 'why', 'how', 'which', 'whose', + 'can', 'could', 'should', 'would', 'will', 'shall', 'may', 'might', 'must', + 'do', 'does', 'did', 'is', 'are', 'am', 'was', 'were', 'be', 'been', 'being', + 'has', 'have', 'had', + // greetings / discourse markers / polite openers + 'hi', 'hey', 'hello', 'thanks', 'thank', 'please', 'yes', 'no', 'ok', 'okay', 'sure', + 'maybe', 'well', 'oh', 'let', "let's", 'lets', +]); + +/** + * SOFT common words — frequent non-entity words that DO get capitalized at + * sentence start. Dropped only when a single-token candidate appears solely at + * sentence start (and is never seen capitalized mid-sentence, which would be a + * strong name signal). Weekdays/months/time words live here. Compared lowercase. + */ +const COMMON_WORDS = new Set([ + 'monday', 'tuesday', 'wednesday', 'thursday', 'friday', 'saturday', 'sunday', + 'january', 'february', 'march', 'april', 'may', 'june', 'july', 'august', + 'september', 'october', 'november', 'december', + 'today', 'tomorrow', 'yesterday', 'now', 'soon', 'later', 'tonight', 'morning', + 'afternoon', 'evening', 'week', 'month', 'year', 'meeting', 'call', 'note', 'task', + 'here', 'there', 'every', 'some', 'any', 'all', 'one', 'two', 'three', 'first', 'last', + 'next', 'new', 'old', 'good', 'bad', 'great', 'nice', 'thing', 'something', 'anything', +]); + +const HANDLE_RE = /@([A-Za-z0-9_]{2,})/g; +// Capitalized token runs: an uppercase-initial word, up to 4 tokens total. +// A token allows internal letters/digits/apostrophes/hyphens, plus internal +// dots ONLY when followed by a letter (so "U.S." keeps its dot but a +// sentence-ending "Apple." does NOT glue into the next sentence's word). +const CAP_TOKEN = `\\p{Lu}[\\p{L}0-9'’\\-]*(?:\\.\\p{L}[\\p{L}0-9'’\\-]*)*`; +const CAP_RUN_RE = new RegExp(`${CAP_TOKEN}(?:\\s+${CAP_TOKEN}){0,3}`, 'gu'); + +/** Strip a trailing possessive ("Garry's" → "Garry", "Jones’" → "Jones"). */ +function stripPossessive(s: string): string { + return s.replace(/['’]s$/i, '').replace(/['’]$/i, ''); +} + +/** True when the match at `idx` is the first non-space char of the text or a sentence. */ +function isAtSentenceStart(text: string, idx: number): boolean { + let i = idx - 1; + // skip immediate whitespace + while (i >= 0 && /\s/.test(text[i])) i--; + if (i < 0) return true; // start of text + // sentence-ending punctuation (or a list bullet / opening bracket) precedes + return /[.!?:;\n\r•\-(["“]/.test(text[i]); +} + +function isPureNumber(s: string): boolean { + return /^[0-9][0-9.,]*$/.test(s); +} + +/** + * Extract candidate entity surface-forms from one turn's text. + * Deterministic, precision-biased, capped at MAX_CANDIDATES. Deduped on the + * normalizeAlias() form (so "Garry" and "garry" collapse), first display wins. + */ +export function extractCandidates(text: string): EntityCandidate[] { + if (!text || typeof text !== 'string') return []; + + // Track, per normalized query, its display + whether it was ever seen + // capitalized mid-sentence (a strong "this is a real name" signal) and how + // many tokens it spans. + interface Acc { + display: string; + query: string; + multiToken: boolean; + seenMidSentence: boolean; + order: number; + } + const acc = new Map(); + let order = 0; + + const consider = (rawDisplay: string, rawQuery: string, midSentence: boolean) => { + const display = rawDisplay.trim(); + const query = stripPossessive(rawQuery.trim()); + if (!query) return; + const norm = normalizeAlias(query); + if (!norm) return; + const existing = acc.get(norm); + if (existing) { + if (midSentence) existing.seenMidSentence = true; + return; + } + acc.set(norm, { + display, + query, + multiToken: /\s/.test(query), + seenMidSentence: midSentence, + order: order++, + }); + }; + + // 1. @handles — strong signal; resolved as aliases. Display keeps the @. + for (const m of text.matchAll(HANDLE_RE)) { + const handle = m[1]; + // handles are intentional references; treat as mid-sentence (never drop on + // the sentence-start heuristic). + consider(`@${handle}`, handle, true); + } + + // 2. Capitalized token runs. + for (const m of text.matchAll(CAP_RUN_RE)) { + const surface = m[0]; + const idx = m.index ?? 0; + consider(surface, surface, !isAtSentenceStart(text, idx)); + } + + // 3. Filter for precision. + const out: EntityCandidate[] = []; + for (const c of Array.from(acc.values()).sort((a, b) => a.order - b.order)) { + const lc = c.query.toLowerCase(); + // Single bare tokens get the strict filters; multi-token runs ("Garry Tan", + // "Initialized Capital") are inherently high-signal and skip the soft list. + if (!c.multiToken) { + if (c.query.length < 2) continue; // single char + if (isPureNumber(c.query)) continue; // "2026" + if (STOPWORDS.has(lc)) continue; // hard: never an entity + // soft: common word AND only seen at sentence start → drop. If it also + // appeared capitalized mid-sentence, keep it (likely a real name like + // "Apple" or a person whose name collides with a common word). + if (COMMON_WORDS.has(lc) && !c.seenMidSentence) continue; + } + out.push({ display: c.display, query: c.query }); + if (out.length >= MAX_CANDIDATES) break; + } + return out; +} diff --git a/src/core/context/reflex.ts b/src/core/context/reflex.ts new file mode 100644 index 000000000..afec4054a --- /dev/null +++ b/src/core/context/reflex.ts @@ -0,0 +1,195 @@ +/** + * Retrieval Reflex — per-turn orchestrator (issue #1981, Layer 1). + * + * Glues the pure extractor to the engine-aware resolver ladder and returns the + * pointer markdown to append to `systemPromptAddition`. Called from the context + * engine's `assemble()` on every turn, so it is: + * - zero-candidate fast path: no brain touched when nothing is salient + * - fully fail-open: any error returns null (the turn never breaks) + * - time-bounded: a hard timeout caps the per-turn cost + * + * Resolver ladder (engine-aware — see plan D1/D9): + * 1. host-injected resolveEntities (ctx.brainQuery) — any engine + * 2. PGLite → serve resolve IPC socket — through the lock holder + * 3. Postgres → cached direct connection — multi-connection, safe + * 4. else → disabled (policy skill carries; doctor reports it) + */ + +import { homedir } from 'node:os'; +import { join } from 'node:path'; +import { mkdirSync, appendFileSync } from 'node:fs'; +import { loadConfig, type GBrainConfig } from '../config.ts'; +import type { BrainEngine } from '../engine.ts'; +import { extractCandidates, type EntityCandidate } from './entity-salience.ts'; +import { + resolveEntitiesToPointers, + DEFAULT_MAX_POINTERS, + type PointerBlock, +} from './retrieval-reflex.ts'; +import { resolveViaIpc, resolveSocketPath, IPC_UNAVAILABLE } from './resolve-ipc.ts'; + +/** Host capability shape (D1=A): candidates in, pointers out. Narrow by design. */ +export type ResolveEntitiesFn = ( + candidates: EntityCandidate[], + opts: { priorContextText?: string; maxPointers?: number }, +) => Promise; + +export interface ReflexParams { + workspaceDir: string; + /** The current turn's user text (drives extraction). */ + currentUserText: string; + /** Joined PRIOR turns + loaded page bodies — EXCLUDES the current turn (suppression). */ + priorContextText: string; + /** Host-provided resolver, if the OpenClaw plugin contract supplied one. */ + resolveEntities?: ResolveEntitiesFn; +} + +const TIMEOUT_MS = 1500; // generous per-turn ceiling; the work is usually <100ms +const HEARTBEAT_PATH = join(homedir(), '.gbrain', 'integrations', 'retrieval-reflex', 'heartbeat.jsonl'); + +/** File-plane + env gate. Default ON. DB-plane does NOT gate (assemble() is sync). */ +export function reflexEnabled(cfg: GBrainConfig | null): boolean { + const env = process.env.GBRAIN_RETRIEVAL_REFLEX; + if (env != null && env !== '') return !(env === 'false' || env === '0'); + return cfg?.retrieval_reflex !== false; +} + +function maxPointers(cfg: GBrainConfig | null): number { + const n = cfg?.retrieval_reflex_max_pointers; + return typeof n === 'number' && n > 0 ? n : DEFAULT_MAX_POINTERS; +} + +/** + * Build the pointer-block markdown for this turn, or null to inject nothing. + * Never throws. + */ +export async function buildReflexAddition(params: ReflexParams): Promise { + try { + const cfg = loadConfig(); + if (!reflexEnabled(cfg)) return null; + + // Zero-candidate fast path: one regex pass, no brain touch. + const candidates = extractCandidates(params.currentUserText); + if (!candidates.length) return null; + + const opts = { priorContextText: params.priorContextText, maxPointers: maxPointers(cfg) }; + const block = await withTimeout(resolve(params, cfg, candidates, opts), TIMEOUT_MS); + if (!block || !block.pointers.length) return null; + + writeHeartbeat(cfg, block.pointers.length); + return block.text; + } catch { + return null; // fail-open: the live-context block still ships + } +} + +async function resolve( + params: ReflexParams, + cfg: GBrainConfig | null, + candidates: EntityCandidate[], + opts: { priorContextText?: string; maxPointers?: number }, +): Promise { + // 1. Host capability (any engine). + if (params.resolveEntities) { + return params.resolveEntities(candidates, opts); + } + // 2. PGLite → serve resolve IPC. + if (cfg?.engine === 'pglite' && cfg.database_path) { + const sock = resolveSocketPath(cfg.database_path); + const r = await resolveViaIpc(sock, { candidates, ...opts }); + return r === IPC_UNAVAILABLE ? null : r; + } + // 3. Postgres → cached direct connection. + if (isPostgres(cfg)) { + const engine = await getPostgresEngine(cfg); + if (!engine) return null; + const { resolveSourceId } = await import('../source-resolver.ts'); + const sourceId = await resolveSourceId(engine, null, params.workspaceDir); + return resolveEntitiesToPointers(engine, sourceId, candidates, opts); + } + // 4. Disabled (PGLite with no serve / unknown engine). Policy skill carries. + return null; +} + +function isPostgres(cfg: GBrainConfig | null): boolean { + if (cfg?.engine === 'postgres') return true; + // engine unset but a database_url present → postgres (createEngine default). + return !cfg?.engine && !!cfg?.database_url; +} + +// ── Postgres process-singleton ────────────────────────────────────────── +// One connection per process, reused across sessions/turns. Avoids the +// connection-multiplication a per-session open would cause (Codex finding). +let _pgEngine: BrainEngine | null = null; +let _pgPending: Promise | null = null; +let _pgFailedUntil = 0; // cooldown so a transient connect failure doesn't storm + +async function getPostgresEngine(cfg: GBrainConfig | null): Promise { + if (_pgEngine) return _pgEngine; + if (Date.now() < _pgFailedUntil) return null; + if (_pgPending) return _pgPending; + _pgPending = (async () => { + try { + const { createEngine } = await import('../engine-factory.ts'); + const engineConfig = { + engine: 'postgres' as const, + database_url: cfg?.database_url, + database_path: cfg?.database_path, + }; + const engine = await createEngine(engineConfig); + await engine.connect(engineConfig); + _pgEngine = engine; + return engine; + } catch { + _pgFailedUntil = Date.now() + 60_000; // 60s cooldown + return null; + } finally { + _pgPending = null; + } + })(); + return _pgPending; +} + +/** + * Warm the Postgres connection ahead of the first salient turn (called by the + * context-engine factory). No-op for PGLite/host paths. Fire-and-forget. + */ +export function warmReflex(): void { + try { + const cfg = loadConfig(); + if (reflexEnabled(cfg) && isPostgres(cfg)) void getPostgresEngine(cfg); + } catch { + /* best effort */ + } +} + +/** Dispose the cached Postgres connection (tests + clean shutdown). */ +export async function disposeReflex(): Promise { + const e = _pgEngine; + _pgEngine = null; + _pgPending = null; + _pgFailedUntil = 0; + if (e) { + try { await e.disconnect(); } catch { /* noop */ } + } +} + +function writeHeartbeat(cfg: GBrainConfig | null, count: number): void { + try { + mkdirSync(join(homedir(), '.gbrain', 'integrations', 'retrieval-reflex'), { recursive: true }); + const engine = cfg?.engine ?? 'unknown'; + appendFileSync( + HEARTBEAT_PATH, + JSON.stringify({ ts: new Date().toISOString(), event: 'inject', pointers: count, engine }) + '\n', + ); + } catch { + /* heartbeat is advisory; never block the turn */ + } +} + +async function withTimeout(p: Promise, ms: number): Promise { + return Promise.race([ + p, + new Promise((r) => setTimeout(() => r(null), ms)), + ]); +} diff --git a/src/core/context/resolve-ipc.ts b/src/core/context/resolve-ipc.ts new file mode 100644 index 000000000..3955ea3e1 --- /dev/null +++ b/src/core/context/resolve-ipc.ts @@ -0,0 +1,148 @@ +/** + * Retrieval Reflex — resolve IPC (issue #1981, D9=C). + * + * PGLite is single-connection: `gbrain serve` holds the one connection for its + * lifetime, so the context engine cannot open its own and must NOT shell out to + * a subprocess (that would force-steal the lock past the 5-min staleness window + * and crash the brain — see plan D9 rejected option). Instead, `serve` + * optionally listens on a local unix-domain socket and answers a NARROW request + * — candidates in, pointers out — using the connection it already owns. Both + * ends are gbrain code; raw SQL never crosses the wire (closes the trust hole). + * + * Protocol: newline-delimited JSON. One request line, one response line. + * req: { candidates, priorContextText?, maxPointers?, sourceId? } + * resp: { ok: true, block: PointerBlock | null } | { ok: false, error } + * + * Local-only (unix socket on the brain's data dir, mode 0600) — no network + * surface. + */ + +import net from 'node:net'; +import { existsSync, unlinkSync, statSync, chmodSync } from 'node:fs'; +import { join } from 'node:path'; +import type { EntityCandidate } from './entity-salience.ts'; +import type { PointerBlock } from './retrieval-reflex.ts'; + +const SOCK_NAME = '.gbrain-resolve.sock'; +const CLIENT_TIMEOUT_MS = 250; +const MAX_MSG_BYTES = 256 * 1024; + +/** Marker the client returns when no server is reachable (vs. a real null result). */ +export const IPC_UNAVAILABLE = Symbol('ipc-unavailable'); + +export interface ResolveRequest { + candidates: EntityCandidate[]; + priorContextText?: string; + maxPointers?: number; + sourceId?: string; +} + +export type ResolveHandler = (req: ResolveRequest) => Promise; + +/** Canonical socket path for a PGLite data dir. */ +export function resolveSocketPath(dataDir: string): string { + return join(dataDir, SOCK_NAME); +} + +/** + * Client: ship candidates to a running serve, get pointers back. Returns + * IPC_UNAVAILABLE when no server is listening (caller falls through the ladder); + * a real PointerBlock | null otherwise. Never throws — fail-soft to UNAVAILABLE. + */ +export async function resolveViaIpc( + socketPath: string, + req: ResolveRequest, +): Promise { + if (!existsSync(socketPath)) return IPC_UNAVAILABLE; + return new Promise((resolve) => { + let settled = false; + let buf = ''; + const finish = (v: PointerBlock | null | typeof IPC_UNAVAILABLE) => { + if (settled) return; + settled = true; + try { sock.destroy(); } catch { /* noop */ } + resolve(v); + }; + const sock = net.createConnection(socketPath); + sock.setTimeout(CLIENT_TIMEOUT_MS); + sock.on('connect', () => { + sock.write(JSON.stringify(req) + '\n'); + }); + sock.on('data', (chunk) => { + buf += chunk.toString('utf8'); + if (buf.length > MAX_MSG_BYTES) return finish(IPC_UNAVAILABLE); + const nl = buf.indexOf('\n'); + if (nl < 0) return; + try { + const resp = JSON.parse(buf.slice(0, nl)); + if (resp && resp.ok) return finish(resp.block ?? null); + return finish(IPC_UNAVAILABLE); + } catch { + return finish(IPC_UNAVAILABLE); + } + }); + // Any error (ENOENT, ECONNREFUSED, stale socket), timeout, or close before + // a response → treat as unavailable, fall through the ladder. + sock.on('timeout', () => finish(IPC_UNAVAILABLE)); + sock.on('error', () => finish(IPC_UNAVAILABLE)); + sock.on('close', () => finish(IPC_UNAVAILABLE)); + }); +} + +/** + * Server: start a resolve listener on `socketPath`. Cleans up a stale socket + * left by a dead owner first. Returns the net.Server (caller closes on + * shutdown). Errors are swallowed (best-effort feature) — returns null if the + * socket can't be bound. + */ +export async function startResolveIpcServer( + socketPath: string, + handler: ResolveHandler, +): Promise { + // Remove a stale socket file if present (a previous serve that didn't clean up). + cleanupStaleSocket(socketPath); + + return new Promise((resolve) => { + const server = net.createServer((conn) => { + let buf = ''; + conn.setEncoding('utf8'); + conn.on('data', async (chunk: string) => { + buf += chunk; + if (buf.length > MAX_MSG_BYTES) { conn.destroy(); return; } + const nl = buf.indexOf('\n'); + if (nl < 0) return; + const line = buf.slice(0, nl); + let resp: string; + try { + const req = JSON.parse(line) as ResolveRequest; + const block = await handler(req); + resp = JSON.stringify({ ok: true, block }); + } catch (e) { + resp = JSON.stringify({ ok: false, error: (e as Error).message }); + } + try { conn.write(resp + '\n'); } catch { /* client gone */ } + conn.end(); + }); + conn.on('error', () => { try { conn.destroy(); } catch { /* noop */ } }); + }); + server.on('error', () => resolve(null)); + server.listen(socketPath, () => { + try { chmodSync(socketPath, 0o600); } catch { /* best effort */ } + resolve(server); + }); + }); +} + +/** Remove a socket file whose owning process is gone (or any leftover file). */ +export function cleanupStaleSocket(socketPath: string): void { + try { + if (existsSync(socketPath)) { + // A unix socket shows up as a socket file; unlink unconditionally — if a + // live server holds it, listen() below would fail and we return null. + const st = statSync(socketPath); + if (st.isSocket() || st.isFIFO() || st.isFile()) unlinkSync(socketPath); + } + } catch { + /* best effort */ + } +} diff --git a/src/core/context/retrieval-reflex.ts b/src/core/context/retrieval-reflex.ts new file mode 100644 index 000000000..d1cb8e3af --- /dev/null +++ b/src/core/context/retrieval-reflex.ts @@ -0,0 +1,248 @@ +/** + * Retrieval Reflex — resolver core (issue #1981, Layer 1). + * + * Given salient candidate surface-forms (from entity-salience.ts) and a real + * BrainEngine, resolve the ones that map to an EXISTING brain page and return a + * compact POINTER block (name → slug → safe one-line synopsis) to inject into + * the system prompt. Detect + point, NEVER auto-dump the body — the agent makes + * the deliberate get_page call when the entity is actually the subject. + * + * This is the NARROW operation behind the resolve capability: it takes + * candidates and returns pointers. The IPC (serve) and host (ctx.brainQuery) + * paths run THIS SAME function server-side so no raw SQL ever crosses a trust + * boundary — the wire only carries candidates in and pointers out. + * + * Deterministic, zero-LLM. Precision-biased resolution (no trgm-fuzzy): + * 1. alias-first — page_aliases exact (unambiguous single-slug only) + * 2. title + slug-suffix — lower(title) exact OR slug suffix match + * (real slugs are namespaced people/alice-example; bare slugify misses). + * + * Privacy (eng-review D5): the synopsis is taken from a SAFE source — + * frontmatter `summary` if present, else the page body with takes/private-fact + * fences STRIPPED (same boundary get_page applies to untrusted readers). Raw + * compiled_truth is never injected. + */ + +import type { BrainEngine } from '../engine.ts'; +import { normalizeAlias } from '../search/alias-normalize.ts'; +import { slugify } from '../entities/resolve.ts'; +import { stripTakesFence } from '../takes-fence.ts'; +import { stripFactsFence } from '../facts-fence.ts'; +import type { EntityCandidate } from './entity-salience.ts'; + +/** Default cap on pointers injected per turn (config: retrieval_reflex_max_pointers). */ +export const DEFAULT_MAX_POINTERS = 3; +const SYNOPSIS_MAX = 160; + +export interface ReflexPointer { + display: string; + slug: string; + synopsis: string; +} + +export interface PointerBlock { + pointers: ReflexPointer[]; + /** Pre-rendered markdown for systemPromptAddition. */ + text: string; +} + +export interface ResolvePointersOpts { + maxPointers?: number; + /** + * Joined text of PRIOR turns + already-loaded page bodies (NOT the current + * user message). Pointers whose slug/title already appear here are suppressed + * — the agent has seen them. MUST exclude the current turn, or the triggering + * message's own mention would suppress every pointer (eng-review/Codex fix). + */ + priorContextText?: string; +} + +interface PageRow { + slug: string; + title: string; + type: string | null; + frontmatter: Record | null; + compiled_truth: string | null; +} + +/** + * Resolve candidates to a pointer block. Returns null when nothing resolves + * (so the caller injects nothing). Never throws for data reasons — each arm is + * independently guarded so a pre-v110 brain (no page_aliases) still gets the + * title/slug arm. + */ +export async function resolveEntitiesToPointers( + engine: BrainEngine, + sourceId: string, + candidates: EntityCandidate[], + opts: ResolvePointersOpts = {}, +): Promise { + if (!candidates.length) return null; + const maxPointers = opts.maxPointers ?? DEFAULT_MAX_POINTERS; + const priorLc = (opts.priorContextText ?? '').toLowerCase(); + + // display lookup keyed by normalized query, so resolved slugs can recover a + // human surface form for the pointer label. + const displayByNorm = new Map(); + const aliasNorms: string[] = []; + const titlesLc: string[] = []; + const exactSlugs: string[] = []; + const slugSuffixes: string[] = []; + for (const c of candidates) { + const norm = normalizeAlias(c.query); + if (!norm) continue; + if (!displayByNorm.has(norm)) displayByNorm.set(norm, c.display); + aliasNorms.push(norm); + titlesLc.push(c.query.toLowerCase()); + const s = slugify(c.query); + if (s) { + exactSlugs.push(s); + slugSuffixes.push(`%/${s}`); + } + } + if (!aliasNorms.length) return null; + + // Ordered set of resolved slugs (alias hits first → higher confidence). + const resolvedSlugs: string[] = []; + const seen = new Set(); + const pushSlug = (slug: string) => { + if (slug && !seen.has(slug)) { + seen.add(slug); + resolvedSlugs.push(slug); + } + }; + + // Arm 1 — alias-first. Unambiguous single-slug hits only. Guarded: pre-v110 + // brains throw "relation page_aliases does not exist" — swallow and continue. + try { + const aliasMap = await engine.resolveAliases(aliasNorms, { sourceId }); + for (const norm of aliasNorms) { + const hits = aliasMap.get(norm); + if (hits && hits.length === 1) pushSlug(hits[0].slug); + } + } catch { + /* no page_aliases table (pre-v110) — degrade to the title/slug arm */ + } + + // Arm 2 — exact title OR slug-suffix. This is the recall fix: a bare "Alice + // Example" slugifies to alice-example, but the real page is people/alice-example, + // so a plain slug = ANY() misses. Match lower(title) exactly or the slug suffix. + let rows: PageRow[] = []; + try { + rows = await engine.executeRaw( + `SELECT slug, title, type, frontmatter, compiled_truth + FROM pages + WHERE deleted_at IS NULL + AND source_id = $1 + AND ( lower(title) = ANY($2::text[]) + OR slug = ANY($3::text[]) + OR slug LIKE ANY($4::text[]) )`, + [sourceId, titlesLc, exactSlugs, slugSuffixes], + ); + } catch { + rows = []; + } + // Hydrate alias-resolved slugs too (their bodies for the synopsis) if not in rows. + const rowBySlug = new Map(); + for (const r of rows) rowBySlug.set(r.slug, r); + const aliasOnly = resolvedSlugs.filter((s) => !rowBySlug.has(s)); + if (aliasOnly.length) { + try { + const extra = await engine.executeRaw( + `SELECT slug, title, type, frontmatter, compiled_truth + FROM pages + WHERE deleted_at IS NULL AND source_id = $1 AND slug = ANY($2::text[])`, + [sourceId, aliasOnly], + ); + for (const r of extra) rowBySlug.set(r.slug, r); + } catch { + /* ignore — alias slug may be stale */ + } + } + // Title/slug matches that weren't alias hits, appended after alias hits. + for (const r of rows) pushSlug(r.slug); + + // Build pointers in confidence order, applying suppression + cap. + const pointers: ReflexPointer[] = []; + for (const slug of resolvedSlugs) { + const row = rowBySlug.get(slug); + if (!row) continue; + // Suppression: already present in PRIOR context (slug or title). The current + // turn is deliberately excluded from priorContextText. + if (priorLc) { + const titleLc = (row.title ?? '').toLowerCase(); + if (priorLc.includes(slug.toLowerCase())) continue; + if (titleLc && wholeWordIncludes(priorLc, titleLc)) continue; + } + const display = displayForRow(row, displayByNorm); + const synopsis = safeSynopsis(row); + pointers.push({ display, slug, synopsis }); + if (pointers.length >= maxPointers) break; + } + + if (!pointers.length) return null; + return { pointers, text: renderPointerBlock(pointers) }; +} + +/** Recover a display label: prefer the matched candidate surface, else the page title. */ +function displayForRow(row: PageRow, displayByNorm: Map): string { + const byTitle = displayByNorm.get(normalizeAlias(row.title ?? '')); + if (byTitle) return byTitle; + // try the slug tail (people/alice-example → alice-example) + const tail = row.slug.includes('/') ? row.slug.slice(row.slug.lastIndexOf('/') + 1) : row.slug; + return row.title || tail; +} + +/** + * Privacy-safe synopsis (eng-review D5). Prefer a curated frontmatter `summary`; + * otherwise strip takes/private-fact fences from the body (the same boundary + * get_page applies to untrusted readers) and take the first sentence. Never + * returns raw compiled_truth. + */ +function safeSynopsis(row: PageRow): string { + const fmSummary = row.frontmatter?.summary; + if (typeof fmSummary === 'string' && fmSummary.trim()) { + return clip(collapse(fmSummary), SYNOPSIS_MAX); + } + const body = row.compiled_truth ?? ''; + if (!body) return ''; + const stripped = stripFactsFence(stripTakesFence(body), { keepVisibility: ['world'] }); + // Drop frontmatter block, markdown headings, and blank lines; first real prose line. + const firstProse = stripped + .replace(/^---[\s\S]*?---\s*/m, '') + .split('\n') + .map((l) => l.trim()) + .find((l) => l && !l.startsWith('#') && !l.startsWith('