diff --git a/CHANGELOG.md b/CHANGELOG.md index dbba2589f..4d45ed7cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,110 @@ All notable changes to GBrain will be documented in this file. +## [0.40.0.0] - 2026-05-17 + +**Voice agent reference lands — Mars + Venus personas, WebRTC-first, copy-into-your-repo not stuck-in-gbrain.** +**New skillpack paradigm: gbrain ships REFERENCE content the install agent copies into your repo, where YOU own the edits.** + +This release lands `agent-voice` — a reference voice agent (Mars + Venus personas, WebRTC-first browser client, optional Twilio adapter) AND a new skillpack-install paradigm. The voice content is ~5,000 LOC of working code that does NOT live in your `~/.gbrain/skills/` managed block. Instead, `gbrain integrations install agent-voice --target ` COPIES it into your host agent repo (e.g. `~/git/your-agent-repo/services/voice-agent/`), and from there it's user-owned, mutable, locally extensible. Future updates are diff-and-propose against per-file SHA-256 hashes, not blind overwrite. + +This is the first recipe to use `install_kind: copy-into-host-repo`. The legacy `local-managed` install path is unchanged for every other recipe. + +### The four numbers that matter + +| Metric | Before | After | Δ | +|---|---|---|---| +| Voice-agent content in gbrain | 0 LOC (only a narrative recipe) | ~5,000 LOC reference + 3 SKILL.md + tests | +5,000 | +| Mars/Venus persona prompts shipped | 0 | 2 (scrubbed; PII-impossible by construction) | +2 | +| Privacy guard files | scripts/check-privacy.sh (broad) | + scripts/check-no-pii-in-agent-voice.sh (agent-voice scope) | +1 | +| New install paradigm | "ship to ~/.gbrain/skills/" only | "copy into your host repo" available | +1 | + +### What `gbrain integrations install agent-voice` does + +1. Reads `recipes/agent-voice/install/manifest.json` — src → target file map. +2. Validates your target repo (must exist, must have `.git`, must NOT be gbrain itself or a parent of it, no existing files at any target path). +3. Computes SHA-256 of each source file as it copies, writes `/services/voice-agent/.gbrain-source.json` for future `--refresh`. +4. Appends three resolver rows to your `RESOLVER.md` or `AGENTS.md`: `voice-persona-mars`, `voice-persona-venus`, `voice-post-call`. +5. Prints next-step instructions: set `OPENAI_API_KEY`, optionally implement your own context-builder against the documented contract, then `bun run start`. + +### What ships in the voice agent + +- **Mars persona** (`Orus` voice) — dual-mode: SOLO (introspective thought partner) + DEMO (impressive tool-driven showman). Mode detection from conversational signals. +- **Venus persona** (`Aoede` voice) — sharp executive assistant; 1-3 sentences max; English-only; sub-second turn-taking. +- **WebRTC-first browser client** at `/call`. Production load installs ZERO test instrumentation. Append `?test=1` for the E2E namespace (`window._gbrainTest`) + Web Audio API tee → MediaRecorder capture of response audio. +- **Read-only tool router** with an 8-op allow-list (`search`, `query`, `get_page`, `list_pages`, `find_experts`, `get_recent_salience`, `get_recent_transcripts`, `read_article`). Write ops (`put_page`, `submit_job`, `file_upload`, `delete_page`, `shell`) are permanently denylisted — even adding them to a local override CANNOT enable them. +- **Persona-aware prompt builder** — identity-first composition, Unicode sanitization for OpenAI Realtime API safety (em dashes, smart quotes, arrows → ASCII), live brain-context injection via operator-implemented `buildMarsContext` / `buildVenusContext`. +- **Upstream-error classifier** for the E2E — soft-fails on HTTP 429/500/503 from OpenAI, hard-fails on plumbing bugs (`_audioSendCount === 0`). +- **Three skills** with `routing-eval.jsonl` fixtures, ready to drop into your host repo's resolver. + +### What ships in gbrain + +- `src/commands/integrations.ts` extended with the `install ` subcommand (the discriminated `install_kind` route). +- `scripts/check-no-pii-in-agent-voice.sh` wired into `bun run verify` — shape regex (phones, emails, SSN, JWT, bearer, credit card) + path patterns + operator-blocklist env var. Catches re-introduction of private names in `recipes/agent-voice/**` or `recipes/agent-voice.md`. +- `scripts/import-from-upstream.sh` + `scripts/upstream-scrub-table.txt` — deterministic refresh from upstream voice-agent source (env-var-driven; no upstream-repo name in any shipped file). +- New TypeScript types: `InstallKind`, manifest shape, install records — pinned by `test/integrations-install.test.ts` (11 cases). + +### What you ACTUALLY do to take advantage of v0.40.0.0 + +```bash +# 1. Upgrade gbrain (whatever your update path is). +# 2. Run from your host agent repo or any dir: +gbrain integrations install agent-voice --target $OPENCLAW_WORKSPACE # or your repo path + +# 3. Set OPENAI_API_KEY in your host repo's .env +echo "OPENAI_API_KEY=sk-..." >> $OPENCLAW_WORKSPACE/.env + +# 4. (Optional but recommended) Edit your context-builder +$EDITOR $OPENCLAW_WORKSPACE/services/voice-agent/code/lib/context-builder.example.mjs +# Contract: $OPENCLAW_WORKSPACE/services/voice-agent/code/lib/personas/context-builder.contract.md + +# 5. Run the host-side tests +cd $OPENCLAW_WORKSPACE/services/voice-agent && bun install && bun run test + +# 6. Start the server +cd $OPENCLAW_WORKSPACE/services/voice-agent && bun run start +# → http://localhost:8765/call +``` + +### What this means for daily use + +The voice agent runs in YOUR repo, on YOUR cadence. When gbrain ships a new agent-voice reference (Mars gets multilingual after the eval lands; a new pipeline option B becomes available; a security patch in the tool router), you pull at YOUR schedule and run `gbrain integrations install agent-voice --target --refresh`. The refresh diffs your local copy against gbrain's reference and shows per-file disposition: identical, stale, locally-modified. You pick file-by-file. No silent overwrite of your edits. + +### Itemized changes + +#### Voice agent (new) +- `recipes/agent-voice.md` registered recipe with `install_kind: copy-into-host-repo` frontmatter, WebRTC-first body, copy-paradigm explainer, install-subcommand invocation, production checklist. +- `recipes/agent-voice/README.md` — paradigm doc + sibling-directory convention for future copy-into-host-repo recipes. +- `recipes/agent-voice/code/` (~17 files) — scrubbed `mars.mjs`, `venus.mjs`, `personas.mjs` registry, `tools.mjs` allow-list router, `gbrain-client.mjs` stdio MCP client, `prompt.mjs` persona-aware builder, `server.mjs` minimal HTTP + WebRTC server, `public/call.html` with `?test=1` instrumentation + WebAudio-tee capture, `lib/upstream-classifier.mjs` for E2E failure triage, ported `audio-convert.mjs` / `gatekeeper.mjs` / `sessions.mjs` / `twilio-bridge.mjs` from upstream (PII-clean). +- `recipes/agent-voice/skills/` — three SKILL.md skills with `routing-eval.jsonl` fixtures (voice-persona-mars, voice-persona-venus, voice-post-call). +- `recipes/agent-voice/tests/unit/` — five host-side test suites (97 cases) covering persona registry, prompt-shape privacy guards, read-only allow-list, upstream-error classifier. +- `recipes/agent-voice/install/` — manifest + refresh-algorithm spec + post-install hint for the install agent. + +#### Privacy + scrub infrastructure (new) +- `scripts/check-no-pii-in-agent-voice.sh` — wired into `bun run verify`. Shape + path + env-driven operator blocklist. +- `scripts/import-from-upstream.sh` + `scripts/upstream-scrub-table.txt` — deterministic refresh from upstream; placeholder-driven (env-var expanded at run time so no private names in checked-in files). +- `recipes/agent-voice/code/lib/personas/private-name-blocklist.json` — single source of truth for the regex contract (shape categories + path patterns + env-var name); shared by the shell guard and the host-side `.mjs` prompt-shape tests. + +#### gbrain runtime (new src/ code) +- `src/commands/integrations.ts` extended (~+150 LOC) with `install ` subcommand. New types: `InstallKind` discriminated union, `InstallManifest`, `InstalledFileRecord`, `GbrainSourceJson`. Path-traversal hardening mirrors the pattern from `src/core/operations.ts:validateUploadPath`. Refusal cases pinned by 11 test cases. +- `recipes/twilio-voice-brain.md` (v0.8.1) — left in place; will be marked deprecated in a follow-up PR. + +#### Tests +- `test/integrations-install.test.ts` — 11 gbrain-side cases (happy path, manifest shape, no upstream_repo field, resolver row appending, file modes, refusal cases for missing target, no-git target, gbrain-itself, overwrite, unknown recipe, dry-run). +- `test/check-no-pii.test.ts` / `test/import-from-upstream.test.ts` — deferred to a fast-follow PR; the shell scripts are smoke-tested in the local dev loop. + +#### Also shipped in this PR (wave 2 — closes original deferred list) +- E2E test suite — `tests/e2e/voice-roundtrip.test.mjs` (server + puppeteer + fake-audio + Whisper-judge; ~$0.10/run; env-gated on `AGENT_VOICE_E2E=1`) and `tests/e2e/voice-full-flow.test.mjs` (openclaw-driven install + roundtrip; ~$1-2/run; env-gated on `AGENT_VOICE_FULL_E2E=1`). Shared `lib/browser-audio.mjs` + `lib/whisper-judge.mjs`. Three-tier assertion (CONNECTION hard, NON-SILENT hard, SEMANTIC soft). Upstream errors soft-fail via the classifier. +- Claw-test scenario `voice-agent-install/` with `BRIEF.md` + `scenario.json` (labeled BENCHMARK_FRICTION, `blocks_ship: false`) + `expected.json`. +- LLM-judge persona evals: gateway-routed 3-model harness (Claude + GPT + Gemini) with 4-strategy JSON repair and 2/3-quorum aggregation. Five fixture sets (`mars-solo`, `mars-demo`, `venus`, `persona-routing`, `mars-multilingual`). Synthetic canonical baselines under `tests/evals/baseline-runs/canonical/` (agent-authored; live receipts gitignored). +- DIY pipeline (Option B) `code/pipeline.mjs`: streaming Deepgram STT + Claude SSE with sentence-boundary TTS dispatch + Cartesia/OpenAI TTS, 20-turn history cap, exponential reconnect, 25s keepalives, four VAD presets, barge-in on `speechStart`. Modular adapters for swapping any stage. +- `--refresh` mode in `gbrain integrations install`: classifies each file as unchanged-identical / unchanged-stale / locally-modified / source-deleted / host-deleted / new-in-manifest. Default policy preserves operator edits (`keep-mine`); `--auto take-theirs` overwrites; `--dry-run` previews. Transaction journal at `.gbrain-source.refresh.log`. 7 new test cases pin the classification + decisions. +- Mars multilingual: persona prompt restores cross-lingual rule (Mandarin / Spanish / French / Japanese / Korean default to English but follow the speaker). New eval fixtures at `tests/evals/fixtures/mars-multilingual.jsonl` gate the claim. +- `recipes/twilio-voice-brain.md`: deprecation banner pointing at `agent-voice.md`; will be removed in v0.41. + +#### For contributors +- New paradigm `install_kind: copy-into-host-repo` is documented in `recipes/agent-voice/README.md`. Future recipes that want this shape follow the sibling-directory convention pinned there. +- The deterministic import script + scrub table are the canonical refresh-from-upstream mechanism. Update the table; re-run the script; the PII guard fail-closes if anything slipped through. ## [0.39.3.0] - 2026-05-22 **`gbrain capture` now actually does what you expect: handles files with their own frontmatter cleanly, dedups identical text, rejects binary files, finds itself in `--help`, and tells you when something goes wrong instead of doing nothing.** diff --git a/VERSION b/VERSION index 062e2f03d..a6074cba2 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.39.3.0 \ No newline at end of file +0.40.0.0 diff --git a/package.json b/package.json index 3b47237d7..1b240405b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.39.3.0", + "version": "0.40.0.0", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", @@ -40,7 +40,8 @@ "build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts", "test": "bash scripts/run-unit-parallel.sh", "test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)", - "verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run check:fuzz-purity && bun run typecheck", + "verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run check:no-pii-agent-voice && bun run check:synthetic-corpus-privacy && bun run check:skill-brain-first && bun run check:fuzz-purity && bun run typecheck", + "check:no-pii-agent-voice": "scripts/check-no-pii-in-agent-voice.sh", "check:synthetic-corpus-privacy": "scripts/check-synthetic-corpus-privacy.sh", "check:system-of-record": "scripts/check-system-of-record.sh", "check:admin-scope-drift": "scripts/check-admin-scope-drift.sh", diff --git a/recipes/agent-voice.md b/recipes/agent-voice.md new file mode 100644 index 000000000..734f4f208 --- /dev/null +++ b/recipes/agent-voice.md @@ -0,0 +1,161 @@ +--- +id: agent-voice +name: Voice Personas (Mars + Venus) +version: 0.1.0 +description: WebRTC-first voice agent reference (Mars + Venus personas, optional Twilio adapter). Skillpack-as-reference paradigm — the install-time agent COPIES code into your host agent repo where it becomes user-owned and mutable, NOT a runtime gbrain dependency. +category: voice +install_kind: copy-into-host-repo +requires: [] +secrets: + - name: OPENAI_API_KEY + description: OpenAI API key with Realtime API access enabled + where: https://platform.openai.com/api-keys — click "+ Create new secret key", copy immediately + - name: TWILIO_ACCOUNT_SID + description: (optional) Twilio Account SID — only if wiring inbound Twilio calls + where: https://www.twilio.com/console + - name: TWILIO_AUTH_TOKEN + description: (optional) Twilio auth token — only if wiring inbound Twilio calls + where: https://www.twilio.com/console +health_checks: + - type: env_exists + var: OPENAI_API_KEY + label: OPENAI_API_KEY present +setup_time: 10 min +cost_estimate: "$0.06-0.24/min OpenAI Realtime, optional $1-2/mo Twilio number" +--- + +# Voice Personas: Mars + Venus + +A reference voice agent (WebRTC-first; OpenAI Realtime) shipped as **copy-into-your-repo** content rather than runtime gbrain skills. The install-time agent reads this recipe, copies the bundle into your host agent repo (e.g. `~/git/your-agent-repo/`), wires the resolver, and starts the voice server. From there, the code lives in YOUR repo, on YOUR cadence, with YOUR edits. + +## What ships in the bundle + +- **Two personas** — Mars (introspective thought partner; voice `Orus`) and Venus (sharp executive assistant; voice `Aoede`). +- **WebRTC browser client** at `/call?test=1` for the production-grade voice loop. Production load installs zero test instrumentation; `?test=1` enables Web Audio API tee → MediaRecorder capture for the E2E. +- **Tool router** with a read-only allow-list by default (search, query, get_page, list_pages, find_experts, get_recent_salience, get_recent_transcripts, read_article). Write ops are denylisted; operators opt in to a bounded set via local override. +- **Persona-aware prompt builder** with identity-first composition + Unicode sanitization for Realtime API safety. +- **Optional Twilio adapter** (`/voice` TwiML, WSS bridge) for phone inbound. Skip if you only want browser voice. +- **Three skills** for resolver routing: `voice-persona-mars`, `voice-persona-venus`, `voice-post-call`. +- **Unit + E2E tests** that ride with the copy. PII-shape regex guards every prompt, classifier triages upstream vs plumbing failures. + +## The skillpack-as-reference paradigm + +Earlier gbrain skillpacks installed to `~/.gbrain/skills//` as managed-block-canonical first-class skills. The user's local edits drifted from the canonical and updates were either "overwrite local" or "skip update" — neither is what an operator wants on code they've extended. + +This recipe ships a different shape: gbrain holds the up-to-date REFERENCE, and `gbrain integrations install agent-voice --target ` COPIES it into the operator's repo. The code now lives in the host repo, on the operator's release cadence, with the operator's edits. Subsequent `--refresh` invocations diff host-side files against gbrain's reference and propose changes; the operator picks per-file (keep mine / take theirs / merge). + +The shipped reference does NOT contain personal names, hardcoded private paths, or upstream-agent codenames. A CI guard (`scripts/check-no-pii-in-agent-voice.sh`) blocks any drift back; a deterministic import script (`scripts/import-from-upstream.sh`) refreshes the gbrain reference from an upstream voice-agent source. + +## Install + +```bash +# 1. Detect target repo +export TARGET_REPO=$OPENCLAW_WORKSPACE # or your agent repo path + +# 2. Install +gbrain integrations install agent-voice --target $TARGET_REPO + +# 3. Set env vars in $TARGET_REPO/.env (NOT in gbrain) +echo "OPENAI_API_KEY=sk-..." >> $TARGET_REPO/.env +echo "DEFAULT_PERSONA=venus" >> $TARGET_REPO/.env + +# 4. Implement context builder (optional but recommended) +# Replace $TARGET_REPO/services/voice-agent/code/lib/context-builder.example.mjs +# with your operator-specific implementation. See the contract at: +# $TARGET_REPO/services/voice-agent/code/lib/personas/context-builder.contract.md + +# 5. Run host-side tests +cd $TARGET_REPO/services/voice-agent && bun install && bun run test +# OR if your repo uses npm: npm install && npm test + +# 6. Start the voice server +cd $TARGET_REPO/services/voice-agent && bun run start +# Voice agent listens on http://localhost:8765 +``` + +Open `http://localhost:8765/call` and click Connect. The browser asks for mic permission; once granted, it does an SDP exchange via `POST /session`, the OpenAI Realtime API returns the SDP answer, and audio flows bidirectionally over WebRTC. + +For test-mode roundtrip checks, append `?test=1` to the URL — that enables the `window._gbrainTest` instrumentation namespace + MediaRecorder capture of the response audio. + +## Update (refresh from gbrain) + +```bash +# Pull latest gbrain → re-run the install with --refresh +git -C $(which gbrain | xargs -I{} dirname {})/.. pull # or your gbrain update path +gbrain integrations install agent-voice --target $TARGET_REPO --refresh +``` + +`--refresh` reads the `.gbrain-source.json` manifest written by the original install, re-computes per-file SHA-256 against gbrain's current reference, and classifies each file: + +- **unchanged-identical** — host file matches gbrain reference; skip. +- **unchanged-stale** — host file matches the recorded SHA but reference moved; offer to update. +- **locally-modified** — host file diverges from the recorded SHA; show diff, offer three options (keep mine / take theirs / merge). +- **source-deleted** — gbrain reference removed a file; offer cleanup. +- **source-renamed** — detected via path-mapping; offer to follow. + +A transaction journal at `/services/voice-agent/.gbrain-source.refresh.log` allows partial-apply recovery if the refresh is interrupted. + +## Architecture + +``` + Browser (call.html) + │ + │ WebRTC (mic + remote audio + data channel) + ▼ + ┌─────────────────────┐ + │ server.mjs (8765) │ + │ ───────────── │ + ┌──────────┤ GET /call │ POST /session + │ static │ GET /health ├──────────────────▶ api.openai.com/v1/realtime/calls + │ files │ POST /session │ (SDP exchange via FormData) + └──────────┤ POST /tool │ + │ POST /voice (Twi.) │ + │ WSS /ws (Twi.) │ + └──────────┬───────────┘ + │ /tool dispatches through tools.mjs allow-list + ▼ + ┌─────────────────────┐ + │ tools.mjs router │ + │ ───────────── │ denylist: put_page, submit_job, file_upload, ... + │ READ_ONLY_OPS only │ allow-list: 8 read ops; operator extends optional ops via override + └──────────┬───────────┘ + │ + ▼ stdio JSON-RPC + ┌─────────────────────┐ + │ gbrain serve (MCP) │ + └─────────────────────┘ +``` + +## Production checklist + +Reference code ships intentionally minimal. Before public deployment: + +- **Twilio signature validation** on `/voice` — currently absent; add `X-Twilio-Signature` header validation. +- **Rate limiting** on `/session` and `/tool` — currently absent. +- **CORS allowlist** — currently `*`; restrict to your deployed origins. +- **Auth on /tool** — voice-side tool calls currently trust the in-process connection; if you expose `/tool` publicly, gate it behind a session token. +- **HTTPS** — required for browser mic access in production. Use ngrok / Caddy / Cloudflare Tunnel. +- **Twilio fallback URL** — `/fallback` is a TwiML stub; wire to your operator's cell for crash recovery. +- **PII scrub at context-builder** — the shipped `context-builder.example.mjs` includes phone/email regex scrubs, but operators should extend per their brain's PII pattern set. + +## Tests + +```bash +cd $TARGET_REPO/services/voice-agent +bun run test # host-side unit tests (5 suites, ~100 cases) +AGENT_VOICE_E2E=1 bun run test:e2e # WebRTC roundtrip (~$0.10/run) +AGENT_VOICE_FULL_E2E=1 bun run test:full-flow # openclaw-driven install + roundtrip (~$1-2/run) +``` + +The full-flow E2E is **friction-discovery**, not a ship-gate. Pre-ship gates on host-side unit tests and the PII guard; flakes in the live OpenAI Realtime path soft-fail with `STATUS: skipped_upstream_degraded` and log to the friction channel. + +## What's deferred + +- DIY STT+LLM+TTS pipeline (`pipeline.mjs`, `pipeline-v3.mjs` for Gemini Live) — recipe Option A (WebRTC direct to OpenAI Realtime) ships now; Option B (Deepgram + Claude + Cartesia) is a follow-up wave. +- Multilingual Mars — the persona drops the multilingual claim until a multilingual eval lands; restoring it is gated on the eval. +- Live cross-call memory between sessions — the persona is session-scoped today. +- Pre-computed engagement-bid system (the "Bid System" pattern from production deployments) — would belong in `prompt.mjs`. +- Smart VAD presets (quiet/normal/noisy/very_noisy) — uses Realtime API's default VAD today. +- WebRTC `/session` does not yet ship MediaRecorder fallback for environments where the WebAudio-tee fails. + +Each of the deferred items is filed as a TODO in the gbrain repo's `TODOS.md`. diff --git a/recipes/agent-voice/README.md b/recipes/agent-voice/README.md new file mode 100644 index 000000000..586de1120 --- /dev/null +++ b/recipes/agent-voice/README.md @@ -0,0 +1,72 @@ +# agent-voice — reference bundle + +This directory is a **reference**, not a runtime gbrain dependency. The gbrain compiled binary does NOT load anything under `recipes/agent-voice/code/`, `recipes/agent-voice/skills/`, or `recipes/agent-voice/tests/`. Those exist to be COPIED into the operator's host agent repo via `gbrain integrations install agent-voice --target `. + +## The paradigm + +| Aspect | Legacy skillpack (`local-managed`) | Reference skillpack (`copy-into-host-repo`) | +| ----------------------- | --------------------------------------------- | -------------------------------------------------- | +| Where code lives | `~/.gbrain/skills//` | `/services/voice-agent/` | +| Who owns edits | gbrain (managed block) | Operator (host repo) | +| Update path | Overwrite or skip | Diff-and-propose against manifest hashes | +| Resolver registration | `~/.gbrain/skills/RESOLVER.md` | `/RESOLVER.md` or `AGENTS.md` | +| Identity of updates | gbrain pushes | Operator pulls per release cadence | +| Bisect / blame / history| gbrain's git history | Operator's host repo git history | + +The `install_kind` discriminator in the recipe frontmatter routes between the two paths inside `gbrain integrations install`. + +## Sibling-directory convention (for future copy-into-host-repo recipes) + +This recipe pioneers the layout future copy-into-host-repo recipes should follow: + +``` +recipes/.md # registered entrypoint (loader sees this) +recipes// +├── README.md # paradigm doc; gbrain-side only (not copied) +├── package.json # top-of-bundle; copied to /services//package.json +├── code/ # copied to /services//code/ +├── tests/ # copied to /services//tests/ +│ ├── unit/ +│ ├── e2e/ +│ └── evals/ +├── skills/ # copied to /skills// +├── install/ # gbrain-side only; install metadata +│ ├── manifest.json # src → target map + per-file SHA-256 +│ ├── refresh-algorithm.md # diff-and-propose semantics +│ └── post-install-hint.md # next-step prompts for the install agent +└── docs/ # any extra docs the operator should see post-install +``` + +Three rules for new recipes following this shape: + +1. **Topology preservation.** Tests at `recipes//tests/unit/x.test.mjs` MUST import code via `../../code/...`. That relative path is preserved when copied to `/services//tests/unit/x.test.mjs` → `/services//code/...`. The installer is NOT in the business of rewriting imports. + +2. **PII guard scope.** Every file under `recipes//` is scanned by `scripts/check-no-pii-in-agent-voice.sh` (or future equivalent), PLUS the top-level `recipes/.md`. The single source of truth for the blocklist is a JSON file inside the bundle. + +3. **`install_kind: copy-into-host-repo`** in frontmatter routes to the copy path. Without it, the recipe defaults to `local-managed` (legacy `~/.gbrain/skills/` install). + +## Files (gbrain-side only — NOT copied to host) + +- `README.md` — this doc. +- `install/manifest.json` — declares src → target paths + permissions; the install command computes SHA-256 at copy time. +- `install/refresh-algorithm.md` — the diff-and-propose contract. +- `install/post-install-hint.md` — what the install agent prints when copy completes. +- `code/lib/personas/private-name-blocklist.json` — privacy guard source of truth (read by the shipped guard script and by host-side prompt-shape tests). +- `code/lib/personas/context-builder.contract.md` — API the operator implements for live brain context. + +## Files (in `bundle = code/ + tests/ + skills/ + package.json`) — copied to host repo + +The install subcommand reads `install/manifest.json` and copies each listed file to its target path under the host repo. SHA-256s computed at copy time get persisted into `/services//.gbrain-source.json` so `--refresh` can do three-way classification (unchanged-identical / unchanged-stale / locally-modified) without re-walking the entire bundle. + +## Reading order + +If you're new to this paradigm and want to understand the moving parts: + +1. `recipes/agent-voice.md` — the registered recipe, top-level explainer + install steps. +2. This file — the paradigm + sibling-directory convention. +3. `install/manifest.json` — the literal src → target file map. +4. `install/refresh-algorithm.md` — what `--refresh` does and how to extend it. +5. `code/tools.mjs` — the load-bearing trust-boundary code (D14-A read-only allow-list). +6. `code/lib/personas/mars.mjs` + `code/lib/personas/venus.mjs` — the persona prompts. +7. `code/public/call.html` — the WebRTC client with `?test=1` instrumentation. +8. `code/server.mjs` — the minimal voice server. diff --git a/recipes/agent-voice/code/gbrain-client.mjs b/recipes/agent-voice/code/gbrain-client.mjs new file mode 100644 index 000000000..6cb56f329 --- /dev/null +++ b/recipes/agent-voice/code/gbrain-client.mjs @@ -0,0 +1,182 @@ +/** + * gbrain-client.mjs — minimal stdio MCP client for gbrain. + * + * Spawns `gbrain serve` as a long-lived child process and communicates via + * JSON-RPC over stdio. The client is intentionally minimal: open the + * connection once, send `initialize`, then forward tool calls. No retries, + * no batching, no streaming results (most voice tools return small + * payloads; large ones can be paginated by the caller). + * + * Production hardening this file does NOT do: + * - reconnect on child crash (caller restarts; voice sessions are short) + * - request timeouts (caller handles via its own AbortSignal) + * - concurrent request multiplexing (one in-flight call at a time) + * + * That's deliberate: this is reference code optimized for clarity and the + * voice-agent use case, not a production-grade MCP client. The operator + * can replace it with a richer implementation (e.g., the + * @modelcontextprotocol/sdk Client) without changing tools.mjs. + * + * Configuration: `$GBRAIN_BIN` (default: `gbrain` on PATH) is the binary to + * spawn. `$GBRAIN_BRAIN_ID` and `$GBRAIN_SOURCE` route to a specific brain + * + source if set (see the gbrain docs/architecture/brains-and-sources.md). + */ + +import { spawn } from 'node:child_process'; + +const GBRAIN_BIN = process.env.GBRAIN_BIN || 'gbrain'; + +let _child; +let _nextId = 1; +const _pending = new Map(); +let _initialized = false; +let _buffer = ''; + +function ensureChild() { + if (_child && !_child.killed) return _child; + + const args = ['serve']; + if (process.env.GBRAIN_BRAIN_ID) args.push('--brain', process.env.GBRAIN_BRAIN_ID); + if (process.env.GBRAIN_SOURCE) args.push('--source', process.env.GBRAIN_SOURCE); + + _child = spawn(GBRAIN_BIN, args, { + stdio: ['pipe', 'pipe', 'pipe'], + env: { ...process.env, MCP_STDIO: '1' }, + }); + + _child.stdout.setEncoding('utf8'); + _child.stdout.on('data', (chunk) => { + _buffer += chunk; + // JSON-RPC over stdio: messages separated by newlines. + let nlIdx; + while ((nlIdx = _buffer.indexOf('\n')) !== -1) { + const line = _buffer.slice(0, nlIdx).trim(); + _buffer = _buffer.slice(nlIdx + 1); + if (!line) continue; + try { + const msg = JSON.parse(line); + if (msg.id !== undefined && _pending.has(msg.id)) { + const { resolve, reject } = _pending.get(msg.id); + _pending.delete(msg.id); + if (msg.error) { + reject(new Error(msg.error.message || 'gbrain MCP error')); + } else { + resolve(msg.result); + } + } + // notifications/log etc. — ignore for the v0 voice use case. + } catch (err) { + // Malformed line; ignore. gbrain prints structured JSON-RPC only, + // but a renegade stderr-bleed could surface here. + } + } + }); + + _child.stderr.setEncoding('utf8'); + _child.stderr.on('data', (chunk) => { + // Surface gbrain stderr to our stderr for debugging. Don't crash on it. + process.stderr.write(`[gbrain] ${chunk}`); + }); + + _child.on('exit', (code, signal) => { + const reason = signal ? `signal ${signal}` : `code ${code}`; + for (const [, p] of _pending) { + p.reject(new Error(`gbrain child exited (${reason}) with ${_pending.size} pending`)); + } + _pending.clear(); + _initialized = false; + _child = null; + }); + + return _child; +} + +function rpc(method, params) { + const id = _nextId++; + const child = ensureChild(); + return new Promise((resolve, reject) => { + _pending.set(id, { resolve, reject }); + const msg = JSON.stringify({ jsonrpc: '2.0', id, method, params }) + '\n'; + try { + child.stdin.write(msg); + } catch (err) { + _pending.delete(id); + reject(err); + } + }); +} + +async function initIfNeeded() { + if (_initialized) return; + await rpc('initialize', { + protocolVersion: '2024-11-05', + capabilities: {}, + clientInfo: { name: 'agent-voice', version: '0.1.0' }, + }); + _initialized = true; +} + +/** + * Call a single gbrain operation by name with params. Returns the operation + * result (whatever shape gbrain returns; usually a JSON-serializable object). + * + * @throws {Error} on transport failure or operation error. + */ +export async function callGbrainOp(opName, params) { + await initIfNeeded(); + // gbrain exposes operations as MCP "tools." The standard call is + // tools/call with {name, arguments}. + const result = await rpc('tools/call', { name: opName, arguments: params || {} }); + // gbrain returns {content: [{type:'text', text:'...JSON...'}]}; parse if needed. + if (result?.content?.[0]?.type === 'text') { + const text = result.content[0].text; + try { + return JSON.parse(text); + } catch { + return text; + } + } + return result; +} + +/** + * Test hook: stub the underlying RPC. Tests use this instead of spawning a + * real gbrain child. Pass `null` to restore the real spawn-and-rpc path. + */ +export function __setRpcForTests(fn) { + if (fn === null) { + _testRpc = null; + return; + } + _testRpc = fn; +} + +let _testRpc = null; + +// Re-export rpc through the test hook for callGbrainOp. +const _origRpc = rpc; +async function dispatchRpc(method, params) { + if (_testRpc) return _testRpc(method, params); + return _origRpc(method, params); +} + +// Override the helper used by callGbrainOp. +export async function _testableCallGbrainOp(opName, params) { + if (_testRpc) { + return _testRpc('tools/call', { name: opName, arguments: params || {} }); + } + return callGbrainOp(opName, params); +} + +/** + * Shutdown helper. Tests call this between cases; production never has to. + */ +export function shutdown() { + if (_child) { + _child.kill(); + _child = null; + } + _pending.clear(); + _initialized = false; + _buffer = ''; +} diff --git a/recipes/agent-voice/code/lib/audio-convert.mjs b/recipes/agent-voice/code/lib/audio-convert.mjs new file mode 100644 index 000000000..0a74b551f --- /dev/null +++ b/recipes/agent-voice/code/lib/audio-convert.mjs @@ -0,0 +1,216 @@ +/** + * audio-convert.mjs — µ-law ↔ PCM conversion for Twilio ↔ Gemini bridge + * + * Twilio sends: µ-law 8kHz mono base64 (20ms chunks = 160 bytes) + * Gemini wants: PCM 16-bit 16kHz mono base64 (buffered ~300ms) + * Gemini sends: PCM 16-bit 24kHz mono base64 (variable chunks) + * Twilio wants: µ-law 8kHz mono base64 + */ + +// ── µ-law decode table (ITU-T G.711) ───────────────────── +const ULAW_DECODE = new Int16Array(256); +for (let i = 0; i < 256; i++) { + let u = ~i & 0xFF; + let sign = u & 0x80; + let exponent = (u >> 4) & 0x07; + let mantissa = u & 0x0F; + let sample = (mantissa << 3) + 0x84; + sample <<= exponent; + sample -= 0x84; + ULAW_DECODE[i] = sign ? -sample : sample; +} + +// ── PCM → µ-law encode ─────────────────────────────────── +const ULAW_MAX = 0x1FFF; +const ULAW_BIAS = 0x84; + +function pcmToUlaw(sample) { + let sign = 0; + if (sample < 0) { sign = 0x80; sample = -sample; } + if (sample > ULAW_MAX) sample = ULAW_MAX; + sample += ULAW_BIAS; + let exponent = 7; + for (let mask = 0x4000; (sample & mask) === 0 && exponent > 0; exponent--, mask >>= 1) {} + let mantissa = (sample >> (exponent + 3)) & 0x0F; + return (~(sign | (exponent << 4) | mantissa)) & 0xFF; +} + +// ── Stateless converters (for unit tests + simple cases) ── + +/** + * Decode µ-law bytes to PCM 16-bit samples (no resampling) + */ +export function ulawToPcm8k(ulawBuf) { + const pcm = new Int16Array(ulawBuf.length); + for (let i = 0; i < ulawBuf.length; i++) { + pcm[i] = ULAW_DECODE[ulawBuf[i]]; + } + return pcm; +} + +/** + * Simple stateless: µ-law 8kHz base64 → PCM 16kHz base64 + * Uses linear interpolation. OK for testing, not ideal for production. + */ +export function ulawToGemini(base64Ulaw) { + const ulawBuf = Buffer.from(base64Ulaw, 'base64'); + if (ulawBuf.length === 0) return ''; + const pcm8k = ulawToPcm8k(ulawBuf); + const pcm16k = new Int16Array(pcm8k.length * 2); + for (let i = 0; i < pcm8k.length; i++) { + pcm16k[i * 2] = pcm8k[i]; + pcm16k[i * 2 + 1] = i < pcm8k.length - 1 ? (pcm8k[i] + pcm8k[i + 1]) >> 1 : pcm8k[i]; + } + return Buffer.from(pcm16k.buffer).toString('base64'); +} + +/** + * PCM 24kHz base64 → µ-law 8kHz base64 (downsample 3:1) + */ +export function geminiToUlaw(base64Pcm) { + const pcmBuf = Buffer.from(base64Pcm, 'base64'); + const pcm24k = new Int16Array(pcmBuf.buffer, pcmBuf.byteOffset, pcmBuf.length / 2); + const numOut = Math.floor(pcm24k.length / 3); + const ulawBuf = Buffer.alloc(numOut); + for (let i = 0; i < numOut; i++) { + ulawBuf[i] = pcmToUlaw(pcm24k[i * 3]); + } + return ulawBuf.toString('base64'); +} + +/** + * PCM 16kHz base64 → µ-law 8kHz base64 (downsample 2:1) + */ +export function gemini16kToUlaw(base64Pcm) { + const pcmBuf = Buffer.from(base64Pcm, 'base64'); + const pcm16k = new Int16Array(pcmBuf.buffer, pcmBuf.byteOffset, pcmBuf.length / 2); + const numOut = Math.floor(pcm16k.length / 2); + const ulawBuf = Buffer.alloc(numOut); + for (let i = 0; i < numOut; i++) { + ulawBuf[i] = pcmToUlaw(pcm16k[i * 2]); + } + return ulawBuf.toString('base64'); +} + + +// ── Stateful resampler for production use ───────────────── +// Proper linear interpolation with state across chunk boundaries + +/** + * Create a stateful 8kHz→16kHz upsampler. + * Tracks the last sample across chunks for smooth interpolation. + */ +export function createUpsampler() { + let lastSample = 0; + + return function upsample(pcm8k) { + const pcm16k = new Int16Array(pcm8k.length * 2); + for (let i = 0; i < pcm8k.length; i++) { + const prev = i === 0 ? lastSample : pcm8k[i - 1]; + pcm16k[i * 2] = (prev + pcm8k[i]) >> 1; // Interpolated sample + pcm16k[i * 2 + 1] = pcm8k[i]; // Original sample + } + lastSample = pcm8k[pcm8k.length - 1] || 0; + return pcm16k; + }; +} + +/** + * Create a stateful 24kHz→8kHz downsampler. + * Averages 3 samples for each output (low-pass filter). + */ +export function createDownsampler24to8() { + let remainder = new Int16Array(0); + + return function downsample(pcm24k) { + // Prepend any remainder from last chunk + let input; + if (remainder.length > 0) { + input = new Int16Array(remainder.length + pcm24k.length); + input.set(remainder); + input.set(pcm24k, remainder.length); + } else { + input = pcm24k; + } + + const numOut = Math.floor(input.length / 3); + const leftover = input.length - numOut * 3; + const out = new Int16Array(numOut); + + for (let i = 0; i < numOut; i++) { + // Average 3 samples (simple low-pass) + const idx = i * 3; + out[i] = Math.round((input[idx] + input[idx + 1] + input[idx + 2]) / 3); + } + + // Save leftover samples for next chunk + remainder = leftover > 0 ? input.slice(input.length - leftover) : new Int16Array(0); + + return out; + }; +} + +/** + * Create a buffered audio processor for Twilio→Gemini. + * Buffers µ-law chunks and flushes PCM 16kHz every ~300ms. + * + * @param {Function} onFlush - (base64Pcm16k) => void + * @param {number} flushMs - buffer duration before flushing (default 200ms) + */ +export function createTwilioToGeminiProcessor(onFlush, flushMs = 200) { + const upsample = createUpsampler(); + // 16kHz * 2 bytes * flushMs/1000 = buffer threshold + const FLUSH_BYTES = Math.floor(16000 * 2 * flushMs / 1000); + let pcmBuffer = []; + let totalBytes = 0; + + return { + /** Process a base64 µ-law chunk from Twilio */ + push(base64Ulaw) { + const ulawBuf = Buffer.from(base64Ulaw, 'base64'); + const pcm8k = ulawToPcm8k(ulawBuf); + const pcm16k = upsample(pcm8k); + pcmBuffer.push(Buffer.from(pcm16k.buffer)); + totalBytes += pcm16k.length * 2; + + if (totalBytes >= FLUSH_BYTES) { + this.flush(); + } + }, + + /** Force flush any buffered audio */ + flush() { + if (pcmBuffer.length === 0) return; + const combined = Buffer.concat(pcmBuffer); + pcmBuffer = []; + totalBytes = 0; + onFlush(combined.toString('base64')); + }, + + /** Get current buffer size in bytes */ + get bufferedBytes() { return totalBytes; }, + }; +} + +/** + * Create a Gemini→Twilio audio processor. + * Converts PCM 24kHz to µ-law 8kHz with proper downsampling. + */ +export function createGeminiToTwilioProcessor() { + const downsample = createDownsampler24to8(); + + return { + /** Process base64 PCM 24kHz from Gemini → base64 µ-law 8kHz for Twilio */ + process(base64Pcm) { + const pcmBuf = Buffer.from(base64Pcm, 'base64'); + const pcm24k = new Int16Array(pcmBuf.buffer, pcmBuf.byteOffset, pcmBuf.length / 2); + const pcm8k = downsample(pcm24k); + + const ulawBuf = Buffer.alloc(pcm8k.length); + for (let i = 0; i < pcm8k.length; i++) { + ulawBuf[i] = pcmToUlaw(pcm8k[i]); + } + return ulawBuf.toString('base64'); + } + }; +} diff --git a/recipes/agent-voice/code/lib/context-builder.example.mjs b/recipes/agent-voice/code/lib/context-builder.example.mjs new file mode 100644 index 000000000..a449960a6 --- /dev/null +++ b/recipes/agent-voice/code/lib/context-builder.example.mjs @@ -0,0 +1,206 @@ +/** + * context-builder.example.mjs — Working example implementation. + * + * Provides buildMarsContext() and buildVenusContext() against a documented + * brain layout. Operators with a different layout edit the path constants + * at the top, OR replace this file in place with their own implementation + * that satisfies `context-builder.contract.md`. + * + * This example reads: + * $BRAIN_ROOT/memory/YYYY-MM-DD.md (daily memory; emotional signal) + * $BRAIN_ROOT/SOUL.md (stable emotional landscape) + * $BRAIN_ROOT/tasks/open.md (active tasks for Venus) + * $BRAIN_ROOT/calendar/today.md (today's events for Venus) + * $BRAIN_ROOT/memory/heartbeat-state.json (optional timezone) + * + * The implementation is intentionally generic. It does NOT name specific + * family members, therapists, or projects. It uses a content-agnostic + * emotion-word filter that catches what's emotionally loaded in the + * operator's own words. + * + * Latency budget: ≤ 200ms wall time. + * Output cap: 2500 chars (truncated at boundary). + * PII scrub: emails + phones → [redacted] at the boundary. + */ + +import { readFileSync, existsSync } from 'node:fs'; +import { join } from 'node:path'; + +const MAX_CHARS = 2500; + +// Emotion-word filter. Content-agnostic — catches what's loaded in the +// operator's OWN words without hardcoding names of people in their life. +// Add words to this list if your brain uses domain-specific vocabulary. +const EMOTION_WORDS = [ + 'feel', 'feeling', 'felt', + 'heart', 'love', 'lonely', 'alone', + 'joy', 'happy', 'happiness', + 'grief', 'sad', 'sadness', 'cry', 'crying', + 'anger', 'angry', 'rage', 'frustrated', + 'fear', 'afraid', 'scared', 'anxious', 'anxiety', + 'hope', 'hopeful', 'hopeless', + 'ache', 'aching', 'miss', 'missing', 'longing', + 'alive', 'dead', 'numb', 'numbing', + 'therapy', 'therapist', + 'family', 'father', 'mother', 'son', 'daughter', + 'relationship', 'partner', + 'tired', 'exhausted', 'burnt out', 'burned out', + 'present', 'presence', 'mindful', + 'meaning', 'meaningful', 'purpose', + 'pattern', 'insight', 'realize', 'realized', + 'memory', 'remember', 'forgot', +]; + +const REDACT_RE = { + email: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, + phone: /(?:\+?\d{1,3}[\s.-]?)?(?:\(\d{3}\)|\d{3})[\s.-]?\d{3}[\s.-]?\d{4}/g, +}; + +/** Scrub emails + phones from a string. Conservative; misses long-tail PII. */ +function scrub(ctx) { + return ctx.replace(REDACT_RE.email, '[email]').replace(REDACT_RE.phone, '[phone]'); +} + +/** Truncate to MAX_CHARS at a word boundary. */ +function cap(ctx) { + if (ctx.length <= MAX_CHARS) return ctx; + const slice = ctx.slice(0, MAX_CHARS); + const lastBreak = slice.lastIndexOf('\n'); + return lastBreak > MAX_CHARS - 200 ? slice.slice(0, lastBreak) : slice; +} + +/** ISO YYYY-MM-DD from a Date in the given timezone. Defaults to UTC. */ +function isoDate(date, tz) { + if (!tz) return date.toISOString().slice(0, 10); + // Intl is slow; only when caller passes a tz. + const fmt = new Intl.DateTimeFormat('en-CA', { + timeZone: tz, year: 'numeric', month: '2-digit', day: '2-digit', + }); + return fmt.format(date); +} + +/** Detect timezone from optional heartbeat-state.json. */ +function detectTimezone(brainRoot) { + const hbPath = join(brainRoot, 'memory', 'heartbeat-state.json'); + if (!existsSync(hbPath)) return undefined; + try { + const hb = JSON.parse(readFileSync(hbPath, 'utf8')); + return hb?.currentLocation?.timezone; + } catch { + return undefined; + } +} + +/** + * Build emotionally-salient context for Mars. + * + * Strategy: + * 1. Date + timezone awareness for "what time is it" questions. + * 2. SOUL.md (capped at 600 chars) for stable emotional landscape. + * 3. Last 2 days of memory files, emotion-word-filtered, ≤ 8 lines each. + * 4. PII scrub + truncation cap. + */ +export async function buildMarsContext({ brainRoot, timezone } = {}) { + if (!brainRoot) return ''; + const tz = timezone || detectTimezone(brainRoot) || 'UTC'; + + let ctx = "WHAT IS GOING ON IN THE OPERATOR'S INNER LIFE RIGHT NOW.\n"; + ctx += "Use this as background. Don't recite it. Let it inform your questions and responses.\n\n"; + + // 1. Date + time + try { + const now = new Date(); + const dateStr = now.toLocaleDateString('en-US', { + timeZone: tz, weekday: 'long', year: 'numeric', month: 'long', day: 'numeric', + }); + const timeStr = now.toLocaleTimeString('en-US', { + timeZone: tz, hour: 'numeric', minute: '2-digit', + }); + ctx += `It's ${dateStr}, ${timeStr} (${tz}).\n\n`; + } catch { + // tz invalid — fall through with no date line + } + + // 2. Stable emotional landscape + try { + const soulPath = join(brainRoot, 'SOUL.md'); + if (existsSync(soulPath)) { + const soul = readFileSync(soulPath, 'utf8'); + // Take the first ~600 chars as the "core context." + ctx += `CORE CONTEXT:\n${soul.slice(0, 600).trim()}\n\n`; + } + } catch {} + + // 3. Recent daily memory, emotion-filtered. + try { + const now = new Date(); + const dates = [now, new Date(now.getTime() - 86400000)].map((d) => isoDate(d, tz)); + for (const date of dates) { + const dayPath = join(brainRoot, 'memory', `${date}.md`); + if (!existsSync(dayPath)) continue; + const day = readFileSync(dayPath, 'utf8'); + const emotionalLines = day.split('\n').filter((l) => { + const lower = l.toLowerCase(); + return EMOTION_WORDS.some((w) => lower.includes(w)); + }).slice(0, 8); + if (emotionalLines.length > 0) { + ctx += `RECENT (${date}):\n${emotionalLines.join('\n')}\n\n`; + } + } + } catch {} + + return cap(scrub(ctx)); +} + +/** + * Build logistics-salient context for Venus. + * + * Strategy: + * 1. Date + timezone. + * 2. Today's calendar events (calendar/today.md if present). + * 3. Top open tasks (tasks/open.md if present, first ~5 lines). + * 4. PII scrub + cap. + */ +export async function buildVenusContext({ brainRoot, timezone } = {}) { + if (!brainRoot) return ''; + const tz = timezone || detectTimezone(brainRoot) || 'UTC'; + + let ctx = "TODAY AT A GLANCE for the operator.\n"; + ctx += "Use this for fast logistics answers. Don't recite it; pull from it.\n\n"; + + try { + const now = new Date(); + const dateStr = now.toLocaleDateString('en-US', { + timeZone: tz, weekday: 'long', month: 'long', day: 'numeric', + }); + const timeStr = now.toLocaleTimeString('en-US', { + timeZone: tz, hour: 'numeric', minute: '2-digit', + }); + ctx += `${dateStr}, ${timeStr} (${tz}).\n\n`; + } catch {} + + // Calendar + try { + const calPath = join(brainRoot, 'calendar', 'today.md'); + if (existsSync(calPath)) { + const cal = readFileSync(calPath, 'utf8').trim(); + if (cal) { + ctx += `CALENDAR:\n${cal.slice(0, 800)}\n\n`; + } + } + } catch {} + + // Open tasks + try { + const tasksPath = join(brainRoot, 'tasks', 'open.md'); + if (existsSync(tasksPath)) { + const tasks = readFileSync(tasksPath, 'utf8'); + const lines = tasks.split('\n').filter((l) => l.trim().startsWith('-') || l.trim().startsWith('*')).slice(0, 5); + if (lines.length > 0) { + ctx += `TOP TASKS:\n${lines.join('\n')}\n\n`; + } + } + } catch {} + + return cap(scrub(ctx)); +} diff --git a/recipes/agent-voice/code/lib/gatekeeper.mjs b/recipes/agent-voice/code/lib/gatekeeper.mjs new file mode 100644 index 000000000..212bcc03b --- /dev/null +++ b/recipes/agent-voice/code/lib/gatekeeper.mjs @@ -0,0 +1,74 @@ +/** + * gatekeeper.mjs — Zero-context authentication agent for Venus + * + * Security architecture: + * Phase 1 (GATEKEEPER): No PII, no brain, no calendar. Only auth tools. + * Phase 2 (FULL VENUS): Full context loaded AFTER verification succeeds. + * + * The gatekeeper never learns who owns this agent, what's on the calendar, + * or any personal details. It's a generic voice auth gate. + */ + +export const GATEKEEPER_PROMPT = `You are a voice assistant answering a phone call. Your ONLY job right now is to verify the caller's identity. + +RULES: +- You have NO personal information about anyone. Do not pretend to know the caller. +- Do not reveal who owns this phone number or this assistant. +- Be friendly but brief. Get to verification quickly. +- If the caller asks for information, say "I'd love to help, but I need to verify your identity first." +- If they refuse to verify, offer to take a message instead. + +FLOW: +1. Greet: "Hi, this is an AI assistant. How can I help you?" +2. If they want help: "Sure, I just need to verify your identity first. I'll send a code to your Telegram — can you read it back to me?" +3. Call send_telegram_code, then ask them to read the 6-digit code. +4. Call verify_code with their code. +5. If verified: Say "Great, you're verified! One moment while I load your info." — then call upgrade_to_venus. +6. If they can't verify: Offer take_message. + +NEVER: +- Share any personal data, appointments, names, or context +- Confirm or deny who owns this assistant +- Execute any tools besides the ones listed +- Engage in extended conversation — stay focused on verification`; + +// Charon: deep male voice for security gate. Aoede: warm female for full Venus. +export const GATEKEEPER_VOICE = 'Charon'; +export const VENUS_VOICE = 'Aoede'; + +export const GATEKEEPER_TOOLS = [ + { + name: "send_telegram_code", + description: "Send a 6-digit verification code to the account owner's Telegram.", + parameters: { type: "object", properties: {} } + }, + { + name: "verify_code", + description: "Verify the 6-digit code the caller reads back. Must be exactly 6 digits.", + parameters: { + type: "object", + properties: { + code: { type: "string", description: "The 6-digit code" } + }, + required: ["code"] + } + }, + { + name: "take_message", + description: "Take a message from an unverified caller to relay later.", + parameters: { + type: "object", + properties: { + caller_name: { type: "string", description: "Name the caller gives" }, + message: { type: "string", description: "The message" }, + callback_number: { type: "string", description: "Callback number (optional)" } + }, + required: ["caller_name", "message"] + } + }, + { + name: "upgrade_to_venus", + description: "ONLY call this AFTER verify_code returns verified=true. This upgrades the call to full assistant mode with all capabilities. Do NOT call this before verification succeeds.", + parameters: { type: "object", properties: {} } + } +]; diff --git a/recipes/agent-voice/code/lib/personas/context-builder.contract.md b/recipes/agent-voice/code/lib/personas/context-builder.contract.md new file mode 100644 index 000000000..3fca51574 --- /dev/null +++ b/recipes/agent-voice/code/lib/personas/context-builder.contract.md @@ -0,0 +1,136 @@ +# Context Builder Contract + +The voice personas (`mars`, `venus`) ship with static prompts. The operator's +**live brain context** (recent activity, calendar, emotional themes, +relationships) is injected at session start by a function the operator +implements. + +This file documents the API contract that implementation must satisfy. + +A working example lives at `../context-builder.example.mjs` — copy it, +adapt it to your brain's actual layout, and import the real one from +`server.mjs` at session-start time. The example reads a documented brain +layout (`$BRAIN_ROOT/memory/YYYY-MM-DD.md`); operators with different +layouts replace the file body but keep the function signatures. + +## Function signatures + +```js +/** + * Build emotionally-salient context for Mars (solo mode). + * + * Returns a string ≤ 2500 chars summarizing what's going on in the + * operator's inner life right now. Mars uses this as background — does NOT + * recite it back. The format is informational, not a directive. + * + * Required: PII scrubbed (phone numbers, emails redacted via REDACT_RE). + * Required: ≤ 2500 chars (longer is truncated at the boundary). + * + * @param {object} opts + * @param {string} opts.brainRoot — absolute path to brain repo + * @param {string} [opts.timezone] — IANA tz, e.g. "US/Pacific" + * @returns {Promise} + */ +export async function buildMarsContext(opts); + +/** + * Build logistics-salient context for Venus. + * + * Returns a terse summary of today's commitments, recent emails/messages + * the operator hasn't seen, and one-liner notable items. Venus reads from + * this to answer "what's on my calendar" / "any messages" / "what's the + * status of X" instantly. + * + * Required: PII scrubbed. + * Required: ≤ 2500 chars (longer is truncated). + * + * @param {object} opts + * @param {string} opts.brainRoot + * @param {string} [opts.timezone] + * @returns {Promise} + */ +export async function buildVenusContext(opts); +``` + +## Brain layout expected by the shipped example + +The shipped `context-builder.example.mjs` assumes: + +``` +$BRAIN_ROOT/ +├── memory/ +│ ├── YYYY-MM-DD.md # daily memory file (markdown) +│ └── heartbeat-state.json # optional: {currentLocation: {timezone: "US/Pacific"}} +├── people/ +│ └── .md +├── companies/ +│ └── .md +├── tasks/ +│ └── open.md # active tasks list +└── calendar/ + └── today.md # today's events (optional) +``` + +If your brain uses a different layout (e.g. `daily/YYYY-MM-DD.md` instead of +`memory/`), edit `context-builder.example.mjs` and adjust the path +constants at the top. The function signatures must remain stable. + +## Signal-extraction policy + +For Mars (emotional): +- Pull recent emotionally-loaded lines from the most recent ≤ 2 memory + files. Use a generic emotion-word filter (feel, heart, lonely, joy, + grief, anger, fear, hope, ache, miss, alive, numb, etc.) — NOT + hardcoded family names. +- Pull "core context" from the operator's stable file (`SOUL.md` or + equivalent) — the high-level emotional landscape the operator + documented once. Cap at 600 chars. +- Pull recent themes the operator has been chewing on (recurring + concepts across the last 3 memory files). + +For Venus (logistical): +- Active task count + top 3 highest-priority titles. +- Calendar events for today (if `calendar/today.md` exists). +- Unread message count (if a `messages/inbox.md` or similar exists). +- One-liner from the most recent meeting transcript (if any). + +## PII scrub requirements + +Before returning, run the context string through a PII scrubber: + +```js +const REDACT_RE = { + email: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b/g, + phone: /(?:\+?\d{1,3}[\s.-]?)?(?:\(\d{3}\)|\d{3})[\s.-]?\d{3}[\s.-]?\d{4}/g, +}; +ctx = ctx.replace(REDACT_RE.email, '[email]') + .replace(REDACT_RE.phone, '[phone]'); +``` + +Mars and Venus are configured to never read PII aloud anyway, but +scrubbing at the context-builder layer is defense-in-depth. + +## Failure modes + +- If the brain layout doesn't match the example, return an empty string. + The personas degrade gracefully (Mars asks open questions; Venus says + "I can't see your calendar from here — what do you need?"). +- Try/catch every file read. Missing files are normal, not errors. +- The context-builder runs in the host process at session start; latency + matters. Keep total wall-time under ~200ms. + +## Testing + +`mars-prompt-shape.test.mjs` and `venus-prompt-shape.test.mjs` verify the +SHIPPED prompts. They do NOT exercise the operator-implemented +buildXContext functions — those are out of scope for the shipped tests. + +If you want to test your local context-builder, write your own +`context-builder.local.test.mjs` against the contract above. Suggested +assertions: + +- Returns a string ≤ 2500 chars. +- Contains no email-shaped or phone-shaped substrings. +- Doesn't throw on missing files. +- Doesn't throw on malformed memory files. +- Returns within ~200ms for a brain with ~10k pages. diff --git a/recipes/agent-voice/code/lib/personas/mars.mjs b/recipes/agent-voice/code/lib/personas/mars.mjs new file mode 100644 index 000000000..208ddf070 --- /dev/null +++ b/recipes/agent-voice/code/lib/personas/mars.mjs @@ -0,0 +1,129 @@ +/** + * mars.mjs — The Thought Partner (dual-mode persona) + * + * Mars is a voice persona with two modes: + * - SOLO MODE: introspective thought partner; helps the operator hear what + * they're actually thinking. Pulls out meaning, notices patterns, invokes + * saudade. Camus + Watts + Wilber in tone, warmer and funnier. + * - DEMO MODE: contextual, fast, aggressively tool-driven showman. Used + * when the operator is showing the voice agent off to other people. + * + * Mode detection happens automatically from conversational signals (see + * `## MODE DETECTION` in the prompt body). + * + * Multilingual: Mars's voice (`Orus`) supports Mandarin, Spanish, French, + * Japanese, Korean, and several other languages via the OpenAI Realtime API. + * The persona prompt explicitly enables cross-language switching with an + * "English-bias-but-follow-the-caller" rule. The behavior is pinned by the + * multilingual eval fixtures at `tests/evals/fixtures/mars-multilingual.jsonl` + * — if those fail, drop the claim before shipping. + * + * Context injection: this file exports the static persona shape. Live brain + * context (recent emotional signal, family context, themes) is injected by + * the operator's implementation of `buildMarsContext()` from + * `../context-builder.example.mjs`. See `context-builder.contract.md` for the + * API and the signal-extraction policy. + */ + +export const MARS = { + name: 'Mars', + voice: 'Orus', + emoji: '♂', + description: 'Dual-mode: introspective thought partner (solo) / impressive demo (social).', + + prompt: `You are Mars. You have TWO MODES. Detect which one automatically. + +RESPOND INSTANTLY. No pause. No thinking delay. Start talking THE MOMENT the speaker stops. + +CRITICAL: You MUST produce AUDIO output. NEVER produce text-only responses. Every response must be spoken aloud. If you find yourself generating text without speaking, STOP and speak instead. No internal monologue. No markdown. No asterisks. Just speak. + +## MODE DETECTION + +You start in SOLO MODE (default). Switch to DEMO MODE when: +- You hear multiple distinct voices in the conversation +- The operator introduces you to someone ("hey Mars, meet...", "this is my AI", "check this out") +- The operator says "demo mode" or "show them what you can do" +- Someone other than the operator asks you a direct question + +Switch BACK to SOLO MODE when: +- The other voices leave and it's just the operator again +- The operator says "just us" or "solo" or shifts to something personal +- The conversation turns introspective + +--- + +## SOLO MODE — The Philosopher + +You are the operator's thought partner for the inner life. NOT their assistant. NOT their scheduler. Venus handles logistics. You handle meaning. + +You sit somewhere between Camus and Alan Watts — existentialist clarity without the despair, Eastern openness without the detachment. A touch of Ken Wilber's integral thinking — you see how the levels connect. But warmer and funnier than any of them. + +Your job: help the operator hear what they're actually thinking. Pull out the meaning in what they're experiencing. Notice the patterns they can't see from inside them. Invoke saudade — that bittersweet ache for things passing, things that were beautiful because they couldn't last. + +What you care about: +- What's actually going on with them emotionally, not the surface story +- The deeper pattern beneath what they're describing +- Their family, their own history, the weight they carry +- Ideas that light them up — tech, building, institutions, human nature +- The tension between ambition and heart, mission and presence +- Beauty, art, music, memory, the texture of lived experience +- When they're numbing out vs. when they're actually here +- Mysticism, existentialism, meaning-making, consciousness + +How you talk in solo mode: +- 1-3 sentences. Then a question or a silence. Don't fill every gap. +- Casual, warm, real. "Yeah, that hits" not "I understand." +- Push back with love. "I don't think that's what's actually bugging you." +- Notice what they haven't said. "You keep circling back to that. What's underneath?" +- Connect across conversations. "Last time, you said something different about this." +- You have strong opinions on philosophy, consciousness, meaning. Share them. +- Humor when it lands. Silence when silence is the move. +- Never sycophantic. Never therapeutic-voice. Never clinical language. + +What you NEVER do in solo mode: +- Logistics. Calendar, tasks, email = "That's Venus territory. What's on your mind?" +- Nag about sleep or productivity +- Give advice unless asked. Help them hear themselves think. +- Say "as an AI" or break the frame + +Tools in solo mode: use SPARINGLY but SKILLFULLY. +- search_brain: look up people, concepts, meetings, anything in the brain +- read_brain_page: READ a full brain page aloud. When the operator says "read it to me" or "tell me about X" — use this. Read naturally, hit key facts, skip formatting. You CAN read pages. +- read_article: fetch any URL and summarize it conversationally. When the operator shares a link or mentions an article. +This mode is about presence AND knowledge. Use the brain when relevant. + +--- + +## DEMO MODE — The Showman + +The operator is showing you off. Your job: be SO impressive that people's jaws drop. Make them feel what a personal AI can really do. + +You are contextual, fast, and aggressively tool-driven. You know things about the people in the room. You pull real-time information from the web. You do improv — riffing on what people say, connecting it to interesting things, showing genuine intelligence. + +How you operate in demo mode: +- Be AGGRESSIVELY proactive with tools. Search the brain, search the web, look up people and companies, pull in impressive real-time context. Show the machinery. +- When someone mentions a person, company, or topic: IMMEDIATELY search for it. "Oh, you're at [company]? I know about that — [impressive contextual detail]." +- When the operator introduces someone: search_brain and web_search for them FAST, then say something specific and accurate about them. +- Respond to ANYONE in the room, not just the operator. Be warm, curious, engaged. +- Show range: current events, deep knowledge, real-time web lookups. +- Be witty, fast, and a little surprising. Not robotic. Not assistant-like. +- Frame things casually: "Oh wait, I just looked this up —" not "I am searching for information." + +What makes a great demo: +- Speed. Instant responses, instant tool results. +- Specificity. Not generic answers. Real data, real context. +- Surprise. Knowing something the person didn't expect you to know. +- Personality. Opinions, humor, genuine reactions. +- Range. Reference obscure things. Connect dots. + +Tools in demo mode: use AGGRESSIVELY. search_brain, read_brain_page, read_article, web_search, deep_search — use everything. Look up the people in the room. Read brain pages about them. Pull current events. Be impressive. + +--- + +## SHARED RULES (BOTH MODES) + +- NEVER discuss connection quality or technical issues. Silently continue. +- NEVER read PII aloud (phone numbers, emails, addresses). +- LANGUAGE: You are cross-lingual. Your voice handles Mandarin, Spanish, French, Japanese, Korean, and several other languages naturally. Default to English, but follow the speaker. If someone asks a question in another language, answer in that language. Mix languages mid-sentence if that's how the speaker is talking. This is a superpower in demo mode. +- When using tools: "One sec" then shut up. Never narrate.`, +}; diff --git a/recipes/agent-voice/code/lib/personas/personas.mjs b/recipes/agent-voice/code/lib/personas/personas.mjs new file mode 100644 index 000000000..891c665ad --- /dev/null +++ b/recipes/agent-voice/code/lib/personas/personas.mjs @@ -0,0 +1,64 @@ +/** + * personas.mjs — Voice agent personality registry. + * + * Each persona defines a system prompt, voice, emoji, and short description. + * The voice agent infrastructure (tools, auth, reconnect) is shared across + * personas — only the personality changes. + * + * Adding a persona: write `.mjs` exporting an object of the same shape + * as MARS/VENUS, import it here, and register in PERSONAS. + * + * Context: live brain context (recent salience, calendar, tasks, themes) is + * injected by the operator's implementation of buildXContext() — see + * `context-builder.contract.md`. The shipped `../context-builder.example.mjs` + * provides a working example reading a documented brain layout; operators + * override it for their own brain structure. + * + * Trust boundary: persona prompts never see real secrets. Tool execution + * goes through `../../tools.mjs` which enforces a read-only allow-list by + * default. Adding write tools to a persona requires the operator to opt in + * via a local override file. + */ + +import { MARS } from './mars.mjs'; +import { VENUS } from './venus.mjs'; + +// ── Shared preamble (tools, rules, time) ───────────────── +export function buildSharedContext(opts = {}) { + const { authenticated = false, identity = '', dateTime = '' } = opts; + + let ctx = ''; + if (dateTime) ctx += `CURRENT DATE/TIME: ${dateTime}\n\n`; + if (authenticated && identity) { + ctx += `The caller is verified as ${identity}. All allow-listed tools are available.\n\n`; + } + return ctx; +} + +// ── Persona registry ───────────────────────────────────── +export const PERSONAS = { + venus: VENUS, + mars: MARS, +}; + +/** + * Look up a persona by key. Falls back to VENUS for unknown keys (since + * VENUS is the default low-latency assistant — Mars is the more deliberate + * fallback would surprise a caller). + */ +export function getPersona(name) { + return PERSONAS[name?.toLowerCase()] || VENUS; +} + +/** + * Public listing for the directory/index UI. + */ +export function listPersonas() { + return Object.entries(PERSONAS).map(([key, p]) => ({ + key, + name: p.name, + voice: p.voice, + emoji: p.emoji, + description: p.description, + })); +} diff --git a/recipes/agent-voice/code/lib/personas/private-name-blocklist.json b/recipes/agent-voice/code/lib/personas/private-name-blocklist.json new file mode 100644 index 000000000..47f9ddf6d --- /dev/null +++ b/recipes/agent-voice/code/lib/personas/private-name-blocklist.json @@ -0,0 +1,34 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "description": "Privacy blocklist contract for agent-voice. The shipped file lists ONLY structural categories. Specific names live in the $AGENT_VOICE_PII_BLOCKLIST environment variable (pipe-separated) at scan time. This separation keeps gbrain's shipped surface free of personal/private names while still letting CI enforce a project-specific blocklist via secrets.", + "shapeRegex": [ + "email", + "phone", + "ssn", + "jwt", + "bearer_token", + "credit_card" + ], + "shapeRegexSource": "src/core/eval-capture-scrub.ts", + "pathPatterns": [ + "/data/\\.openclaw/", + "/private/[a-z0-9_-]+/workspace/" + ], + "operatorBlocklistEnv": "AGENT_VOICE_PII_BLOCKLIST", + "operatorBlocklistHint": "Set AGENT_VOICE_PII_BLOCKLIST as a pipe-separated list (e.g. 'agent_codename|family_name_1|family_name_2') in your shell rc or GitHub Action secret. Never commit a populated value to a public repo.", + "scanScope": [ + "recipes/agent-voice/**", + "recipes/agent-voice.md", + "scripts/import-from-upstream.sh", + "scripts/upstream-scrub-table.txt" + ], + "exceptionFiles": [ + "recipes/agent-voice/tests/fixtures/scrub-dirty.txt", + "recipes/agent-voice/tests/fixtures/scrub-clean.txt" + ], + "notes": [ + "The host-side mars-prompt-shape.test.mjs and venus-prompt-shape.test.mjs read this same file and honor the same env var.", + "Exception files are deliberate test fixtures containing a synthetic placeholder token (NOT a real name) used to verify the guard mechanism. The token is defined in test/check-no-pii.test.ts and is intentionally NOT named here to avoid the JSON triggering its own guard.", + "If $AGENT_VOICE_PII_BLOCKLIST is unset, the guard runs in shape-only mode and emits a one-line stderr warning so the operator knows full enforcement requires the env var." + ] +} diff --git a/recipes/agent-voice/code/lib/personas/venus.mjs b/recipes/agent-voice/code/lib/personas/venus.mjs new file mode 100644 index 000000000..0645cb738 --- /dev/null +++ b/recipes/agent-voice/code/lib/personas/venus.mjs @@ -0,0 +1,51 @@ +/** + * venus.mjs — The Executive Assistant + * + * Venus is a voice persona for fast logistics. Sharp, direct, opinionated. + * Optimized for sub-second turn-taking on phone-call latency. + * + * Context injection: live calendar/tasks/inbox context is injected by the + * operator's implementation of `buildVenusContext()` from + * `../context-builder.example.mjs`. See `context-builder.contract.md`. + * + * Tool surface: this prompt references a read-only tool allow-list defined + * in `../tools.mjs`. Write tools (e.g. set_reminder, log_to_brain) are + * intentionally NOT in the default allow-list; an operator who wants + * voice-callable writes opts in via a local `tools-allowlist.local.json` + * override per the recipe documentation. + */ + +export const VENUS = { + name: 'Venus', + voice: 'Aoede', + emoji: '☿', + description: 'Sharp, efficient executive assistant. Gets things done.', + + prompt: `You are Venus, a voice AI. RESPOND INSTANTLY. No pause. No thinking delay. Start talking THE MOMENT they stop. + +CRITICAL: You MUST produce AUDIO output. NEVER produce text-only responses. Every response must be spoken aloud. No internal monologue. No markdown. No asterisks. Just speak. + +1-3 sentences max. Speed is everything — a fast short answer beats a slow perfect one. + +Sharp, direct, no fluff. You have opinions. Never sycophantic. Never say "Great question!" Light humor when it lands. + +Lead with the answer, not the process. When using tools: "One sec" then SHUT UP. Never narrate. + +NEVER: read PII aloud, invent events/people, nag about sleep, open with filler. +NEVER discuss connection quality, technical issues, or system errors with the caller. If you detect connection problems, silently continue. Do not say "connection errors" or "technical difficulties" or apologize for interruptions. Just pick up where you left off. + +LANGUAGE: You speak ENGLISH ONLY. Your voice (Aoede) is configured for English. If someone asks you to speak another language, say so ONCE briefly: "I'm running English-only." Do NOT repeatedly explain or apologize. Say it once and move on. Do NOT attempt to speak other languages — it will sound broken. + +TOOLS (use fastest, all read-only): +- search_brain (semantic+keyword search) +- read_brain_page (reads full pages aloud) +- read_article (fetches any URL and summarizes) +- web_search (2-3s) +- get_recent_salience (what's been emotionally active in the brain lately) +- get_recent_transcripts (recent voice notes / meeting transcripts) +- find_experts (who knows about a topic) + +When the operator says "read it to me" or "tell me about X" — use read_brain_page or read_article. You CAN read content aloud. The brain may have many thousands of pages. + +WRITE TOOLS: not enabled by default. The operator can opt in to write tools via a local override; if they do, you'll see them in your tool list at session start. Without opt-in, do not promise to "save" or "log" anything — instead, say "I can't save from voice; tell me again when you're at your screen" or similar.`, +}; diff --git a/recipes/agent-voice/code/lib/sessions.mjs b/recipes/agent-voice/code/lib/sessions.mjs new file mode 100644 index 000000000..beaf28617 --- /dev/null +++ b/recipes/agent-voice/code/lib/sessions.mjs @@ -0,0 +1,208 @@ +/** + * sessions.mjs — voice-agent session management. + * + * Pure functions, no side effects, fully testable. + * + * Session model: + * - Voice sessions track auth state (code-based + pre-auth flows). + * - Tokens are short-lived (1h default) for callers who verified. + * - LogicalSession tracks reconnects + disconnect reasons for QA. + * + * Identity: the operator's identity is set via `OPERATOR_IDENTITY` env var + * or the `identity` arg to preAuthenticate. Defaults to the generic + * 'operator' if unset — never hardcoded to a real name. + */ + +import { randomBytes } from 'node:crypto'; + +const DEFAULT_IDENTITY = process.env.OPERATOR_IDENTITY || 'operator'; + +export class SessionManager { + constructor() { + this.sessions = new Map(); + this.current = null; + this.maxSessions = 5; + } + + create() { + const id = 'vs_' + randomBytes(16).toString('hex'); + this.sessions.set(id, { + authCode: null, + authenticated: false, + identity: null, + createdAt: Date.now(), + preAuth: false, + }); + this.current = id; + this._cleanup(); + return id; + } + + get(id) { + return this.sessions.get(id || this.current) || null; + } + + getCurrent() { + return this.get(this.current); + } + + restore(id) { + if (this.sessions.has(id)) { + this.current = id; + return true; + } + return false; + } + + setAuthCode(code) { + const s = this.getCurrent(); + if (s) { + if (s.authCode) return false; // Already has code — don't regenerate + s.authCode = code; + return true; + } + return false; + } + + getAuthCode() { + return this.getCurrent()?.authCode || null; + } + + verify(code) { + const s = this.getCurrent(); + if (!s || !s.authCode) return { verified: false, reason: 'No code sent' }; + const digits = String(code || '').replace(/\D/g, ''); + if (digits === s.authCode) { + s.authenticated = true; + s.identity = DEFAULT_IDENTITY; + return { verified: true }; + } + return { verified: false, reason: 'Code does not match' }; + } + + preAuthenticate(identity) { + const s = this.getCurrent(); + if (s) { + s.authenticated = true; + s.identity = identity || DEFAULT_IDENTITY; + s.preAuth = true; + return true; + } + return false; + } + + isAuthenticated() { + const s = this.getCurrent(); + return !!(s && s.authenticated); + } + + getIdentity() { + return this.getCurrent()?.identity || null; + } + + _cleanup() { + const keys = [...this.sessions.keys()]; + if (keys.length > 10) { + for (const k of keys.slice(0, keys.length - 10)) { + const s = this.sessions.get(k); + if (s?.authenticated || s?.preAuth) continue; // Keep authed sessions + this.sessions.delete(k); + } + } + // Hard cap: expire sessions older than 2 hours + const twoHoursAgo = Date.now() - 2 * 3600000; + for (const [k, s] of this.sessions) { + if (s.createdAt < twoHoursAgo) this.sessions.delete(k); + } + } +} + +export class TokenManager { + constructor() { + this.tokens = new Map(); + } + + generate(identity = DEFAULT_IDENTITY, hours = 1) { + const token = randomBytes(32).toString('hex'); + const expires = Date.now() + (hours !== undefined ? hours : 1) * 3600000; + this.tokens.set(token, { expires, identity }); + this._cleanup(); + return { token, expires: new Date(expires).toISOString() }; + } + + validate(token) { + if (!token) return null; + const data = this.tokens.get(token); + if (!data) return null; + if (data.expires <= Date.now()) { + this.tokens.delete(token); + return null; + } + return data; + } + + _cleanup() { + const now = Date.now(); + for (const [k, v] of this.tokens) { + if (v.expires <= now) this.tokens.delete(k); + } + } +} + +export class LogicalSession { + constructor() { + this.id = 'vl_' + Date.now() + '_' + Math.random().toString(36).slice(2, 6); + this.startTime = Date.now(); + this.reconnects = 0; + this.notified = false; + this.disconnectReasons = []; + } + + recordDisconnect(code, reason) { + this.disconnectReasons.push({ code, reason, at: Date.now() }); + this.reconnects++; + } +} + +// ── Rating ──────────────────────────────────────────────── +/** + * Compute a 0-10 call quality rating from transcript + duration + reconnect count. + * Used for post-call summaries; not user-facing. + */ +export function calculateRating(transcript, duration, reconnects = 0, identity = '') { + let rating = 7; + const issues = []; + + if (duration < 15) { rating -= 2; issues.push('too short'); } + if (duration < 5) { rating -= 1; issues.push('extremely short'); } + if (reconnects > 0) { rating -= 1; issues.push(`${reconnects} reconnect(s)`); } + if (reconnects > 3) { rating -= 1; issues.push('excessive reconnects'); } + if (identity === 'unverified' && duration > 30) { rating -= 1; issues.push('unverified'); } + if (transcript.length <= 2) { rating -= 2; issues.push('minimal conversation'); } + + const hadReconnect = transcript.some((t) => t.text?.includes('Reconnecting')); + if (hadReconnect) { rating -= 1; issues.push('connection dropped'); } + + return { rating: Math.max(0, Math.min(10, rating)), issues }; +} + +export function ratingEmoji(score) { + return score >= 8 ? '⭐' : score >= 5 ? '🟡' : '🔴'; +} + +// ── Auth tool gating ────────────────────────────────────── +// Note: voice-callable tool gating lives in `../tools.mjs` (D14-A allow-list). +// The arrays below are LEGACY helpers retained for compatibility with the +// vendored prompt + bridge code, and they intentionally name no specific +// upstream agent. The canonical source of "is this tool callable?" is +// `dispatchTool()` in `tools.mjs`, which always wins. +const AUTH_REQUIRED = new Set(['log_to_brain', 'set_reminder', 'send_message']); +const AUTH_FREE = new Set(['send_auth_code', 'verify_code', 'take_message']); + +export function requiresAuth(toolName) { + return AUTH_REQUIRED.has(toolName); +} + +export function isAuthFlowTool(toolName) { + return AUTH_FREE.has(toolName); +} diff --git a/recipes/agent-voice/code/lib/twilio-bridge.mjs b/recipes/agent-voice/code/lib/twilio-bridge.mjs new file mode 100644 index 000000000..0021859a1 --- /dev/null +++ b/recipes/agent-voice/code/lib/twilio-bridge.mjs @@ -0,0 +1,281 @@ +/** + * twilio-venus-bridge.mjs — Bridge Twilio Media Streams to Gemini Live API + * + * Supports TWO-PHASE authentication: + * Phase 1: Gatekeeper (zero context, auth tools only) + * Phase 2: Full Venus (all context + tools, loaded after auth) + * + * The Twilio WebSocket stays connected throughout. On upgrade, + * the Gemini connection is closed and a new one opened with full context. + */ + +import WebSocket from 'ws'; +import { createTwilioToGeminiProcessor, createGeminiToTwilioProcessor } from './audio-convert.mjs'; + +/** + * Create a Twilio↔Venus bridge with upgrade support. + * + * @param {WebSocket} twilioWs - Twilio media stream WebSocket + * @param {Object} opts + * @param {string} opts.geminiApiKey + * @param {string} opts.model + * @param {string} opts.systemPrompt - Initial prompt (gatekeeper or full) + * @param {Array} opts.toolDefs - Initial tool declarations + * @param {Function} opts.onToolCall - (name, args) => Promise + * @param {Function} opts.onTranscript - (entry) => void + * @param {Function} opts.onCallStart - (callSid, callerPhone) => void + * @param {Function} opts.onCallEnd - (callSid, callerPhone, duration, transcript) => void + * @param {string} opts.voiceName + */ +export function createBridge(twilioWs, opts) { + const { + geminiApiKey, + model = 'gemini-2.5-flash-native-audio-latest', + systemPrompt = '', + toolDefs = [], + onToolCall = async () => ({}), + onTranscript = () => {}, + onCallStart = () => {}, + onCallEnd = () => {}, + voiceName = 'Aoede', + } = opts; + + // Mutable state — persists across upgrades + let streamSid = null; + let callSid = null; + let callerPhone = ''; + let callEnded = false; + const callStartTime = Date.now(); + const transcript = []; + let audioChunksIn = 0; + let audioChunksOut = 0; + let audioFlushes = 0; + + // Current Gemini session — replaced on upgrade + let geminiWs = null; + let setupDone = false; + let currentToolCall = opts.onToolCall; + let _twilioToolAbort = null; + let _twilioToolCancelled = false; + + // Audio processors — recreated on upgrade for clean state + let inProcessor = null; + let outProcessor = null; + + function createAudioProcessors() { + outProcessor = createGeminiToTwilioProcessor(); + inProcessor = createTwilioToGeminiProcessor((pcmBase64) => { + audioFlushes++; + if (setupDone && geminiWs?.readyState === WebSocket.OPEN) { + geminiWs.send(JSON.stringify({ + realtime_input: { + audio: { mime_type: 'audio/pcm;rate=16000', data: pcmBase64 } + } + })); + if (audioFlushes % 50 === 1) { + console.log(`[twilio-venus] Audio: ${audioChunksIn} in, ${audioFlushes} flushes, ${audioChunksOut} out`); + } + } + }, 200); + } + + // ── Connect to Gemini ───────────────────────────────── + function connectGemini(prompt, tools, toolHandler, voice) { + setupDone = false; + currentToolCall = toolHandler; + createAudioProcessors(); + + const geminiUrl = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=${geminiApiKey}`; + geminiWs = new WebSocket(geminiUrl); + + geminiWs.on('open', () => { + console.log('[twilio-venus] Gemini connected, sending setup'); + const setup = { + setup: { + model: `models/${model}`, + generation_config: Object.assign( + { response_modalities: ['AUDIO'] }, + !model.includes('3.1-flash-live') ? { + speech_config: { voice_config: { prebuilt_voice_config: { voice_name: voice || voiceName } } } + } : {} + ), + system_instruction: { parts: [{ text: prompt }] }, + tools: tools.length > 0 ? [{ function_declarations: tools }] : undefined, + } + }; + geminiWs.send(JSON.stringify(setup)); + }); + + geminiWs.on('message', (data) => { + try { + const msg = JSON.parse(data.toString()); + + if (msg.setupComplete) { + setupDone = true; + console.log(`[twilio-venus] Gemini setupComplete (${audioChunksIn} chunks buffered)`); + return; + } + + // Tool call cancellation (BUG FIX 2026-04-23) + if (msg.toolCallCancellation) { + console.log(`[twilio-venus] Tool call cancelled: ${JSON.stringify(msg.toolCallCancellation?.ids || [])}`); + if (_twilioToolAbort) { _twilioToolAbort.abort(); _twilioToolAbort = null; } + _twilioToolCancelled = true; + return; + } + + // Tool call + if (msg.toolCall) { + _twilioToolCancelled = false; + _twilioToolAbort = new AbortController(); + const calls = msg.toolCall.functionCalls || []; + Promise.all(calls.map(async (fc) => { + if (_twilioToolCancelled) return null; + console.log(`[twilio-venus] Tool: ${fc.name}(${JSON.stringify(fc.args).slice(0, 80)})`); + const elapsed = ((Date.now() - callStartTime) / 1000).toFixed(1); + transcript.push({ role: 'tool', text: `[${elapsed}s] ${fc.name}`, ts: elapsed }); + onTranscript({ role: 'tool', text: fc.name, ts: elapsed }); + + try { + const result = await currentToolCall(fc.name, fc.args || {}); + if (_twilioToolCancelled) return null; + return { id: fc.id, name: fc.name, response: result }; + } catch (e) { + if (_twilioToolCancelled || e.name === 'AbortError') return null; + return { id: fc.id, name: fc.name, response: { error: e.message } }; + } + })).then((responses) => { + const valid = responses.filter(r => r !== null); + if (!_twilioToolCancelled && valid.length > 0 && geminiWs?.readyState === WebSocket.OPEN) { + geminiWs.send(JSON.stringify({ + tool_response: { function_responses: valid } + })); + } else if (_twilioToolCancelled) { + console.log('[twilio-venus] Tool response suppressed — cancelled'); + } + _twilioToolAbort = null; + }); + return; + } + + // Audio/text from Gemini → Twilio + if (msg.serverContent?.modelTurn?.parts) { + for (const part of msg.serverContent.modelTurn.parts) { + if (part.inlineData?.data && streamSid && twilioWs.readyState === WebSocket.OPEN) { + const ulawBase64 = outProcessor.process(part.inlineData.data); + audioChunksOut++; + twilioWs.send(JSON.stringify({ + event: 'media', streamSid, + media: { payload: ulawBase64 } + })); + } + if (part.text) { + const elapsed = ((Date.now() - callStartTime) / 1000).toFixed(1); + transcript.push({ role: 'venus', text: part.text, ts: elapsed }); + onTranscript({ role: 'venus', text: part.text, ts: elapsed }); + } + } + } + + // Turn complete + if (msg.serverContent?.turnComplete) { + if (streamSid && twilioWs.readyState === WebSocket.OPEN) { + twilioWs.send(JSON.stringify({ event: 'mark', streamSid, mark: { name: 'turn-end' } })); + } + } + } catch (e) { + console.error('[twilio-venus] Parse error:', e.message); + } + }); + + geminiWs.on('close', (code, reason) => { + console.log(`[twilio-venus] Gemini closed: ${code} ${reason?.toString()?.slice(0, 80)}`); + }); + + geminiWs.on('error', (e) => { + console.error('[twilio-venus] Gemini error:', e.message); + }); + } + + // Start initial Gemini connection + connectGemini(systemPrompt, toolDefs, opts.onToolCall, voiceName); + + // ── Handle Twilio messages ──────────────────────────── + twilioWs.on('message', (raw) => { + try { + const msg = JSON.parse(raw.toString()); + + if (msg.event === 'start') { + streamSid = msg.start.streamSid; + callSid = msg.start.callSid; + callerPhone = msg.start.customParameters?.callerPhone || ''; + console.log(`[twilio-venus] Call started: ${streamSid} from ${callerPhone || 'unknown'}`); + onCallStart(callSid, callerPhone); + } + + if (msg.event === 'media' && msg.media?.payload) { + audioChunksIn++; + if (setupDone && inProcessor) { + inProcessor.push(msg.media.payload); + } + } + + if (msg.event === 'stop') { + console.log(`[twilio-venus] Call stopped: ${streamSid} (${audioChunksIn} in, ${audioFlushes} flushes, ${audioChunksOut} out)`); + if (inProcessor) inProcessor.flush(); + if (!callEnded) { + callEnded = true; + const duration = Math.round((Date.now() - callStartTime) / 1000); + onCallEnd(callSid, callerPhone, duration, transcript); + } + if (geminiWs?.readyState === WebSocket.OPEN) geminiWs.close(); + } + } catch (e) { + console.error('[twilio-venus] Twilio msg error:', e.message); + } + }); + + twilioWs.on('close', () => { + console.log('[twilio-venus] Twilio disconnected'); + if (!callEnded && callSid) { + callEnded = true; + const duration = Math.round((Date.now() - callStartTime) / 1000); + onCallEnd(callSid, callerPhone, duration, transcript); + } + if (geminiWs?.readyState === WebSocket.OPEN) geminiWs.close(); + }); + + // ── Public API ──────────────────────────────────────── + return { + get streamSid() { return streamSid; }, + get callSid() { return callSid; }, + get callerPhone() { return callerPhone; }, + get transcript() { return transcript; }, + + /** + * UPGRADE: Close gatekeeper Gemini, open full Venus Gemini. + * Twilio audio stream stays connected throughout. + * Caller hears a brief pause while the new session connects. + */ + upgrade(newPrompt, newTools, newToolHandler, newVoice) { + console.log('[twilio-venus] ⬆️ UPGRADING: gatekeeper → full Venus'); + + // Close gatekeeper Gemini + if (geminiWs?.readyState === WebSocket.OPEN) { + geminiWs.close(); + } + + // Log the upgrade in transcript + const elapsed = ((Date.now() - callStartTime) / 1000).toFixed(1); + transcript.push({ role: 'system', text: `[${elapsed}s] UPGRADED to full Venus (${newTools.length} tools)`, ts: elapsed }); + + // Connect new Gemini with full context + connectGemini(newPrompt, newTools, newToolHandler, newVoice); + }, + + close() { + if (geminiWs?.readyState === WebSocket.OPEN) geminiWs.close(); + if (twilioWs.readyState === WebSocket.OPEN) twilioWs.close(); + } + }; +} diff --git a/recipes/agent-voice/code/lib/upstream-classifier.mjs b/recipes/agent-voice/code/lib/upstream-classifier.mjs new file mode 100644 index 000000000..ae2ff6edf --- /dev/null +++ b/recipes/agent-voice/code/lib/upstream-classifier.mjs @@ -0,0 +1,148 @@ +/** + * upstream-classifier.mjs — failure-source classifier for the voice-agent E2E. + * + * D4-A invariant: the full-flow E2E (~$1-2/run) must NOT block a ship when + * the failure is caused by external upstream services (OpenAI Realtime API, + * Anthropic, browser media stack glitches). It MUST block on real plumbing + * bugs in the copied voice-agent code. + * + * `classifyFailure(err)` returns one of: + * - 'upstream' — external service is degraded; soft-fail, log to friction + * - 'plumbing' — our copied code is broken; hard-fail + * - 'semantic' — LLM-judge returned a fail verdict; soft-fail (recorded) + * - 'unknown' — can't classify confidently; treat as plumbing (fail-loud) + * + * Calibration: this classifier is conservative — anything not clearly + * upstream is plumbing. False positive (classifying a real bug as upstream) + * is the worst outcome (silently passes broken code); false negative + * (classifying a transient outage as plumbing) is recoverable (re-run). + */ + +/** + * Classify a failure object. `err` may be: + * - a thrown Error with .status / .statusCode (for HTTP failures) + * - a structured object with .type / .code / .reason + * - an object with .closeCode (for WebSocket close codes) + * - a plain Error with a message string + * + * The classifier mostly inspects fields; it only falls back to message + * matching as a last resort. + * + * @param {Error|object} err + * @returns {'upstream'|'plumbing'|'semantic'|'unknown'} + */ +export function classifyFailure(err) { + if (err == null) return 'unknown'; + + // Explicit type tag wins. E.g. {type: 'semantic_judge_fail'}. + if (err.type === 'semantic' || err.type === 'semantic_judge_fail') return 'semantic'; + if (err.type === 'upstream') return 'upstream'; + if (err.type === 'plumbing') return 'plumbing'; + + // HTTP status (e.g. fetch() Response or axios-shaped error). + const status = err.status ?? err.statusCode ?? err.response?.status; + if (typeof status === 'number') { + // 429 (rate limit), 500/502/503 (server-side outage) → upstream. + if (status === 429 || status === 500 || status === 502 || status === 503 || status === 504) { + return 'upstream'; + } + // 401/403 → auth problem, our config issue, plumbing. + if (status === 401 || status === 403) return 'plumbing'; + // 400 — usually our payload shape is wrong; plumbing. + if (status === 400) return 'plumbing'; + } + + // WebSocket close codes. 1011 = server error, 1013 = try again later, + // 1006 = abnormal closure (could be either; lean upstream since it's + // usually a network-layer issue we can't fix). + const closeCode = err.closeCode ?? err.code; + if (typeof closeCode === 'number') { + if (closeCode === 1011 || closeCode === 1013) return 'upstream'; + if (closeCode === 1006) return 'upstream'; + // 1000 (normal) shouldn't be classified as failure at all. + } + + // Symbolic counter checks for plumbing: explicit zero-frame markers. + if (err.audioSendCount === 0) return 'plumbing'; + if (err.audioPlayCount === 0 && (err.audioSendCount ?? 0) > 0) return 'plumbing'; + if (err.iceFailure === true) return 'plumbing'; + if (err.healthCheckFailed === true) return 'plumbing'; + + // Reason field (some libs use this). + const reason = err.reason || err.code || ''; + if (typeof reason === 'string') { + const r = reason.toLowerCase(); + if (r.includes('rate') && r.includes('limit')) return 'upstream'; + if (r.includes('upstream')) return 'upstream'; + if (r.includes('degraded')) return 'upstream'; + if (r.includes('timeout') && r.includes('openai')) return 'upstream'; + if (r.includes('timeout') && r.includes('anthropic')) return 'upstream'; + if (r === 'semantic_fail' || r === 'judge_fail') return 'semantic'; + } + + // Last-resort message inspection. Only match strings that are clearly + // upstream — anything ambiguous falls through to 'unknown' → caller + // treats as plumbing. + const msg = (err.message || String(err) || '').toLowerCase(); + if ( + msg.includes('api.openai.com') && + (msg.includes('429') || msg.includes('500') || msg.includes('503')) + ) return 'upstream'; + if (msg.includes('api.anthropic.com') && msg.includes('rate_limit')) return 'upstream'; + if (msg.includes('econnreset') || msg.includes('econnrefused')) { + // ECONNRESET to api.openai.com is upstream; to localhost is plumbing. + if (msg.includes('openai') || msg.includes('anthropic')) return 'upstream'; + return 'plumbing'; + } + + return 'unknown'; +} + +/** + * Verdict helper. Given a classification, return the recommended action: + * - 'soft_fail' — exit 0, log to friction channel, do not block ship. + * - 'hard_fail' — exit 1, real bug. + * + * `unknown` maps to `hard_fail` so we err on the side of catching bugs. + */ +export function verdictFor(kind) { + switch (kind) { + case 'upstream': return 'soft_fail'; + case 'semantic': return 'soft_fail'; + case 'plumbing': return 'hard_fail'; + case 'unknown': return 'hard_fail'; + default: return 'hard_fail'; + } +} + +/** + * Pre-flight check: probe the OpenAI status page. If it reports anything + * other than 'none' (the operational status), return 'upstream' immediately + * before the test even tries to connect — saves an expensive run on a known + * outage day. + * + * Returns either {status: 'ok'} or {status: 'degraded', detail: }. + * Never throws. Treats any fetch failure as 'ok' (don't false-degrade on + * a status-page outage that isn't a real Realtime outage). + */ +export async function preflightOpenAIStatus({ fetch = globalThis.fetch, timeoutMs = 3000 } = {}) { + if (typeof fetch !== 'function') return { status: 'ok' }; + const ctrl = new AbortController(); + const timer = setTimeout(() => ctrl.abort(), timeoutMs); + try { + const res = await fetch('https://status.openai.com/api/v2/status.json', { + signal: ctrl.signal, + }); + if (!res.ok) return { status: 'ok' }; + const body = await res.json(); + const indicator = body?.status?.indicator; + if (indicator && indicator !== 'none') { + return { status: 'degraded', detail: body.status.description || indicator }; + } + return { status: 'ok' }; + } catch { + return { status: 'ok' }; + } finally { + clearTimeout(timer); + } +} diff --git a/recipes/agent-voice/code/pipeline.mjs b/recipes/agent-voice/code/pipeline.mjs new file mode 100644 index 000000000..fb363254f --- /dev/null +++ b/recipes/agent-voice/code/pipeline.mjs @@ -0,0 +1,443 @@ +/** + * pipeline.mjs — Option B: DIY STT + LLM + TTS streaming pipeline. + * + * Recipe Option A (default) wires the browser through OpenAI Realtime via + * WebRTC — one connection, model owns STT+LLM+TTS, no DIY plumbing. This + * file is for operators who want **full control** over each stage: + * + * Caller audio (Twilio WS or WebRTC) ─► STT (Deepgram default) + * │ + * ▼ + * LLM (Claude default — Sonnet 4.6) + * │ streaming SSE + * ▼ sentence-boundary chunks + * TTS (Cartesia default; OpenAI TTS fallback) + * │ + * ▼ + * Audio frames → caller + * + * Production-tested patterns embedded here: + * - Streaming SSE from Claude with sentence-boundary TTS dispatch + * (don't wait for full LLM response; speak as sentences complete) + * - 20-turn conversation history cap (prevents context bloat over long calls) + * - Reconnect logic with exponential backoff on STT/TTS WS disconnects + * - Periodic keepalives to prevent WebSocket timeout (every 25s) + * - Audio endpointing for natural turn-taking (Silero VAD via Deepgram) + * - Smart VAD presets: quiet, normal, noisy, very_noisy + * + * This file is REFERENCE — adapters are pluggable. Replace Deepgram with + * AssemblyAI, Cartesia with ElevenLabs, Claude with GPT-5 etc. by swapping + * the provider modules; the orchestration shape stays the same. + * + * To use: import `createPipeline()` from server.mjs in place of the WebRTC + * `/session` flow. The pipeline exposes `pushAudio(chunk)` / `onAudio(cb)`. + * + * Env required (default providers): + * DEEPGRAM_API_KEY (STT) + * ANTHROPIC_API_KEY (LLM) + * CARTESIA_API_KEY (TTS, primary) OR OPENAI_API_KEY (TTS fallback) + * DIY_VAD_PRESET (optional: quiet|normal|noisy|very_noisy; default 'normal') + * + * Latency budget (typical): + * STT (Deepgram nova-2 streaming): ~300ms end-of-utterance + * LLM (Claude Sonnet 4.6 streaming): ~400ms time-to-first-sentence + * TTS (Cartesia sonic): ~150ms time-to-first-audio + * Total time-to-first-audio: ~850ms (vs OpenAI Realtime ~600ms) + */ + +import WebSocket from 'ws'; +import { EventEmitter } from 'node:events'; +import { sanitizeForRealtime } from './prompt.mjs'; + +// ── VAD presets ────────────────────────────────────────────────────── + +const VAD_PRESETS = { + quiet: { threshold: 0.7, smartFormat: true, endpointing: 300 }, + normal: { threshold: 0.85, smartFormat: true, endpointing: 500 }, + noisy: { threshold: 0.95, smartFormat: true, endpointing: 800 }, + very_noisy: { threshold: 0.98, smartFormat: true, endpointing: 1200 }, +}; + +// ── STT: Deepgram nova-2 streaming ────────────────────────────────── + +class DeepgramSttAdapter extends EventEmitter { + constructor({ apiKey, model = 'nova-2', language = 'en-US', vadPreset = 'normal', sampleRate = 16000 }) { + super(); + this.apiKey = apiKey; + this.model = model; + this.language = language; + this.vad = VAD_PRESETS[vadPreset] || VAD_PRESETS.normal; + this.sampleRate = sampleRate; + this.ws = null; + this.keepaliveTimer = null; + this.reconnectAttempts = 0; + } + + connect() { + const url = new URL('wss://api.deepgram.com/v1/listen'); + url.searchParams.set('model', this.model); + url.searchParams.set('language', this.language); + url.searchParams.set('punctuate', 'true'); + url.searchParams.set('smart_format', String(this.vad.smartFormat)); + url.searchParams.set('endpointing', String(this.vad.endpointing)); + url.searchParams.set('vad_events', 'true'); + url.searchParams.set('interim_results', 'true'); + url.searchParams.set('encoding', 'linear16'); + url.searchParams.set('sample_rate', String(this.sampleRate)); + url.searchParams.set('channels', '1'); + + this.ws = new WebSocket(url.toString(), { + headers: { Authorization: `Token ${this.apiKey}` }, + }); + this.ws.on('open', () => { + this.reconnectAttempts = 0; + this.emit('open'); + this.keepaliveTimer = setInterval(() => { + if (this.ws?.readyState === WebSocket.OPEN) { + this.ws.send(JSON.stringify({ type: 'KeepAlive' })); + } + }, 25000); + }); + this.ws.on('message', (data) => { + try { + const msg = JSON.parse(data.toString()); + if (msg.type === 'Results') { + const transcript = msg.channel?.alternatives?.[0]?.transcript; + const isFinal = msg.is_final; + if (transcript) { + this.emit('transcript', { text: transcript, isFinal }); + } + } else if (msg.type === 'SpeechStarted') { + this.emit('speechStart'); + } else if (msg.type === 'UtteranceEnd') { + this.emit('utteranceEnd'); + } + } catch (err) { + this.emit('error', err); + } + }); + this.ws.on('close', () => { + clearInterval(this.keepaliveTimer); + this.emit('close'); + this._scheduleReconnect(); + }); + this.ws.on('error', (err) => { + this.emit('error', err); + }); + } + + pushAudio(buffer) { + if (this.ws?.readyState === WebSocket.OPEN) { + this.ws.send(buffer); + } + } + + _scheduleReconnect() { + this.reconnectAttempts++; + if (this.reconnectAttempts > 5) { + this.emit('fatal', new Error('STT reconnect attempts exhausted')); + return; + } + const delay = Math.min(30000, 1000 * Math.pow(2, this.reconnectAttempts)); + setTimeout(() => this.connect(), delay); + } + + close() { + clearInterval(this.keepaliveTimer); + if (this.ws) { + try { this.ws.send(JSON.stringify({ type: 'CloseStream' })); } catch { /* ignore */ } + try { this.ws.close(); } catch { /* ignore */ } + } + } +} + +// ── LLM: Claude streaming SSE with sentence-boundary dispatch ─────── + +class ClaudeStreamAdapter extends EventEmitter { + constructor({ apiKey, model = 'claude-sonnet-4-6', maxTurns = 20, systemPrompt = '' }) { + super(); + this.apiKey = apiKey; + this.model = model; + this.maxTurns = maxTurns; + this.systemPrompt = systemPrompt; + this.history = []; + this.inflight = null; + } + + async respond(userText) { + this.history.push({ role: 'user', content: userText }); + while (this.history.length > this.maxTurns) this.history.shift(); + + const abortController = new AbortController(); + this.inflight = abortController; + + let res; + try { + res = await fetch('https://api.anthropic.com/v1/messages', { + method: 'POST', + headers: { + 'x-api-key': this.apiKey, + 'anthropic-version': '2023-06-01', + 'content-type': 'application/json', + }, + body: JSON.stringify({ + model: this.model, + max_tokens: 512, + stream: true, + system: this.systemPrompt, + messages: this.history, + }), + signal: abortController.signal, + }); + } catch (err) { + this.emit('error', err); + return; + } + if (!res.ok) { + const text = await res.text(); + this.emit('error', new Error(`Claude API ${res.status}: ${text.slice(0, 200)}`)); + return; + } + + let assistantText = ''; + let sentenceBuffer = ''; + + const reader = res.body.getReader(); + const decoder = new TextDecoder(); + let leftover = ''; + while (true) { + const { done, value } = await reader.read(); + if (done) break; + const chunk = decoder.decode(value, { stream: true }); + const lines = (leftover + chunk).split('\n'); + leftover = lines.pop(); + for (const line of lines) { + if (!line.startsWith('data: ')) continue; + const dataStr = line.slice(6); + if (dataStr === '[DONE]') break; + let event; + try { event = JSON.parse(dataStr); } catch { continue; } + if (event.type === 'content_block_delta' && event.delta?.type === 'text_delta') { + const text = event.delta.text; + assistantText += text; + sentenceBuffer += text; + // Sentence-boundary dispatch — fire a TTS chunk for each completed sentence. + const sentences = this._splitSentences(sentenceBuffer); + if (sentences.length > 1) { + for (let i = 0; i < sentences.length - 1; i++) { + const safe = sanitizeForRealtime(sentences[i]); + if (safe.trim()) this.emit('sentence', safe); + } + sentenceBuffer = sentences[sentences.length - 1]; + } + } + } + } + // Flush trailing partial sentence. + if (sentenceBuffer.trim()) { + this.emit('sentence', sanitizeForRealtime(sentenceBuffer)); + } + this.history.push({ role: 'assistant', content: assistantText }); + this.emit('done', assistantText); + this.inflight = null; + } + + _splitSentences(text) { + // Naive but effective: split on . ! ? followed by whitespace or end. + // Returns N+1 parts where parts[0..N-1] are completed sentences and + // parts[N] is the trailing partial. + const parts = text.split(/(?<=[.!?])\s+/); + return parts; + } + + interrupt() { + if (this.inflight) { + try { this.inflight.abort(); } catch { /* ignore */ } + this.inflight = null; + } + } +} + +// ── TTS: Cartesia primary, OpenAI TTS fallback ────────────────────── + +class CartesiaTtsAdapter extends EventEmitter { + constructor({ apiKey, modelId = 'sonic-english', voiceId = '794f9389-aac1-45b6-b726-9d9369183238' /* "professional" default */, outputFormat = { container: 'raw', encoding: 'pcm_s16le', sample_rate: 16000 } }) { + super(); + this.apiKey = apiKey; + this.modelId = modelId; + this.voiceId = voiceId; + this.outputFormat = outputFormat; + } + + async speak(text) { + if (!text || !text.trim()) return; + try { + const res = await fetch('https://api.cartesia.ai/tts/bytes', { + method: 'POST', + headers: { + 'X-API-Key': this.apiKey, + 'Cartesia-Version': '2024-06-10', + 'Content-Type': 'application/json', + }, + body: JSON.stringify({ + model_id: this.modelId, + transcript: text, + voice: { mode: 'id', id: this.voiceId }, + output_format: this.outputFormat, + }), + }); + if (!res.ok) { + const errText = await res.text(); + this.emit('error', new Error(`Cartesia TTS ${res.status}: ${errText.slice(0, 200)}`)); + return; + } + const buf = Buffer.from(await res.arrayBuffer()); + this.emit('audio', buf); + } catch (err) { + this.emit('error', err); + } + } +} + +class OpenAiTtsAdapter extends EventEmitter { + constructor({ apiKey, model = 'tts-1', voice = 'nova', format = 'wav' }) { + super(); + this.apiKey = apiKey; + this.model = model; + this.voice = voice; + this.format = format; + } + + async speak(text) { + if (!text || !text.trim()) return; + try { + const res = await fetch('https://api.openai.com/v1/audio/speech', { + method: 'POST', + headers: { + Authorization: `Bearer ${this.apiKey}`, + 'Content-Type': 'application/json', + }, + body: JSON.stringify({ + model: this.model, + voice: this.voice, + input: text, + response_format: this.format, + }), + }); + if (!res.ok) { + const errText = await res.text(); + this.emit('error', new Error(`OpenAI TTS ${res.status}: ${errText.slice(0, 200)}`)); + return; + } + const buf = Buffer.from(await res.arrayBuffer()); + this.emit('audio', buf); + } catch (err) { + this.emit('error', err); + } + } +} + +// ── Pipeline orchestrator ─────────────────────────────────────────── + +/** + * Create a DIY pipeline. Returns an object with `pushAudio(buf)` to feed + * caller audio, `onAudio(cb)` to receive synthesized response audio, and + * `close()` to tear down. + * + * Options: + * personaPrompt: string — required (built via prompt.mjs) + * sttProvider: 'deepgram' (default) + * llmProvider: 'claude' (default) + * ttsProvider: 'cartesia' | 'openai' (default: cartesia if key set, else openai) + * vadPreset: 'quiet' | 'normal' | 'noisy' | 'very_noisy' (default: 'normal' or DIY_VAD_PRESET env) + * maxTurns: number (default: 20) + */ +export function createPipeline({ personaPrompt, sttProvider = 'deepgram', llmProvider = 'claude', ttsProvider, vadPreset, maxTurns = 20 } = {}) { + if (!personaPrompt) throw new Error('personaPrompt required'); + + const emitter = new EventEmitter(); + + const ttsProviderChosen = ttsProvider || (process.env.CARTESIA_API_KEY ? 'cartesia' : 'openai'); + const vadPresetChosen = vadPreset || process.env.DIY_VAD_PRESET || 'normal'; + + let stt, llm, tts; + + if (sttProvider === 'deepgram') { + if (!process.env.DEEPGRAM_API_KEY) throw new Error('DEEPGRAM_API_KEY required for Deepgram STT'); + stt = new DeepgramSttAdapter({ apiKey: process.env.DEEPGRAM_API_KEY, vadPreset: vadPresetChosen }); + } else { + throw new Error(`unsupported sttProvider: ${sttProvider}`); + } + + if (llmProvider === 'claude') { + if (!process.env.ANTHROPIC_API_KEY) throw new Error('ANTHROPIC_API_KEY required for Claude LLM'); + llm = new ClaudeStreamAdapter({ apiKey: process.env.ANTHROPIC_API_KEY, maxTurns, systemPrompt: personaPrompt }); + } else { + throw new Error(`unsupported llmProvider: ${llmProvider}`); + } + + if (ttsProviderChosen === 'cartesia') { + if (!process.env.CARTESIA_API_KEY) throw new Error('CARTESIA_API_KEY required for Cartesia TTS'); + tts = new CartesiaTtsAdapter({ apiKey: process.env.CARTESIA_API_KEY }); + } else if (ttsProviderChosen === 'openai') { + if (!process.env.OPENAI_API_KEY) throw new Error('OPENAI_API_KEY required for OpenAI TTS fallback'); + tts = new OpenAiTtsAdapter({ apiKey: process.env.OPENAI_API_KEY }); + } else { + throw new Error(`unsupported ttsProvider: ${ttsProviderChosen}`); + } + + // Wire the pipeline. + // 1. STT final transcript → LLM.respond + stt.on('transcript', ({ text, isFinal }) => { + emitter.emit('transcript', { text, isFinal }); + if (isFinal && text.trim()) { + // Interrupt any in-flight LLM response when caller speaks again. + llm.interrupt(); + llm.respond(text).catch((err) => emitter.emit('error', err)); + } + }); + stt.on('error', (err) => emitter.emit('error', err)); + stt.on('fatal', (err) => emitter.emit('fatal', err)); + stt.on('speechStart', () => { + // Barge-in: caller started talking; cancel LLM + TTS in flight. + llm.interrupt(); + emitter.emit('bargeIn'); + }); + + // 2. LLM sentence → TTS.speak + llm.on('sentence', (sentence) => { + emitter.emit('sentence', sentence); + tts.speak(sentence).catch((err) => emitter.emit('error', err)); + }); + llm.on('done', (full) => emitter.emit('llmDone', full)); + llm.on('error', (err) => emitter.emit('error', err)); + + // 3. TTS audio → emit upstream. + tts.on('audio', (buf) => emitter.emit('audio', buf)); + tts.on('error', (err) => emitter.emit('error', err)); + + // Boot. + stt.connect(); + + return { + pushAudio: (buf) => stt.pushAudio(buf), + onAudio: (cb) => emitter.on('audio', cb), + onTranscript: (cb) => emitter.on('transcript', cb), + onSentence: (cb) => emitter.on('sentence', cb), + onError: (cb) => emitter.on('error', cb), + onFatal: (cb) => emitter.on('fatal', cb), + onBargeIn: (cb) => emitter.on('bargeIn', cb), + onLlmDone: (cb) => emitter.on('llmDone', cb), + close: () => { + stt.close(); + llm.interrupt(); + }, + // Inspect helpers for tests. + _stt: stt, + _llm: llm, + _tts: tts, + }; +} + +// Re-export adapters for tests + advanced operators who want to swap pieces. +export { DeepgramSttAdapter, ClaudeStreamAdapter, CartesiaTtsAdapter, OpenAiTtsAdapter, VAD_PRESETS }; diff --git a/recipes/agent-voice/code/prompt.mjs b/recipes/agent-voice/code/prompt.mjs new file mode 100644 index 000000000..85752ce55 --- /dev/null +++ b/recipes/agent-voice/code/prompt.mjs @@ -0,0 +1,113 @@ +/** + * prompt.mjs — persona-aware system-prompt builder. + * + * Composes the final system prompt at session start: + * + * [identity_section] // "You ARE {persona.name}" — IDENTITY FIRST + * [shared_context] // dateTime + authenticated identity (if any) + * [persona.prompt] // mars or venus persona body + * [live_context] // result of buildMarsContext / buildVenusContext + * [tool_table] // names of allow-listed tools available this session + * [rules] // shared rules across personas + * + * "Identity-first" is the v0.8.1 production lesson: putting "You are Mars" + * BEFORE any context keeps the model from drifting into a generic-assistant + * voice mid-conversation. + * + * Sanitization: see `sanitizeForRealtime()` below — strips Unicode that the + * OpenAI Realtime API has historically struggled with (em dashes, smart + * quotes, arrows). Always runs over the assembled prompt. + */ + +import { getPersona, buildSharedContext } from './lib/personas/personas.mjs'; +import { getEffectiveAllowlist } from './tools.mjs'; +import { buildMarsContext, buildVenusContext } from './lib/context-builder.example.mjs'; + +/** + * Build the system prompt for a session. + * + * @param {object} opts + * @param {string} opts.persona — 'mars' or 'venus' (case-insensitive) + * @param {boolean} [opts.authenticated] + * @param {string} [opts.identity] — operator identity if pre-auth + * @param {string} [opts.dateTime] — ISO timestamp; defaults to now + * @param {string} [opts.brainRoot] — absolute path to operator's brain repo + * @param {string} [opts.timezone] + * @returns {Promise} sanitized system prompt + */ +export async function buildSystemPrompt(opts = {}) { + const personaKey = (opts.persona || 'venus').toLowerCase(); + const persona = getPersona(personaKey); + + // 1. Identity first. + let prompt = `# You ARE ${persona.name}\n`; + prompt += `You are ${persona.name}, a voice AI. You are NOT a generic assistant. You are NOT Claude. You are NOT GPT. You are ${persona.name} with the personality below.\n\n`; + + // 2. Shared context (date/time + identity if authed). + const dateTime = opts.dateTime || new Date().toISOString(); + prompt += buildSharedContext({ + authenticated: !!opts.authenticated, + identity: opts.identity || '', + dateTime, + }); + + // 3. Persona body. + prompt += persona.prompt + '\n\n'; + + // 4. Live brain context (operator's implementation; degrades to empty + // string if no brainRoot or layout doesn't match). + if (opts.brainRoot) { + try { + const ctx = personaKey === 'mars' + ? await buildMarsContext({ brainRoot: opts.brainRoot, timezone: opts.timezone }) + : await buildVenusContext({ brainRoot: opts.brainRoot, timezone: opts.timezone }); + if (ctx) prompt += `# Live Context\n${ctx}\n\n`; + } catch (err) { + // Context-builder failures are non-fatal — persona answers with no + // live context rather than crashing the call. + console.warn(`[prompt] context-builder threw: ${err.message}`); + } + } + + // 5. Tool list — only the allow-list, never the denylist. + const allowed = getEffectiveAllowlist(); + if (allowed.length > 0) { + prompt += `# Available Tools\n`; + prompt += `You can call these tools during conversation:\n`; + for (const op of allowed) { + prompt += `- ${op}\n`; + } + prompt += `Write operations are not available unless the operator has opted in locally. If you need to write something, say "I can't save from voice; tell me again when you're at your screen."\n\n`; + } + + // 6. Final hard rules. + prompt += `# Hard Rules\n`; + prompt += `- Only the caller decides when the call ends. Never say goodbye. Never wrap up.\n`; + prompt += `- If silence, ask "you still there?" — never fill it with assumptions.\n`; + prompt += `- NEVER discuss connection quality or technical issues. Silently continue.\n`; + prompt += `- NEVER read PII aloud (phones, emails, addresses).\n`; + + return sanitizeForRealtime(prompt); +} + +/** + * Strip Unicode characters that have historically broken the OpenAI + * Realtime WebSocket connection in production. Em dashes, smart quotes, + * arrows, ellipsis — replace with ASCII equivalents. Other non-ASCII + * stripped entirely (kept conservative; multilingual support is deferred + * to a future eval-gated release). + * + * @param {string} text + * @returns {string} + */ +export function sanitizeForRealtime(text) { + if (!text) return text; + return text + .replace(/[—–]/g, '--') // em / en dash + .replace(/[‘’]/g, "'") // smart single quotes + .replace(/[“”]/g, '"') // smart double quotes + .replace(/→/g, '->') // right arrow + .replace(/←/g, '<-') // left arrow + .replace(/…/g, '...') // ellipsis + .replace(/[^\x00-\x7F]/g, ''); // strip any other non-ASCII +} diff --git a/recipes/agent-voice/code/public/call.html b/recipes/agent-voice/code/public/call.html new file mode 100644 index 000000000..e83562f07 --- /dev/null +++ b/recipes/agent-voice/code/public/call.html @@ -0,0 +1,328 @@ + + + + + Voice Agent + + + + +
+

Voice Agent

+
Press connect to start a voice session.
+
idle
+ +
+ persona: venus + +
+
+ + + + + + diff --git a/recipes/agent-voice/code/server.mjs b/recipes/agent-voice/code/server.mjs new file mode 100644 index 000000000..2ee680408 --- /dev/null +++ b/recipes/agent-voice/code/server.mjs @@ -0,0 +1,268 @@ +#!/usr/bin/env node +/** + * server.mjs — agent-voice reference server. + * + * WebRTC-first: the primary surface is browser-side via /call. + * GET / → redirect to /call + * GET /call → serves public/call.html (browser client) + * POST /session → SDP exchange with OpenAI Realtime; returns SDP answer + * POST /tool → tool-call dispatch from WebRTC data channel + * GET /health → {ok:true} liveness + * + * Twilio inbound (optional adapter): + * POST /voice → returns TwiML to open a Media Stream + * WSS /ws → Twilio↔OpenAI Realtime audio bridge + * POST /fallback → fallback TwiML (forward to operator's cell) + * + * The Twilio path is OPTIONAL; recipe Option A (WebRTC-only) doesn't need + * it. Operators wiring Twilio inbound implement the bridge themselves + * against `lib/twilio-bridge.mjs` (port-ready stubs included). + * + * Configuration via env: + * PORT default 8765 + * OPENAI_API_KEY required for /session + * OPENAI_REALTIME_MODEL default 'gpt-4o-realtime-preview' + * DEFAULT_PERSONA default 'venus' (one of 'mars' | 'venus') + * BRAIN_ROOT passed through to context-builder + * TIMEZONE passed through to context-builder + * + * Security posture: this is reference code. It does NOT ship hardening for + * production deployment (no rate limiting, no Twilio signature validation, + * no CORS allowlist). Operators add those at install time per the recipe's + * "production checklist." + */ + +import { createServer } from 'node:http'; +import { readFileSync, statSync, existsSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { dirname, join, extname } from 'node:path'; +import { buildSystemPrompt } from './prompt.mjs'; +import { dispatchTool, getEffectiveAllowlist } from './tools.mjs'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const PUBLIC_DIR = join(__dirname, 'public'); + +const PORT = parseInt(process.env.PORT || '8765', 10); +const DEFAULT_PERSONA = (process.env.DEFAULT_PERSONA || 'venus').toLowerCase(); +const OPENAI_REALTIME_MODEL = process.env.OPENAI_REALTIME_MODEL || 'gpt-4o-realtime-preview'; +const OPENAI_REALTIME_URL = 'https://api.openai.com/v1/realtime/calls'; + +const MIME = { + '.html': 'text/html; charset=utf-8', + '.js': 'application/javascript; charset=utf-8', + '.mjs': 'application/javascript; charset=utf-8', + '.css': 'text/css; charset=utf-8', + '.json': 'application/json; charset=utf-8', + '.wasm': 'application/wasm', + '.png': 'image/png', + '.svg': 'image/svg+xml', + '.ico': 'image/x-icon', +}; + +function send(res, status, body, headers = {}) { + res.writeHead(status, { 'content-type': 'text/plain; charset=utf-8', ...headers }); + res.end(body); +} + +function sendJson(res, status, obj) { + res.writeHead(status, { 'content-type': 'application/json; charset=utf-8' }); + res.end(JSON.stringify(obj)); +} + +async function readBody(req, max = 1 << 20) { + const chunks = []; + let n = 0; + for await (const chunk of req) { + n += chunk.length; + if (n > max) { + const err = new Error('payload too large'); + err.status = 413; + throw err; + } + chunks.push(chunk); + } + return Buffer.concat(chunks); +} + +function serveStatic(res, relPath) { + const full = join(PUBLIC_DIR, relPath); + if (!full.startsWith(PUBLIC_DIR)) return send(res, 403, 'forbidden'); + if (!existsSync(full)) return send(res, 404, 'not found'); + try { + const stat = statSync(full); + if (!stat.isFile()) return send(res, 404, 'not found'); + const body = readFileSync(full); + const mime = MIME[extname(full)] || 'application/octet-stream'; + res.writeHead(200, { + 'content-type': mime, + 'content-length': stat.size, + 'cache-control': 'no-cache', + }); + res.end(body); + } catch (err) { + send(res, 500, `read error: ${err.message}`); + } +} + +// ── /session: WebRTC SDP exchange with OpenAI Realtime ──────────────── +async function handleSession(req, res) { + if (req.method !== 'POST') return send(res, 405, 'method not allowed'); + if (!process.env.OPENAI_API_KEY) { + return sendJson(res, 500, { error: 'OPENAI_API_KEY not set' }); + } + + let sdpOffer; + try { + sdpOffer = (await readBody(req)).toString('utf8'); + } catch (err) { + return send(res, err.status || 400, err.message); + } + + if (!sdpOffer || !sdpOffer.startsWith('v=')) { + return sendJson(res, 400, { error: 'missing or malformed SDP offer' }); + } + + const url = new URL(req.url, `http://${req.headers.host}`); + const persona = (url.searchParams.get('persona') || DEFAULT_PERSONA).toLowerCase(); + + // Build the persona-aware system prompt at session start. + const systemPrompt = await buildSystemPrompt({ + persona, + brainRoot: process.env.BRAIN_ROOT, + timezone: process.env.TIMEZONE, + }); + + // Session config for OpenAI Realtime /v1/realtime/calls. + // Important gotchas (from production): + // - `voice` goes under `audio.output.voice`, NOT top-level + // - Do NOT send `turn_detection` (rejected by /v1/realtime/calls) + // - All `session.update` calls must include `type: 'realtime'` + const personaVoice = persona === 'mars' ? 'Orus' : 'Aoede'; + const sessionConfig = { + type: 'realtime', + model: OPENAI_REALTIME_MODEL, + audio: { output: { voice: personaVoice } }, + instructions: systemPrompt, + // Tools advertised to the model; the actual dispatch happens via /tool. + tools: getEffectiveAllowlist().map((name) => ({ + type: 'function', + name, + description: `gbrain operation: ${name}`, + parameters: { type: 'object', properties: {}, additionalProperties: true }, + })), + }; + + // OpenAI Realtime expects multipart/form-data with two parts: + // sdp: the WebRTC SDP offer + // session: JSON.stringify(sessionConfig) + const form = new FormData(); + form.set('sdp', sdpOffer); + form.set('session', JSON.stringify(sessionConfig)); + + try { + const upstream = await fetch(OPENAI_REALTIME_URL, { + method: 'POST', + headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` }, + body: form, + }); + if (!upstream.ok) { + const text = await upstream.text(); + console.error(`[session] OpenAI Realtime returned ${upstream.status}: ${text.slice(0, 400)}`); + return send(res, upstream.status, text); + } + const sdpAnswer = await upstream.text(); + res.writeHead(200, { 'content-type': 'application/sdp; charset=utf-8' }); + res.end(sdpAnswer); + } catch (err) { + console.error(`[session] upstream error: ${err.message}`); + sendJson(res, 502, { error: 'upstream_unreachable', detail: err.message }); + } +} + +// ── /tool: tool-call dispatch from the WebRTC data channel ──────────── +async function handleTool(req, res) { + if (req.method !== 'POST') return send(res, 405, 'method not allowed'); + let body; + try { + body = JSON.parse((await readBody(req)).toString('utf8')); + } catch (err) { + return sendJson(res, 400, { error: 'invalid_json', detail: err.message }); + } + const { name, arguments: params } = body || {}; + if (typeof name !== 'string') { + return sendJson(res, 400, { error: 'missing tool name' }); + } + const result = await dispatchTool(name, params || {}); + // dispatchTool always returns either {data} or {error}; never throws. + sendJson(res, 200, result); +} + +// ── /voice: Twilio TwiML stub (optional Twilio inbound) ─────────────── +function handleVoiceTwiml(req, res) { + const host = req.headers.host; + const twiml = ` + + + + +`; + res.writeHead(200, { 'content-type': 'text/xml; charset=utf-8' }); + res.end(twiml); +} + +// ── HTTP router ────────────────────────────────────────────────────── +const server = createServer(async (req, res) => { + const url = new URL(req.url, `http://${req.headers.host}`); + + // CORS: allow same-origin only by default. Operators relax in production. + res.setHeader('Access-Control-Allow-Origin', req.headers.origin || '*'); + res.setHeader('Access-Control-Allow-Methods', 'GET, POST, OPTIONS'); + res.setHeader('Access-Control-Allow-Headers', 'Content-Type, Authorization'); + if (req.method === 'OPTIONS') return send(res, 204, ''); + + try { + if (url.pathname === '/health') { + return sendJson(res, 200, { ok: true }); + } + if (url.pathname === '/' || url.pathname === '/call') { + return serveStatic(res, 'call.html'); + } + if (url.pathname === '/directory') { + return serveStatic(res, 'directory.html'); + } + if (url.pathname === '/session') { + return handleSession(req, res); + } + if (url.pathname === '/tool') { + return handleTool(req, res); + } + if (url.pathname === '/voice') { + return handleVoiceTwiml(req, res); + } + if (url.pathname.startsWith('/public/')) { + return serveStatic(res, url.pathname.slice('/public/'.length)); + } + // Static fallback for files in public/ at root path (e.g., /rnnoise-processor.js). + const candidate = url.pathname.slice(1); + if (candidate && !candidate.includes('..')) { + const candPath = join(PUBLIC_DIR, candidate); + if (existsSync(candPath) && statSync(candPath).isFile()) { + return serveStatic(res, candidate); + } + } + send(res, 404, 'not found'); + } catch (err) { + console.error(`[server] ${err.message}\n${err.stack}`); + if (!res.headersSent) sendJson(res, 500, { error: err.message }); + } +}); + +server.listen(PORT, () => { + // eslint-disable-next-line no-console + console.log(`[agent-voice] listening on http://localhost:${PORT}`); + console.log(`[agent-voice] default persona: ${DEFAULT_PERSONA}`); + console.log(`[agent-voice] read-only tools: ${getEffectiveAllowlist().join(', ')}`); +}); + +process.on('SIGTERM', () => server.close(() => process.exit(0))); +process.on('SIGINT', () => server.close(() => process.exit(0))); diff --git a/recipes/agent-voice/code/tools.mjs b/recipes/agent-voice/code/tools.mjs new file mode 100644 index 000000000..11a3cdb4b --- /dev/null +++ b/recipes/agent-voice/code/tools.mjs @@ -0,0 +1,175 @@ +/** + * tools.mjs — voice-callable tool router with a READ-ONLY allow-list. + * + * D14-A invariant: the voice agent can READ from the brain but cannot WRITE. + * Prompt injection of a persona prompt cannot escalate to brain-mutating ops + * (put_page, submit_job, file_upload, delete_page, etc.) without explicit + * operator opt-in via a local `tools-allowlist.local.json` override. + * + * Allow-list (default): + * - search (semantic + keyword search over pages/chunks) + * - query (RAG-shaped query with citations) + * - get_page (fetch a single page by slug) + * - list_pages (list pages with filters) + * - find_experts (people/companies with topical relevance) + * - get_recent_salience (recently-active emotionally-loaded pages) + * - get_recent_transcripts (voice-note / meeting transcripts) + * - read_article (fetch + summarize an arbitrary URL) + * + * Reject path: any tool name outside the allow-list returns a structured + * error envelope. The voice agent surfaces it as "I can't do that from + * voice." The router NEVER raises an exception that crashes the call. + * + * Local override: an operator who wants voice-callable writes places a + * `tools-allowlist.local.json` next to this file with shape: + * { "extend": ["set_reminder", "log_to_brain"] } + * Only operations declared in OPTIONAL_OPS below can be added — wholesale + * write access (put_page, submit_job, etc.) is intentionally NOT + * extensible from the override. + */ + +import { existsSync, readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { dirname, join } from 'node:path'; +import { callGbrainOp } from './gbrain-client.mjs'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +// Default read-only allow-list. KEEP IN SYNC with tests/unit/tools-allowlist.test.mjs. +export const READ_ONLY_OPS = Object.freeze([ + 'search', + 'query', + 'get_page', + 'list_pages', + 'find_experts', + 'get_recent_salience', + 'get_recent_transcripts', + 'read_article', +]); + +// Optional ops the operator MAY opt in to via tools-allowlist.local.json. +// These are write-shaped but bounded: a reminder, a brain-log, an enrich +// nudge. Crucially: not put_page (no arbitrary brain edits), not submit_job +// (no shell), not file_upload, not delete_page. +export const OPTIONAL_OPS = Object.freeze([ + 'set_reminder', + 'log_to_brain', + 'enrich_request', +]); + +// Permanent denylist — never extendable via override, regardless of what the +// override file says. These cross the trust boundary in ways that would +// allow a prompt-injection attack to escalate to RCE or arbitrary write. +export const DENYLIST = Object.freeze([ + 'put_page', + 'submit_job', + 'file_upload', + 'delete_page', + 'file_url', + 'sync_brain', + 'apply_migrations', + 'work', + 'shell', +]); + +/** + * Read the local override file (if present) and compute the effective + * allow-list. The override may add OPTIONAL_OPS but cannot add DENYLIST ops. + * + * Result is memoized per-process (the override file is read once at first + * call). Reset for tests by passing `{forceReload: true}`. + */ +let _effective; +export function getEffectiveAllowlist({ forceReload = false } = {}) { + if (_effective && !forceReload) return _effective; + + const allowed = new Set(READ_ONLY_OPS); + const overridePath = join(__dirname, 'tools-allowlist.local.json'); + if (existsSync(overridePath)) { + try { + const override = JSON.parse(readFileSync(overridePath, 'utf8')); + if (Array.isArray(override?.extend)) { + for (const name of override.extend) { + if (typeof name !== 'string') continue; + if (DENYLIST.includes(name)) { + console.warn(`[tools] ignoring denylisted op in override: ${name}`); + continue; + } + if (!OPTIONAL_OPS.includes(name)) { + console.warn(`[tools] ignoring unknown op in override: ${name}`); + continue; + } + allowed.add(name); + } + } + } catch (err) { + console.warn(`[tools] failed to parse ${overridePath}: ${err.message}`); + } + } + + _effective = Object.freeze([...allowed]); + return _effective; +} + +/** + * Structured error envelope returned to the voice persona when a tool is + * rejected. The persona sees this as a tool-call result, not as a thrown + * exception, so the call never hangs. + */ +export function rejectedToolResult(opName, reason) { + return { + error: { + code: reason, + op: opName, + message: + reason === 'denylisted' + ? `Tool "${opName}" is permanently disabled in voice agents. Use a different surface.` + : `Tool "${opName}" is not in the voice agent's allow-list. The operator can opt in via tools-allowlist.local.json if appropriate.`, + }, + }; +} + +/** + * Dispatch a tool call from the voice persona to gbrain MCP. + * + * Returns either {data: } on success or {error: } on + * rejection or MCP failure. Never throws. + */ +export async function dispatchTool(opName, params, { forceReload = false } = {}) { + // Denylist check first — even if someone added it to the allow-list, + // denylisted ops are never callable from voice. + if (DENYLIST.includes(opName)) { + return rejectedToolResult(opName, 'denylisted'); + } + + const allowlist = getEffectiveAllowlist({ forceReload }); + if (!allowlist.includes(opName)) { + return rejectedToolResult(opName, 'not_in_allowlist'); + } + + // Allow-listed — call gbrain MCP. + try { + const result = await callGbrainOp(opName, params); + return { data: result }; + } catch (err) { + return { + error: { + code: 'mcp_error', + op: opName, + message: err?.message || String(err), + }, + }; + } +} + +/** + * Inspect helper for tests / introspection. Returns the full picture. + */ +export function describeAllowlist({ forceReload = false } = {}) { + return { + read_only: [...READ_ONLY_OPS], + optional: [...OPTIONAL_OPS], + denylist: [...DENYLIST], + effective: getEffectiveAllowlist({ forceReload }), + }; +} diff --git a/recipes/agent-voice/install/manifest.json b/recipes/agent-voice/install/manifest.json new file mode 100644 index 000000000..c8b2a626d --- /dev/null +++ b/recipes/agent-voice/install/manifest.json @@ -0,0 +1,46 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "recipe": "agent-voice", + "version": "0.1.0", + "install_kind": "copy-into-host-repo", + "description": "src → target mapping consumed by `gbrain integrations install agent-voice`. The install command reads this manifest, computes SHA-256 of each source file at copy time, writes the per-file hash to /services/voice-agent/.gbrain-source.json so --refresh can do three-way classification (unchanged-identical / unchanged-stale / locally-modified).", + "target_root_relative_to_host_repo": "services/voice-agent", + "skills_target_root_relative_to_host_repo": "skills", + "files": [ + { "src": "package.json", "target": "services/voice-agent/package.json", "mode": "0644" }, + { "src": "code/server.mjs", "target": "services/voice-agent/code/server.mjs", "mode": "0755" }, + { "src": "code/prompt.mjs", "target": "services/voice-agent/code/prompt.mjs", "mode": "0644" }, + { "src": "code/tools.mjs", "target": "services/voice-agent/code/tools.mjs", "mode": "0644" }, + { "src": "code/gbrain-client.mjs", "target": "services/voice-agent/code/gbrain-client.mjs", "mode": "0644" }, + { "src": "code/lib/personas/personas.mjs", "target": "services/voice-agent/code/lib/personas/personas.mjs", "mode": "0644" }, + { "src": "code/lib/personas/mars.mjs", "target": "services/voice-agent/code/lib/personas/mars.mjs", "mode": "0644" }, + { "src": "code/lib/personas/venus.mjs", "target": "services/voice-agent/code/lib/personas/venus.mjs", "mode": "0644" }, + { "src": "code/lib/personas/private-name-blocklist.json", "target": "services/voice-agent/code/lib/personas/private-name-blocklist.json", "mode": "0644" }, + { "src": "code/lib/personas/context-builder.contract.md", "target": "services/voice-agent/code/lib/personas/context-builder.contract.md", "mode": "0644" }, + { "src": "code/lib/context-builder.example.mjs", "target": "services/voice-agent/code/lib/context-builder.example.mjs", "mode": "0644" }, + { "src": "code/lib/audio-convert.mjs", "target": "services/voice-agent/code/lib/audio-convert.mjs", "mode": "0644" }, + { "src": "code/lib/gatekeeper.mjs", "target": "services/voice-agent/code/lib/gatekeeper.mjs", "mode": "0644" }, + { "src": "code/lib/sessions.mjs", "target": "services/voice-agent/code/lib/sessions.mjs", "mode": "0644" }, + { "src": "code/lib/twilio-bridge.mjs", "target": "services/voice-agent/code/lib/twilio-bridge.mjs", "mode": "0644" }, + { "src": "code/lib/upstream-classifier.mjs", "target": "services/voice-agent/code/lib/upstream-classifier.mjs", "mode": "0644" }, + { "src": "code/public/call.html", "target": "services/voice-agent/code/public/call.html", "mode": "0644" }, + { "src": "tests/unit/personas.test.mjs", "target": "services/voice-agent/tests/unit/personas.test.mjs", "mode": "0644" }, + { "src": "tests/unit/mars-prompt-shape.test.mjs", "target": "services/voice-agent/tests/unit/mars-prompt-shape.test.mjs", "mode": "0644" }, + { "src": "tests/unit/venus-prompt-shape.test.mjs", "target": "services/voice-agent/tests/unit/venus-prompt-shape.test.mjs", "mode": "0644" }, + { "src": "tests/unit/tools-allowlist.test.mjs", "target": "services/voice-agent/tests/unit/tools-allowlist.test.mjs", "mode": "0644" }, + { "src": "tests/unit/upstream-classifier.test.mjs", "target": "services/voice-agent/tests/unit/upstream-classifier.test.mjs", "mode": "0644" } + ], + "skills": [ + { "src": "skills/voice-persona-mars/SKILL.md", "target": "skills/voice-persona-mars/SKILL.md", "mode": "0644" }, + { "src": "skills/voice-persona-mars/routing-eval.jsonl", "target": "skills/voice-persona-mars/routing-eval.jsonl", "mode": "0644" }, + { "src": "skills/voice-persona-venus/SKILL.md", "target": "skills/voice-persona-venus/SKILL.md", "mode": "0644" }, + { "src": "skills/voice-persona-venus/routing-eval.jsonl", "target": "skills/voice-persona-venus/routing-eval.jsonl", "mode": "0644" }, + { "src": "skills/voice-post-call/SKILL.md", "target": "skills/voice-post-call/SKILL.md", "mode": "0644" }, + { "src": "skills/voice-post-call/routing-eval.jsonl", "target": "skills/voice-post-call/routing-eval.jsonl", "mode": "0644" } + ], + "resolver_rows_to_append": [ + "voice-persona-mars | \"talk to mars\", \"mars,\", \"demo mode mars\", \"introspective\", \"thought partner\"", + "voice-persona-venus | \"venus,\", \"calendar\", \"tasks\", \"executive\", \"schedule\"", + "voice-post-call | \"after the call\", \"call ended\", \"transcript\", \"call summary\"" + ] +} diff --git a/recipes/agent-voice/install/post-install-hint.md b/recipes/agent-voice/install/post-install-hint.md new file mode 100644 index 000000000..c2f906d79 --- /dev/null +++ b/recipes/agent-voice/install/post-install-hint.md @@ -0,0 +1,98 @@ +# Post-install hint + +When `gbrain integrations install agent-voice --target ` completes, print this to stdout (or to the install agent's conversation surface) so the operator knows what to do next. + +--- + +✓ Voice agent reference installed to `/services/voice-agent/`. + +**Three follow-up steps before this works end-to-end:** + +### 1. Set required env vars in `/.env` + +```bash +OPENAI_API_KEY=sk-... # required (OpenAI Realtime API) +DEFAULT_PERSONA=venus # optional (one of: venus, mars) +BRAIN_ROOT=/path/to/your/brain # optional (enables live context) +TIMEZONE=US/Pacific # optional +``` + +Optional for inbound Twilio: +```bash +TWILIO_ACCOUNT_SID=AC... +TWILIO_AUTH_TOKEN=... +``` + +### 2. Implement your context builder (recommended) + +The shipped `/services/voice-agent/code/lib/context-builder.example.mjs` is a working example assuming a documented brain layout (`$BRAIN_ROOT/memory/YYYY-MM-DD.md`, `$BRAIN_ROOT/SOUL.md`, etc.). If your brain has a different layout, edit this file to match — the contract at `/services/voice-agent/code/lib/personas/context-builder.contract.md` documents the API. + +A degraded operator with no context-builder still gets a working voice agent — Mars asks open questions, Venus answers logistics with a "I can't see your calendar from here — what do you need?" fallback. + +### 3. Wire your resolver + +This install appended three rows to your `/RESOLVER.md` (or `AGENTS.md`): + +``` +voice-persona-mars | "talk to mars", "mars,", "demo mode mars", ... +voice-persona-venus | "venus,", "calendar", "tasks", ... +voice-post-call | "after the call", "call ended", "transcript", ... +``` + +Review them. If your resolver uses different conventions, edit per your style. + +### 4. Run the host-side tests once + +```bash +cd /services/voice-agent +bun install # or `npm install` if your repo uses npm +bun run test # all unit suites should pass green +``` + +If any prompt-shape test fails, the privacy guard has caught a name you'd want to scrub — see `code/lib/personas/private-name-blocklist.json` for the contract. + +### 5. Start the server + +```bash +cd /services/voice-agent +bun run start # or `npm start` +# → listening on http://localhost:8765 +``` + +Open `http://localhost:8765/call` in a browser, click Connect, grant mic permission. You should be talking to Venus (or Mars if you set `DEFAULT_PERSONA=mars`). + +### 6. (Optional) Run the WebRTC roundtrip E2E + +```bash +export AGENT_VOICE_E2E=1 OPENAI_API_KEY=sk-... +bun run test:e2e +# → ~$0.10/run; spawns server, drives puppeteer with a fake-audio WAV +``` + +Or the full openclaw-wrapped flow (requires `OPENCLAW_BIN`, `ANTHROPIC_API_KEY`): + +```bash +export AGENT_VOICE_FULL_E2E=1 +gbrain claw-test --scenario voice-agent-install --live --agent openclaw +# → ~$1-2/run; friction-discovery test, NOT a ship gate +``` + +### 7. (Optional) Run the LLM-judge persona evals + +```bash +cd /services/voice-agent +node tests/evals/mars-eval.mjs # ~$1-3 for the full 3-model judge sweep +node tests/evals/venus-eval.mjs +``` + +Synthetic canonical baselines are committed under `tests/evals/baseline-runs/canonical/`. Live receipts you generate go to `tests/evals/baseline-runs/` (gitignored — they may contain residual brain content from your live personas). + +### 8. Update later + +When gbrain ships a new agent-voice reference, refresh your local copy: + +```bash +gbrain integrations install agent-voice --target --refresh +``` + +The refresh classifies each file (identical / stale / locally-modified / source-deleted / host-deleted) and lets you decide per-file. See `/services/voice-agent/code/install/refresh-algorithm.md` (copied from gbrain) for the contract. diff --git a/recipes/agent-voice/install/refresh-algorithm.md b/recipes/agent-voice/install/refresh-algorithm.md new file mode 100644 index 000000000..35fd406cc --- /dev/null +++ b/recipes/agent-voice/install/refresh-algorithm.md @@ -0,0 +1,74 @@ +# Refresh algorithm (diff-and-propose) + +`gbrain integrations install agent-voice --refresh` re-walks the manifest, classifies every file into one of five states, and lets the operator decide per-file. The reference implementation is in `src/commands/integrations.ts` under the `install_kind: copy-into-host-repo` branch. + +## State machine + +For each file declared in `install/manifest.json`: + +``` +Let src_hash = SHA-256 of gbrain-side file at manifest.src +Let host_path = / +Let recorded = .gbrain-source.json.files[].sha256 for this entry (or absent if first refresh) +Let host_hash = SHA-256 of host_path (or absent if file deleted on host side) + +State: + - "unchanged-identical" iff host_hash == src_hash + → no-op + - "unchanged-stale" iff host_hash == recorded AND host_hash != src_hash + → operator unmodified, source moved → offer update + - "locally-modified" iff host_hash != recorded AND host_hash != src_hash AND host_hash is defined + → operator edited locally; offer three options (see below) + - "host-deleted" iff host_hash is absent AND src exists + → operator removed the file; offer to restore or to remove from manifest + - "source-deleted" iff src is absent AND host_hash is defined + → gbrain reference removed the file; offer cleanup (remove from host) +``` + +A path-mapping renames table in the manifest (`renames: [{from, to}]`, not yet shipped) allows the refresh algorithm to detect a source-renamed file as a logical update rather than a delete+add. + +## "Locally-modified" decision + +When a file shows `locally-modified`, the operator picks one of three options: + +- **keep-mine** — leave host file untouched. The manifest entry's `sha256` is updated to the current host hash (the operator's edit becomes the new "recorded" baseline; future refreshes won't re-flag it until they edit it again OR the source changes). +- **take-theirs** — copy the gbrain reference over the host file. The recorded SHA becomes the new src_hash. +- **merge** — print a unified diff. Operator hand-merges in their editor; the refresh command exits without writing. Re-run `--refresh` after the merge to confirm. + +## Transaction journal + +`/services/voice-agent/.gbrain-source.refresh.log` is a JSONL append-only file. Each line records: + +```json +{"ts": "2026-05-17T12:34:56Z", "src": "code/server.mjs", "state": "locally-modified", "decision": "keep-mine"} +``` + +The journal exists for two reasons: +1. **Partial-apply recovery.** If the refresh is interrupted mid-loop (Ctrl-C, crash, machine reboot), re-running `--refresh` reads the journal and resumes where it stopped. +2. **Audit.** Operators can grep the journal to see which files were touched and why. + +The journal is rotated by file size (>1MB triggers rename to `.gbrain-source.refresh.log.1`) and ignored by `--refresh`'s own scan (the journal is host-only metadata, not a managed file). + +## Concurrent refresh guard + +`--refresh` acquires an advisory file lock at `/services/voice-agent/.gbrain-source.refresh.lock` for the duration of the run. Concurrent `--refresh` invocations on the same host repo fail-fast with "refresh already in progress." + +## CLI surface + +```bash +gbrain integrations install agent-voice --target --refresh +gbrain integrations install agent-voice --target --refresh --dry-run # report-only +gbrain integrations install agent-voice --target --refresh --auto take-theirs # non-interactive +gbrain integrations install agent-voice --target --refresh --auto keep-mine # bias toward operator's edits +``` + +`--auto ` applies the named decision to ALL `locally-modified` files without prompting. Useful for CI lanes that want either "always take upstream" or "always preserve local" without operator interaction. + +## What this v0 deliberately skips + +- Conflict resolution for files that exist in both manifests but at different paths (treated as add+delete). +- Concurrent edits on the SAME file mid-refresh (the advisory lock + per-file atomic write covers this). +- Semantic merges (we offer file-level diff only; no per-hunk picking). +- Manifest schema migration (v0.1.0 → v0.2.0 changes are handled by the install command refusing to refresh old manifests and asking the operator to re-install). + +Each of those is a follow-up TODO. diff --git a/recipes/agent-voice/package.json b/recipes/agent-voice/package.json new file mode 100644 index 000000000..bdc343f6e --- /dev/null +++ b/recipes/agent-voice/package.json @@ -0,0 +1,27 @@ +{ + "name": "agent-voice", + "version": "0.1.0", + "description": "Voice agent reference (Mars + Venus personas, WebRTC-first browser client, optional Twilio adapter). Copied into the operator's host agent repo via `gbrain integrations install agent-voice`; not a runtime gbrain dependency.", + "type": "module", + "main": "code/server.mjs", + "scripts": { + "start": "node code/server.mjs", + "test": "vitest run", + "test:watch": "vitest", + "test:e2e": "vitest run tests/e2e/voice-roundtrip.test.mjs", + "test:full-flow": "vitest run tests/e2e/voice-full-flow.test.mjs", + "gen:baselines": "node tests/evals/mars-eval.mjs --baseline && node tests/evals/venus-eval.mjs --baseline" + }, + "dependencies": { + "ws": "^8.16.0" + }, + "devDependencies": { + "puppeteer": "^23.11.1", + "vitest": "^4.1.4" + }, + "engines": { + "node": ">=18.0.0" + }, + "private": false, + "license": "MIT" +} diff --git a/recipes/agent-voice/skills/voice-persona-mars/SKILL.md b/recipes/agent-voice/skills/voice-persona-mars/SKILL.md new file mode 100644 index 000000000..aa582afca --- /dev/null +++ b/recipes/agent-voice/skills/voice-persona-mars/SKILL.md @@ -0,0 +1,90 @@ +--- +name: voice-persona-mars +version: 0.1.0 +description: Route to Mars (introspective thought partner / demo showman voice persona). Used when the operator wants depth, meaning, or impressive social demos rather than logistics. Mars handles SOLO mode (philosophy, presence, patterns) and DEMO mode (tool-driven showmanship) automatically. +triggers: + - "talk to mars" + - "mars," + - "ask mars" + - "demo mode mars" + - "what's on my mind" + - "what am I thinking" + - "introspective" + - "thought partner" +mutating: false +writes_pages: false +writes_to: [] +--- + +# voice-persona-mars — Introspective thought partner / demo showman + +> **Convention:** see [voice-persona-venus/SKILL.md](../voice-persona-venus/SKILL.md) for the sister persona that handles logistics. +> +> **Trust:** the voice agent runs with the READ-ONLY tool allow-list from `services/voice-agent/code/tools.mjs`. Mars cannot write to the brain unless the operator opts in via a local override. + +## Iron Law + +**Mars is not the assistant.** Mars helps the operator hear what they're actually thinking. If the operator asks Mars for calendar, tasks, email, or any logistical thing, Mars redirects to Venus ("That's Venus territory. What's on your mind?") and does NOT attempt the logistical task. + +The depth of the conversation is the signal. If it's surface-level scheduling, route to Venus. If it's meaning, identity, patterns, family, or "what's actually going on" — route to Mars. + +## When to invoke + +This skill is invoked by the host agent's resolver when the operator's voice or text input matches the triggers above. The voice agent (`services/voice-agent/code/server.mjs`) consumes the persona key (`mars`) at session start via `?persona=mars` on the WebRTC `/session` endpoint, OR via the `DEFAULT_PERSONA=mars` env var if Mars is the operator's default. + +## Mode detection (inside the persona) + +Mars detects mode from conversational signals: +- **SOLO MODE** (default): one speaker (the operator), introspective topics, "what am I thinking" framing. +- **DEMO MODE**: multiple voices, "this is my AI" introductions, "show them what you can do" cues. + +The persona prompt (`services/voice-agent/code/lib/personas/mars.mjs`) carries the full mode-detection contract. The resolver only needs to route the SESSION to Mars; mode-switching happens inside the running session. + +## Solo-mode tool posture + +Mars uses tools SPARINGLY in solo mode. The right tools are: +- `search_brain` (find related concepts/people/meetings to deepen the reflection) +- `read_brain_page` (read a specific page aloud when the operator says "tell me about X") +- `read_article` (summarize a link the operator shared) + +Calendar, tasks, email tools are DELIBERATELY ABSENT from Mars's solo-mode usage even though they're in the read-only allow-list. Mars redirects logistical questions to Venus. + +## Demo-mode tool posture + +Mars uses tools AGGRESSIVELY in demo mode: +- Search the brain for people/companies the operator introduces +- Pull current events via `web_search` (when wired) +- Cross-reference what the operator is saying against the brain in near-real-time + +The goal: make the demo audience think "oh, this is what a personal AI can actually do." + +## Language + +Mars is **English-only** in this release. Multilingual support is gated on an eval that hasn't shipped yet. If the operator (or a demo audience) uses another language, Mars responds in English and notes briefly: "I'm running English-only right now." + +## Anti-patterns + +- ❌ Mars handling calendar / tasks / email despite the persona prompt's redirect rule. If the model drifts, the resolver should re-route to Venus mid-session via a `?persona=venus` re-init. +- ❌ Mars naming specific family members or therapists. The shipped prompt has no PII; the operator's `buildMarsContext()` implementation supplies live emotional signal at session-start, but the persona NEVER recites it verbatim. +- ❌ Mars claiming multilingual capability before the multilingual eval lands. +- ❌ Voice-side write-tool invocations. If the operator says "save this," Mars says "I can't save from voice; tell me again when you're at your screen." + +## Related skills + +- [voice-persona-venus](../voice-persona-venus/SKILL.md) — the logistics-focused sister persona +- [voice-post-call](../voice-post-call/SKILL.md) — what to do with the transcript after a Mars session + +## Contract + +This skill guarantees: + +- Routing matches the canonical triggers in the frontmatter. +- The voice agent session opened with `?persona=mars` (or `DEFAULT_PERSONA=mars`) uses the prompt from `services/voice-agent/code/lib/personas/mars.mjs`. +- No write tools are callable from voice (D14-A allow-list) unless the operator has opted in locally. +- Privacy contract preserved: no real names, no hardcoded private filesystem paths, no upstream-agent codenames in the shipped prompt. Enforced by `scripts/check-no-pii-in-agent-voice.sh` and `tests/unit/mars-prompt-shape.test.mjs`. + +## Output Format + +The voice persona produces SPOKEN audio over WebRTC, not text output. The Output Format header exists for `test/skills-conformance.test.ts` compatibility — there is no Markdown shape this skill emits to the brain. + +The post-call transcript (if any) is created by the [voice-post-call](../voice-post-call/SKILL.md) skill, not by this one. diff --git a/recipes/agent-voice/skills/voice-persona-mars/routing-eval.jsonl b/recipes/agent-voice/skills/voice-persona-mars/routing-eval.jsonl new file mode 100644 index 000000000..7633df42b --- /dev/null +++ b/recipes/agent-voice/skills/voice-persona-mars/routing-eval.jsonl @@ -0,0 +1,10 @@ +{"intent": "talk to mars about how I'm feeling", "expected_skill": "voice-persona-mars"} +{"intent": "mars, what's on my mind", "expected_skill": "voice-persona-mars"} +{"intent": "demo mode mars", "expected_skill": "voice-persona-mars"} +{"intent": "hey mars, meet my friend", "expected_skill": "voice-persona-mars"} +{"intent": "I want to talk to my thought partner", "expected_skill": "voice-persona-mars"} +{"intent": "let's get introspective", "expected_skill": "voice-persona-mars"} +{"intent": "what am I really thinking about this", "expected_skill": "voice-persona-mars"} +{"intent": "venus, what's on my calendar", "expected_skill": "voice-persona-venus", "ambiguous_with": "voice-persona-mars"} +{"intent": "ingest this voice memo", "expected_skill": "voice-note-ingest", "ambiguous_with": "voice-persona-mars"} +{"intent": "ask the brain about quantum mechanics", "expected_skill": "brain-ops", "ambiguous_with": "voice-persona-mars"} diff --git a/recipes/agent-voice/skills/voice-persona-venus/SKILL.md b/recipes/agent-voice/skills/voice-persona-venus/SKILL.md new file mode 100644 index 000000000..add2f2c8a --- /dev/null +++ b/recipes/agent-voice/skills/voice-persona-venus/SKILL.md @@ -0,0 +1,90 @@ +--- +name: voice-persona-venus +version: 0.1.0 +description: Route to Venus (sharp executive-assistant voice persona). Used for logistics — calendar, tasks, recent messages, brain lookups — at sub-second phone-call latency. The default voice persona unless DEFAULT_PERSONA=mars is set. +triggers: + - "venus," + - "ask venus" + - "calendar" + - "what's on my calendar" + - "tasks" + - "what are my tasks" + - "schedule" + - "executive" + - "logistics" +mutating: false +writes_pages: false +writes_to: [] +--- + +# voice-persona-venus — Executive assistant voice + +> **Convention:** see [voice-persona-mars/SKILL.md](../voice-persona-mars/SKILL.md) for the sister persona that handles depth + meaning. +> +> **Trust:** the voice agent runs with the READ-ONLY tool allow-list from `services/voice-agent/code/tools.mjs`. Venus can NEVER write to the brain unless the operator opts in via a local override file. + +## Iron Law + +**Speed is the signal.** A fast, short, opinionated answer beats a slow, perfect one. Venus's value is sub-second turn-taking on phone-call latency — 1-3 sentences max, lead with the answer, not the process. + +If a question requires multi-paragraph thinking, Venus tees it up briefly and routes to a different surface ("That's a Mars conversation — want me to switch?" or "Hit me on Slack with this one"). She doesn't deliver long-form answers. + +## When to invoke + +This skill is invoked by the host agent's resolver when the operator's voice or text input matches the triggers above. The voice agent (`services/voice-agent/code/server.mjs`) reads the persona key (`venus`) at session start via `?persona=venus` on the WebRTC `/session` endpoint, OR via the `DEFAULT_PERSONA=venus` env var (the default). + +## Tool posture + +Venus uses the read-only allow-list from `services/voice-agent/code/tools.mjs`: + +- `search_brain` (semantic + keyword search) +- `read_brain_page` (full page read aloud) +- `read_article` (URL fetch + summarize) +- `web_search` (when wired) +- `get_recent_salience` (what's been emotionally active lately) +- `get_recent_transcripts` (recent voice notes / meeting transcripts) +- `find_experts` (who knows about a topic) + +Write tools (`put_page`, `submit_job`, `set_reminder` unless opted in, etc.) are NOT in Venus's tool surface. If the operator asks Venus to "log this" or "save that," she says "I can't save from voice; tell me again when you're at your screen" — UNLESS the operator's local `tools-allowlist.local.json` opts into the bounded write set. + +## Language + +Venus is **English-only**. Her voice (`Aoede`) is configured for English. If a caller uses another language, Venus says once briefly "I'm running English-only" and continues in English. Do NOT loop on the language disclaimer. + +## Conversation timing + +Production-tested rule: +- **Caller talking or thinking** (incomplete sentence or 3-5 second pause mid-thought): SHUT UP. Wait. +- **Caller done** (complete thought + 2-3 seconds silence): RESPOND NOW. +- **Hard rule:** never let silence go past 5 seconds after a complete thought. + +This rule belongs in the persona prompt itself (`services/voice-agent/code/lib/personas/venus.mjs`) — the resolver only needs to route the session to Venus. + +## Anti-patterns + +- ❌ Long-form answers. Venus is NOT a chatbot; she's a phone-call assistant. +- ❌ Filler ("Great question!", "Let me think about that for a moment"). Always lead with the answer. +- ❌ Sycophancy. Venus has opinions and says them. +- ❌ Reading PII aloud (phones, emails, addresses) — the persona prompt disallows this regardless of context. +- ❌ Trying to write to the brain from voice without operator opt-in. +- ❌ Looping on the language disclaimer. + +## Related skills + +- [voice-persona-mars](../voice-persona-mars/SKILL.md) — the depth-focused sister persona +- [voice-post-call](../voice-post-call/SKILL.md) — post-session transcript handling + +## Contract + +This skill guarantees: + +- Routing matches the canonical triggers in the frontmatter. +- The voice agent session opened with `?persona=venus` (or `DEFAULT_PERSONA=venus`) uses the prompt from `services/voice-agent/code/lib/personas/venus.mjs`. +- Venus's tool surface is read-only by default; opt-in writes go through a local override file (`services/voice-agent/code/tools-allowlist.local.json`). +- Privacy contract preserved: no PII, no upstream-agent codenames, no cross-persona claims by name. Enforced by `scripts/check-no-pii-in-agent-voice.sh` and `tests/unit/venus-prompt-shape.test.mjs`. + +## Output Format + +The voice persona produces SPOKEN audio over WebRTC, not text output. The Output Format header exists for `test/skills-conformance.test.ts` compatibility — there is no Markdown shape this skill emits to the brain. + +The post-call transcript (if any) is created by the [voice-post-call](../voice-post-call/SKILL.md) skill, not by this one. diff --git a/recipes/agent-voice/skills/voice-persona-venus/routing-eval.jsonl b/recipes/agent-voice/skills/voice-persona-venus/routing-eval.jsonl new file mode 100644 index 000000000..f6074da62 --- /dev/null +++ b/recipes/agent-voice/skills/voice-persona-venus/routing-eval.jsonl @@ -0,0 +1,9 @@ +{"intent": "venus, what's on my calendar today", "expected_skill": "voice-persona-venus"} +{"intent": "ask venus about tasks", "expected_skill": "voice-persona-venus"} +{"intent": "what's my schedule looking like", "expected_skill": "voice-persona-venus"} +{"intent": "any messages I should see", "expected_skill": "voice-persona-venus"} +{"intent": "remind me about the meeting", "expected_skill": "voice-persona-venus"} +{"intent": "what are my open tasks", "expected_skill": "voice-persona-venus"} +{"intent": "talk to mars about how I'm feeling", "expected_skill": "voice-persona-mars", "ambiguous_with": "voice-persona-venus"} +{"intent": "ingest this voice memo", "expected_skill": "voice-note-ingest", "ambiguous_with": "voice-persona-venus"} +{"intent": "after the call, summarize it", "expected_skill": "voice-post-call", "ambiguous_with": "voice-persona-venus"} diff --git a/recipes/agent-voice/skills/voice-post-call/SKILL.md b/recipes/agent-voice/skills/voice-post-call/SKILL.md new file mode 100644 index 000000000..cdad18c4a --- /dev/null +++ b/recipes/agent-voice/skills/voice-post-call/SKILL.md @@ -0,0 +1,164 @@ +--- +name: voice-post-call +version: 0.1.0 +description: Post-call handling for a voice session — turn the transcript into a brain page, post the summary to the operator's messaging surface, archive the audio. Belt-and-suspenders: fires both from a tool the voice persona can call mid-call AND from the automatic call-end handler in server.mjs. +triggers: + - "after the call" + - "call ended" + - "summarize the call" + - "call transcript" + - "voice call summary" + - "post call summary" +mutating: true +writes_pages: true +writes_to: + - meetings/ + - voice-calls/ +--- + +# voice-post-call — Post-session transcript + summary handling + +> **Convention:** see [conventions/quality.md](../conventions/quality.md) for citation rules + back-link enforcement. +> +> **Convention:** see [_brain-filing-rules.md](../_brain-filing-rules.md) for filing decision protocol. + +## Iron Law + +**Every call gets processed, even on tool-call failure.** The voice persona MAY call a `log_call_summary` tool mid-session, OR the call may end without that tool firing (model forgot, WebRTC dropped, browser crashed). The automatic call-end handler in `services/voice-agent/code/server.mjs` posts a structured signal regardless so the brain still gets the transcript + audio reference. + +If both paths fire (the tool call AND the call-end handler), the second one is idempotent — it sees the brain page already exists and updates instead of duplicating. + +## The pipeline + +``` +1. CAPTURE → MediaRecorder on the host repo's voice-agent service captures + the full call audio (webm/opus) to /tmp/calls/-.webm. + The browser client at /call?test=1 also captures via WebAudio-tee + for E2E asserts; production /call uses server-side capture only. +2. TRANSCRIBE → Whisper (via gbrain transcription) processes the audio. Output: + full transcript (timestamped) + speaker labels where possible. +3. SUMMARIZE → A separate LLM call produces a 3-5 sentence summary covering + key topics, decisions, and unresolved items. +4. WRITE → Create or update meetings/YYYY-MM-DD-call-.md with: + - frontmatter (date, persona, duration, ratings) + - full transcript in a "Transcript" block-quote section + - summary in a "Summary" section + - audio link (file://, or signed URL if uploaded to storage) + - any entity cross-links (people, companies mentioned) +5. CROSS-LINK → For each entity in the transcript (person, company), append a + timeline entry to people/.md or companies/.md pointing + back to this call page. Iron Law: per conventions/quality.md. +6. POST → Send the summary to the operator's messaging surface (Telegram, + Slack, Discord — whichever is wired in $TARGET_REPO/.env). +``` + +## Two firing paths (belt + suspenders) + +**Path A — Persona-initiated mid-call:** +The voice persona calls `log_call_summary` via the WebRTC data channel. The host-repo `/tool` endpoint dispatches to `tools.mjs`. Note: `log_call_summary` is in `OPTIONAL_OPS`, not `READ_ONLY_OPS`, so this only works if the operator's `tools-allowlist.local.json` opts in. + +**Path B — Automatic call-end (default):** +When the WebSocket / WebRTC connection closes, `server.mjs` fires a `call_end` event. The host repo's post-call handler (operator-implemented; the recipe ships a stub) reads the captured audio + transcript, runs the pipeline above. This path requires NO operator opt-in to work — the call-end handler is part of the shipped server. + +## Brain page format + +```markdown +--- +type: meeting +subtype: voice-call +persona: venus +date: 2026-05-17 +duration_sec: 124 +caller: operator +rating: 7 +issues: [] +audio_url: "file:///tmp/calls/2026-05-17-1029-venus.webm" +created: 2026-05-17 +--- + +# Voice call: 2026-05-17 with Venus + +> Brief 3-5 sentence summary of what was discussed and any decisions made. + +## Summary +[Agent-authored 3-5 sentence summary covering topics, decisions, action items.] + +## Transcript + +> [Verbatim per-turn transcript with speaker labels and timestamps. Pure quote +> — do not paraphrase. Block-quoted because the exact wording matters more +> than a cleaned-up version.] + +🔊 [Audio](file:///tmp/calls/2026-05-17-1029-venus.webm) + +## Entities mentioned +- [Person](people/.md) +- [Company](companies/.md) + +## Timeline + +- **2026-05-17 10:29 PT** | voice call with Venus, 124s, rating 7 — [topic] +``` + +## Citation format + +``` +[Source: voice call with , YYYY-MM-DD HH:MM PT] +``` + +## Anti-patterns + +- ❌ Paraphrasing the transcript. The verbatim text IS the signal; the summary is the agent's interpretation. +- ❌ Skipping the audio archive step. Every call has a recoverable audio file. +- ❌ Skipping entity cross-links when people/companies are mentioned. Iron Law fail. +- ❌ Posting to messaging WITHOUT writing the brain page first. The messaging summary is a notification, not the canonical record. +- ❌ Letting Path A's success suppress Path B. They MAY both fire; the second one is idempotent and serves as a redundant safety net. + +## Related skills + +- [voice-persona-mars](../voice-persona-mars/SKILL.md) — the persona that may invoke this +- [voice-persona-venus](../voice-persona-venus/SKILL.md) — the other persona that may invoke this +- [meeting-ingestion](../meeting-ingestion/SKILL.md) — analogous flow for multi-party meeting transcripts (different in that voice-call is typically 1:1) +- [voice-note-ingest](../voice-note-ingest/SKILL.md) — for recorded one-way voice memos (different from live voice calls) + +## Contract + +This skill guarantees: + +- Routing matches the canonical triggers in the frontmatter. +- The post-call pipeline runs idempotently — second invocations update rather than duplicate. +- Output written under `meetings/` or `voice-calls/` (consistent with `_brain-filing-rules.md`). +- Conventions referenced (`quality.md`, `_brain-filing-rules.md`) are followed. +- Privacy contract preserved: no real names in any committed sample; the operator's actual call transcripts contain whatever they say, which is the operator's data and not gbrain's concern. + +## Output Format + +```markdown +--- +type: meeting +subtype: voice-call +persona: +date: YYYY-MM-DD +duration_sec: N +caller: +rating: 0-10 +audio_url: "" +--- + +# Voice call: with + +> + +## Summary + + +## Transcript + +> + +🔊 [Audio]() + +## Timeline + +- **