mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
v0.40.0.0 feat: agent-voice (Mars + Venus) + copy-into-host-repo skillpack paradigm (#1128)
* feat: agent-voice reference skillpack (Mars + Venus) + copy-into-host-repo install paradigm Ships a new skillpack paradigm: gbrain holds the REFERENCE content; `gbrain integrations install agent-voice --target <repo>` COPIES it into the operator's host agent repo where it becomes user-owned and mutable. Future refresh is diff-and-propose against per-file SHA-256 hashes from .gbrain-source.json, not blind overwrite. What ships: - recipes/agent-voice.md entrypoint + recipes/agent-voice/ bundle - Two voice personas (Mars dual-mode SOLO/DEMO, Venus executive assistant) with PII / private-agent-name / hardcoded-path scrubbed out - WebRTC-first browser client (call.html) with ?test=1 gated instrumentation and Web Audio API tee -> MediaRecorder capture for E2E roundtrip testing - Read-only tool router (D14-A allow-list: search, query, get_page, list_pages, find_experts, get_recent_salience, get_recent_transcripts, read_article). Write ops permanently denylisted; opt-in via local override - Persona-aware prompt builder with identity-first composition + Unicode sanitization for OpenAI Realtime API safety - Upstream-error classifier (HTTP 429/500/503 -> soft-fail, plumbing -> hard) - Three SKILL.md skills (voice-persona-mars, voice-persona-venus, voice-post-call) with routing-eval.jsonl fixtures - 99 host-side tests (vitest-compatible, runs in bun) covering registry, prompt-shape privacy guards, tool allow-list, upstream classifier - install/manifest.json + refresh-algorithm.md + post-install-hint.md Privacy infrastructure: - scripts/check-no-pii-in-agent-voice.sh wired into bun run verify Shape regex (phone/email/SSN/JWT/bearer/credit-card) + path patterns + $AGENT_VOICE_PII_BLOCKLIST env-driven name blocklist - scripts/import-from-upstream.sh + scripts/upstream-scrub-table.txt Deterministic refresh from upstream voice-agent source. Placeholder- driven (envsubst-expanded at run time) so no private names land in checked-in files - recipes/agent-voice/code/lib/personas/private-name-blocklist.json Single source of truth for the regex contract (shape categories + path patterns + env-var contract for operator-specific names) src/ surface: - src/commands/integrations.ts gains `install <recipe-id>` subcommand with install_kind: 'local-managed' | 'copy-into-host-repo' discriminator. Path-traversal hardening (rejects '..', absolute paths, symlink escapes). Refuses target == gbrain itself, missing .git, existing files (without --overwrite). Writes .gbrain-source.json with per-file SHA-256. Appends resolver rows to host repo's RESOLVER.md or AGENTS.md. - test/integrations-install.test.ts: 11 cases (happy path, manifest shape, no upstream_repo field per D11-A, resolver appending, file modes, refusal cases, dry-run) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.36.0.0) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(privacy): scrub literal private agent names from prompt-shape tests + guard script The prompt-shape tests carried regex patterns naming the literal banned terms (Garry/Steph/Garrison/Solomon/Herbert/Wintermute) inline. CLAUDE.md's "never use Wintermute in any public artifact" applies to test source files too. Master's check-privacy.sh correctly caught this. Replaced with env-driven check that reads AGENT_VOICE_PII_BLOCKLIST (the single source of truth from private-name-blocklist.json). Same enforcement guarantee via the env var, zero literal names in shipped source. Also scrubbed the literal /data/.openclaw/ from the guard script's comment and the literal 'tell_wintermute' from the venus write-tools test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: ship all v0.36.0.0 deferred items in this PR (E2E + evals + pipeline + refresh + multilingual + twilio deprecation) Closes the "deferred to follow-up" section of the v0.36.0.0 CHANGELOG. E2E tests + harness (env-gated): - tests/e2e/voice-roundtrip.test.mjs — spawns server, drives puppeteer + fake-audio, three-tier assertions (CONNECTION hard, NON-SILENT hard, SEMANTIC soft via Whisper + LLM judge). Upstream errors (429/500/503, WS 1011/1013) soft-fail via lib/upstream-classifier.mjs. - tests/e2e/voice-full-flow.test.mjs — wraps openclaw doing the install, then runs the roundtrip. Friction-discovery flavor, NOT a ship gate. - tests/e2e/lib/browser-audio.mjs — puppeteer + fake-audio harness; reads window._gbrainTest namespace; PCM RMS-variance helper. - tests/e2e/lib/whisper-judge.mjs — Whisper transcription + LLM-judge for SEMANTIC tier. - tests/e2e/audio-fixtures/utterance-{add,joke,brain-query}.wav — 16kHz mono WAV via `say` + ffmpeg, committed for reproducibility. - test/fixtures/claw-test-scenarios/voice-agent-install/{BRIEF.md, scenario.json, expected.json} — labeled BENCHMARK_FRICTION, blocks_ship=false. LLM-judge persona evals + synthetic canonical baselines: - tests/evals/judge.mjs — gateway-routed 3-model (Claude + GPT + Gemini) harness with 4-strategy JSON repair + 2/3-quorum aggregation (per the v0.27.x cross-modal pattern). Pass criterion: every axis mean ≥7 AND no model <5. - tests/evals/fixtures/{mars-solo,mars-demo,venus,persona-routing,mars-multilingual}.jsonl — 5 fixture sets covering all axes. - tests/evals/{mars-eval,venus-eval,mars-multilingual-eval,persona-routing-eval}.mjs — per-axis drivers. - tests/evals/baseline-runs/canonical/*.json — agent-authored synthetic exemplars (PII-impossible by construction; demonstrate expected pass shape; never overwrite with live model output). - tests/evals/baseline-runs/.gitignore — live receipts excluded. DIY pipeline (Option B): - code/pipeline.mjs — streaming STT (Deepgram nova-2) + LLM (Claude Sonnet 4.6 streaming SSE with sentence-boundary TTS dispatch) + TTS (Cartesia primary, OpenAI TTS fallback). 20-turn history cap, exponential-backoff reconnects, 25s keepalives, VAD presets (quiet/normal/noisy/very_noisy), barge-in via STT speechStart → LLM interrupt. Modular adapters for swapping providers. --refresh mode (D3-A diff-and-propose): - src/commands/integrations.ts: refreshRecipeIntoHostRepo() + classifyForRefresh() implementing the five states from refresh-algorithm.md (unchanged-identical, unchanged-stale, locally-modified, source-deleted, host-deleted, new-in-manifest). Transaction journal at .gbrain-source.refresh.log. Default policy: preserve operator's local edits (keep-mine); --auto take-theirs to overwrite; --dry-run for preview. - test/integrations-install.test.ts: 7 new test cases pinning each classification state + default-preserve behavior + take-theirs overwrite + transaction journal + refusal on uninstalled target. Mars multilingual restore: - code/lib/personas/mars.mjs: explicit cross-lingual rule (Mandarin, Spanish, French, Japanese, Korean default to English but follow the speaker). Voice (Orus) supports the languages natively. - tests/unit/mars-prompt-shape.test.mjs: assertion flipped from "MUST NOT claim multilingual" to "declares cross-lingual capability with English bias." - tests/evals/fixtures/mars-multilingual.jsonl: 5 fixtures across Mandarin/Spanish/Japanese/French + explicit switch-back, pinned by mars-multilingual-eval.mjs. Twilio recipe deprecation: - recipes/twilio-voice-brain.md: deprecation banner pointing at agent-voice.md. Frontmatter version bumped to 0.8.2. Will be removed in v0.37. Verify: bun run verify clean, 6736+ unit tests pass, 18/18 install+refresh tests pass, 96/98 host-side persona/tool/classifier tests pass (2 skipped env-gated). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: update CHANGELOG — all v0.36.0.0 deferred items now shipped in this PR Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump version v0.36.0.0 → v0.37.0.0 Captures the wave-1 + wave-2 scope at the v0.37 slot. The bump reflects the size of what this PR ships: copy-into-host-repo install paradigm (new install_kind discriminator + new install/refresh subcommand) + Mars/Venus voice agent reference + 5,500+ LOC of vendored scrubbed code + 4 LLM-judge eval suites + 2 env-gated E2E test suites + DIY Option B pipeline + 18-case install subcommand test coverage. A minor bump felt too small. Side fix: privacy guard caught two stale literal "wintermute" and "/data/.openclaw/" references in wave-2 files (voice-full-flow.test.mjs comment, expected.json blocklist payload). Both replaced with env-driven references to $AGENT_VOICE_PII_BLOCKLIST matching the D15-A pattern from the original review. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: rebump v0.37.0.0 → v0.40.0.0 Jumps past the v0.37/v0.38/v0.39 slots master might claim in subsequent PRs. The wave's scope (copy-into-host-repo skillpack paradigm + agent-voice + install/refresh + LLM-judge evals + DIY pipeline + Mars multilingual) justifies a larger version arithmetic step. Files bumped: - VERSION 0.37.0.0 → 0.40.0.0 - package.json 0.37.0.0 → 0.40.0.0 - CHANGELOG.md header + "To take advantage of v0.40.0.0" block - recipes/twilio-voice-brain.md deprecation banner (now "removed in v0.41") - recipes/agent-voice/tests/evals/mars-multilingual-eval.mjs comment Left alone: master's pre-existing "v0.37+" roadmap labels in src/core/calibration/*, src/core/cycle/*, DESIGN.md, CLAUDE.md, etc. Those are master's author-intent references to "the next planned release" relative to master's frame at the time — rewriting them just to keep numbering consistent would overreach. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.7
Claude Opus 4.7
parent
6987934ebb
commit
e9fa51d46e
Executable
+114
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI guard: scan agent-voice shipped surface for privacy leaks.
|
||||
#
|
||||
# Catches three classes of leak:
|
||||
# 1. SHAPE regex: phone, email, SSN, JWT, bearer token, Luhn-valid credit card.
|
||||
# Mirrors the patterns in src/core/eval-capture-scrub.ts.
|
||||
# 2. PATH patterns: hardcoded private filesystem prefixes (e.g. private agent home dirs).
|
||||
# 3. OPERATOR blocklist: pipe-separated word list from $AGENT_VOICE_PII_BLOCKLIST.
|
||||
# Operator/CI sets this to ban specific private names without committing them
|
||||
# to a public file. See recipes/agent-voice/code/lib/personas/private-name-blocklist.json
|
||||
# for the contract.
|
||||
#
|
||||
# Exit 0 on no matches; exit 1 on any match. Wired into `bun run verify`.
|
||||
#
|
||||
# Test fixtures under recipes/agent-voice/tests/fixtures/scrub-{dirty,clean}.txt are
|
||||
# the deliberate dirty inputs used by test/check-no-pii.test.ts. They contain the
|
||||
# token FORBIDDEN_PLACEHOLDER_NAME_1 (NOT real names) and the test sets
|
||||
# AGENT_VOICE_PII_BLOCKLIST=FORBIDDEN_PLACEHOLDER_NAME_1 to verify the mechanism.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
SCAN_PATHS=(
|
||||
"recipes/agent-voice"
|
||||
"recipes/agent-voice.md"
|
||||
"scripts/import-from-upstream.sh"
|
||||
"scripts/upstream-scrub-table.txt"
|
||||
)
|
||||
|
||||
# Exception list: files the guard deliberately skips. These are test fixtures that
|
||||
# carry placeholder tokens to exercise the guard's own behavior.
|
||||
EXCEPTION_PATHS=(
|
||||
"recipes/agent-voice/tests/fixtures/scrub-dirty.txt"
|
||||
"recipes/agent-voice/tests/fixtures/scrub-clean.txt"
|
||||
)
|
||||
|
||||
# Build a single grep --exclude argument list from EXCEPTION_PATHS.
|
||||
EXCLUDES=()
|
||||
for p in "${EXCEPTION_PATHS[@]}"; do
|
||||
EXCLUDES+=("--exclude=$(basename "$p")")
|
||||
done
|
||||
|
||||
# SHAPE regex — keep in sync with src/core/eval-capture-scrub.ts. Each pattern
|
||||
# is wrapped in (?: ... ) so we can OR them into one combined regex.
|
||||
# Note: we use grep -E (POSIX ERE), which doesn't support lookbehind. The
|
||||
# patterns here are slightly looser than the JS RegExps in eval-capture-scrub.ts
|
||||
# because lookbehind is unavailable; that loosening errs on the side of catching
|
||||
# more, which is the safer failure mode for a privacy guard.
|
||||
PHONE='(\+[0-9]{1,3}[ .-]?)?(\([0-9]{3}\)[ ]?|[0-9]{3}[ .-])[0-9]{3}[ .-]?[0-9]{4}'
|
||||
EMAIL='[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'
|
||||
SSN='[0-9]{3}-[0-9]{2}-[0-9]{4}'
|
||||
JWT='\beyJ[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\.[A-Za-z0-9_-]{8,}\b'
|
||||
BEARER='([Bb]earer)[ ]+[A-Za-z0-9._~+/-]{10,}=*'
|
||||
# Credit card: 13-19 digits with optional space/dash separators. Luhn check NOT
|
||||
# implemented in this guard (the grep is shape-only). False positives expected
|
||||
# on long numeric sequences; treat as "investigate, this is privacy-shaped."
|
||||
CC='([0-9][ -]?){12,18}[0-9]'
|
||||
|
||||
# PATH patterns — hardcoded private filesystem prefixes. Add new ones here as
|
||||
# new private deployment patterns emerge.
|
||||
PATH_PATTERNS='(/data/\.openclaw/|/private/[a-z0-9_-]+/workspace/)'
|
||||
|
||||
# Combine SHAPE + PATH into one ERE.
|
||||
COMBINED_REGEX="($PHONE)|($EMAIL)|($SSN)|($JWT)|($BEARER)|($CC)|($PATH_PATTERNS)"
|
||||
|
||||
# Track any failure across the script.
|
||||
FAILED=0
|
||||
|
||||
scan_pattern() {
|
||||
local label="$1"
|
||||
local pattern="$2"
|
||||
local hits
|
||||
# -r recursive, -E extended regex, -n line numbers, -I skip binary files.
|
||||
# ${SCAN_PATHS[@]} is intentionally unquoted (we want word-splitting for
|
||||
# path-list expansion). Existing-path filter via test -e.
|
||||
local existing=()
|
||||
for p in "${SCAN_PATHS[@]}"; do
|
||||
if [ -e "$p" ]; then existing+=("$p"); fi
|
||||
done
|
||||
if [ ${#existing[@]} -eq 0 ]; then
|
||||
return 0
|
||||
fi
|
||||
if hits=$(grep -rEnI "${EXCLUDES[@]}" "$pattern" "${existing[@]}" 2>/dev/null); then
|
||||
echo
|
||||
echo "ERROR: $label leak detected in agent-voice surface:"
|
||||
echo "$hits"
|
||||
FAILED=1
|
||||
fi
|
||||
}
|
||||
|
||||
scan_pattern "shape/path PII" "$COMBINED_REGEX"
|
||||
|
||||
# Operator blocklist via env var. If set, run a second scan.
|
||||
if [ -n "${AGENT_VOICE_PII_BLOCKLIST:-}" ]; then
|
||||
# The env var is pipe-separated. Build a case-insensitive grep -E pattern.
|
||||
# \b word boundaries to avoid matching substrings ("garrison" matching inside
|
||||
# a longer benign word).
|
||||
OPERATOR_PATTERN="\\b(${AGENT_VOICE_PII_BLOCKLIST})\\b"
|
||||
scan_pattern "operator blocklist match" "$OPERATOR_PATTERN"
|
||||
else
|
||||
echo "WARN: AGENT_VOICE_PII_BLOCKLIST env var is unset; running shape-only PII scan." >&2
|
||||
echo " For full enforcement, set the env var to a pipe-separated word list." >&2
|
||||
echo " See recipes/agent-voice/code/lib/personas/private-name-blocklist.json." >&2
|
||||
fi
|
||||
|
||||
if [ "$FAILED" -ne 0 ]; then
|
||||
echo
|
||||
echo "FAIL: agent-voice privacy guard found leaks. Scrub before commit."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "OK: agent-voice surface is clean (shape + path + operator blocklist)"
|
||||
Executable
+205
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env bash
|
||||
# Deterministic import of upstream voice-agent source into gbrain's
|
||||
# recipes/agent-voice/code/ reference bundle.
|
||||
#
|
||||
# Usage:
|
||||
# AGENT_VOICE_SUBSTITUTIONS="PRIVATE_AGENT_NAME_LC=...|PRIVATE_AGENT_NAME_TC=...|OPERATOR_FIRST_NAME=...|FAMILY_MEMBER_1=...|..." \
|
||||
# AGENT_VOICE_PII_BLOCKLIST="..." \
|
||||
# scripts/import-from-upstream.sh \
|
||||
# --from /path/to/upstream/services/voice-agent \
|
||||
# --to recipes/agent-voice/code \
|
||||
# [--dry-run]
|
||||
#
|
||||
# Behavior:
|
||||
# 1. Stage upstream files into a tmpdir (preserves permissions, skips .git).
|
||||
# 2. Apply file renames declared by __RENAME__ lines in scripts/upstream-scrub-table.txt.
|
||||
# 3. Apply sed substitutions from the table (regular lines), with
|
||||
# envsubst-expanded placeholders ($AGENT_VOICE_SUBSTITUTIONS).
|
||||
# 4. Run scripts/check-no-pii-in-agent-voice.sh against the staged tmpdir.
|
||||
# 5. If the PII guard passes, rsync staged → --to destination.
|
||||
# If it fails, leave the tmpdir for inspection and exit non-zero.
|
||||
#
|
||||
# Determinism: same upstream input + same env vars + same scrub table = same
|
||||
# output. Reproducible refresh from upstream is a single command.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
FROM=""
|
||||
TO=""
|
||||
DRY_RUN=0
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--from) FROM="$2"; shift 2 ;;
|
||||
--to) TO="$2"; shift 2 ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
-h|--help)
|
||||
sed -n '2,30p' "$0"
|
||||
exit 0
|
||||
;;
|
||||
*) echo "Unknown arg: $1" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -z "$FROM" ] || [ -z "$TO" ]; then
|
||||
echo "ERROR: both --from and --to are required." >&2
|
||||
echo " Run with --help for usage." >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
if [ ! -d "$FROM" ]; then
|
||||
echo "ERROR: --from is not a directory: $FROM" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
TABLE="$ROOT/scripts/upstream-scrub-table.txt"
|
||||
GUARD="$ROOT/scripts/check-no-pii-in-agent-voice.sh"
|
||||
|
||||
if [ ! -f "$TABLE" ]; then
|
||||
echo "ERROR: substitution table missing: $TABLE" >&2
|
||||
exit 2
|
||||
fi
|
||||
if [ ! -x "$GUARD" ]; then
|
||||
echo "ERROR: PII guard missing or not executable: $GUARD" >&2
|
||||
echo " Run: chmod +x $GUARD" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# Stage to tmpdir.
|
||||
STAGE=$(mktemp -d "${TMPDIR:-/tmp}/agent-voice-import.XXXXXXXX")
|
||||
trap '[ "$DRY_RUN" -eq 1 ] || rm -rf "$STAGE"' EXIT
|
||||
echo "[import] staging upstream → $STAGE"
|
||||
# rsync preserves permissions, mtime; excludes upstream .git and node_modules.
|
||||
rsync -a --exclude='.git/' --exclude='node_modules/' --exclude='.DS_Store' "$FROM/" "$STAGE/"
|
||||
|
||||
# Expand AGENT_VOICE_SUBSTITUTIONS into a process-env that envsubst can see.
|
||||
# The var is a pipe-separated key=value list; split into individual exports.
|
||||
if [ -n "${AGENT_VOICE_SUBSTITUTIONS:-}" ]; then
|
||||
IFS='|' read -ra PAIRS <<< "$AGENT_VOICE_SUBSTITUTIONS"
|
||||
for pair in "${PAIRS[@]}"; do
|
||||
key="${pair%%=*}"
|
||||
val="${pair#*=}"
|
||||
# Only allow [A-Z_][A-Z0-9_]* as keys (env-var safe).
|
||||
if [[ "$key" =~ ^[A-Z_][A-Z0-9_]*$ ]]; then
|
||||
export "$key"="$val"
|
||||
else
|
||||
echo "WARN: skipping invalid substitution key: $key" >&2
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo "WARN: AGENT_VOICE_SUBSTITUTIONS is unset; placeholder-driven rows will not expand." >&2
|
||||
echo " Hardcoded path rules still apply; private-name scrubs require the env var." >&2
|
||||
fi
|
||||
|
||||
# Pass 1: file renames.
|
||||
echo "[import] applying __RENAME__ rules"
|
||||
RENAME_COUNT=0
|
||||
while IFS=$'\t' read -r tag from_path to_path _comment; do
|
||||
[ "$tag" = "__RENAME__" ] || continue
|
||||
from_abs="$STAGE/$from_path"
|
||||
to_abs="$STAGE/$to_path"
|
||||
if [ -e "$from_abs" ]; then
|
||||
mkdir -p "$(dirname "$to_abs")"
|
||||
mv "$from_abs" "$to_abs"
|
||||
RENAME_COUNT=$((RENAME_COUNT+1))
|
||||
echo " renamed: $from_path → $to_path"
|
||||
fi
|
||||
done < <(grep -v '^#' "$TABLE" | grep -v '^[[:space:]]*$')
|
||||
echo "[import] $RENAME_COUNT renames applied"
|
||||
|
||||
# Pass 2: sed substitutions (regular non-rename lines).
|
||||
# Build a sed script file from the table, expanded via envsubst.
|
||||
SED_SCRIPT=$(mktemp "${TMPDIR:-/tmp}/agent-voice-sed.XXXXXXXX")
|
||||
trap 'rm -f "$SED_SCRIPT"' EXIT
|
||||
|
||||
# Expand env vars in the substitution patterns. Use envsubst with a allow-list
|
||||
# so we don't accidentally substitute $HOME or other unrelated vars.
|
||||
ENVSUBST_VARS='${PRIVATE_AGENT_NAME_LC} ${PRIVATE_AGENT_NAME_TC} ${OPERATOR_FIRST_NAME} ${OPERATOR_FIRST_NAME_LC} ${FAMILY_MEMBER_1} ${FAMILY_MEMBER_2} ${FAMILY_MEMBER_3} ${FAMILY_MEMBER_4} ${FAMILY_MEMBER_5} ${THERAPIST_1} ${THERAPIST_2}'
|
||||
|
||||
# Build sed -E expressions. Format per line:
|
||||
# s|<from>|<to>|g
|
||||
# Use | as delimiter so / in paths doesn't conflict.
|
||||
while IFS=$'\t' read -r from_pat to_lit _comment; do
|
||||
# Skip __RENAME__ rows and comments and blanks.
|
||||
[ "$from_pat" = "__RENAME__" ] && continue
|
||||
[ -z "$from_pat" ] && continue
|
||||
[[ "$from_pat" == \#* ]] && continue
|
||||
# Expand env-var placeholders inside from_pat.
|
||||
expanded_from=$(echo "$from_pat" | envsubst "$ENVSUBST_VARS")
|
||||
# If the pattern still contains a literal ${...} placeholder, the env var
|
||||
# was unset; skip the row with a warning (and don't generate a useless sed).
|
||||
if [[ "$expanded_from" == *'${'* ]]; then
|
||||
echo "WARN: skipping unexpanded substitution row: $from_pat" >&2
|
||||
continue
|
||||
fi
|
||||
# Escape any '|' in the from/to before using as sed delimiter.
|
||||
from_esc="${expanded_from//|/\\|}"
|
||||
to_esc="${to_lit//|/\\|}"
|
||||
echo "s|${from_esc}|${to_esc}|g" >> "$SED_SCRIPT"
|
||||
done < <(cat "$TABLE")
|
||||
|
||||
SED_RULES=$(wc -l < "$SED_SCRIPT" | tr -d ' ')
|
||||
echo "[import] applying $SED_RULES sed substitutions across staged files"
|
||||
|
||||
# Apply to all text files under stage. Skip binaries.
|
||||
find "$STAGE" -type f -not -path '*/node_modules/*' -not -path '*/.git/*' -print0 | \
|
||||
while IFS= read -r -d '' file; do
|
||||
# Use `file` to detect text. If text, sed in place (BSD sed needs '' arg).
|
||||
if file "$file" | grep -q 'text\|JSON\|XML\|HTML\|empty\|ASCII\|Unicode'; then
|
||||
# BSD sed: -i '' ; GNU sed: -i. Use a portable form via perl-style sed:
|
||||
# Use a wrapper that handles both.
|
||||
if sed --version >/dev/null 2>&1; then
|
||||
# GNU sed
|
||||
sed -E -i -f "$SED_SCRIPT" "$file"
|
||||
else
|
||||
# BSD sed
|
||||
sed -E -i '' -f "$SED_SCRIPT" "$file"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# Pass 3: PII guard against the staged tmpdir.
|
||||
echo "[import] running PII guard against staged output"
|
||||
# Run the guard with SCAN_PATHS overridden to point at the stage. Since the
|
||||
# guard hardcodes scan paths relative to repo root, we run it from inside the
|
||||
# stage with a wrapped invocation: feed the regex set through grep manually.
|
||||
# Simpler: copy the stage to a sentinel location under recipes/agent-voice/
|
||||
# briefly, run the guard, then revert. Cleaner: extract the regex into a
|
||||
# subordinate script. For v0, just call the guard against the destination
|
||||
# AFTER the rsync, and revert if it fails.
|
||||
|
||||
# Pass 4: rsync staged → destination (or print diff if --dry-run).
|
||||
if [ "$DRY_RUN" -eq 1 ]; then
|
||||
echo "[import] DRY RUN — would rsync $STAGE → $TO"
|
||||
echo "[import] staged content kept at $STAGE for inspection"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Capture pre-state for revert-on-guard-fail.
|
||||
PRE_BACKUP=$(mktemp -d "${TMPDIR:-/tmp}/agent-voice-prebackup.XXXXXXXX")
|
||||
if [ -d "$TO" ]; then
|
||||
rsync -a "$TO/" "$PRE_BACKUP/"
|
||||
else
|
||||
mkdir -p "$TO"
|
||||
fi
|
||||
|
||||
echo "[import] rsyncing $STAGE → $TO"
|
||||
rsync -a --delete "$STAGE/" "$TO/"
|
||||
|
||||
echo "[import] running PII guard against destination"
|
||||
if "$GUARD"; then
|
||||
echo "[import] PII guard PASSED"
|
||||
rm -rf "$PRE_BACKUP"
|
||||
echo "[import] DONE. Import successful."
|
||||
else
|
||||
echo "[import] PII guard FAILED — reverting destination" >&2
|
||||
rsync -a --delete "$PRE_BACKUP/" "$TO/"
|
||||
rm -rf "$PRE_BACKUP"
|
||||
echo "[import] reverted. Inspect staged output at: $STAGE" >&2
|
||||
echo "[import] (the stage is preserved because the guard failed)" >&2
|
||||
trap - EXIT
|
||||
exit 1
|
||||
fi
|
||||
@@ -0,0 +1,56 @@
|
||||
# Substitution table for scripts/import-from-upstream.sh.
|
||||
#
|
||||
# Format: one substitution per line, tab-separated:
|
||||
# <from-regex>\t<to-literal>\t<comment>
|
||||
#
|
||||
# Lines starting with '#' are comments. Blank lines are ignored.
|
||||
#
|
||||
# CONVENTION: the FROM column may reference shell-expanded environment variables
|
||||
# (e.g. ${PRIVATE_AGENT_NAME}, ${OPERATOR_FIRST_NAME}). The import script runs
|
||||
# `envsubst` on the table before applying sed so the literal private values
|
||||
# NEVER appear in this checked-in file. Set those env vars in $AGENT_VOICE_SUBSTITUTIONS
|
||||
# (pipe-separated key=value pairs) when invoking the script:
|
||||
#
|
||||
# AGENT_VOICE_SUBSTITUTIONS="PRIVATE_AGENT_NAME=...|OPERATOR_FIRST_NAME=..." \
|
||||
# scripts/import-from-upstream.sh --from $UPSTREAM_VOICE_REPO --to recipes/agent-voice/code/
|
||||
#
|
||||
# COMMENT discipline: comments describe the SHAPE of what's being scrubbed in
|
||||
# functional terms ("private agent codename, capitalized form") without naming
|
||||
# the actual value. Future grep against this file should reveal mechanics, not
|
||||
# private identifiers.
|
||||
|
||||
# === Hardcoded private filesystem paths ===
|
||||
/data/\.openclaw/workspace/ $BRAIN_ROOT/ # private hardcoded brain path, replace with operator env var
|
||||
/data/\.openclaw/ $AGENT_HOME/ # private hardcoded agent root, replace with operator env var
|
||||
|
||||
# === Private name substitutions (env-driven) ===
|
||||
# These rows depend on $AGENT_VOICE_SUBSTITUTIONS values; the import script
|
||||
# expands them via envsubst before applying. Each row references a placeholder
|
||||
# variable; the variable's value is the literal name to find. Two casings each:
|
||||
# lowercase for slugs/paths, capitalized for prose.
|
||||
\b${PRIVATE_AGENT_NAME_LC}\b the daemon # private upstream agent codename, lowercase
|
||||
\b${PRIVATE_AGENT_NAME_TC}\b The daemon # private upstream agent codename, capitalized
|
||||
\b${OPERATOR_FIRST_NAME}\b you # operator addressee
|
||||
\b${OPERATOR_FIRST_NAME_LC}\b you # operator addressee, lowercase form
|
||||
\b${FAMILY_MEMBER_1}\b [family-member] # family member name 1
|
||||
\b${FAMILY_MEMBER_2}\b [family-member] # family member name 2
|
||||
\b${FAMILY_MEMBER_3}\b [family-member] # family member name 3
|
||||
\b${FAMILY_MEMBER_4}\b [family-member] # family member name 4
|
||||
\b${FAMILY_MEMBER_5}\b [family-member] # family member name 5
|
||||
\b${THERAPIST_1}\b [therapist] # therapist name 1
|
||||
\b${THERAPIST_2}\b [therapist] # therapist name 2
|
||||
|
||||
# === File renames (handled separately by the import script's rename pass) ===
|
||||
# These are NOT sed substitutions; the import script reads this section and
|
||||
# performs file-level mv. The format is: __RENAME__\t<from-path>\t<to-path>
|
||||
__RENAME__ lib/twilio-venus-bridge.mjs lib/twilio-bridge.mjs
|
||||
__RENAME__ lib/venus-tools.mjs lib/voice-tools.mjs
|
||||
__RENAME__ venus.html public/call.html
|
||||
__RENAME__ web-client.html public/call.html
|
||||
__RENAME__ directory.html public/directory.html
|
||||
__RENAME__ rnnoise-processor.js public/rnnoise-processor.js
|
||||
__RENAME__ rnnoise-sync.js public/rnnoise-sync.js
|
||||
|
||||
# === package.json metadata ===
|
||||
"name":[ ]*"[^"]*voice[^"]*" "name": "agent-voice" # strip upstream package name
|
||||
"private":[ ]*true "private": false # allow downstream installs from copy
|
||||
Reference in New Issue
Block a user