mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 21:19:18 +00:00
* v0.41.15.0 feat: conversation parser cathedral + progressive-batch primitive (closes #1461) Replaces PR #1461's single-format Telegram regex with a 12-pattern built-in registry covering iMessage/Slack, Telegram (×2), Discord (×2), WhatsApp (×2 locales), Signal, Matrix/Element, IRC (×2), Teams. Each pattern is hand-vetted from public format docs (signal-cli, DiscordChatExporter, Telegram Desktop, WhatsApp export docs, Element matrix-archive, irssi/weechat defaults); module-load validation runs test_positive[] + test_negative[] for every pattern at startup so a typo makes gbrain refuse to start. PR #1461 contributor's BRACKET_TIME_RX + cleanSpeaker survive verbatim as the `telegram-bracket` built-in pattern + DEFAULT_SPEAKER_CLEAN export. All 33 of their test cases pass against the new orchestrator. Three layers per page (orchestrator chooses): 1. Built-in pattern registry (zero-cost, deterministic) 2. User-declared simple_pattern via config (deferred to v0.42+) 3. Opt-IN LLM polish + fallback (privacy-first; chat content goes to Anthropic only when user explicitly enables) D18 priority scoring picks the highest-match-rate pattern across the first 10 lines (not first-wins) so overlapping formats don't silently mis-route. D5 multi_line per-pattern + D11 quick_reject prefix screen + D19 timezone_policy per-pattern complete the registry shape. Companion: src/core/progressive-batch/ primitive (rule of three satisfied across 12+ ad-hoc cost-prompt sites). Wintermute-inspired ramp shape (trial 10 → 100 → 500 → full with verification at each stage), productionized with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). D3 fail-closed budget gate: null tracker + null Policy.maxCostUsd → abort_cost_cap reason='no_budget_safety_net'. D20 discriminated Verifier union (output_count | idempotent_mutation | noop). extract-conversation-facts is the one proven consumer in v0.41.15.0; 9-site retrofit deferred to v0.41.16.0+ per TODOS.md. Codex outside-voice review absorbed 8 substantive findings: - Privacy posture (LLM polish/fallback flipped to opt-IN) - ReDoS theater (dropped arbitrary user regex; v0.42+ uses RE2) - LLM-inferred-regex persistence as silent-corruption machine - Pattern priority scoring across first 10 lines - Timezone policy on every PatternEntry - Verifier shape discriminated union - Behavior parity for sites that "jumped straight to full" - Real-corpus-redacted fixture gap (v0.42+ TODO) CI gates: - bun run check:conversation-parser (13 fixtures, --no-llm, deterministic) - bun run check:fixture-privacy (banned-token grep) Doctor surfaces 3 new checks: conversation_format_coverage, progressive_batch_audit_health, conversation_parser_probe_health. Tests: 198/198 across primitive + parser + LLM + nightly probe + eval CLI + debug CLI + doctor checks + migration v97 round-trip + E2E parser ↔ engine integration. Real bug caught + fixed during gap audit: IdempotentMutationVerifier was comparing absolute mutated-count vs per-stage expected (failed silently on stage 2+); now uses per-stage delta semantics matching OutputCountVerifier. Schema migration v97: conversation_parser_llm_cache table with (content_sha256, model_id, call_shape) composite key. NO inferred_patterns table (D17: silent-corruption machine). Plan + 23 decisions + codex outside-voice absorption at ~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md. Co-Authored-By: garrytan-agents (PR #1461) <noreply@github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(check-privacy): allowlist scripts/check-fixture-privacy.sh The new sibling privacy guard literally names the banned tokens in its BANNED_TOKENS array — same meta-exception that check-privacy.sh itself gets. Without this allowlist entry, bun run verify rejects the file post-merge because the banned name appears in the rule-definition script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: renumber v0.41.15.0 → v0.41.16.0 (queue drift) Mechanical rename across all surfaces: VERSION, package.json, CHANGELOG (header + body refs), CLAUDE.md, TODOS.md, src/core/ migrate.ts (migration v98 comment), all src/core/conversation-parser/* and src/core/progressive-batch/* file headers, all test/ headers, scripts/check-privacy.sh allowlist comment, llms-full.txt regenerated. Audit clean: VERSION + package.json + CHANGELOG header all show 0.41.16.0. verify 24/24, touched tests 179/179. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents (PR #1461) <noreply@github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
65 lines
2.0 KiB
Bash
Executable File
65 lines
2.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# v0.41.13.0 — Privacy guard for test/fixtures/conversation-formats/.
|
|
#
|
|
# Per CLAUDE.md privacy rule: "Never reference real people, companies,
|
|
# funds, or private agent names in any public-facing artifact."
|
|
# Test fixtures ship in the repo; they ARE public.
|
|
#
|
|
# This script greps for known real-name signals and fails the build if
|
|
# any leak. Add to bun run verify so the gate runs every PR.
|
|
#
|
|
# Banned tokens (case-insensitive substring match):
|
|
# - 'wintermute' / 'openclaw' (real downstream agent names)
|
|
# - 'palantir' (real company per Garry's history)
|
|
# - common real-fund names (sequoia, andreessen, founders fund, etc.)
|
|
# - 'ycombinator' / 'y combinator' (the org running gbrain)
|
|
#
|
|
# Allowed (placeholder convention):
|
|
# - alice-example / bob-example / charlie-example / diana-example
|
|
# - widget-co / acme-example
|
|
# - fund-a / fund-b / fund-c
|
|
|
|
set -euo pipefail
|
|
|
|
FIXTURE_DIR="test/fixtures/conversation-formats"
|
|
|
|
if [ ! -d "$FIXTURE_DIR" ]; then
|
|
echo "[check-fixture-privacy] $FIXTURE_DIR does not exist; nothing to check"
|
|
exit 0
|
|
fi
|
|
|
|
# Real-name signals. Add to this list when new banned tokens surface.
|
|
BANNED_TOKENS=(
|
|
"wintermute"
|
|
"openclaw"
|
|
"palantir"
|
|
"sequoia"
|
|
"andreessen"
|
|
"founders fund"
|
|
"founders\\.fund"
|
|
"ycombinator"
|
|
"y combinator"
|
|
"garry tan"
|
|
"garrytan"
|
|
)
|
|
|
|
errors=0
|
|
for token in "${BANNED_TOKENS[@]}"; do
|
|
matches=$(grep -ril "$token" "$FIXTURE_DIR" 2>/dev/null || true)
|
|
if [ -n "$matches" ]; then
|
|
echo "[check-fixture-privacy] BANNED token '$token' found in:"
|
|
echo "$matches" | sed 's/^/ - /'
|
|
errors=$((errors + 1))
|
|
fi
|
|
done
|
|
|
|
if [ "$errors" -gt 0 ]; then
|
|
echo ""
|
|
echo "[check-fixture-privacy] FAIL: $errors banned token(s) found in fixtures."
|
|
echo "[check-fixture-privacy] Fixtures must use placeholder names (alice-example, widget-co, fund-a, ...)."
|
|
echo "[check-fixture-privacy] See CLAUDE.md \"Privacy rule\" section."
|
|
exit 1
|
|
fi
|
|
|
|
echo "[check-fixture-privacy] OK: no banned tokens found in $FIXTURE_DIR"
|