mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* v0.41.15.0 feat: conversation parser cathedral + progressive-batch primitive (closes #1461) Replaces PR #1461's single-format Telegram regex with a 12-pattern built-in registry covering iMessage/Slack, Telegram (×2), Discord (×2), WhatsApp (×2 locales), Signal, Matrix/Element, IRC (×2), Teams. Each pattern is hand-vetted from public format docs (signal-cli, DiscordChatExporter, Telegram Desktop, WhatsApp export docs, Element matrix-archive, irssi/weechat defaults); module-load validation runs test_positive[] + test_negative[] for every pattern at startup so a typo makes gbrain refuse to start. PR #1461 contributor's BRACKET_TIME_RX + cleanSpeaker survive verbatim as the `telegram-bracket` built-in pattern + DEFAULT_SPEAKER_CLEAN export. All 33 of their test cases pass against the new orchestrator. Three layers per page (orchestrator chooses): 1. Built-in pattern registry (zero-cost, deterministic) 2. User-declared simple_pattern via config (deferred to v0.42+) 3. Opt-IN LLM polish + fallback (privacy-first; chat content goes to Anthropic only when user explicitly enables) D18 priority scoring picks the highest-match-rate pattern across the first 10 lines (not first-wins) so overlapping formats don't silently mis-route. D5 multi_line per-pattern + D11 quick_reject prefix screen + D19 timezone_policy per-pattern complete the registry shape. Companion: src/core/progressive-batch/ primitive (rule of three satisfied across 12+ ad-hoc cost-prompt sites). Wintermute-inspired ramp shape (trial 10 → 100 → 500 → full with verification at each stage), productionized with verifier+policy injection (callers describe HOW TO MEASURE SUCCESS, not WHEN TO WAIT FOR CTRL-C). D3 fail-closed budget gate: null tracker + null Policy.maxCostUsd → abort_cost_cap reason='no_budget_safety_net'. D20 discriminated Verifier union (output_count | idempotent_mutation | noop). extract-conversation-facts is the one proven consumer in v0.41.15.0; 9-site retrofit deferred to v0.41.16.0+ per TODOS.md. Codex outside-voice review absorbed 8 substantive findings: - Privacy posture (LLM polish/fallback flipped to opt-IN) - ReDoS theater (dropped arbitrary user regex; v0.42+ uses RE2) - LLM-inferred-regex persistence as silent-corruption machine - Pattern priority scoring across first 10 lines - Timezone policy on every PatternEntry - Verifier shape discriminated union - Behavior parity for sites that "jumped straight to full" - Real-corpus-redacted fixture gap (v0.42+ TODO) CI gates: - bun run check:conversation-parser (13 fixtures, --no-llm, deterministic) - bun run check:fixture-privacy (banned-token grep) Doctor surfaces 3 new checks: conversation_format_coverage, progressive_batch_audit_health, conversation_parser_probe_health. Tests: 198/198 across primitive + parser + LLM + nightly probe + eval CLI + debug CLI + doctor checks + migration v97 round-trip + E2E parser ↔ engine integration. Real bug caught + fixed during gap audit: IdempotentMutationVerifier was comparing absolute mutated-count vs per-stage expected (failed silently on stage 2+); now uses per-stage delta semantics matching OutputCountVerifier. Schema migration v97: conversation_parser_llm_cache table with (content_sha256, model_id, call_shape) composite key. NO inferred_patterns table (D17: silent-corruption machine). Plan + 23 decisions + codex outside-voice absorption at ~/.claude/plans/system-instruction-you-are-working-cuddly-hollerith.md. Co-Authored-By: garrytan-agents (PR #1461) <noreply@github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(check-privacy): allowlist scripts/check-fixture-privacy.sh The new sibling privacy guard literally names the banned tokens in its BANNED_TOKENS array — same meta-exception that check-privacy.sh itself gets. Without this allowlist entry, bun run verify rejects the file post-merge because the banned name appears in the rule-definition script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: renumber v0.41.15.0 → v0.41.16.0 (queue drift) Mechanical rename across all surfaces: VERSION, package.json, CHANGELOG (header + body refs), CLAUDE.md, TODOS.md, src/core/ migrate.ts (migration v98 comment), all src/core/conversation-parser/* and src/core/progressive-batch/* file headers, all test/ headers, scripts/check-privacy.sh allowlist comment, llms-full.txt regenerated. Audit clean: VERSION + package.json + CHANGELOG header all show 0.41.16.0. verify 24/24, touched tests 179/179. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents (PR #1461) <noreply@github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
218 lines
7.9 KiB
Bash
Executable File
218 lines
7.9 KiB
Bash
Executable File
#!/bin/bash
|
|
#
|
|
# check-privacy.sh — CLAUDE.md:550 enforcement.
|
|
#
|
|
# CLAUDE.md forbids the private OpenClaw fork name in public artifacts:
|
|
# CHANGELOG.md, README.md, docs/, skills/, PR titles + bodies, commit
|
|
# messages, and comments in checked-in code. This script greps for the
|
|
# banned name in either the staged index (for pre-commit hooks) or the
|
|
# working tree (for CI) and fails loudly if found.
|
|
#
|
|
# The allow-list below whitelists files where the name is legitimate
|
|
# — specifically, this script itself (where we reference the name to
|
|
# describe the rule) and upgrade guides that historically referenced
|
|
# the pre-rename fork name.
|
|
#
|
|
# Usage:
|
|
# scripts/check-privacy.sh # scan working tree
|
|
# scripts/check-privacy.sh --staged # scan git staged index
|
|
# scripts/check-privacy.sh --help
|
|
#
|
|
# Exit codes:
|
|
# 0 clean
|
|
# 1 banned name found (or --help printed)
|
|
# 2 git / grep not available
|
|
|
|
set -euo pipefail
|
|
|
|
BANNED_NAME='wintermute'
|
|
# v0.25.1 (codex T7): additional patterns from wintermute-specific filesystem
|
|
# layouts that would leak private fork context if they slipped through a port.
|
|
# `wintermute_only` already matches via the case-insensitive `wintermute` regex
|
|
# above; this list is for orthogonal patterns.
|
|
BANNED_PATHS=(
|
|
'/data/brain/'
|
|
'/data/.openclaw/'
|
|
)
|
|
|
|
usage() {
|
|
cat <<EOF
|
|
scripts/check-privacy.sh — scan for the banned OpenClaw fork name.
|
|
|
|
USAGE:
|
|
scripts/check-privacy.sh Scan all tracked files in the working tree.
|
|
scripts/check-privacy.sh --staged Scan only files staged for commit.
|
|
scripts/check-privacy.sh --help Show this message.
|
|
|
|
The script greps for '${BANNED_NAME}' (case-insensitive) in:
|
|
- CHANGELOG.md, README.md
|
|
- docs/**
|
|
- skills/**
|
|
- src/**
|
|
- test/**
|
|
- scripts/**
|
|
|
|
Allow-list (references to the name are permitted):
|
|
- scripts/check-privacy.sh itself
|
|
- docs/UPGRADING_DOWNSTREAM_AGENTS.md (historical context for pre-rename upgrades)
|
|
- .git/** (branch names, commit history — not checked in artifacts)
|
|
|
|
Exit codes: 0 clean, 1 banned name found, 2 setup error.
|
|
EOF
|
|
}
|
|
|
|
MODE=working
|
|
for arg in "$@"; do
|
|
case "$arg" in
|
|
--staged) MODE=staged ;;
|
|
--help|-h) usage; exit 1 ;;
|
|
*)
|
|
echo "Unknown argument: $arg" >&2
|
|
usage >&2
|
|
exit 2
|
|
;;
|
|
esac
|
|
done
|
|
|
|
if ! command -v git >/dev/null 2>&1; then
|
|
echo "check-privacy: git not found" >&2
|
|
exit 2
|
|
fi
|
|
|
|
# Build the file list by scanning-mode.
|
|
if [ "$MODE" = staged ]; then
|
|
FILES=$(git diff --cached --name-only --diff-filter=ACMR 2>/dev/null || true)
|
|
else
|
|
FILES=$(git ls-files 2>/dev/null || true)
|
|
fi
|
|
|
|
if [ -z "$FILES" ]; then
|
|
exit 0
|
|
fi
|
|
|
|
# Allow-list: files in which the banned name is legitimate.
|
|
# Meta-rule docs (define the rule itself), auto-generated LLM indexes,
|
|
# historical upgrade guides, and the test that enforces the rule
|
|
# against recipes/ all reference the banned name by necessity.
|
|
ALLOW_LIST=(
|
|
'scripts/check-privacy.sh'
|
|
# v0.41.16.0: sibling rule-enforcement script for test/fixtures/
|
|
# conversation-formats/. Same meta-exception as check-privacy.sh
|
|
# itself — the script's BANNED_TOKENS array literally names the
|
|
# tokens it forbids.
|
|
'scripts/check-fixture-privacy.sh'
|
|
'CLAUDE.md'
|
|
'llms-full.txt'
|
|
'docs/UPGRADING_DOWNSTREAM_AGENTS.md'
|
|
'test/integrations.test.ts'
|
|
# v0.25.1 (codex T7) BANNED_PATHS allow-list:
|
|
# Historical docs, frozen migration files, test fixtures, and env-var
|
|
# fallbacks where /data/brain/ or /data/.openclaw/ appears legitimately.
|
|
# New skills/, src/, and tests must NOT slip onto this list — extend the
|
|
# banned check above instead.
|
|
'docs/GBRAIN_RECOMMENDED_SCHEMA.md'
|
|
'docs/GBRAIN_V0.md'
|
|
'docs/guides/minions-shell-jobs.md'
|
|
'scripts/smoke-test.sh'
|
|
'skills/migrations/v0.9.0.md'
|
|
'skills/migrations/v0.14.0.md'
|
|
'test/storage-status.test.ts'
|
|
# CHANGELOG.md documents the rule (the v0.25.1 entry references the
|
|
# banned literals in describing what's banned). Same exception status
|
|
# as CLAUDE.md and this script itself: meta-documentation needs to
|
|
# name the patterns it forbids.
|
|
'CHANGELOG.md'
|
|
# skills/migrations/v0.25.1.md is the agent-readable upgrade
|
|
# walkthrough; it explains the privacy-guard extension to the
|
|
# operating agent and references the banned literals while doing so.
|
|
'skills/migrations/v0.25.1.md'
|
|
# v0.29.1: the recency-decay default-map test asserts that
|
|
# DEFAULT_RECENCY_DECAY's keys do NOT include fork-specific path
|
|
# prefixes. The test must name the banned tokens to assert their
|
|
# absence — same exception status as scripts/check-privacy.sh,
|
|
# CHANGELOG.md, and CLAUDE.md (meta-rule enforcement requires
|
|
# mentioning what the rule forbids).
|
|
'test/recency-decay.test.ts'
|
|
# v0.32.5: the sibling check-test-real-names.sh enforces the same
|
|
# privacy rule for test fixtures and lists the banned names literally
|
|
# (Wintermute, Hermes, etc) inside its BANNED_NAMES + ALLOWLIST arrays.
|
|
# Same meta-rule-enforcement exception as scripts/check-privacy.sh itself.
|
|
'scripts/check-test-real-names.sh'
|
|
# v0.34 / Lane CI: scripts/check-proposal-pii.sh and its test list the
|
|
# banned literal as part of the structural denylist they enforce against
|
|
# docs/proposals/*.md. Same meta-rule-enforcement exception as the two
|
|
# entries above — describing what the rule forbids requires naming it.
|
|
'scripts/check-proposal-pii.sh'
|
|
'test/scripts/check-proposal-pii.test.ts'
|
|
# v0.32.3.0: the functional-area-resolver skill's behavior-contract
|
|
# section describes the privacy guarantees the skill preserves and
|
|
# references the banned literals while doing so (line 306). Same
|
|
# meta-rule-enforcement exception as scripts/check-privacy.sh and
|
|
# CHANGELOG.md — describing what the rule forbids requires naming it.
|
|
'skills/functional-area-resolver/SKILL.md'
|
|
# v0.36.0.0: the gbrain skillpack harvest privacy linter's whole job
|
|
# is to catch the banned literal leaking into gbrain. The regex
|
|
# pattern in harvest-lint.ts is `\bWintermute\b` by necessity; the
|
|
# tests verify that pattern fires by feeding it the banned string;
|
|
# the harvest skill markdown describes the substitution policy
|
|
# ("Wintermute → your OpenClaw") as part of the genericization
|
|
# checklist. Same meta-rule-enforcement exception as the privacy
|
|
# checks themselves.
|
|
'src/core/skillpack/harvest-lint.ts'
|
|
'test/skillpack-harvest-lint.test.ts'
|
|
'test/skillpack-harvest.test.ts'
|
|
'test/e2e/skillpack-flow.test.ts'
|
|
'skills/skillpack-harvest/SKILL.md'
|
|
# v0.40.1.0 Track D / T5: the qrels gate test contains a privacy-grep
|
|
# regression guard whose block list names the banned literal to assert
|
|
# it's NOT in the qrels fixture. Same meta-rule-enforcement exception
|
|
# as the other test entries above.
|
|
'test/eval-replay-gate.test.ts'
|
|
)
|
|
|
|
is_allowed() {
|
|
local f="$1"
|
|
for a in "${ALLOW_LIST[@]}"; do
|
|
if [ "$f" = "$a" ]; then
|
|
return 0
|
|
fi
|
|
done
|
|
return 1
|
|
}
|
|
|
|
FOUND=0
|
|
while IFS= read -r file; do
|
|
[ -z "$file" ] && continue
|
|
[ ! -f "$file" ] && continue
|
|
if is_allowed "$file"; then
|
|
continue
|
|
fi
|
|
# Case-insensitive grep; only specific extensions + known docs.
|
|
case "$file" in
|
|
*.md|*.ts|*.mjs|*.js|*.py|*.sh|*.json|*.yaml|*.yml|*.txt|README*|CHANGELOG*|CLAUDE*|AGENTS*)
|
|
if grep -in "$BANNED_NAME" "$file" >/dev/null 2>&1; then
|
|
echo "[check-privacy] BANNED NAME in $file:" >&2
|
|
grep -in "$BANNED_NAME" "$file" | sed 's|^| |' >&2
|
|
FOUND=1
|
|
fi
|
|
# Banned wintermute-specific filesystem paths (codex T7).
|
|
for path in "${BANNED_PATHS[@]}"; do
|
|
if grep -nF "$path" "$file" >/dev/null 2>&1; then
|
|
echo "[check-privacy] BANNED PATH '$path' in $file:" >&2
|
|
grep -nF "$path" "$file" | sed 's|^| |' >&2
|
|
FOUND=1
|
|
fi
|
|
done
|
|
;;
|
|
esac
|
|
done <<< "$FILES"
|
|
|
|
if [ "$FOUND" -eq 1 ]; then
|
|
echo "" >&2
|
|
echo "The private OpenClaw fork name is banned in public artifacts." >&2
|
|
echo "CLAUDE.md:550. Replace with 'your OpenClaw', 'OpenClaw reference deployment', or 'openclaw-reference'." >&2
|
|
exit 1
|
|
fi
|
|
|
|
exit 0
|