Files
gbrain/scripts/check-privacy.sh
T
dd1cc121d8 v0.35.3.1 feat(eval): temporal-aware contradiction probe + verdict enum (#1052)
* rfc: temporal axis for contradiction probe

Field report on residual HIGH findings from gbrain eval suspected-contradictions
and proposal for a 4-phase fix (Phase 1 = judge prompt + verdict enum is the
recommended starting point).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): pass effective_date to judge prompt; bump PROMPT_VERSION

Lane A1 of the temporal-contradiction-probe wave. Threads page-level
effective_date through the search projection into the contradiction judge so
the LLM can reason about supersession instead of treating every dated pair as
a contradiction.

Changes:
- SearchResult interface adds optional effective_date + effective_date_source
  fields; rowToSearchResult populates them from the row data with date-only
  YYYY-MM-DD normalization (handles both postgres.js Date and PGLite string).
- 8 SELECT projection sites (3 in postgres-engine, 5 in pglite-engine) now
  carry p.effective_date + p.effective_date_source through their inner CTEs
  and outer SELECTs so search results expose the field on both engines.
- PairMember (eval-contradictions/types.ts) gets the two fields as required
  (string | null) so the type forces every constructor to think about temporal
  anchoring. Runner's searchResultToMember + takeToMember handle the
  normalization; takes inherit the chunk's page-level date.
- buildJudgePrompt emits `Statement A (from: YYYY-MM-DD)` when effective_date
  is non-null, else `(date unknown)`. Prompt instructions explain the tag so
  the model knows what to do with it.
- PROMPT_VERSION bumps '1' → '2'. Cache-key tuple shape unchanged; old rows
  miss naturally on first run against the new prompt.

Test fixtures in 5 files updated to include the new required fields. All 205
eval-contradictions unit tests + 101 search-related tests pass. Typecheck
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): replace contradicts:boolean with verdict:enum (6 members)

Lane A2 of the temporal-contradiction-probe wave. Expands the judge's
classification vocabulary from a binary contradicts:bool to a six-member
verdict enum so the probe can distinguish "this changed" from "this is wrong".

Verdict taxonomy:
  no_contradiction       — drop from findings
  contradiction          — genuine conflict at same point in time
  temporal_supersession  — newer claim updates/replaces older; not an error
  temporal_regression    — metric/status went backwards over time (signal)
  temporal_evolution     — legitimate change, neither supersession nor regression
  negation_artifact      — judge misread an explicit negation

Changes:
- types.ts: Verdict union (6 members); Severity gains 'info'; ResolutionKind
  extended with temporal_supersede, flag_for_review, log_timeline_change;
  JudgeVerdict.contradicts → verdict; ContradictionFinding now carries verdict;
  ProbeReport adds queries_with_any_finding + verdict_breakdown (additive).
- judge.ts: parseResolutionKind + parseVerdict guards; normalizeVerdict reads
  the new field and applies the C1 confidence floor only to verdict='contradiction'
  (the new verdicts are informational classifications, no floor). Prompt rubric
  rewritten to ask for verdict + extended severity scale.
- severity-classify.ts: 'info' joins the rank with value 0; defaultSeverityForVerdict
  maps each verdict to its baseline severity (D7 — supersession=info, regression=high,
  etc.). parseSeverity gains a fallback param so consumers can override 'low' default.
- auto-supersession.ts: classifyResolution + renderResolutionCommand handle the
  three new resolution kinds. Probe still NEVER auto-mutates — the new kinds
  render paste-ready commands or informational lines.
- cache.ts: isJudgeVerdict shape check matches the new verdict field; old v1
  rows fail the guard and treat as misses.
- runner.ts: emit predicate at cache-hit and judge-success branches changes
  from `verdict.contradicts` to `verdict.verdict !== 'no_contradiction'`.
  Without this, the new verdicts vanish from the report. Added per-verdict
  tally + queriesWithAnyFinding alongside the strict queriesWithContradiction.
- trends.ts: latest run verdict breakdown surfaces in the trend chart.

Test fixtures updated across 8 test files. All 210 eval-contradictions unit
tests pass. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): relax date-filter rule 3 when both sides dated

Lane B of the temporal-contradiction-probe wave. The v1 date pre-filter
skipped pairs whose chunk-text-extracted dates differed by >30 days as a
cost-saving heuristic. That heuristic silently killed exactly the cases the
new verdict taxonomy exists to surface — role transitions across years
(e.g. a 2017 historical record vs. a 2025 current state), MRR claims years
apart, status changes recorded over time.

Lane A1+A2 made temporal supersession explicit and cheap to classify. The
filter no longer needs to skip these pairs; the judge can label them.

Changes:
- date-filter.ts: shouldSkipForDateMismatch accepts optional effectiveDateA
  and effectiveDateB. When BOTH are non-null, returns skip=false with the new
  'both_have_effective_date' reason — the judge will see the dates via the
  (from: YYYY-MM-DD) prompt tag from Lane A1. Other rules (same-paragraph
  dual-date override, missing-date fallback) preserved verbatim and still
  run first.
- runner.ts: threads pair.{a,b}.effective_date into the date-filter call.
  Pairs that previously vanished into the skip bucket now reach the judge.

Tests (R1 IRON RULE regression suite, 6 new cases):
- both sides effective_date → not skipped
- both sides effective_date overrides >30d chunk-text rule
- rule 1 (same-paragraph dual-date) still wins over effective_date relaxation
- rule 2 (missing chunk dates) still applies when effective_date partially present
- undefined effective_dates fall through to v1 behavior (back-compat)
- empty-string effective_date treated as missing (only real dates enable the relaxation)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): cost-estimate prompt + --budget-usd + Haiku routing

Lane C of the temporal-contradiction-probe wave. Three layers of cost
guardrail, all stacked:

(a) cost-estimate prompt at probe-run-time. Before the runner spends any
    tokens after a PROMPT_VERSION change, eval-suspected-contradictions
    reads the most recent persisted prompt_version from
    eval_contradictions_runs and compares. When they differ:
      - TTY: prints an upper-bound estimate + Ctrl-C window (default 10s,
        override via GBRAIN_PROBE_PROMPT_GRACE_SECONDS).
      - non-TTY: prints the estimate + auto-proceeds (autopilot path).
      - --yes override or GBRAIN_NO_PROBE_PROMPT=1: skip entirely.
    Mirrors the v0.32.7 runPostUpgradeReembedPrompt pattern.

(b) --budget-usd N hard cap (pre-existing; PreFlightBudgetError surfaces
    when the estimate alone exceeds the cap, and CostTracker halts the
    run mid-flight when cumulative cost exceeds it). Documented in the
    help text alongside (a).

(c) Judge model now routes through resolveModel() with configKey
    'models.eval.contradictions_judge', tier 'utility' (Haiku-class
    default), and env var GBRAIN_CONTRADICTIONS_JUDGE_MODEL. The legacy
    --judge CLI flag still wins as the highest-precedence override.
    Doctor's model touchpoint registry (src/commands/models.ts:50) carries
    the new key so `gbrain models` and `gbrain models doctor` surface it.

Also in this lane:
- CLI: --severity accepts 'info' (the new Severity member from Lane A2).
- CLI: --severity output shows [verdict] tag alongside slug pairs so
  operators distinguish genuine contradictions from temporal classifications.
- Human summary: prints the new queries_with_any_finding metric and the
  per-verdict breakdown table.
- Help text: explains the cost-prompt + budget-cap + model-routing
  interactions in one paragraph.

New tests (9 cases on the cost-prompt helper):
- --yes override skips
- GBRAIN_NO_PROBE_PROMPT=1 skips
- prompt_version unchanged → skips
- non-TTY auto-proceeds with stderr note
- TTY proceeds after grace
- TTY aborts on Ctrl-C
- fresh brain (no prior runs) fires the prompt
- GBRAIN_PROBE_PROMPT_GRACE_SECONDS override honored
- estimate banner contains query count + judge model + dollar amount

All 225 eval-contradictions tests + 25 model-config tests pass. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(eval): R4/R5/R6 IRON-RULE regressions for the verdict-enum wave

Lane D of the temporal-contradiction-probe wave. The Lanes A1/A2/B/C lanes
landed the behavior; this lane pins the regressions that protect the wave
against future drift.

R4 (runner emit predicate): five new tests, one per non-no_contradiction
verdict, prove the runner.ts emit rule surfaces each one as a finding with
the correct verdict tag, and that:
  - queries_with_contradiction (Wilson-CI denominator) ONLY counts verdict
    ='contradiction' — the strict metric is preserved
  - queries_with_any_finding counts every non-no_contradiction verdict
  - verdict_breakdown tallies correctly
Plus one negative case: verdict='no_contradiction' produces zero findings.
Without R4, a future runner refactor could collapse the new verdicts back
to /dev/null and the report would silently shrink.

R5 (cache key shape): direct shape assertion on buildCacheKey output. The
key tuple is exactly 5 fields (chunk_a_hash, chunk_b_hash, model_id,
prompt_version, truncation_policy). Adding a 6th field would silently break
every operator's brain (no migration path).

R6 (contradiction severity unchanged): four tests on normalizeVerdict pin
the legacy semantics — judge-supplied severity wins (whether 'high' or
'low'), and on garbage severity input the fallback is 'medium' (per
defaultSeverityForVerdict('contradiction')) NOT 'low'. The contradiction
verdict's severity must never default to 'low', which would silently mask
genuine conflicts as cosmetic naming issues. The temporal_regression case
is included for parity (garbage → 'high' since regressions are real
investor red flags).

236 eval-contradictions tests pass (211 + 6 R4 + 1 R5 + 4 R6 + 9 cost-prompt
from Lane C).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ci): privacy lint for docs/proposals/*.md

Captures the residual TODO from the temporal-contradiction-probe wave's
plan: prevent the bug class where an RFC lands in docs/proposals/ with
PII that should never appear in a public technical artifact. The
original RFC had to be scrubbed at force-push time (Step 0); this lint
catches the same patterns at CI time so the next one can't slip through.

Sibling to scripts/check-privacy.sh:
- check-privacy.sh: bans the literal "Wintermute" repo-wide.
- check-proposal-pii.sh: focuses on docs/proposals/*.md and the OTHER
  PII classes — personal-relationship vocabulary, private repo refs.

Design contract: the denylist names PATTERNS, not real people. Naming
specific real names (deceased relatives, therapist first names,
dealflow contacts) inside this script would leak PII into the repo
just by appearing here. The structural patterns below catch the
SURROUNDING vocabulary that always accompanies such content in
personal RFC prose. Trade-off: a future RFC that names a real person
without any contextual markers won't be caught — accepted as residual
risk handled by human review.

Patterns flagged in docs/proposals/*.md:
- garrytan/brain (private repo reference)
- trial separation, permanent separation
- couples session, couples therapist
- divorce attorney(s)
- grandmother's funeral, aunt's funeral
- wintermute (also caught by check-privacy.sh; listed here for
  proposal-scoped clarity)

Bare common words (separation, funeral) are NOT banned — only the
combined personal-context phrases. "Separation of concerns" and other
software vocabulary survives.

Wired into:
- `bun run verify` (gates every push)
- `bun run check:all`
- `bun run check:proposal-pii` (standalone)

Tests: 15 cases in test/scripts/check-proposal-pii.test.ts.
- Each pattern flagged when present, plus exit-code + stderr signal.
- Two negative cases (separation-of-concerns, funeral metaphor) prove
  the lint doesn't false-positive on legitimate software prose.
- No-proposals-dir → exit 0 (not a failure).
- Multi-hit case proves all patterns surface together with a summary
  count.
- The two test fixtures that name "Wintermute" / "WINTERMUTE" as
  sentinel literals are allowlisted in check-test-real-names.sh per
  the same meta-rule-enforcement exception as check-privacy.sh itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(privacy): allowlist new privacy-guard files in check-privacy.sh

check-privacy.sh bans the literal Wintermute repo-wide. The two new files
from the v0.34 privacy lint (scripts/check-proposal-pii.sh and its test)
necessarily name the token to do their job. Same meta-rule-enforcement
exception as scripts/check-privacy.sh itself, scripts/check-test-real-names.sh,
test/recency-decay.test.ts, and the existing entries — describing what
the rule forbids requires naming it.

Without this allowlist, `bun run verify` fails on check:privacy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.35.1.0)

Temporal-contradiction-probe wave — Phase 1 of the RFC at
docs/proposals/temporal-contradiction-probe.md.

Headline: the contradiction probe now classifies pairs into a 6-member
verdict enum (no_contradiction, contradiction, temporal_supersession,
temporal_regression, temporal_evolution, negation_artifact) and sees the
page-level effective_date for each chunk via a (from: YYYY-MM-DD) tag in
the prompt. The pre-judge date filter no longer skips dated wide-gap pairs,
so the role-transition class (e.g. a 2017 historical record vs. a 2025
current state) reaches the judge and gets classified as
temporal_supersession instead of vanishing into the skip bucket.

PROMPT_VERSION bumped 1 → 2 (cache fully invalidated). Three-layer cost
guardrail: TTY-only cost-estimate prompt with Ctrl-C window, --budget-usd
hard cap, Haiku-tier routing via new models.eval.contradictions_judge
config key.

Also adds a CI privacy lint (scripts/check-proposal-pii.sh) wired into
bun run verify that catches PII patterns in docs/proposals/*.md so future
RFCs can't ship with personal-context vocabulary the way this wave's
source RFC did at draft time.

Phases 2-4 deferred to follow-up RFCs per the plan.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 08:32:03 -07:00

195 lines
6.7 KiB
Bash
Executable File

#!/bin/bash
#
# check-privacy.sh — CLAUDE.md:550 enforcement.
#
# CLAUDE.md forbids the private OpenClaw fork name in public artifacts:
# CHANGELOG.md, README.md, docs/, skills/, PR titles + bodies, commit
# messages, and comments in checked-in code. This script greps for the
# banned name in either the staged index (for pre-commit hooks) or the
# working tree (for CI) and fails loudly if found.
#
# The allow-list below whitelists files where the name is legitimate
# — specifically, this script itself (where we reference the name to
# describe the rule) and upgrade guides that historically referenced
# the pre-rename fork name.
#
# Usage:
# scripts/check-privacy.sh # scan working tree
# scripts/check-privacy.sh --staged # scan git staged index
# scripts/check-privacy.sh --help
#
# Exit codes:
# 0 clean
# 1 banned name found (or --help printed)
# 2 git / grep not available
set -euo pipefail
BANNED_NAME='wintermute'
# v0.25.1 (codex T7): additional patterns from wintermute-specific filesystem
# layouts that would leak private fork context if they slipped through a port.
# `wintermute_only` already matches via the case-insensitive `wintermute` regex
# above; this list is for orthogonal patterns.
BANNED_PATHS=(
'/data/brain/'
'/data/.openclaw/'
)
usage() {
cat <<EOF
scripts/check-privacy.sh — scan for the banned OpenClaw fork name.
USAGE:
scripts/check-privacy.sh Scan all tracked files in the working tree.
scripts/check-privacy.sh --staged Scan only files staged for commit.
scripts/check-privacy.sh --help Show this message.
The script greps for '${BANNED_NAME}' (case-insensitive) in:
- CHANGELOG.md, README.md
- docs/**
- skills/**
- src/**
- test/**
- scripts/**
Allow-list (references to the name are permitted):
- scripts/check-privacy.sh itself
- docs/UPGRADING_DOWNSTREAM_AGENTS.md (historical context for pre-rename upgrades)
- .git/** (branch names, commit history — not checked in artifacts)
Exit codes: 0 clean, 1 banned name found, 2 setup error.
EOF
}
MODE=working
for arg in "$@"; do
case "$arg" in
--staged) MODE=staged ;;
--help|-h) usage; exit 1 ;;
*)
echo "Unknown argument: $arg" >&2
usage >&2
exit 2
;;
esac
done
if ! command -v git >/dev/null 2>&1; then
echo "check-privacy: git not found" >&2
exit 2
fi
# Build the file list by scanning-mode.
if [ "$MODE" = staged ]; then
FILES=$(git diff --cached --name-only --diff-filter=ACMR 2>/dev/null || true)
else
FILES=$(git ls-files 2>/dev/null || true)
fi
if [ -z "$FILES" ]; then
exit 0
fi
# Allow-list: files in which the banned name is legitimate.
# Meta-rule docs (define the rule itself), auto-generated LLM indexes,
# historical upgrade guides, and the test that enforces the rule
# against recipes/ all reference the banned name by necessity.
ALLOW_LIST=(
'scripts/check-privacy.sh'
'CLAUDE.md'
'llms-full.txt'
'docs/UPGRADING_DOWNSTREAM_AGENTS.md'
'test/integrations.test.ts'
# v0.25.1 (codex T7) BANNED_PATHS allow-list:
# Historical docs, frozen migration files, test fixtures, and env-var
# fallbacks where /data/brain/ or /data/.openclaw/ appears legitimately.
# New skills/, src/, and tests must NOT slip onto this list — extend the
# banned check above instead.
'docs/GBRAIN_RECOMMENDED_SCHEMA.md'
'docs/GBRAIN_V0.md'
'docs/guides/minions-shell-jobs.md'
'scripts/smoke-test.sh'
'skills/migrations/v0.9.0.md'
'skills/migrations/v0.14.0.md'
'test/storage-status.test.ts'
# CHANGELOG.md documents the rule (the v0.25.1 entry references the
# banned literals in describing what's banned). Same exception status
# as CLAUDE.md and this script itself: meta-documentation needs to
# name the patterns it forbids.
'CHANGELOG.md'
# skills/migrations/v0.25.1.md is the agent-readable upgrade
# walkthrough; it explains the privacy-guard extension to the
# operating agent and references the banned literals while doing so.
'skills/migrations/v0.25.1.md'
# v0.29.1: the recency-decay default-map test asserts that
# DEFAULT_RECENCY_DECAY's keys do NOT include fork-specific path
# prefixes. The test must name the banned tokens to assert their
# absence — same exception status as scripts/check-privacy.sh,
# CHANGELOG.md, and CLAUDE.md (meta-rule enforcement requires
# mentioning what the rule forbids).
'test/recency-decay.test.ts'
# v0.32.5: the sibling check-test-real-names.sh enforces the same
# privacy rule for test fixtures and lists the banned names literally
# (Wintermute, Hermes, etc) inside its BANNED_NAMES + ALLOWLIST arrays.
# Same meta-rule-enforcement exception as scripts/check-privacy.sh itself.
'scripts/check-test-real-names.sh'
# v0.34 / Lane CI: scripts/check-proposal-pii.sh and its test list the
# banned literal as part of the structural denylist they enforce against
# docs/proposals/*.md. Same meta-rule-enforcement exception as the two
# entries above — describing what the rule forbids requires naming it.
'scripts/check-proposal-pii.sh'
'test/scripts/check-proposal-pii.test.ts'
# v0.32.3.0: the functional-area-resolver skill's behavior-contract
# section describes the privacy guarantees the skill preserves and
# references the banned literals while doing so (line 306). Same
# meta-rule-enforcement exception as scripts/check-privacy.sh and
# CHANGELOG.md — describing what the rule forbids requires naming it.
'skills/functional-area-resolver/SKILL.md'
)
is_allowed() {
local f="$1"
for a in "${ALLOW_LIST[@]}"; do
if [ "$f" = "$a" ]; then
return 0
fi
done
return 1
}
FOUND=0
while IFS= read -r file; do
[ -z "$file" ] && continue
[ ! -f "$file" ] && continue
if is_allowed "$file"; then
continue
fi
# Case-insensitive grep; only specific extensions + known docs.
case "$file" in
*.md|*.ts|*.mjs|*.js|*.py|*.sh|*.json|*.yaml|*.yml|*.txt|README*|CHANGELOG*|CLAUDE*|AGENTS*)
if grep -in "$BANNED_NAME" "$file" >/dev/null 2>&1; then
echo "[check-privacy] BANNED NAME in $file:" >&2
grep -in "$BANNED_NAME" "$file" | sed 's|^| |' >&2
FOUND=1
fi
# Banned wintermute-specific filesystem paths (codex T7).
for path in "${BANNED_PATHS[@]}"; do
if grep -nF "$path" "$file" >/dev/null 2>&1; then
echo "[check-privacy] BANNED PATH '$path' in $file:" >&2
grep -nF "$path" "$file" | sed 's|^| |' >&2
FOUND=1
fi
done
;;
esac
done <<< "$FILES"
if [ "$FOUND" -eq 1 ]; then
echo "" >&2
echo "The private OpenClaw fork name is banned in public artifacts." >&2
echo "CLAUDE.md:550. Replace with 'your OpenClaw', 'OpenClaw reference deployment', or 'openclaw-reference'." >&2
exit 1
fi
exit 0