mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* rfc: temporal axis for contradiction probe Field report on residual HIGH findings from gbrain eval suspected-contradictions and proposal for a 4-phase fix (Phase 1 = judge prompt + verdict enum is the recommended starting point). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): pass effective_date to judge prompt; bump PROMPT_VERSION Lane A1 of the temporal-contradiction-probe wave. Threads page-level effective_date through the search projection into the contradiction judge so the LLM can reason about supersession instead of treating every dated pair as a contradiction. Changes: - SearchResult interface adds optional effective_date + effective_date_source fields; rowToSearchResult populates them from the row data with date-only YYYY-MM-DD normalization (handles both postgres.js Date and PGLite string). - 8 SELECT projection sites (3 in postgres-engine, 5 in pglite-engine) now carry p.effective_date + p.effective_date_source through their inner CTEs and outer SELECTs so search results expose the field on both engines. - PairMember (eval-contradictions/types.ts) gets the two fields as required (string | null) so the type forces every constructor to think about temporal anchoring. Runner's searchResultToMember + takeToMember handle the normalization; takes inherit the chunk's page-level date. - buildJudgePrompt emits `Statement A (from: YYYY-MM-DD)` when effective_date is non-null, else `(date unknown)`. Prompt instructions explain the tag so the model knows what to do with it. - PROMPT_VERSION bumps '1' → '2'. Cache-key tuple shape unchanged; old rows miss naturally on first run against the new prompt. Test fixtures in 5 files updated to include the new required fields. All 205 eval-contradictions unit tests + 101 search-related tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): replace contradicts:boolean with verdict:enum (6 members) Lane A2 of the temporal-contradiction-probe wave. Expands the judge's classification vocabulary from a binary contradicts:bool to a six-member verdict enum so the probe can distinguish "this changed" from "this is wrong". Verdict taxonomy: no_contradiction — drop from findings contradiction — genuine conflict at same point in time temporal_supersession — newer claim updates/replaces older; not an error temporal_regression — metric/status went backwards over time (signal) temporal_evolution — legitimate change, neither supersession nor regression negation_artifact — judge misread an explicit negation Changes: - types.ts: Verdict union (6 members); Severity gains 'info'; ResolutionKind extended with temporal_supersede, flag_for_review, log_timeline_change; JudgeVerdict.contradicts → verdict; ContradictionFinding now carries verdict; ProbeReport adds queries_with_any_finding + verdict_breakdown (additive). - judge.ts: parseResolutionKind + parseVerdict guards; normalizeVerdict reads the new field and applies the C1 confidence floor only to verdict='contradiction' (the new verdicts are informational classifications, no floor). Prompt rubric rewritten to ask for verdict + extended severity scale. - severity-classify.ts: 'info' joins the rank with value 0; defaultSeverityForVerdict maps each verdict to its baseline severity (D7 — supersession=info, regression=high, etc.). parseSeverity gains a fallback param so consumers can override 'low' default. - auto-supersession.ts: classifyResolution + renderResolutionCommand handle the three new resolution kinds. Probe still NEVER auto-mutates — the new kinds render paste-ready commands or informational lines. - cache.ts: isJudgeVerdict shape check matches the new verdict field; old v1 rows fail the guard and treat as misses. - runner.ts: emit predicate at cache-hit and judge-success branches changes from `verdict.contradicts` to `verdict.verdict !== 'no_contradiction'`. Without this, the new verdicts vanish from the report. Added per-verdict tally + queriesWithAnyFinding alongside the strict queriesWithContradiction. - trends.ts: latest run verdict breakdown surfaces in the trend chart. Test fixtures updated across 8 test files. All 210 eval-contradictions unit tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): relax date-filter rule 3 when both sides dated Lane B of the temporal-contradiction-probe wave. The v1 date pre-filter skipped pairs whose chunk-text-extracted dates differed by >30 days as a cost-saving heuristic. That heuristic silently killed exactly the cases the new verdict taxonomy exists to surface — role transitions across years (e.g. a 2017 historical record vs. a 2025 current state), MRR claims years apart, status changes recorded over time. Lane A1+A2 made temporal supersession explicit and cheap to classify. The filter no longer needs to skip these pairs; the judge can label them. Changes: - date-filter.ts: shouldSkipForDateMismatch accepts optional effectiveDateA and effectiveDateB. When BOTH are non-null, returns skip=false with the new 'both_have_effective_date' reason — the judge will see the dates via the (from: YYYY-MM-DD) prompt tag from Lane A1. Other rules (same-paragraph dual-date override, missing-date fallback) preserved verbatim and still run first. - runner.ts: threads pair.{a,b}.effective_date into the date-filter call. Pairs that previously vanished into the skip bucket now reach the judge. Tests (R1 IRON RULE regression suite, 6 new cases): - both sides effective_date → not skipped - both sides effective_date overrides >30d chunk-text rule - rule 1 (same-paragraph dual-date) still wins over effective_date relaxation - rule 2 (missing chunk dates) still applies when effective_date partially present - undefined effective_dates fall through to v1 behavior (back-compat) - empty-string effective_date treated as missing (only real dates enable the relaxation) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): cost-estimate prompt + --budget-usd + Haiku routing Lane C of the temporal-contradiction-probe wave. Three layers of cost guardrail, all stacked: (a) cost-estimate prompt at probe-run-time. Before the runner spends any tokens after a PROMPT_VERSION change, eval-suspected-contradictions reads the most recent persisted prompt_version from eval_contradictions_runs and compares. When they differ: - TTY: prints an upper-bound estimate + Ctrl-C window (default 10s, override via GBRAIN_PROBE_PROMPT_GRACE_SECONDS). - non-TTY: prints the estimate + auto-proceeds (autopilot path). - --yes override or GBRAIN_NO_PROBE_PROMPT=1: skip entirely. Mirrors the v0.32.7 runPostUpgradeReembedPrompt pattern. (b) --budget-usd N hard cap (pre-existing; PreFlightBudgetError surfaces when the estimate alone exceeds the cap, and CostTracker halts the run mid-flight when cumulative cost exceeds it). Documented in the help text alongside (a). (c) Judge model now routes through resolveModel() with configKey 'models.eval.contradictions_judge', tier 'utility' (Haiku-class default), and env var GBRAIN_CONTRADICTIONS_JUDGE_MODEL. The legacy --judge CLI flag still wins as the highest-precedence override. Doctor's model touchpoint registry (src/commands/models.ts:50) carries the new key so `gbrain models` and `gbrain models doctor` surface it. Also in this lane: - CLI: --severity accepts 'info' (the new Severity member from Lane A2). - CLI: --severity output shows [verdict] tag alongside slug pairs so operators distinguish genuine contradictions from temporal classifications. - Human summary: prints the new queries_with_any_finding metric and the per-verdict breakdown table. - Help text: explains the cost-prompt + budget-cap + model-routing interactions in one paragraph. New tests (9 cases on the cost-prompt helper): - --yes override skips - GBRAIN_NO_PROBE_PROMPT=1 skips - prompt_version unchanged → skips - non-TTY auto-proceeds with stderr note - TTY proceeds after grace - TTY aborts on Ctrl-C - fresh brain (no prior runs) fires the prompt - GBRAIN_PROBE_PROMPT_GRACE_SECONDS override honored - estimate banner contains query count + judge model + dollar amount All 225 eval-contradictions tests + 25 model-config tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(eval): R4/R5/R6 IRON-RULE regressions for the verdict-enum wave Lane D of the temporal-contradiction-probe wave. The Lanes A1/A2/B/C lanes landed the behavior; this lane pins the regressions that protect the wave against future drift. R4 (runner emit predicate): five new tests, one per non-no_contradiction verdict, prove the runner.ts emit rule surfaces each one as a finding with the correct verdict tag, and that: - queries_with_contradiction (Wilson-CI denominator) ONLY counts verdict ='contradiction' — the strict metric is preserved - queries_with_any_finding counts every non-no_contradiction verdict - verdict_breakdown tallies correctly Plus one negative case: verdict='no_contradiction' produces zero findings. Without R4, a future runner refactor could collapse the new verdicts back to /dev/null and the report would silently shrink. R5 (cache key shape): direct shape assertion on buildCacheKey output. The key tuple is exactly 5 fields (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy). Adding a 6th field would silently break every operator's brain (no migration path). R6 (contradiction severity unchanged): four tests on normalizeVerdict pin the legacy semantics — judge-supplied severity wins (whether 'high' or 'low'), and on garbage severity input the fallback is 'medium' (per defaultSeverityForVerdict('contradiction')) NOT 'low'. The contradiction verdict's severity must never default to 'low', which would silently mask genuine conflicts as cosmetic naming issues. The temporal_regression case is included for parity (garbage → 'high' since regressions are real investor red flags). 236 eval-contradictions tests pass (211 + 6 R4 + 1 R5 + 4 R6 + 9 cost-prompt from Lane C). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ci): privacy lint for docs/proposals/*.md Captures the residual TODO from the temporal-contradiction-probe wave's plan: prevent the bug class where an RFC lands in docs/proposals/ with PII that should never appear in a public technical artifact. The original RFC had to be scrubbed at force-push time (Step 0); this lint catches the same patterns at CI time so the next one can't slip through. Sibling to scripts/check-privacy.sh: - check-privacy.sh: bans the literal "Wintermute" repo-wide. - check-proposal-pii.sh: focuses on docs/proposals/*.md and the OTHER PII classes — personal-relationship vocabulary, private repo refs. Design contract: the denylist names PATTERNS, not real people. Naming specific real names (deceased relatives, therapist first names, dealflow contacts) inside this script would leak PII into the repo just by appearing here. The structural patterns below catch the SURROUNDING vocabulary that always accompanies such content in personal RFC prose. Trade-off: a future RFC that names a real person without any contextual markers won't be caught — accepted as residual risk handled by human review. Patterns flagged in docs/proposals/*.md: - garrytan/brain (private repo reference) - trial separation, permanent separation - couples session, couples therapist - divorce attorney(s) - grandmother's funeral, aunt's funeral - wintermute (also caught by check-privacy.sh; listed here for proposal-scoped clarity) Bare common words (separation, funeral) are NOT banned — only the combined personal-context phrases. "Separation of concerns" and other software vocabulary survives. Wired into: - `bun run verify` (gates every push) - `bun run check:all` - `bun run check:proposal-pii` (standalone) Tests: 15 cases in test/scripts/check-proposal-pii.test.ts. - Each pattern flagged when present, plus exit-code + stderr signal. - Two negative cases (separation-of-concerns, funeral metaphor) prove the lint doesn't false-positive on legitimate software prose. - No-proposals-dir → exit 0 (not a failure). - Multi-hit case proves all patterns surface together with a summary count. - The two test fixtures that name "Wintermute" / "WINTERMUTE" as sentinel literals are allowlisted in check-test-real-names.sh per the same meta-rule-enforcement exception as check-privacy.sh itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(privacy): allowlist new privacy-guard files in check-privacy.sh check-privacy.sh bans the literal Wintermute repo-wide. The two new files from the v0.34 privacy lint (scripts/check-proposal-pii.sh and its test) necessarily name the token to do their job. Same meta-rule-enforcement exception as scripts/check-privacy.sh itself, scripts/check-test-real-names.sh, test/recency-decay.test.ts, and the existing entries — describing what the rule forbids requires naming it. Without this allowlist, `bun run verify` fails on check:privacy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.35.1.0) Temporal-contradiction-probe wave — Phase 1 of the RFC at docs/proposals/temporal-contradiction-probe.md. Headline: the contradiction probe now classifies pairs into a 6-member verdict enum (no_contradiction, contradiction, temporal_supersession, temporal_regression, temporal_evolution, negation_artifact) and sees the page-level effective_date for each chunk via a (from: YYYY-MM-DD) tag in the prompt. The pre-judge date filter no longer skips dated wide-gap pairs, so the role-transition class (e.g. a 2017 historical record vs. a 2025 current state) reaches the judge and gets classified as temporal_supersession instead of vanishing into the skip bucket. PROMPT_VERSION bumped 1 → 2 (cache fully invalidated). Three-layer cost guardrail: TTY-only cost-estimate prompt with Ctrl-C window, --budget-usd hard cap, Haiku-tier routing via new models.eval.contradictions_judge config key. Also adds a CI privacy lint (scripts/check-proposal-pii.sh) wired into bun run verify that catches PII patterns in docs/proposals/*.md so future RFCs can't ship with personal-context vocabulary the way this wave's source RFC did at draft time. Phases 2-4 deferred to follow-up RFCs per the plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
149 lines
5.9 KiB
Bash
Executable File
149 lines
5.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# CI guard: fail if any test fixture references a real person's name.
|
|
#
|
|
# CLAUDE.md's "Privacy rule" section is unambiguous: never reference real
|
|
# people, companies, funds, or private agent names in any public-facing
|
|
# artifact. Tests are checked-in code distributed with every release and
|
|
# indexed by GitHub search. This guard catches the patterns the rule names.
|
|
#
|
|
# Design (post-Codex F4 review):
|
|
# - Banned names: exact-string allowlist of known real identifiers. Adding
|
|
# a name when CLAUDE.md flags one is a one-line edit.
|
|
# - Banned emails: specific addresses that identify real contacts. NOT a
|
|
# broad corporate-email regex — those would catch legitimate fixture
|
|
# domains in billing/auth tests (`customer@stripe.com` etc.).
|
|
# - Allowlist: exact "file:offending-string" pairs that are intentional
|
|
# and pre-existing (e.g., the user's own email is not a "contact").
|
|
#
|
|
# Scope: test/**/*.test.ts only. Historical CHANGELOG entries, doc examples,
|
|
# and skill READMEs each have their own scrub status and are out of scope
|
|
# for this guard.
|
|
#
|
|
# Usage: scripts/check-test-real-names.sh
|
|
# Exit: 0 clean, 1 banned reference found, 2 setup error (rg + grep missing).
|
|
|
|
set -euo pipefail
|
|
|
|
ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
|
|
cd "$ROOT"
|
|
|
|
# Banned real-name strings (matched as whole words, case-insensitive).
|
|
# Add an entry when CLAUDE.md flags a new real-person name.
|
|
BANNED_NAMES=(
|
|
'Diana' # Diana Hu, named in CLAUDE.md privacy example
|
|
'Wintermute' # private OpenClaw fork name (CLAUDE.md rule)
|
|
'Hermes' # downstream agent fork name
|
|
'Technium' # real GP handle
|
|
'McGrew' # ex-OpenAI exec
|
|
'YC Labs' # internal team name
|
|
)
|
|
|
|
# Banned specific email addresses. NOT a generic corporate-email regex —
|
|
# those would catch legitimate fixture domains in billing/auth tests
|
|
# (`customer@stripe.com`, `account@openai.com` etc).
|
|
BANNED_EMAILS=(
|
|
'diana@ycombinator.com'
|
|
)
|
|
|
|
# Exact "file:offending-string" pairs that are intentional and pre-existing.
|
|
# These pre-date the rule, the file's own author confirmed the use, the
|
|
# string identifies the user themselves (not a contact), OR the reference
|
|
# is structural (e.g., a regression test that ASSERTS the banned name does
|
|
# NOT appear in production code — the name MUST be in the test file as a
|
|
# literal).
|
|
ALLOWLIST=(
|
|
"test/writer.test.ts:garry@ycombinator.com" # user's own email — CLAUDE.md rule does not apply
|
|
"test/integrations.test.ts:Wintermute" # regex pattern in personal-info filter test (structural)
|
|
"test/recency-decay.test.ts:Wintermute" # regression-prevention test asserting wintermute is absent (structural)
|
|
"test/scripts/check-proposal-pii.test.ts:Wintermute" # privacy-guard test asserting docs/proposals/ rejects wintermute (structural; same meta-rule exception as check-privacy.sh)
|
|
"test/scripts/check-proposal-pii.test.ts:WINTERMUTE" # case-insensitive sentinel literal for the same privacy-guard test
|
|
"test/serve-stdio-lifecycle.test.ts:Hermes" # comment naming a downstream-agent scenario — pre-existing, low signal
|
|
"test/extract.test.ts:Hermes" # markdown-link extraction test fixture — pre-existing, ambiguous (Greek god vs fork)
|
|
)
|
|
|
|
# Build the combined regex. Names matched as whole words (\b), emails matched
|
|
# literally with dot escapes.
|
|
PATTERN_PARTS=()
|
|
for n in "${BANNED_NAMES[@]}"; do
|
|
# Escape any regex metacharacters in the name (defensive — most are bare
|
|
# words but YC Labs has a space).
|
|
escaped="${n//./\\.}"
|
|
escaped="${escaped// /\\s}"
|
|
PATTERN_PARTS+=("\\b${escaped}\\b")
|
|
done
|
|
for e in "${BANNED_EMAILS[@]}"; do
|
|
escaped="${e//./\\.}"
|
|
PATTERN_PARTS+=("${escaped}")
|
|
done
|
|
|
|
# Join with |.
|
|
IFS='|' eval 'PATTERN="${PATTERN_PARTS[*]}"'
|
|
|
|
# Find tool.
|
|
if command -v rg >/dev/null 2>&1; then
|
|
matches="$(rg -niH --no-heading -t ts "$PATTERN" test/ 2>/dev/null || true)"
|
|
elif command -v grep >/dev/null 2>&1; then
|
|
matches="$(grep -rniE --include='*.test.ts' "$PATTERN" test/ 2>/dev/null || true)"
|
|
else
|
|
echo "check-test-real-names: ERROR: neither rg nor grep available." >&2
|
|
exit 2
|
|
fi
|
|
|
|
if [ -z "$matches" ]; then
|
|
exit 0
|
|
fi
|
|
|
|
# Apply allowlist. Each line is "file:lineno:content"; check whether
|
|
# "file:<needle>" appears in ALLOWLIST for any needle in BANNED_EMAILS+NAMES
|
|
# that matches the content.
|
|
filtered=""
|
|
while IFS= read -r line; do
|
|
[ -z "$line" ] && continue
|
|
# Extract filename and content (everything after second :).
|
|
file="${line%%:*}"
|
|
rest="${line#*:}"
|
|
# rest is "lineno:content" — strip lineno.
|
|
content="${rest#*:}"
|
|
|
|
matched_needle=""
|
|
for needle in "${BANNED_EMAILS[@]}" "${BANNED_NAMES[@]}"; do
|
|
if echo "$content" | grep -qi -- "$needle"; then
|
|
matched_needle="$needle"
|
|
break
|
|
fi
|
|
done
|
|
|
|
allow_key="${file}:${matched_needle}"
|
|
allowed=0
|
|
for allow_entry in "${ALLOWLIST[@]}"; do
|
|
if [ "$allow_entry" = "$allow_key" ]; then
|
|
allowed=1
|
|
break
|
|
fi
|
|
done
|
|
|
|
if [ "$allowed" = "0" ]; then
|
|
filtered+="${line}"$'\n'
|
|
fi
|
|
done <<< "$matches"
|
|
|
|
if [ -z "$filtered" ]; then
|
|
exit 0
|
|
fi
|
|
|
|
echo "check-test-real-names: banned real-name references found in test/ fixtures." >&2
|
|
echo "" >&2
|
|
echo "$filtered" >&2
|
|
echo "" >&2
|
|
echo "Fix: replace with canonical placeholders per CLAUDE.md 'Name mapping' table." >&2
|
|
echo " alice-example / @alice-example for people" >&2
|
|
echo " bob-example / charlie-example for additional people" >&2
|
|
echo " alice@example.com for emails (example.com is RFC 6761 reserved)" >&2
|
|
echo " acme-example / widget-co for companies" >&2
|
|
echo " fund-a / fund-b for funds" >&2
|
|
echo " a-team / agent-fork for teams / OpenClaw forks" >&2
|
|
echo "" >&2
|
|
echo "If the match is intentional (e.g., the user's own identifier, not a contact)," >&2
|
|
echo "add an exact 'file:string' entry to ALLOWLIST in scripts/check-test-real-names.sh." >&2
|
|
exit 1
|