# gbrain eval suspected-contradictions (v0.32.6) The contradiction probe samples retrieval results, asks an LLM judge whether any pair contradicts on a factual claim relevant to the user's query, and aggregates into a calibrated report. The output is data — the operator decides what to act on. This doc covers the architecture, severity rubric, how to interpret the headline number, and when to act. ## Why this exists gbrain handles contradictions for *curated* pages via compiled-truth-plus- timeline and source-boost: when `companies/acme.md` says MRR is $2M and a chat transcript from 2024 says MRR was $50K, the curated page outranks the chat. `takes.active` filtering hides explicitly-superseded takes. Recency decay biases ranking toward fresher content per source-tier. What none of those mechanisms measure: how often do unmarked semantic contradictions actually surface in retrieval? Without a probe, every "should we build the bigger swing (chunk-level `revises` field + ranking change)" decision is vibes. The probe produces evidence. ## Architecture ``` ┌──────────────────────────────────────┐ │ gbrain eval suspected-contradictions │ └──────────────────┬───────────────────┘ │ ┌──────────────────▼───────────────────┐ │ For each query: hybridSearch top-K │ │ → cross_slug_chunks + intra_page │ │ chunk-vs-take pairs │ └──────────────────┬───────────────────┘ │ ┌──────────────────▼───────────────────┐ │ Date pre-filter: skip pairs whose │ │ dates are >30d apart (Codex fix: │ │ same-paragraph-dual-date overrides) │ └──────────────────┬───────────────────┘ │ ┌──────────────────▼───────────────────┐ │ Persistent cache lookup │ │ (chunk_a_hash, chunk_b_hash, model, │ │ prompt_version, truncation_policy) │ └────────┬─────────┬────────────────────┘ hit│ │miss │ ▼ │ ┌─────────────────────────┐ │ │ LLM judge call │ │ │ → JudgeVerdict │ │ │ confidence floor ≥ 0.7 │ │ └─────────┬───────────────┘ │ │ ▼ ▼ ┌──────────────────────────────────────┐ │ Aggregate per-query + global stats │ │ Wilson 95% CI on headline % │ │ source-tier breakdown │ │ hot pages + resolution proposals │ └──────────────────┬───────────────────┘ │ ▼ ProbeReport JSON │ ┌──────────────────┼──────────────────────┬───────────────┐ ▼ ▼ ▼ ▼ doctor (M1) MCP (M3) synthesize (M2) trend (M5) surfaces find_contradictions informational persistent findings op for agents block in prompt tracking ``` ## Severity rubric The judge assigns severity per finding: | Level | Rubric | Example | |---|---|---| | `low` | naming/format differences | "Alice Smith" vs "A. Smith" | | `medium` | factual values that may be stale | revenue figure, headcount, valuation | | `high` | identity / structural claims | founder/CEO/CFO role, company status | Doctor sorts findings by severity DESC. The MCP op accepts a severity filter so agents can fetch just the high-priority items. ## How to interpret the headline number The probe outputs `queries_with_contradiction / queries_evaluated` with a Wilson 95% confidence interval: ``` Queries with >=1 contradiction: 12 / 50 (24%) Wilson CI 95%: 14–37% ``` What this says: with 95% confidence, the true rate is between 14% and 37%. The 24% point estimate is the most-likely-value but bounded by sampling noise. **`small_sample_note` fires when n < 30** — at that scale the CI is too wide to act on. Decision criteria for the bigger swing (chunk-level `revises` field): | Wilson CI lower bound | What it says | Action | |---|---|---| | < 5% | Source-boost + recency-decay + curated pages handle the load | Stop here; this is the right scope | | 5–15% | Real but bounded | Operator decides whether the cost justifies the swing | | > 15% | Real and substantial | Plan the bigger swing in v0.34+ | ## When to act on findings Each finding ships with a `resolution_command` field — paste-ready: - `gbrain takes supersede --row N` — newer take should replace the older chunk text on the same page (intra_page kind). - `gbrain dream --phase synthesize --slug ` — compiled_truth for the curated entity needs an update (cross_slug curated-vs-bulk). - `gbrain takes mark-debate --row N` — intentional disagreement (e.g., two opinions you want to keep both of). - `# manual review: vs ` — judge wasn't sure; operator decides. Run `gbrain eval suspected-contradictions review --severity high` to inspect findings without re-running the probe. ## Cost model Default judge is `claude-haiku-4-5` at ~$1/Mtok in, $5/Mtok out. With the v0.32.6 truncation at 1500 chars per pair, ~500 input + 80 output tokens per judge call. Budget cap defaults to $5 in TTY / $1 non-TTY. - ~$0.0006 per judge call - ~$0.005 per query (after date pre-filter + cache hits) - ~$0.50 per 100 queries The persistent cache means nightly runs against the same query set pay near-zero on re-runs (until you bump PROMPT_VERSION). ## Trust posture - Probe never mutates the brain. Runs only read pages/takes/chunks. Writes go only to `eval_contradictions_runs` and `eval_contradictions_cache`. - MCP `find_contradictions` is read-scope. NOT in the subagent allowlist — user-initiated only, not autonomous-action surface. - Build-fixture script is local-only. The redactor + `isCleanForCommit` gate makes accidental private-data commits hard, but the operator MUST inspect every redaction before commit. ## See also - Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md` - CHANGELOG: `## [0.32.6]` entry covers the whole release. - Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence + trend-tracking workflow. - **Temporal axis follow-on (v0.35.3.1 + v0.35.7):** v0.35.3.1 added a six-member verdict enum (`no_contradiction | contradiction | temporal_supersession | temporal_regression | temporal_evolution | negation_artifact`) and threaded `pages.effective_date` into the judge prompt so the probe stops crying wolf on legitimate change-over-time. v0.35.7 lands the trajectory substrate the probe pointed at: `gbrain eval trajectory ` shows the chronological typed-claim history with regressions flagged inline; `gbrain founder scorecard ` rolls up four signals (accuracy, consistency, growth direction, red flags) into a stable JSON contract. MCP op `find_trajectory` (read scope, visibility-filtered for remote callers) exposes the same data to agents. The probe's `temporal_supersession` verdict and the consolidate phase's `valid_until` writeback both preserve the `auto-supersession.ts:4` "NEVER auto-applies" invariant — the probe still emits paste-ready commands, only `consolidate` writes `valid_until` (R1+R8 grep guard pins this).