gbrain/docs/contradictions.md

gbrain eval suspected-contradictions (v0.32.6)

The contradiction probe samples retrieval results, asks an LLM judge whether any pair contradicts on a factual claim relevant to the user's query, and aggregates into a calibrated report. The output is data — the operator decides what to act on. This doc covers the architecture, severity rubric, how to interpret the headline number, and when to act.

Why this exists

gbrain handles contradictions for curated pages via compiled-truth-plus- timeline and source-boost: when companies/acme.md says MRR is $2M and a chat transcript from 2024 says MRR was $50K, the curated page outranks the chat. takes.active filtering hides explicitly-superseded takes. Recency decay biases ranking toward fresher content per source-tier.

What none of those mechanisms measure: how often do unmarked semantic contradictions actually surface in retrieval? Without a probe, every "should we build the bigger swing (chunk-level revises field + ranking change)" decision is vibes. The probe produces evidence.

Architecture

        ┌──────────────────────────────────────┐
        │ gbrain eval suspected-contradictions │
        └──────────────────┬───────────────────┘
                           │
        ┌──────────────────▼───────────────────┐
        │ For each query: hybridSearch top-K   │
        │ → cross_slug_chunks + intra_page     │
        │   chunk-vs-take pairs                │
        └──────────────────┬───────────────────┘
                           │
        ┌──────────────────▼───────────────────┐
        │ Date pre-filter: skip pairs whose    │
        │ dates are >30d apart (Codex fix:     │
        │ same-paragraph-dual-date overrides)  │
        └──────────────────┬───────────────────┘
                           │
        ┌──────────────────▼───────────────────┐
        │ Persistent cache lookup              │
        │ (chunk_a_hash, chunk_b_hash, model,  │
        │  prompt_version, truncation_policy)  │
        └────────┬─────────┬────────────────────┘
              hit│         │miss
                 │         ▼
                 │   ┌─────────────────────────┐
                 │   │ LLM judge call          │
                 │   │ → JudgeVerdict          │
                 │   │ confidence floor ≥ 0.7  │
                 │   └─────────┬───────────────┘
                 │             │
                 ▼             ▼
        ┌──────────────────────────────────────┐
        │ Aggregate per-query + global stats   │
        │ Wilson 95% CI on headline %          │
        │ source-tier breakdown                │
        │ hot pages + resolution proposals     │
        └──────────────────┬───────────────────┘
                           │
                           ▼
                  ProbeReport JSON
                           │
        ┌──────────────────┼──────────────────────┬───────────────┐
        ▼                  ▼                      ▼               ▼
   doctor (M1)         MCP (M3)             synthesize (M2)   trend (M5)
   surfaces           find_contradictions    informational     persistent
   findings           op for agents          block in prompt   tracking

Severity rubric

The judge assigns severity per finding:

Level	Rubric	Example
low	naming/format differences	"Alice Smith" vs "A. Smith"
medium	factual values that may be stale	revenue figure, headcount, valuation
high	identity / structural claims	founder/CEO/CFO role, company status

Doctor sorts findings by severity DESC. The MCP op accepts a severity filter so agents can fetch just the high-priority items.

How to interpret the headline number

The probe outputs queries_with_contradiction / queries_evaluated with a Wilson 95% confidence interval:

Queries with >=1 contradiction: 12 / 50 (24%)  Wilson CI 95%: 14–37%

What this says: with 95% confidence, the true rate is between 14% and 37%. The 24% point estimate is the most-likely-value but bounded by sampling noise. small_sample_note fires when n < 30 — at that scale the CI is too wide to act on.

Decision criteria for the bigger swing (chunk-level revises field):

Wilson CI lower bound	What it says	Action
< 5%	Source-boost + recency-decay + curated pages handle the load	Stop here; this is the right scope
5–15%	Real but bounded	Operator decides whether the cost justifies the swing
> 15%	Real and substantial	Plan the bigger swing in v0.34+

When to act on findings

Each finding ships with a resolution_command field — paste-ready:

• gbrain takes supersede <slug> --row N — newer take should replace the older chunk text on the same page (intra_page kind).
• gbrain dream --phase synthesize --slug <slug> — compiled_truth for the curated entity needs an update (cross_slug curated-vs-bulk).
• gbrain takes mark-debate <slug> --row N — intentional disagreement (e.g., two opinions you want to keep both of).
• # manual review: <a> vs <b> — judge wasn't sure; operator decides.

Run gbrain eval suspected-contradictions review --severity high to inspect findings without re-running the probe.

Cost model

Default judge is claude-haiku-4-5 at ~$1/Mtok in, $5/Mtok out. With the v0.32.6 truncation at 1500 chars per pair, ~500 input + 80 output tokens per judge call. Budget cap defaults to $5 in TTY / $1 non-TTY.

• ~$0.0006 per judge call
• ~$0.005 per query (after date pre-filter + cache hits)
• ~$0.50 per 100 queries

The persistent cache means nightly runs against the same query set pay near-zero on re-runs (until you bump PROMPT_VERSION).

Trust posture

• Probe never mutates the brain. Runs only read pages/takes/chunks. Writes go only to eval_contradictions_runs and eval_contradictions_cache.
• MCP find_contradictions is read-scope. NOT in the subagent allowlist — user-initiated only, not autonomous-action surface.
• Build-fixture script is local-only. The redactor + isCleanForCommit gate makes accidental private-data commits hard, but the operator MUST inspect every redaction before commit.

See also

• Plan: ~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md
• CHANGELOG: ## [0.32.6] entry covers the whole release.
• Cost discipline: docs/eval-bench.md for the recommended nightly cadence
  • trend-tracking workflow.
• Temporal axis follow-on (v0.35.3.1 + v0.35.7): v0.35.3.1 added a six-member verdict enum (no_contradiction | contradiction | temporal_supersession | temporal_regression | temporal_evolution | negation_artifact) and threaded pages.effective_date into the judge prompt so the probe stops crying wolf on legitimate change-over-time. v0.35.7 lands the trajectory substrate the probe pointed at: gbrain eval trajectory <entity> shows the chronological typed-claim history with regressions flagged inline; gbrain founder scorecard <entity> rolls up four signals (accuracy, consistency, growth direction, red flags) into a stable JSON contract. MCP op find_trajectory (read scope, visibility-filtered for remote callers) exposes the same data to agents. The probe's temporal_supersession verdict and the consolidate phase's valid_until writeback both preserve the auto-supersession.ts:4 "NEVER auto-applies" invariant — the probe still emits paste-ready commands, only consolidate writes valid_until (R1+R8 grep guard pins this).