diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d8bd94a7..328de4b71 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,74 @@ All notable changes to GBrain will be documented in this file. +## [0.35.3.1] - 2026-05-15 + +**The contradiction probe stops crying wolf on time. Six-member verdict enum + page-level date in the prompt + a new privacy lint for proposals.** + +`gbrain eval suspected-contradictions` used to treat every claim as timeless. On a brain with conversation transcripts, dated meeting pages, evolving takes, and historical entity records, that meant ~60% of residual HIGH findings were temporal false positives: a status change recorded as "trial" in April and "confirmed" in May got flagged as a contradiction; a role transition recorded in 2017 vs. an updated role in 2025 got flagged. None of those are bugs in the brain; they're features of a brain that records history. + +The judge now sees the page-level `effective_date` for each chunk via a `(from: YYYY-MM-DD)` tag in the prompt, and it classifies into a six-member verdict taxonomy instead of `contradicts: true/false`. The same wave adds a CI privacy lint that catches PII patterns in `docs/proposals/*.md` so future RFCs can't ship with personal-context vocabulary the way this wave's source RFC did at draft time. + +### What you can now do + +**Get verdict-aware findings out of the probe.** The judge now returns one of six verdicts: `no_contradiction`, `contradiction` (genuine conflict at the same point in time), `temporal_supersession` (newer claim updates the older), `temporal_regression` (a metric or status went backwards), `temporal_evolution` (legitimate change over time), or `negation_artifact` (judge misread an explicit negation). `gbrain eval suspected-contradictions trend` shows the per-verdict breakdown in the chart; `gbrain eval suspected-contradictions review` prints a `[verdict]` tag inline with each finding. The Wilson-CI denominator stays anchored to the strict `verdict === 'contradiction'` count so the headline metric is preserved. A new `queries_with_any_finding` sibling metric counts queries that produced any non-`no_contradiction` verdict. + +**Skip the >30d skip rule when both sides have dates.** The pre-judge date filter previously dropped pairs whose chunk-text-extracted dates differed by more than 30 days. That heuristic silently killed exactly the cases the new verdicts exist to surface (the 2017 vs. 2025 role-transition class). The filter now passes those pairs through when both sides have a non-null page-level `effective_date`, so the judge gets to classify them as `temporal_supersession` instead of the pair vanishing into the skip bucket. Same-paragraph dual-date override and missing-date fallback rules are preserved verbatim. + +**Cap the cost of the post-bump re-judge.** The `PROMPT_VERSION` change invalidates the entire judge cache (the prompt shape changed, old verdicts no longer apply). Before that re-judge spends any tokens, the runner prints an upper-bound cost estimate and waits 10 seconds in TTY for Ctrl-C; set `GBRAIN_NO_PROBE_PROMPT=1` to skip or `GBRAIN_PROBE_PROMPT_GRACE_SECONDS=0` to proceed immediately. Non-TTY (autopilot) auto-proceeds with a stderr note. `--budget-usd N` hard-caps the run when cumulative cost exceeds the cap. The judge model now resolves through `models.eval.contradictions_judge` (defaults to Haiku tier), so `gbrain config set models.eval.contradictions_judge` works the same way as every other model config key. `gbrain models doctor` surfaces the new touchpoint. + +**Block PII in `docs/proposals/*.md` at CI time.** A new `scripts/check-proposal-pii.sh` (wired into `bun run verify` and `bun run check:all`) catches the specific PII classes that surfaced in past RFC drafts: private repo references (`garrytan/brain`), personal-relationship vocabulary (`trial separation`, `couples session`, `divorce attorney`, etc.), death-context phrases, and the literal `Wintermute` agent name. Bare common words (`separation`, `funeral`) are intentionally not banned. The denylist names patterns rather than real names, so the script's own source doesn't re-introduce PII into the repo. + +### Itemized changes + +- `SearchResult` interface gains optional `effective_date` + `effective_date_source` fields. Eight SQL projection sites updated (3 in postgres-engine, 5 in pglite-engine — `searchKeyword`, `searchKeywordChunks`, `searchVector` across both engines). `rowToSearchResult` normalizes both Postgres `Date` and PGLite string returns to `YYYY-MM-DD`. Three-state read pattern (undefined when not selected, null when selected-but-empty, string when populated). +- `PairMember` (`src/core/eval-contradictions/types.ts`) gets required `effective_date: string | null` and `effective_date_source: string | null`. `runner.ts` threads the fields through `searchResultToMember` and `takeToMember`; the judge call passes them via the new optional fields on `JudgeInput.a/b`. +- `buildJudgePrompt` now emits `Statement A (from: YYYY-MM-DD)` when `effective_date` is non-null, else `(date unknown)`. Prompt instructions explain the tag and the new verdict semantics. +- `PROMPT_VERSION` bumped `'1' → '2'`. Cache-key tuple shape unchanged (still 5 fields); old rows naturally miss on first run. Cache type guard updated to check `verdict` instead of `contradicts`. +- `JudgeVerdict.contradicts: boolean` replaced with `verdict: Verdict` (6-member union). `Severity` extended with `'info'`. `ResolutionKind` extended with `temporal_supersede`, `flag_for_review`, `log_timeline_change`. `normalizeVerdict` applies the C1 confidence floor only to `verdict === 'contradiction'` (other verdicts are informational classifications, no floor). `defaultSeverityForVerdict` maps each verdict to its baseline severity (D7 map: supersession→info, regression→high, evolution→info, negation_artifact→low, contradiction→medium). +- `runner.ts` emit predicate changes from `if (verdict.contradicts)` to `if (verdict.verdict !== 'no_contradiction')`. Per-verdict tally feeds `ProbeReport.verdict_breakdown`. `queries_with_contradiction` (strict, Wilson-CI denominator) and `queries_with_any_finding` (broad) tracked separately. +- `auto-supersession.ts`: `classifyResolution` and `renderResolutionCommand` extended for new verdicts. Probe still NEVER auto-mutates — new kinds render paste-ready commands (`gbrain takes supersede --since `) or informational lines (`# flag_for_review:`). The `auto-supersession.ts:4` "NEVER auto-applies" invariant preserved verbatim. +- `date-filter.ts`: `shouldSkipForDateMismatch` accepts optional `effectiveDateA` and `effectiveDateB`. When both are non-null, returns `skip=false` with new `'both_have_effective_date'` reason. Other rules preserved. +- `src/commands/eval-suspected-contradictions.ts`: new `--budget-usd N` hard cap (was pre-existing for runtime; help text now explains it), new cost-estimate prompt wired via `src/core/eval-contradictions/cost-prompt.ts`. `--severity` accepts `info`. `--severity` output shows `[verdict]` tag. Judge model routes through `resolveModel({configKey: 'models.eval.contradictions_judge', tier: 'utility', envVar: 'GBRAIN_CONTRADICTIONS_JUDGE_MODEL'})`. Human summary and trend chart show the per-verdict breakdown. +- `src/commands/models.ts` registers `models.eval.contradictions_judge` as a tracked per-task model key so `gbrain models` and `gbrain models doctor` surface it. +- `scripts/check-proposal-pii.sh` + denylist (structural patterns only, no real names) + 15-case test in `test/scripts/check-proposal-pii.test.ts`. Wired into `bun run verify`, `bun run check:all`, and as the new `bun run check:proposal-pii` standalone target. Two privacy-guard test fixtures naming `Wintermute` allowlisted in `scripts/check-test-real-names.sh`; the new privacy-guard scripts themselves allowlisted in `scripts/check-privacy.sh`. +- Six IRON-RULE regression test suites pin the wave's invariants: R1 date-filter rule 3 relaxation preserves rules 1+2 (6 cases), R2 cache invalidation on `PROMPT_VERSION` bump, R3 verdict-enum migration (compile-time via `tsc --noEmit`), R4 runner emit predicate fires for every non-`no_contradiction` verdict (6 cases), R5 cache key tuple stays 5 fields, R6 contradiction severity unchanged (4 cases). Plus 9 cases on the cost-prompt helper. +- The source RFC (`docs/proposals/temporal-contradiction-probe.md`) and PR title/body/commit/branch-name all scrubbed of PII at draft time. The new CI lint prevents the next one from slipping through. + +### Phase 2/3/4 deferred (per the plan) + +The RFC proposed four phases. This wave ships Phase 1 only. Deferred: +- Phase 2 (structured claim extraction + `gbrain trajectory` view): blocked on a separate RFC that maps the existing `extract_facts` + `consolidate` cycle pipeline before extending the facts table. Codex review of the original plan caught factual errors about this pipeline; the deferral is deliberate. +- Phase 3 (auto-write `valid_until` from `temporal_supersession` verdicts): would violate `auto-supersession.ts:4`'s "NEVER auto-applies" invariant. Reframed as paste-ready commands in Phase 1. +- Phase 4 (founder scorecard / Argus integration): needs concrete Argus spec. + +## To take advantage of v0.35.3.1 + +`gbrain upgrade` should do this automatically. The wave is schema-compatible (no migrations), so `gbrain apply-migrations` is a no-op. + +1. **Re-run the contradiction probe to see the new verdicts:** + ```bash + gbrain eval suspected-contradictions run --budget-usd 5 + ``` + Expect a one-time cost-estimate prompt + 10-second Ctrl-C window since `PROMPT_VERSION` changed; the cache is invalidated. Non-TTY runs auto-proceed. +2. **(Optional) Pin the judge model:** + ```bash + gbrain config set models.eval.contradictions_judge anthropic:claude-haiku-4-5 + gbrain models doctor + ``` +3. **(Optional) Adjust the cost-prompt grace window** for CI / autopilot: + ```bash + export GBRAIN_NO_PROBE_PROMPT=1 # skip entirely + # or + export GBRAIN_PROBE_PROMPT_GRACE_SECONDS=0 # auto-proceed in TTY + ``` +4. **If `bun run verify` fails on `check:proposal-pii`,** the lint caught PII in `docs/proposals/*.md`. Replace with generic placeholders (alice-example, acme-corp, fund-a) per CLAUDE.md "Privacy rule: scrub real names from public docs." +5. **If any step fails or the numbers look wrong,** please file an issue: + https://github.com/garrytan/gbrain/issues with: + - output of `gbrain doctor` + - contents of `~/.gbrain/upgrade-errors.jsonl` if it exists + - which step broke + ## [0.35.3.0] - 2026-05-15 **Fix wave: 19 stale community PRs land as one bisect-friendly PR with the architectural fix none of them surfaced.** @@ -46,6 +114,7 @@ Two real bugs, both in shipping code on master since v0.28+, both fixed at the a 4. If you use remote-source git clones (`gbrain config get sources` shows any with a remote URL), they'll start working again on the next `gbrain sync`. Pre-fix, every clone of a remote-source repository was silently failing with git exit 129. If `gbrain doctor` flags anything after upgrade, please file an issue with the doctor output. This was a 7-month silent bug — the doctor's structural guards should catch the next one before it ships. + ## [0.35.1.1] - 2026-05-16 **Fix wave: `gbrain eval longmemeval` actually runs against the public _s split.** diff --git a/VERSION b/VERSION index 1de48e265..80bac5fbd 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.35.3.0 \ No newline at end of file +0.35.3.1 diff --git a/docs/proposals/temporal-contradiction-probe.md b/docs/proposals/temporal-contradiction-probe.md new file mode 100644 index 000000000..f4bbcf2a8 --- /dev/null +++ b/docs/proposals/temporal-contradiction-probe.md @@ -0,0 +1,213 @@ +# Proposal: Temporal Axis for Contradiction Probe + +**Status:** Report / RFC +**Date:** 2026-05-14 +**Context:** A large production run of `gbrain eval suspected-contradictions` surfaced ~115 HIGH findings. Walking through them by hand exposed a structural limitation in the probe. + +## The Problem + +The contradiction probe (`gbrain eval suspected-contradictions`) treats all claims as timeless. When two chunks make conflicting statements, the judge flags a contradiction regardless of whether both statements were true at their respective points in time. + +This worked fine when the brain was mostly static wiki pages. It breaks now that the brain contains: +- Conversation transcripts with claims that were true when spoken +- Meeting pages capturing what people said on specific dates +- Takes that evolve (a founder's ARR claim in January vs. July) +- Status records that supersede each other (a state moves from "trial" to "confirmed") + +The probe can't distinguish "this changed" from "this is wrong." + +## Bug-class examples (synthetic placeholders) + +### 1. Temporal Evolution (False Positive) + +``` +Finding: HIGH + A: [daily/transcripts/2026/2026-04-28] "status: trial" + B: [meetings/2026-05-07-session] "status: confirmed" + Axis: Whether status is trial or confirmed +``` + +Both are correct as of their respective dates. April 28: trial. May 7: confirmed. The probe flags this because it has no concept of "this claim was valid from X until Y." The May 7 record didn't make the April 28 transcript wrong; it recorded a change. + +### 2. Negation Parsing (False Positive) + +``` +Finding: HIGH + A: [people/alice-example] "person traveled to city-a for alice-example's event — NOT bob-example's event" + B: [meetings/2026-05-11-context] mentions of bob-example's event in city-b + Axis: Whose event the city-a trip was for +``` + +The disambiguation fact contains "NOT bob-example's event" as an explicit negation. The judge reads "bob-example's event" as a positive claim and flags it against the alice-example context. The data is correct; the probe can't parse negation. + +### 3. Role Changes (True Positive That Needs Time Awareness) + +``` +Finding: HIGH + A: [sources/notes/2017-03-28] advisor-example: "Partner, venture-firm-a" + B: [people/advisor-example] advisor-example: "Senior Policy Advisor, gov-org-b" +``` + +Both true at their respective times. 2017: partner at venture-firm-a. 2025: gov-org-b advisor. The current probe correctly flags this as a contradiction, but the resolution should be "superseded by time" not "one side is wrong." The 2017 note isn't wrong; it's a historical record. + +## Scenario #1: Founder Tracking (the big one) + +This is the use case that makes a time axis transformative rather than incremental. + +The brain holds hundreds of company pages and thousands of meeting pages. Founders make claims: + +- "We're at $50K MRR" (January OH) +- "We hit $200K MRR" (April OH) +- "We're at $150K MRR" (July OH — what happened?) + +Today the probe would flag January vs. April as a contradiction. The real signal is April vs. July: **a claimed metric went backwards.** That's not a data quality issue; that's intelligence. + +What a time-aware probe could surface: + +**Claim trajectory tracking:** +``` +Company: Acme Corp + 2026-01: "$50K MRR" (source: OH transcript) + 2026-04: "$200K MRR" (source: OH transcript) + 2026-07: "$150K MRR" (source: OH transcript) ← REGRESSION DETECTED + 2026-07: "$2M ARR" (source: investor update) ← INCONSISTENT WITH MRR +``` + +**Prediction vs. outcome:** +``` +Founder: Jane Doe (Acme Corp) + 2026-01: "We'll hit $1M ARR by June" (source: batch kickoff) + 2026-06: Actual ARR: $400K (source: investor update) + → Prediction accuracy: 40% + → Pattern: consistently 2-3x optimistic on timeline +``` + +**Narrative consistency:** +``` +Founder: John Smith (WidgetCo) + 2026-01: "Our moat is proprietary data" (source: interview) + 2026-03: "We're pivoting to an API-first model" (source: OH) + 2026-06: "Our moat is network effects" (source: Demo Day) + → Moat narrative changed 3x in 6 months — flag for review +``` + +This isn't adversarial. It's the kind of pattern an experienced operator notices intuitively across hundreds of conversations. GBrain can make it systematic. + +## Scenario #2: Event Disambiguation + +Two distinct events within a short window can conflate during ingestion because the probe has no temporal frame to say "event A is a different event from event B." + +Time-aware facts would store (synthetic placeholders): +``` +fact: "alice-example milestone" valid_from: 2026-04-15 valid_until: 2026-04-15 +fact: "alice-example event in city-a" valid_from: 2026-04-17 valid_until: 2026-04-19 +fact: "bob-example milestone" valid_from: 2026-05-04 valid_until: 2026-05-04 +fact: "bob-example event in city-b" valid_from: 2026-05-12 valid_until: 2026-05-12 +``` + +The probe should recognize these as two distinct events with non-overlapping time windows, not as contradictions about "whose event." + +## Scenario #3: Role and Status Changes + +People change roles. Companies change status. The brain records history. Synthetic examples representative of the cases observed in production: + +- advisor-example: venture-firm-a partner (2019) → gov-org-b advisor (2025) +- investor-example: fund-a partner → fund-b CEO (2023) +- agent-fork: provider restriction event (2026-04-04) ≠ shutdown +- fund-c: "interesting fund" (early) → "declined" (later) → "losing confidence" (latest) + +All of these are correct historical records. The probe should classify them as **temporal supersession** rather than **contradiction.** + +## Scenario #4: Decision Tracking + +Multi-step decisions that supersede earlier framings example (synthetic): +``` +2026-04-24: "status: trial" (initial framing) +2026-04-25: "status: in progress" (confirmed, no longer "trial") +2026-05-07: "status: finalized" (session record) +2026-05-11: follow-up actions taken +``` + +Each step supersedes the previous. A time-aware probe would show the **evolution chain** rather than flagging each pair as a contradiction. + +## What Exists Today + +The probe already has some temporal infrastructure: + +1. **`date-filter.ts`** — `shouldSkipForDateMismatch()` pre-filters pairs, but only checks whether dates are "too far apart" (a coarse heuristic). It doesn't reason about which claim is newer or whether one supersedes the other. + +2. **`auto-supersession.ts`** — proposes resolution commands, checks `since_date` on takes. But this is post-hoc (after the judge flags a contradiction). The judge itself doesn't see dates. + +3. **Facts table** has `valid_from` and `valid_until` columns. These exist but are sparsely populated and not used by the probe. + +4. **Takes table** has `since_date`. Also sparsely populated. + +## What Would Need to Change + +### Phase 1: Judge prompt enhancement (smallest change, biggest impact) + +Pass the source dates to the judge. The current judge prompt shows two text chunks and asks "are these contradictory?" If it also showed: + +``` +Statement A (from: 2026-04-28): + "status: trial" + +Statement B (from: 2026-05-07): + "status: confirmed" +``` + +The judge could output a `temporal_supersession` verdict instead of `contradiction`. New verdict taxonomy: + +- `no_contradiction` — statements are compatible +- `contradiction` — genuinely conflicting claims at the same point in time +- `temporal_supersession` — newer claim updates/replaces older claim (not an error) +- `temporal_regression` — a metric or status went backwards (potential signal) +- `temporal_evolution` — legitimate change over time, neither supersession nor regression +- `negation_artifact` — one side contains an explicit negation the judge misread + +### Phase 2: Claim trajectory view (new command) + +```bash +gbrain eval trajectory "Acme Corp MRR" +gbrain eval trajectory "advisor-example role" +gbrain eval trajectory "deal-x status" +``` + +Pull all time-stamped claims about an entity+attribute, sort chronologically, detect: +- Regressions (metric went down) +- Contradictions within the same time window +- Prediction vs. outcome gaps +- Narrative drift (moat story changed 3x) + +### Phase 3: Automatic `valid_from`/`valid_until` population + +During `extract_facts`, infer temporal bounds from source context: +- Meeting page dated 2026-04-28 → claims valid_from 2026-04-28 +- Takes from transcripts → valid_from = transcript date +- Imported notes → valid_from = note date +- Entity pages with no date → valid_from = page created date (weakest signal) + +### Phase 4: Founder scorecard + +For founders specifically, a temporal probe could generate: +- **Claim accuracy score** — what they predicted vs. what happened +- **Consistency score** — how stable their narrative is over time +- **Growth trajectory** — whether the numbers are actually moving +- **Red flag detector** — metrics going backwards, story changing, timeline slipping + +## Recommendation + +Start with Phase 1. The judge prompt change is small. It immediately eliminates the temporal false positives (which were a majority of the residual HIGH findings in the production audit) and gives the probe a new vocabulary for time-aware reasoning. + +Phase 2 (trajectory view) is the one that would change how operators use the brain for founder evaluation. Worth scoping as a standalone feature. + +Phases 3–4 are downstream and can wait. + +## Appendix: Production probe stats (2026-05-14) + +- ~107K pages, ~257K chunks +- Previous run: ~115 HIGH findings across 50 queries +- After manual resolution: ~25 residual findings +- Of those ~25: roughly two-thirds temporal false positives, the remainder probe artifacts (self-contradiction, negation parsing) +- 0 genuine data contradictions remained on the queries tested +- Fresh targeted probe on a representative entity-role query: 0 contradictions (was 14+ before fixes) diff --git a/package.json b/package.json index 470821e09..c0595c451 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gbrain", - "version": "0.35.3.0", + "version": "0.35.3.1", "description": "Postgres-native personal knowledge brain with hybrid RAG search", "type": "module", "main": "src/core/index.ts", @@ -37,11 +37,11 @@ "build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts", "test": "bash scripts/run-unit-parallel.sh", "test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)", - "verify": "bun run check:privacy && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run typecheck", + "verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run typecheck", "check:system-of-record": "scripts/check-system-of-record.sh", "check:admin-scope-drift": "scripts/check-admin-scope-drift.sh", "check:cli-exec": "scripts/check-cli-executable.sh", - "check:all": "scripts/check-privacy.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh", + "check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh", "check:wasm": "scripts/check-wasm-embedded.sh", "check:newlines": "scripts/check-trailing-newline.sh", "test:e2e": "bash scripts/run-e2e.sh", @@ -55,6 +55,7 @@ "check:jsonb": "scripts/check-jsonb-pattern.sh", "check:source-id-projection": "scripts/check-source-id-projection.sh", "check:privacy": "scripts/check-privacy.sh", + "check:proposal-pii": "scripts/check-proposal-pii.sh", "check:eval-glossary": "scripts/check-eval-glossary-fresh.sh", "check:test-names": "scripts/check-test-real-names.sh", "check:progress": "scripts/check-progress-to-stdout.sh", diff --git a/scripts/check-privacy.sh b/scripts/check-privacy.sh index 8901461a5..cbef84516 100755 --- a/scripts/check-privacy.sh +++ b/scripts/check-privacy.sh @@ -133,6 +133,12 @@ ALLOW_LIST=( # (Wintermute, Hermes, etc) inside its BANNED_NAMES + ALLOWLIST arrays. # Same meta-rule-enforcement exception as scripts/check-privacy.sh itself. 'scripts/check-test-real-names.sh' + # v0.34 / Lane CI: scripts/check-proposal-pii.sh and its test list the + # banned literal as part of the structural denylist they enforce against + # docs/proposals/*.md. Same meta-rule-enforcement exception as the two + # entries above — describing what the rule forbids requires naming it. + 'scripts/check-proposal-pii.sh' + 'test/scripts/check-proposal-pii.test.ts' # v0.32.3.0: the functional-area-resolver skill's behavior-contract # section describes the privacy guarantees the skill preserves and # references the banned literals while doing so (line 306). Same diff --git a/scripts/check-proposal-pii.sh b/scripts/check-proposal-pii.sh new file mode 100755 index 000000000..a2a8c997d --- /dev/null +++ b/scripts/check-proposal-pii.sh @@ -0,0 +1,166 @@ +#!/bin/bash +# +# check-proposal-pii.sh — privacy guard for `docs/proposals/*.md`. +# +# Sibling to check-privacy.sh: that script bans the `Wintermute` literal +# everywhere. This one focuses on `docs/proposals/*.md` and the OTHER PII +# classes that have surfaced in past RFC drafts — personal-relationship +# vocabulary, private repo references, etc. +# +# Why two scripts: the patterns this lint flags would be too noisy if +# applied repo-wide (e.g. a test fixture mentioning "trial" is fine). +# Restricting to `docs/proposals/` keeps the lint surgical — proposals are +# public-facing RFC documents that should never contain personal context, +# so the false-positive rate is near zero. +# +# Design note: the denylist names PATTERNS, not real people. Specific +# real names (deceased relatives, therapist names, dealflow contacts) +# would leak PII into the repo just by appearing in this script's +# denylist. The structural patterns below catch the SURROUNDING context +# of personal-event prose. The trade-off: a future RFC that names a real +# person without any of the contextual markers won't be caught — that's +# accepted as a residual risk handled by human review. +# +# Usage: +# scripts/check-proposal-pii.sh # scan working tree +# scripts/check-proposal-pii.sh --staged # scan git staged index +# scripts/check-proposal-pii.sh --help +# +# Exit codes: +# 0 clean +# 1 PII pattern found +# 2 setup error + +set -euo pipefail + +REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)" +PROPOSALS_DIR="$REPO_ROOT/docs/proposals" + +# Structural patterns. One per line. Matched case-insensitively, fixed-string +# (no regex). Comments start with #. Blank lines OK. +# +# IMPORTANT — design contract: this list MUST NOT contain real personal +# names (deceased relatives, therapist first names, dealflow contacts). +# Naming those would leak PII into scripts/. The patterns below catch the +# SURROUNDING VOCABULARY that always accompanies such content in personal +# RFC prose. Maintainers extending this list: prefer adding a phrase that +# captures the context (e.g. `couples session`) rather than a specific +# person's name. +read -r -d '' PATTERNS <<'EOF' || true +# Private repo references (zero false-positive risk) +garrytan/brain + +# Personal relationship vocabulary (extremely unlikely in technical RFCs) +trial separation +permanent separation +couples session +couples therapist +divorce attorney +divorce attorneys + +# Death/funeral vocabulary in personal contexts (combined phrases — bare +# "funeral" alone would false-positive in legitimate metaphorical use) +grandmother's funeral +grandmother funeral +aunt's funeral +aunt funeral + +# Private agent / fork name (also enforced repo-wide by check-privacy.sh +# but listed here for proposal-scoped clarity) +wintermute +EOF + +usage() { + cat <&2 + usage >&2 + exit 2 + ;; + esac +done + +if [ ! -d "$PROPOSALS_DIR" ]; then + # No proposals dir yet — nothing to lint. Not a failure. + exit 0 +fi + +# Build the file list. Staged mode filters git's staged set down to +# docs/proposals/*.md; working mode globs the directory directly. +if [ "$MODE" = staged ]; then + if ! command -v git >/dev/null 2>&1; then + echo "check-proposal-pii: git not found" >&2 + exit 2 + fi + FILES=$(git diff --cached --name-only --diff-filter=ACMR 2>/dev/null \ + | grep -E '^docs/proposals/.+\.md$' || true) +else + FILES=$(find "$PROPOSALS_DIR" -maxdepth 1 -type f -name '*.md' 2>/dev/null \ + | sed "s|^$REPO_ROOT/||") +fi + +if [ -z "$FILES" ]; then + exit 0 +fi + +FOUND=0 +# Iterate patterns; for each non-comment line, scan the file list. +while IFS= read -r raw_line; do + # Strip leading/trailing whitespace. + pat="${raw_line#"${raw_line%%[![:space:]]*}"}" + pat="${pat%"${pat##*[![:space:]]}"}" + # Skip empty and comment lines. + [ -z "$pat" ] && continue + case "$pat" in '#'*) continue ;; esac + + while IFS= read -r file; do + [ -z "$file" ] && continue + full="$REPO_ROOT/$file" + [ ! -f "$full" ] && continue + # Fixed-string (-F), case-insensitive (-i), with line numbers (-n). + if matches=$(grep -nFi -- "$pat" "$full" 2>/dev/null); then + if [ -n "$matches" ]; then + echo "[check-proposal-pii] PII pattern in $file:" >&2 + echo " pattern: $pat" >&2 + echo "$matches" | sed 's|^| |' >&2 + FOUND=$((FOUND + 1)) + fi + fi + done <<< "$FILES" +done <<< "$PATTERNS" + +if [ "$FOUND" -gt 0 ]; then + echo "" >&2 + echo "[check-proposal-pii] $FOUND PII pattern hit(s) in docs/proposals/*.md." >&2 + echo "[check-proposal-pii] See CLAUDE.md 'Privacy rule: scrub real names from public docs'." >&2 + echo "[check-proposal-pii] Use generic placeholders: alice-example, acme-corp, fund-a, etc." >&2 + exit 1 +fi + +exit 0 diff --git a/scripts/check-test-real-names.sh b/scripts/check-test-real-names.sh index aaffdcdf6..64c291a27 100755 --- a/scripts/check-test-real-names.sh +++ b/scripts/check-test-real-names.sh @@ -55,6 +55,8 @@ ALLOWLIST=( "test/writer.test.ts:garry@ycombinator.com" # user's own email — CLAUDE.md rule does not apply "test/integrations.test.ts:Wintermute" # regex pattern in personal-info filter test (structural) "test/recency-decay.test.ts:Wintermute" # regression-prevention test asserting wintermute is absent (structural) + "test/scripts/check-proposal-pii.test.ts:Wintermute" # privacy-guard test asserting docs/proposals/ rejects wintermute (structural; same meta-rule exception as check-privacy.sh) + "test/scripts/check-proposal-pii.test.ts:WINTERMUTE" # case-insensitive sentinel literal for the same privacy-guard test "test/serve-stdio-lifecycle.test.ts:Hermes" # comment naming a downstream-agent scenario — pre-existing, low signal "test/extract.test.ts:Hermes" # markdown-link extraction test fixture — pre-existing, ambiguous (Greek god vs fork) ) diff --git a/src/commands/eval-suspected-contradictions.ts b/src/commands/eval-suspected-contradictions.ts index 2b7578e60..39761e68f 100644 --- a/src/commands/eval-suspected-contradictions.ts +++ b/src/commands/eval-suspected-contradictions.ts @@ -19,6 +19,7 @@ import { readFileSync } from 'node:fs'; import type { BrainEngine } from '../core/engine.ts'; +import { resolveModel } from '../core/model-config.ts'; import { PreFlightBudgetError, runContradictionProbe, @@ -28,6 +29,7 @@ import { bucketBySeverity, compareSeverityDesc, } from '../core/eval-contradictions/severity-classify.ts'; +import { maybePromptForCostBeforeProbe } from '../core/eval-contradictions/cost-prompt.ts'; import type { ContradictionFinding, Severity, @@ -40,7 +42,13 @@ interface ParsedFlags { query?: string; fromCapture?: boolean; topK: number; - judge: string; + /** + * v0.34 / Lane C: now optional. When unset, runRun routes through + * resolveModel({configKey: 'models.eval.contradictions_judge', tier: 'utility'}) + * so the user's config keys + tier defaults govern. When set, the CLI flag + * wins per resolveModel's 6-tier precedence chain. + */ + judge?: string; limit?: number; budgetUsd: number; output?: string; @@ -77,7 +85,8 @@ function parseFlags(args: string[]): ParsedFlags { const f: ParsedFlags = { sub, topK: 5, - judge: 'anthropic:claude-haiku-4-5', + // judge intentionally undefined here — resolved in runRun via resolveModel + // so config keys + tier defaults govern. CLI --judge flag wins when set. budgetUsd: isTty ? 5 : 1, maxPairChars: 1500, sampling: 'deterministic', @@ -119,8 +128,9 @@ function parseFlags(args: string[]): ParsedFlags { else if (arg === '--days') f.days = Number.parseInt(next(), 10); else if (arg === '--severity') { const v = next(); - if (v !== 'low' && v !== 'medium' && v !== 'high') { - throw new Error('--severity must be low|medium|high'); + // v0.34 / Lane A2: 'info' joins the rank as a valid severity. + if (v !== 'info' && v !== 'low' && v !== 'medium' && v !== 'high') { + throw new Error('--severity must be info|low|medium|high'); } f.severity = v; } @@ -136,7 +146,9 @@ function printHelp(): void { console.error(`Usage: gbrain eval suspected-contradictions [run] [--queries-file FILE.jsonl | --query "..." | --from-capture] - [--top-k N=5] [--judge MODEL=claude-haiku-4-5] + [--top-k N=5] [--judge MODEL] (default routes via resolveModel → + models.eval.contradictions_judge → + utility-tier (Haiku) fallback) [--limit N] [--budget-usd N] [--output FILE] [--max-pair-chars N=1500] [--sampling deterministic|score-first] [--no-cache] [--refresh-cache] [--json] [--yes] @@ -144,11 +156,19 @@ function printHelp(): void { gbrain eval suspected-contradictions trend [--days N=30] [--json] gbrain eval suspected-contradictions review - [--severity low|medium|high] [--since YYYY-MM-DD] + [--severity info|low|medium|high] [--since YYYY-MM-DD] -The probe samples top-K retrieval pairs and asks an LLM judge whether -any pair contradicts on a factual claim relevant to the query. Outputs -JSON (stable schema_version: 1) and a human summary. +The probe samples top-K retrieval pairs and asks an LLM judge to classify +each pair as one of: no_contradiction, contradiction, temporal_supersession, +temporal_regression, temporal_evolution, negation_artifact. Outputs JSON +(stable schema_version: 1) and a human summary with per-verdict breakdown. + +Cost guardrails: + - When PROMPT_VERSION changes, the runner prints a cost estimate and waits + 10s in TTY for Ctrl-C (auto-proceeds non-TTY). Skip with GBRAIN_NO_PROBE_PROMPT=1. + - --budget-usd N halts the run when cumulative cost exceeds the cap. + - --judge MODEL overrides the resolveModel chain; pair with --yes when + automating. `); } @@ -221,10 +241,36 @@ async function runRun(engine: BrainEngine, f: ParsedFlags): Promise { process.exit(1); } + // v0.34 / Lane C: route the judge model through resolveModel so the user's + // models.eval.contradictions_judge config key + Haiku-tier default + global + // models.default override + env var all compose correctly. The --judge CLI + // flag still wins as the highest-precedence override. + const judgeModel = await resolveModel(engine, { + cliFlag: f.judge, + configKey: 'models.eval.contradictions_judge', + tier: 'utility', + envVar: 'GBRAIN_CONTRADICTIONS_JUDGE_MODEL', + fallback: 'anthropic:claude-haiku-4-5', + }); + console.error( - `Contradiction probe: ${queries.length} queries, top-${f.topK}, judge=${f.judge}, budget=$${f.budgetUsd.toFixed(2)}.`, + `Contradiction probe: ${queries.length} queries, top-${f.topK}, judge=${judgeModel}, budget=$${f.budgetUsd.toFixed(2)}.`, ); + // v0.34 / Lane C: cost-estimate prompt — TTY-only Ctrl-C window before + // the runner spends any tokens when PROMPT_VERSION changed since the last + // persisted run. Non-TTY and env-skip paths auto-proceed. + const promptResult = await maybePromptForCostBeforeProbe({ + engine, + queryCount: queries.length, + topK: f.topK, + judgeModel, + yesOverride: f.yes, + }); + if (promptResult.kind === 'abort') { + process.exit(0); // intentional Ctrl-C — not an error + } + // Refresh-cache: sweep before run so the cache misses on this pass. if (f.refreshCache) { const swept = await engine.sweepContradictionCache(); @@ -235,7 +281,7 @@ async function runRun(engine: BrainEngine, f: ParsedFlags): Promise { const out = await runContradictionProbe({ engine, queries, - judgeModel: f.judge, + judgeModel, topK: f.topK, sampling: f.sampling, budgetUsd: f.budgetUsd, @@ -254,11 +300,23 @@ async function runRun(engine: BrainEngine, f: ParsedFlags): Promise { lines.push(``); lines.push(`Results: ${r.queries_evaluated} queries, top-${r.top_k} each, judge=${r.judge_model}`); lines.push(` Queries with >=1 contradiction: ${r.queries_with_contradiction} / ${r.queries_evaluated} (${pct(r.queries_with_contradiction / Math.max(1, r.queries_evaluated))}%)`); + // v0.34 / Lane A2: broader finding count alongside the strict contradiction count. + lines.push(` Queries with >=1 finding (any verdict): ${r.queries_with_any_finding} / ${r.queries_evaluated}`); lines.push(` Wilson CI 95%: ${pct(r.calibration.wilson_ci_95.lower)}–${pct(r.calibration.wilson_ci_95.upper)}%`); if (r.calibration.small_sample_note) { lines.push(` Note: ${r.calibration.small_sample_note}`); } - lines.push(` Total contradictions flagged: ${r.total_contradictions_flagged}`); + lines.push(` Total findings flagged: ${r.total_contradictions_flagged}`); + // v0.34 / Lane A2: per-verdict breakdown surfaces what kinds of finding + // dominated the run — distinguishes temporal noise from genuine conflicts. + const vb = r.verdict_breakdown; + lines.push(` Verdict breakdown:`); + lines.push(` contradiction: ${vb.contradiction}`); + lines.push(` temporal_supersession: ${vb.temporal_supersession}`); + lines.push(` temporal_regression: ${vb.temporal_regression}`); + lines.push(` temporal_evolution: ${vb.temporal_evolution}`); + lines.push(` negation_artifact: ${vb.negation_artifact}`); + lines.push(` no_contradiction: ${vb.no_contradiction}`); lines.push(` Judge errors: ${r.judge_errors.total} (parse_fail=${r.judge_errors.parse_fail} timeout=${r.judge_errors.timeout} http_5xx=${r.judge_errors.http_5xx} refusal=${r.judge_errors.refusal})`); lines.push(` Cache: ${r.cache.hits} hits / ${r.cache.misses} misses (${pct(r.cache.hit_rate)}% hit-rate)`); lines.push(` Source-tier breakdown:`); @@ -331,12 +389,16 @@ async function runReview(engine: BrainEngine, f: ParsedFlags): Promise { } filtered.sort((a, b) => compareSeverityDesc(a.severity, b.severity)); const buckets = bucketBySeverity(filtered); - for (const sev of ['high', 'medium', 'low'] as const) { + // v0.34 / Lane A2: 'info' is the lowest severity bucket; iterate + // high → medium → low → info so the report lands worst-first. + for (const sev of ['high', 'medium', 'low', 'info'] as const) { const items = buckets[sev]; if (items.length === 0) continue; console.error(`\n${sev.toUpperCase()} severity (${items.length}):`); for (const item of items) { - console.error(` - ${item.a.slug} vs ${item.b.slug}`); + // v0.34 / Lane A2: include verdict so the operator distinguishes + // genuine contradictions from temporal classifications at a glance. + console.error(` - [${item.verdict}] ${item.a.slug} vs ${item.b.slug}`); if (item.axis) console.error(` axis: ${item.axis}`); console.error(` → ${item.resolution_command}`); } diff --git a/src/commands/models.ts b/src/commands/models.ts index 9a8ba084c..ca5a447e7 100644 --- a/src/commands/models.ts +++ b/src/commands/models.ts @@ -47,6 +47,7 @@ const PER_TASK_KEYS: Array<{ key: string; tier: ModelTier; description: string } { key: 'models.subagent', tier: 'subagent', description: '`gbrain agent run` subagent loop' }, { key: 'facts.extraction_model', tier: 'reasoning', description: 'Real-time facts extraction during sync' }, { key: 'models.eval.longmemeval', tier: 'reasoning', description: 'LongMemEval benchmark answer-gen' }, + { key: 'models.eval.contradictions_judge', tier: 'utility', description: 'Contradiction probe judge (v0.34 temporal-aware)' }, { key: 'models.expansion', tier: 'utility', description: 'Query expansion for hybrid search' }, { key: 'models.chat', tier: 'reasoning', description: 'Default `gateway.chat()` model' }, ]; diff --git a/src/core/eval-contradictions/auto-supersession.ts b/src/core/eval-contradictions/auto-supersession.ts index 8dd275c26..08ac88756 100644 --- a/src/core/eval-contradictions/auto-supersession.ts +++ b/src/core/eval-contradictions/auto-supersession.ts @@ -28,6 +28,7 @@ import type { ContradictionPair, JudgeVerdict, ResolutionKind, + Verdict, } from './types.ts'; export interface ResolutionProposal { @@ -46,28 +47,54 @@ function isCuratedEntitySlug(slug: string): boolean { * wins for cross_slug pairs because it has semantic context this rule-based * pass doesn't. For intra_page pairs we trust the structural heuristic since * the judge can't see take_id metadata directly. + * + * v0.34 / Lane A2: extended with verdict-aware mapping. The new verdicts + * (temporal_supersession, temporal_regression, temporal_evolution, + * negation_artifact) have their own resolution_kinds — the v1 four-kind + * mapping only applies to `verdict === 'contradiction'`. The probe still + * NEVER auto-mutates; the new kinds render paste-ready commands or + * informational lines just like the old ones. */ export function classifyResolution( pair: ContradictionPair, judgeHint: ResolutionKind | null, + verdict: Verdict = 'contradiction', ): ResolutionKind { + // Verdict-driven routing for the new (non-contradiction) verdicts. The + // judge hint can still override when it specifies something compatible + // with the new verdict's scope. + if (verdict === 'temporal_supersession') { + return judgeHint === 'flag_for_review' || judgeHint === 'log_timeline_change' + ? judgeHint + : 'temporal_supersede'; + } + if (verdict === 'temporal_regression') { + // Always flag — no auto-mutation, just surface to the user. A future + // founder-scorecard surface (Phase 4) consumes the flagged set. + return 'flag_for_review'; + } + if (verdict === 'temporal_evolution') { + return judgeHint === 'flag_for_review' ? 'flag_for_review' : 'log_timeline_change'; + } + if (verdict === 'negation_artifact') { + // Informational — the data is correct; the judge misread a negation. + // Operator action is to wait for the next prompt_version bump. + return 'flag_for_review'; + } + // verdict === 'contradiction' (or no_contradiction, which shouldn't reach + // this fn — runner filters before calling pairToFinding) falls through to + // the v1 mapping below. if (pair.kind === 'intra_page_chunk_take') { - // One side is a take (b, by convention in the runner). If the take is - // active and the chunk text is older, supersede makes sense. We default - // to takes_supersede; if context is ambiguous the user can pick manual. if (pair.b.take_id !== null) return 'takes_supersede'; if (pair.a.take_id !== null) return 'takes_supersede'; return 'manual_review'; } - // cross_slug: judge hint wins if it's specific. if (judgeHint === 'dream_synthesize' || judgeHint === 'takes_mark_debate') { return judgeHint; } if (judgeHint === 'takes_supersede' || judgeHint === 'manual_review') { return judgeHint; } - // Structural fallback: if either side is a curated entity page, propose - // a synthesize run on the curated slug to reconcile. if (isCuratedEntitySlug(pair.a.slug) || isCuratedEntitySlug(pair.b.slug)) { return 'dream_synthesize'; } @@ -100,6 +127,37 @@ export function renderResolutionCommand( const takeId = takeSide.take_id ?? ''; return `gbrain takes mark-debate ${takeSide.slug} --row ${takeId}`; } + case 'temporal_supersede': { + // v0.34 / Lane A2: pick the newer-dated side as the survivor; render a + // supersede command on the older-dated side. If both sides have takes + // we prefer the take that's NOT on the newer page. Falls back to a + // hint with both slugs when the dates can't be ordered. + const aDate = pair.a.effective_date; + const bDate = pair.b.effective_date; + if (aDate && bDate) { + const olderSide = aDate < bDate ? pair.a : pair.b; + const newerDate = aDate < bDate ? bDate : aDate; + const olderTakeId = olderSide.take_id; + if (olderTakeId !== null) { + return `gbrain takes supersede ${olderSide.slug} --row ${olderTakeId} --since ${newerDate}`; + } + return `# temporal_supersession: ${olderSide.slug} (${aDate < bDate ? aDate : bDate}) superseded by ${newerDate}`; + } + return `# temporal_supersession: ${pair.a.slug} vs ${pair.b.slug} (date order unclear)`; + } + case 'log_timeline_change': { + // v0.34 / Lane A2: timeline-writer subcommand is deferred (see plan + // TODOs). Render a hint pointing at the future command shape so the + // operator can paste a follow-up note manually for now. + const aDate = pair.a.effective_date ?? ''; + const bDate = pair.b.effective_date ?? ''; + return `# temporal_evolution: ${pair.a.slug} (${aDate}) → ${pair.b.slug} (${bDate}); record in timeline when the gbrain timeline writer lands`; + } + case 'flag_for_review': { + // v0.34 / Lane A2: informational; covers temporal_regression and + // negation_artifact. The flag itself surfaces in trends; no command. + return `# flag_for_review: ${pair.a.slug} vs ${pair.b.slug}`; + } case 'manual_review': default: return `# manual review: ${pair.a.slug} vs ${pair.b.slug}`; @@ -110,8 +168,9 @@ export function renderResolutionCommand( export function proposeResolution( pair: ContradictionPair, judgeHint: ResolutionKind | null, + verdict: Verdict = 'contradiction', ): ResolutionProposal { - const kind = classifyResolution(pair, judgeHint); + const kind = classifyResolution(pair, judgeHint, verdict); return { resolution_kind: kind, resolution_command: renderResolutionCommand(pair, kind), @@ -120,16 +179,20 @@ export function proposeResolution( /** * Promote a ContradictionPair + JudgeVerdict to a ContradictionFinding by - * filling in severity/axis/confidence + resolution proposal. Used by the - * runner aggregation pass. + * filling in verdict/severity/axis/confidence + resolution proposal. Used + * by the runner aggregation pass. + * + * v0.34 / Lane A2: threads the new `verdict: Verdict` enum into the finding + * shape and the resolution classifier so the new verdicts route correctly. */ export function pairToFinding( pair: ContradictionPair, verdict: JudgeVerdict, ): ContradictionFinding { - const prop = proposeResolution(pair, verdict.resolution_kind); + const prop = proposeResolution(pair, verdict.resolution_kind, verdict.verdict); return { ...pair, + verdict: verdict.verdict, severity: verdict.severity, axis: verdict.axis, confidence: verdict.confidence, diff --git a/src/core/eval-contradictions/cache.ts b/src/core/eval-contradictions/cache.ts index ebffa63a6..e7620828b 100644 --- a/src/core/eval-contradictions/cache.ts +++ b/src/core/eval-contradictions/cache.ts @@ -62,8 +62,12 @@ export function buildCacheKey(opts: { function isJudgeVerdict(raw: unknown): raw is JudgeVerdict { if (!raw || typeof raw !== 'object') return false; const v = raw as Record; + // v0.34 / Lane A2: shape check matches the new `verdict: Verdict` enum. + // Old v1-shaped rows (with `contradicts: boolean` instead) fail this guard + // and get treated as cache misses — same effect as the PROMPT_VERSION '1' + // rows already being filtered out by the cache-key tuple, but doubly safe. return ( - typeof v.contradicts === 'boolean' && + typeof v.verdict === 'string' && typeof v.severity === 'string' && typeof v.confidence === 'number' && typeof v.axis === 'string' diff --git a/src/core/eval-contradictions/cost-prompt.ts b/src/core/eval-contradictions/cost-prompt.ts new file mode 100644 index 000000000..de47a3881 --- /dev/null +++ b/src/core/eval-contradictions/cost-prompt.ts @@ -0,0 +1,133 @@ +/** + * eval-contradictions/cost-prompt — Lane C cost-estimate prompt. + * + * Fires before the probe runs when PROMPT_VERSION has changed since the most + * recent persisted run (so the operator sees the one-time re-judge cost up + * front instead of being surprised by it inside an autopilot cycle). Pattern + * mirrors `runPostUpgradeReembedPrompt` in post-upgrade-reembed.ts: + * + * - TTY-only Ctrl-C window (default 10s; override via + * GBRAIN_PROBE_PROMPT_GRACE_SECONDS). + * - Non-TTY auto-proceeds with a stderr note (autopilot path). + * - GBRAIN_NO_PROBE_PROMPT=1 skips entirely. + * + * Independent of the runner's `--budget-usd` hard cap: this prompt informs; + * the cap enforces. Both layers compose — operator sees the estimate, then + * the runner halts mid-run if the live cost exceeds the cap. + */ + +import type { BrainEngine } from '../engine.ts'; +import { estimateUpperBoundCost } from './cost-tracker.ts'; +import { PROMPT_VERSION } from './types.ts'; + +export interface CostPromptOpts { + engine: BrainEngine; + queryCount: number; + topK: number; + judgeModel: string; + /** True when the user passed --yes; skip the prompt entirely. */ + yesOverride?: boolean; + /** Override TTY detection for testing. */ + isTtyOverride?: boolean; + /** Override stderr writer for testing. */ + stderrWriter?: (text: string) => void; + /** Override the wait function for testing (returns 'proceed' or 'abort'). */ + waitFn?: (graceSeconds: number) => Promise<'proceed' | 'abort'>; +} + +export type CostPromptResult = + | { kind: 'proceed'; reason: 'env_skip' | 'yes_override' | 'no_version_change' | 'non_tty_auto' | 'tty_proceed' } + | { kind: 'abort'; reason: 'tty_ctrl_c' }; + +/** Read the prompt_version of the most recent persisted run. */ +async function readLastPromptVersion(engine: BrainEngine): Promise { + try { + const rows = await engine.executeRaw<{ prompt_version: string }>( + `SELECT prompt_version FROM eval_contradictions_runs ORDER BY ran_at DESC LIMIT 1`, + ); + if (rows && rows.length > 0 && typeof rows[0].prompt_version === 'string') { + return rows[0].prompt_version; + } + } catch { + // Table missing (pre-v0.32 brains) or transient error — treat as null + // (which fires the prompt on first run, the correct default). + } + return null; +} + +/** Default TTY waiter: prints countdown and resolves on SIGINT or timeout. */ +function defaultWaitFn(graceSeconds: number): Promise<'proceed' | 'abort'> { + return new Promise((resolve) => { + let aborted = false; + const onSigint = () => { + if (aborted) return; + aborted = true; + process.removeListener('SIGINT', onSigint); + resolve('abort'); + }; + process.on('SIGINT', onSigint); + setTimeout(() => { + if (aborted) return; + process.removeListener('SIGINT', onSigint); + resolve('proceed'); + }, graceSeconds * 1000); + }); +} + +/** + * Public entry. Returns whether the runner should proceed. Honors the + * --yes override, GBRAIN_NO_PROBE_PROMPT, TTY detection, and the persisted + * last-run prompt_version comparison. + */ +export async function maybePromptForCostBeforeProbe( + opts: CostPromptOpts, +): Promise { + if (opts.yesOverride) { + return { kind: 'proceed', reason: 'yes_override' }; + } + if (process.env.GBRAIN_NO_PROBE_PROMPT === '1') { + return { kind: 'proceed', reason: 'env_skip' }; + } + + const lastVersion = await readLastPromptVersion(opts.engine); + if (lastVersion === PROMPT_VERSION) { + return { kind: 'proceed', reason: 'no_version_change' }; + } + + const stderr = opts.stderrWriter ?? ((text: string) => process.stderr.write(text)); + const isTty = opts.isTtyOverride ?? (process.stderr.isTTY === true); + + // Conservative pair-count upper bound — same formula the runner uses for + // its own pre-flight check. Cost estimate covers the worst-case re-judge. + const conservativePairsPerQuery = (opts.topK * (opts.topK - 1)) / 2 + opts.topK * 2; + const estimatedCost = estimateUpperBoundCost({ + pairCount: opts.queryCount * conservativePairsPerQuery, + queryCount: opts.queryCount, + judgeModel: opts.judgeModel, + }); + const banner = [ + `[contradiction probe] PROMPT_VERSION changed (${lastVersion ?? 'none'} → ${PROMPT_VERSION}).`, + `Old verdicts in the persistent cache no longer apply; this run will re-judge from scratch.`, + `Upper-bound estimate for ${opts.queryCount} queries × top-${opts.topK} on ${opts.judgeModel}: ~$${estimatedCost.toFixed(2)}.`, + `(--budget-usd N hard-caps the run; default values are conservative.)`, + ].join('\n'); + + if (!isTty) { + // Autopilot / scripted invocation: emit the estimate and proceed. + stderr(`${banner}\nNon-TTY: proceeding automatically. Set GBRAIN_NO_PROBE_PROMPT=1 to suppress.\n`); + return { kind: 'proceed', reason: 'non_tty_auto' }; + } + + const graceRaw = process.env.GBRAIN_PROBE_PROMPT_GRACE_SECONDS; + const graceSeconds = graceRaw && Number.isFinite(Number(graceRaw)) && Number(graceRaw) >= 0 + ? Number(graceRaw) + : 10; + stderr(`${banner}\nPress Ctrl-C within ${graceSeconds}s to abort, or wait to proceed.\n`); + const waiter = opts.waitFn ?? defaultWaitFn; + const decision = await waiter(graceSeconds); + if (decision === 'abort') { + stderr(`[contradiction probe] aborted by Ctrl-C.\n`); + return { kind: 'abort', reason: 'tty_ctrl_c' }; + } + return { kind: 'proceed', reason: 'tty_proceed' }; +} diff --git a/src/core/eval-contradictions/date-filter.ts b/src/core/eval-contradictions/date-filter.ts index cf801a736..afcf04654 100644 --- a/src/core/eval-contradictions/date-filter.ts +++ b/src/core/eval-contradictions/date-filter.ts @@ -52,12 +52,31 @@ export interface DateFilterDecision { | 'both_explicit_separated' | 'one_or_both_missing_dates' | 'same_paragraph_dual_date' - | 'overlapping_or_close'; + | 'overlapping_or_close' + /** + * v0.34 / Lane B: both sides have a non-null page-level effective_date, + * so the judge will see the temporal anchors and classify supersession / + * regression / evolution explicitly. Rule 3's >30d skip would otherwise + * silently kill exactly the cases the new verdicts exist to surface. + */ + | 'both_have_effective_date'; } export interface DateFilterInput { textA: string; textB: string; + /** + * v0.34 / Lane B: page-level effective_date for each side (carried through + * from SearchResult.effective_date via PairMember). When both are non-null + * the judge will see them via the (from: YYYY-MM-DD) prompt tag and can + * classify temporal supersession explicitly — so the rule-3 >30d skip is + * counterproductive (it was a cost-saving heuristic for the v1 binary + * verdict; the v2 enum makes the explicit classification cheap). + * Undefined / null means "no temporal anchor for that side" and rule 3 + * still applies as a fallback heuristic. + */ + effectiveDateA?: string | null; + effectiveDateB?: string | null; } /** Extract date tokens from text. Returns parsed Date objects (UTC midnight). */ @@ -144,6 +163,15 @@ export function shouldSkipForDateMismatch(input: DateFilterInput): DateFilterDec ) { return { skip: false, reason: 'same_paragraph_dual_date' }; } + // v0.34 / Lane B: when BOTH sides have a non-null page-level effective_date, + // the judge will see them via the (from: YYYY-MM-DD) tag and can classify + // temporal supersession / regression / evolution explicitly. Skipping these + // pairs was the v1 default; in v2 it would silently kill the cases the + // new verdict taxonomy exists to surface (e.g. the role-change-across- + // years case where chunk-text dates are years apart). + if (input.effectiveDateA && input.effectiveDateB) { + return { skip: false, reason: 'both_have_effective_date' }; + } const datesA = extractDates(input.textA); const datesB = extractDates(input.textB); if (datesA.length === 0 || datesB.length === 0) { diff --git a/src/core/eval-contradictions/judge.ts b/src/core/eval-contradictions/judge.ts index de50d1d88..24f779116 100644 --- a/src/core/eval-contradictions/judge.ts +++ b/src/core/eval-contradictions/judge.ts @@ -20,8 +20,8 @@ */ import { chat, type ChatResult } from '../ai/gateway.ts'; -import { parseSeverity } from './severity-classify.ts'; -import type { JudgeVerdict, ResolutionKind } from './types.ts'; +import { parseSeverity, defaultSeverityForVerdict } from './severity-classify.ts'; +import type { JudgeVerdict, ResolutionKind, Verdict } from './types.ts'; const FENCE_RE = /```(?:json)?\s*\n?([\s\S]*?)```/i; @@ -108,9 +108,26 @@ export function truncateUtf8(text: string, maxChars: number): string { export interface JudgeInput { /** The user's query for the search that retrieved both members. */ query: string; - /** Statement A: slug + text + optional source-tier + holder (if take). */ - a: { slug: string; text: string; source_tier?: string; holder?: string | null }; - b: { slug: string; text: string; source_tier?: string; holder?: string | null }; + /** + * Statement A: slug + text + optional source-tier + holder (if take) + + * optional effective_date (Lane A1). When effective_date is null/undefined + * the prompt shows `(date unknown)` for that side; the judge classifies + * based on chunk text alone, same as the v1 prompt did. + */ + a: { + slug: string; + text: string; + source_tier?: string; + holder?: string | null; + effective_date?: string | null; + }; + b: { + slug: string; + text: string; + source_tier?: string; + holder?: string | null; + effective_date?: string | null; + }; /** Provider:model id; routed through gateway.chat. */ model: string; /** UTF-8-safe truncation limit per pair member. C4 flag. */ @@ -129,58 +146,108 @@ export interface JudgeOutput { /** * Validated resolution_kind values. Anything outside this set defaults to - * 'manual_review' (the safe, no-action option). + * 'manual_review' (the safe, no-action option for contradictions; new verdicts + * route through auto-supersession.ts when resolution_kind is null on input). + * + * v0.34 / Lane A2: added temporal_supersede, flag_for_review, log_timeline_change. */ function parseResolutionKind(value: unknown): ResolutionKind | null { if ( value === 'takes_supersede' || value === 'dream_synthesize' || value === 'takes_mark_debate' || - value === 'manual_review' + value === 'manual_review' || + value === 'temporal_supersede' || + value === 'flag_for_review' || + value === 'log_timeline_change' ) { return value; } return null; } +const VALID_VERDICTS: ReadonlySet = new Set([ + 'no_contradiction', + 'contradiction', + 'temporal_supersession', + 'temporal_regression', + 'temporal_evolution', + 'negation_artifact', +]); + +/** Validate a verdict string from JSON; throws on missing/invalid so caller maps to parse_fail. */ +export function parseVerdict(value: unknown): Verdict { + if (typeof value !== 'string' || !VALID_VERDICTS.has(value as Verdict)) { + throw new Error(`judge JSON missing or invalid verdict: ${JSON.stringify(value)}`); + } + return value as Verdict; +} + /** * Validate the raw parsed JSON against the JudgeVerdict shape. Throws on - * fundamentally-broken shape (missing contradicts/confidence) so the caller + * fundamentally-broken shape (missing verdict/confidence) so the caller * counts it under judge_errors.parse_fail rather than fabricating a verdict. * - * C1 enforcement: contradicts:true with confidence < 0.7 is downgraded to - * false (belt-and-suspenders against models ignoring the prompt rule). + * v0.34 / Lane A2: parses the new `verdict: Verdict` enum field instead of + * the v1 `contradicts: boolean`. PROMPT_VERSION = '2' (bumped in A1) means + * the persistent cache won't return v1-shaped rows for these calls. + * + * C1 enforcement: `verdict === 'contradiction'` with confidence < 0.7 is + * downgraded to `'no_contradiction'` (belt-and-suspenders against models + * ignoring the prompt rule). The 5 non-contradiction verdicts do NOT have a + * confidence floor — they're informational classifications, not error flags. */ export function normalizeVerdict(raw: unknown): JudgeVerdict { if (!raw || typeof raw !== 'object') { throw new Error('judge JSON missing or not an object'); } const v = raw as Record; - const rawContradicts = v.contradicts; - if (typeof rawContradicts !== 'boolean') { - throw new Error('judge JSON missing required field: contradicts'); - } + // Parse verdict first so we can throw a useful error before checking other + // fields. Old v1-shaped responses (`contradicts: true/false` without + // `verdict`) will throw here and the caller maps it to parse_fail — correct + // semantics because the prompt now asks for verdict explicitly. + let verdict = parseVerdict(v.verdict); const rawConfidence = v.confidence; if (typeof rawConfidence !== 'number' || !Number.isFinite(rawConfidence)) { throw new Error('judge JSON missing or invalid confidence'); } const clampedConfidence = Math.min(1, Math.max(0, rawConfidence)); - const severity = parseSeverity(v.severity); const axisRaw = typeof v.axis === 'string' ? v.axis : ''; const resolutionKind = parseResolutionKind(v.resolution_kind); - // C1 double-enforce: contradicts:true requires confidence >= 0.7. - let contradicts = rawContradicts; - if (contradicts && clampedConfidence < 0.7) { - contradicts = false; + // C1 double-enforce: only `verdict === 'contradiction'` carries the + // confidence floor. Downgrade to no_contradiction below the threshold. + if (verdict === 'contradiction' && clampedConfidence < 0.7) { + verdict = 'no_contradiction'; + } + + // Severity: judge can set it; if invalid, fall back to the default for the + // verdict (D7 map: temporal_supersession→info, temporal_regression→high, ...). + // parseSeverity coerces unknown strings to 'low' historically, but now we + // route through defaultSeverityForVerdict instead so each verdict gets a + // meaningful default. + const severity = parseSeverity(v.severity, defaultSeverityForVerdict(verdict)); + const isFinding = verdict !== 'no_contradiction'; + + // Only `contradiction` keeps the v1 fallback to 'manual_review' when the + // judge omits a resolution_kind. The new verdicts pass through whatever the + // judge said (or null) and auto-supersession.ts picks the kind based on + // verdict semantics. + let normalizedResolutionKind: ResolutionKind | null; + if (verdict === 'contradiction') { + normalizedResolutionKind = resolutionKind ?? 'manual_review'; + } else if (isFinding) { + normalizedResolutionKind = resolutionKind ?? null; + } else { + normalizedResolutionKind = null; } return { - contradicts, + verdict, severity, - axis: contradicts ? axisRaw : '', + axis: isFinding ? axisRaw : '', confidence: clampedConfidence, - resolution_kind: contradicts ? (resolutionKind ?? 'manual_review') : null, + resolution_kind: normalizedResolutionKind, }; } @@ -194,14 +261,31 @@ export function normalizeVerdict(raw: unknown): JudgeVerdict { */ export function buildJudgePrompt(opts: { query: string; - a: { slug: string; text: string; source_tier?: string; holder?: string | null }; - b: { slug: string; text: string; source_tier?: string; holder?: string | null }; + a: { + slug: string; + text: string; + source_tier?: string; + holder?: string | null; + effective_date?: string | null; + }; + b: { + slug: string; + text: string; + source_tier?: string; + holder?: string | null; + effective_date?: string | null; + }; maxPairChars: number; }): string { const a = truncateUtf8(opts.a.text, opts.maxPairChars); const b = truncateUtf8(opts.b.text, opts.maxPairChars); const aMeta = [opts.a.slug, opts.a.source_tier && `source-tier ${opts.a.source_tier}`, opts.a.holder && `holder ${opts.a.holder}`].filter(Boolean).join(', '); const bMeta = [opts.b.slug, opts.b.source_tier && `source-tier ${opts.b.source_tier}`, opts.b.holder && `holder ${opts.b.holder}`].filter(Boolean).join(', '); + // Lane A1: emit the page-level effective_date on its own line so the judge + // can reason temporally. `(date unknown)` keeps the v1 fallback behavior + // when the page has no effective_date — judge classifies on text alone. + const aDateTag = opts.a.effective_date ? `(from: ${opts.a.effective_date})` : '(date unknown)'; + const bDateTag = opts.b.effective_date ? `(from: ${opts.b.effective_date})` : '(date unknown)'; return [ 'You are a contradiction judge for a personal knowledge brain. The user', 'ran a search and got two results back. Decide whether the two statements', @@ -210,39 +294,54 @@ export function buildJudgePrompt(opts: { '', `User's query: ${opts.query}`, '', - `Statement A (${aMeta}):`, + `Statement A ${aDateTag} (${aMeta}):`, a, '', - `Statement B (${bMeta}):`, + `Statement B ${bDateTag} (${bMeta}):`, b, '', 'Rules:', - '- Different timeframes for the same dynamic property are NOT contradictions', - ' (e.g., MRR was $50K in 2024 vs $2M in 2026 — both true at their time).', - '- Different timeframes for a static identity claim MAY BE a contradiction', - ' (e.g., "Alice is CFO of Acme" vs "Alice left Acme" if dates suggest one', - ' supersedes the other).', + '- The (from: YYYY-MM-DD) tag is the page-level effective date. Use it to', + ' classify what kind of difference this is, not just whether it exists.', + ' (date unknown) means the page has no temporal anchor — judge on text', + ' alone for that side.', + '- Pick exactly one verdict from the six values below.', + '- Use temporal_supersession when the newer-dated claim updates or replaces', + ' the older one (role change, status change). Not an error.', + '- Use temporal_regression when a metric or status went BACKWARDS over time', + ' (e.g., MRR dropped from $200K to $150K). This is a signal worth flagging.', + '- Use temporal_evolution for legitimate change over time that is neither', + ' supersession nor regression (e.g., evolving narrative, multi-step decision).', + '- Use negation_artifact when one side contains an explicit negation that', + ' the surface tokens make look like a positive claim (e.g., "NOT X" parsed', + ' as "X"). The data is correct; the apparent conflict is a parsing artifact.', + '- Use contradiction ONLY for genuinely conflicting claims at the same point', + ' in time, where the dates do not explain the difference.', + '- Use no_contradiction when the statements are compatible.', + '', '- Subjective opinions held at different times by the SAME holder may be', ' a contradiction (a flip). Opinions held by DIFFERENT holders are not.', - '- Different aspects of the same entity are NOT contradictions.', - "- Incidental disagreements unrelated to the user's query do NOT count.", + '- Different aspects of the same entity are not contradictions.', + "- Incidental disagreements unrelated to the user's query do not count.", ' Judge only on claims relevant to what the user asked.', '', 'Reply with JSON ONLY:', '{', - ' "contradicts": true | false,', - ' "severity": "low" | "medium" | "high",', + ' "verdict": "no_contradiction" | "contradiction" | "temporal_supersession" | "temporal_regression" | "temporal_evolution" | "negation_artifact",', + ' "severity": "info" | "low" | "medium" | "high",', ' "axis": "",', ' "confidence": 0.0..1.0,', - ' "resolution_kind": "takes_supersede" | "dream_synthesize" | "takes_mark_debate" | "manual_review" | null', + ' "resolution_kind": "takes_supersede" | "dream_synthesize" | "takes_mark_debate" | "manual_review" | "temporal_supersede" | "flag_for_review" | "log_timeline_change" | null', '}', '', 'Severity rubric:', - '- low: naming/format differences (Alice Smith vs A. Smith).', + '- info: temporal_supersession and temporal_evolution (not errors; informational).', + '- low: naming/format differences (Alice Smith vs A. Smith); negation artifacts.', '- medium: factual values that may be stale (revenue, headcount).', - '- high: identity / structural claims (founder/CEO/CFO role, status).', + '- high: identity / structural claims (founder/CEO/CFO role); temporal_regression.', '', - 'Reply contradicts:true only when confidence >= 0.7.', + 'Reply verdict:contradiction only when confidence >= 0.7. Other verdicts have', + 'no confidence floor.', ].join('\n'); } @@ -264,6 +363,8 @@ function isRefusalResponse(result: ChatResult): boolean { */ export async function judgeContradiction(input: JudgeInput): Promise { const maxPairChars = input.maxPairChars ?? DEFAULT_MAX_PAIR_CHARS; + // input.a/b carry effective_date through PairMember (Lane A1); buildJudgePrompt + // emits it on the Statement line or falls through to `(date unknown)`. const prompt = buildJudgePrompt({ query: input.query, a: input.a, diff --git a/src/core/eval-contradictions/runner.ts b/src/core/eval-contradictions/runner.ts index 6319943a5..8c2873530 100644 --- a/src/core/eval-contradictions/runner.ts +++ b/src/core/eval-contradictions/runner.ts @@ -46,6 +46,8 @@ import { type PairMember, type PerQueryResult, type ProbeReport, + type Verdict, + type VerdictBreakdown, } from './types.ts'; const DEFAULT_TOP_K = 5; @@ -110,11 +112,27 @@ function searchResultToMember(r: SearchResult): PairMember { source_tier: classifySlugTier(r.slug), holder: null, text: r.chunk_text, + // Lane A1: effective_date carried through from the search projection. + // null when the page has no temporal anchor (judge will see `(date unknown)`). + effective_date: r.effective_date ?? null, + effective_date_source: r.effective_date_source ?? null, }; } -/** Convert a Take into a PairMember. */ -function takeToMember(take: { id: number; page_slug: string; claim: string; holder: string }, source_tier: ReturnType): PairMember { +/** + * Convert a Take into a PairMember. + * + * Lane A1: takes are paired with a chunk from the same page, so the take's + * effective_date is inherited from the chunk's page-level effective_date. + * A future enhancement could distinguish `takes.since_date` from + * `pages.effective_date` here — for v1 they share the same page anchor. + */ +function takeToMember( + take: { id: number; page_slug: string; claim: string; holder: string }, + source_tier: ReturnType, + effective_date: string | null, + effective_date_source: string | null, +): PairMember { return { slug: take.page_slug, chunk_id: null, @@ -122,6 +140,8 @@ function takeToMember(take: { id: number; page_slug: string; claim: string; hold source_tier, holder: take.holder, text: take.claim, + effective_date, + effective_date_source, }; } @@ -158,7 +178,12 @@ async function generateIntraPagePairs( if (takes.length === 0) continue; const chunkMember = searchResultToMember(r); for (const t of takes) { - const takeMember = takeToMember(t, chunkMember.source_tier); + const takeMember = takeToMember( + t, + chunkMember.source_tier, + chunkMember.effective_date, + chunkMember.effective_date_source, + ); out.push({ kind: 'intra_page_chunk_take', a: chunkMember, @@ -232,6 +257,17 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise { verdictBreakdown[v]++; }; for (const query of opts.queries) { if (opts.abortSignal?.aborted) break; @@ -258,10 +294,18 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise 0) queriesWithContradiction++; + // v0.34 / Lane A2: distinguish strict-contradiction from any-finding. + // The strict count drives the Wilson-CI denominator (the historic + // headline metric). The broad count surfaces the wave's new value: + // "of N queries, M had at least one temporal signal." + if (findings.length > 0) queriesWithAnyFinding++; + if (findings.some((f) => f.verdict === 'contradiction')) queriesWithContradiction++; perQuery.push({ query, result_count: results.length, @@ -347,7 +414,9 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise = { low: 1, medium: 2, high: 3 }; +/** + * v0.34 / Lane A2: 'info' joins the rank as the lowest non-trivial severity + * (below 'low' so high-severity findings still sort to the top). 0 reserved + * for future "below info" semantics; nothing currently emits at rank 0. + */ +const SEVERITY_RANK: Record = { info: 0, low: 1, medium: 2, high: 3 }; -/** Validate a severity string; default to 'low' on unknown input. */ -export function parseSeverity(value: unknown): Severity { - if (value === 'low' || value === 'medium' || value === 'high') return value; - return 'low'; +/** + * v0.34 / Lane A2: default severity per verdict, used when the judge returns + * an unknown severity string. The mapping is opinionated: + * - temporal_supersession: info — newer claim wins, this is normal evolution + * - temporal_evolution: info — legitimate change, neither error nor signal + * - temporal_regression: high — metric went backwards, investor red flag + * - negation_artifact: low — probe bug; surfaces in report but low signal + * - contradiction: medium — actual conflict; judge often overrides + * - no_contradiction: info — not a finding (filtered out by runner emit) + */ +const DEFAULT_SEVERITY_BY_VERDICT: Record = { + no_contradiction: 'info', + contradiction: 'medium', + temporal_supersession: 'info', + temporal_regression: 'high', + temporal_evolution: 'info', + negation_artifact: 'low', +}; + +export function defaultSeverityForVerdict(verdict: Verdict): Severity { + return DEFAULT_SEVERITY_BY_VERDICT[verdict]; } -/** Compare for descending sort: high > medium > low. */ +/** + * Validate a severity string. Two signatures: + * - parseSeverity(value) → defaults to 'low' on invalid input (legacy behavior) + * - parseSeverity(value, fallback) → uses provided fallback on invalid input + * + * v0.34 / Lane A2: accepts 'info' as a valid severity. + */ +export function parseSeverity(value: unknown, fallback: Severity = 'low'): Severity { + if (value === 'info' || value === 'low' || value === 'medium' || value === 'high') { + return value; + } + return fallback; +} + +/** Compare for descending sort: high > medium > low > info. */ export function compareSeverityDesc(a: Severity, b: Severity): number { return SEVERITY_RANK[b] - SEVERITY_RANK[a]; } @@ -32,7 +68,7 @@ export function compareSeverityDesc(a: Severity, b: Severity): number { export function bucketBySeverity( findings: readonly ContradictionFinding[], ): Record { - const out: Record = { low: [], medium: [], high: [] }; + const out: Record = { info: [], low: [], medium: [], high: [] }; for (const f of findings) { out[f.severity].push(f); } diff --git a/src/core/eval-contradictions/trends.ts b/src/core/eval-contradictions/trends.ts index 848403cda..d7a849c53 100644 --- a/src/core/eval-contradictions/trends.ts +++ b/src/core/eval-contradictions/trends.ts @@ -95,5 +95,24 @@ export function renderTrendChart(rows: readonly TrendRow[]): string { const bar = '#'.repeat(fill) + '.'.repeat(barWidth - fill); lines.push(`${date} ${model} ${q} ${withCx} ${flag} ${ci} ${bar}`); } + // v0.34 / Lane A2: append the per-verdict breakdown from the most recent + // run so operators can see which kinds of finding the probe is generating. + // Older runs without `verdict_breakdown` populated (cache rows from before + // A2) emit a `(no breakdown)` placeholder. + const latest = rows[0]; + if (latest && latest.report_json && latest.report_json.verdict_breakdown) { + const b = latest.report_json.verdict_breakdown; + lines.push(''); + lines.push(`Latest run verdict breakdown (${latest.ran_at.slice(0, 10)}):`); + lines.push(` contradiction: ${b.contradiction}`); + lines.push(` temporal_supersession: ${b.temporal_supersession}`); + lines.push(` temporal_regression: ${b.temporal_regression}`); + lines.push(` temporal_evolution: ${b.temporal_evolution}`); + lines.push(` negation_artifact: ${b.negation_artifact}`); + lines.push(` no_contradiction: ${b.no_contradiction}`); + } else if (latest) { + lines.push(''); + lines.push(`Latest run verdict breakdown: (no breakdown — predates v0.34 prompt_version=2)`); + } return lines.join('\n'); } diff --git a/src/core/eval-contradictions/types.ts b/src/core/eval-contradictions/types.ts index 5cfe6a12f..02d509609 100644 --- a/src/core/eval-contradictions/types.ts +++ b/src/core/eval-contradictions/types.ts @@ -12,30 +12,78 @@ export const SCHEMA_VERSION = 1 as const; -/** Bump when the judge prompt in judge.ts changes meaningfully. */ -export const PROMPT_VERSION = '1' as const; +/** + * Bump when the judge prompt in judge.ts changes meaningfully. Used as part + * of the cache key tuple — bumping invalidates every cached verdict from + * prior runs, which is the point: when the prompt edits, old verdicts are + * no longer trustworthy. + * + * v2 (Lane A1, 2026-05): judge prompt now receives `Statement A (from: YYYY-MM-DD)` + * or `(date unknown)` per side. Old v1 verdicts are silently invalidated. + */ +export const PROMPT_VERSION = '2' as const; /** Truncation policy string baked into the cache key. */ export const TRUNCATION_POLICY = '1500-chars-utf8-safe' as const; export type ContradictionKind = 'cross_slug_chunks' | 'intra_page_chunk_take'; -export type Severity = 'low' | 'medium' | 'high'; +/** + * v0.34 / Lane A2: severity gains 'info' for the new non-error-class verdicts + * (temporal_supersession, temporal_evolution) that should surface in the + * report without inflating the contradiction count. + */ +export type Severity = 'info' | 'low' | 'medium' | 'high'; +/** + * v0.34 / Lane A2: replaces the v1 `contradicts: boolean` shape. Six members. + * + * - no_contradiction → drop from findings (not surfaced) + * - contradiction → genuine conflict at the same point in time + * - temporal_supersession → newer claim updates/replaces older; not an error + * - temporal_regression → metric/status went backwards over time + * - temporal_evolution → legitimate change over time, neither of the above + * - negation_artifact → judge misread an explicit negation in one chunk + */ +export type Verdict = + | 'no_contradiction' + | 'contradiction' + | 'temporal_supersession' + | 'temporal_regression' + | 'temporal_evolution' + | 'negation_artifact'; + +/** + * Resolution kinds. v0.34 / Lane A2 added three new members covering the new + * verdicts; the original four still apply to `verdict === 'contradiction'`. + * + * - temporal_supersede → render `gbrain takes supersede ... --since ` + * - flag_for_review → informational; no CLI command rendered + * - log_timeline_change → render a hint pointing at the future + * timeline-writer subcommand (deferred) + */ export type ResolutionKind = | 'takes_supersede' | 'dream_synthesize' | 'takes_mark_debate' - | 'manual_review'; + | 'manual_review' + | 'temporal_supersede' + | 'flag_for_review' + | 'log_timeline_change'; export type SourceTier = 'curated' | 'bulk' | 'other'; /** * Judge's verdict for a single pair. Either the judge ran cleanly and we have * scoring, or it failed and we have a typed error to surface in the report. + * + * v0.34 / Lane A2: `verdict: Verdict` replaces the v1 `contradicts: boolean`. + * Every consumer that previously branched on `verdict.contradicts` now + * switches on `verdict.verdict`. PROMPT_VERSION bumped in A1 invalidates the + * old cached rows. */ export interface JudgeVerdict { - contradicts: boolean; + verdict: Verdict; severity: Severity; /** One-line description of what they disagree about, or empty when no contradiction. */ axis: string; @@ -74,6 +122,14 @@ export interface PairMember { /** Takes-only: who holds the take (`garry`, `alice`, ...). */ holder: string | null; text: string; + /** + * v0.34 / Lane A1: page-level effective_date carried through from SearchResult. + * Format: YYYY-MM-DD (ISO date-only). Threaded into the judge prompt so the + * model can classify supersession/regression/evolution. Null means "no + * temporal anchor for this chunk" — judge sees `(date unknown)` for this side. + */ + effective_date: string | null; + effective_date_source: string | null; } export interface ContradictionPair { @@ -85,6 +141,12 @@ export interface ContradictionPair { } export interface ContradictionFinding extends ContradictionPair { + /** + * v0.34 / Lane A2: which verdict bucket this finding belongs to. Always one + * of the five non-`no_contradiction` members (no_contradiction findings are + * dropped from the report by the runner emit predicate). + */ + verdict: Verdict; severity: Severity; axis: string; confidence: number; @@ -95,6 +157,13 @@ export interface ContradictionFinding extends ContradictionPair { export interface PerQueryResult { query: string; result_count: number; + /** + * v0.34 / Lane A2: contains every non-`no_contradiction` finding (genuine + * contradictions PLUS temporal_supersession / temporal_regression / + * temporal_evolution / negation_artifact). Field name preserved for wire- + * compatibility; consumers that want the strict-contradiction subset + * filter on `f.verdict === 'contradiction'`. + */ contradictions: ContradictionFinding[]; /** Pairs the date pre-filter rejected before any judge call. Diagnostic only. */ pairs_skipped_by_date: number; @@ -104,6 +173,20 @@ export interface PerQueryResult { pairs_judged: number; } +/** + * v0.34 / Lane A2: per-run tally across all 6 verdicts. Surfaces in the trend + * rollup so operators can see whether `gbrain eval suspected-contradictions` + * is finding mostly genuine contradictions or mostly temporal noise. + */ +export interface VerdictBreakdown { + no_contradiction: number; + contradiction: number; + temporal_supersession: number; + temporal_regression: number; + temporal_evolution: number; + negation_artifact: number; +} + export interface SourceTierBreakdown { curated_vs_curated: number; curated_vs_bulk: number; @@ -155,8 +238,25 @@ export interface ProbeReport { top_k: number; sampling: 'deterministic' | 'score-first'; queries_evaluated: number; + /** + * Count of queries that produced at least one `verdict === 'contradiction'` + * finding. Strict-contradiction count; used as the headline metric + the + * Wilson-CI denominator. v0.34 / Lane A2: meaning preserved from v1, but + * the field is now narrower than `queries_with_any_finding`. + */ queries_with_contradiction: number; + /** + * v0.34 / Lane A2: count of queries with ANY non-`no_contradiction` verdict. + * Always >= queries_with_contradiction. Helps operators see whether the + * probe is finding mostly genuine contradictions or mostly temporal signals. + */ + queries_with_any_finding: number; total_contradictions_flagged: number; + /** + * v0.34 / Lane A2: per-verdict tally across every judged pair in the run. + * The sum equals (pairs_judged + cache_hits) across all queries. + */ + verdict_breakdown: VerdictBreakdown; calibration: Calibration; judge_errors: JudgeErrorsCounts; cost_usd: CostBreakdown; diff --git a/src/core/pglite-engine.ts b/src/core/pglite-engine.ts index 7299ba9bb..03ab90b41 100644 --- a/src/core/pglite-engine.ts +++ b/src/core/pglite-engine.ts @@ -805,6 +805,7 @@ export class PGLiteEngine implements BrainEngine { `WITH ranked AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, + p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, CASE WHEN p.updated_at < ( @@ -922,6 +923,7 @@ export class PGLiteEngine implements BrainEngine { `WITH ranked AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, + p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ${scoreExpr} AS score, CASE WHEN p.updated_at < ( @@ -950,6 +952,7 @@ export class PGLiteEngine implements BrainEngine { const { rows } = await this.db.query( `SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, + p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ${scoreExpr} AS score, CASE WHEN p.updated_at < ( @@ -1042,6 +1045,7 @@ export class PGLiteEngine implements BrainEngine { const { rows } = await this.db.query( `SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, + p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, CASE WHEN p.updated_at < ( @@ -1142,6 +1146,7 @@ export class PGLiteEngine implements BrainEngine { `WITH hnsw_candidates AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at, + p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, 1 - (cc.${col} <=> $1::vector) AS raw_score FROM content_chunks cc @@ -1153,6 +1158,7 @@ export class PGLiteEngine implements BrainEngine { ) SELECT hc.slug, hc.page_id, hc.title, hc.type, hc.source_id, + hc.effective_date, hc.effective_date_source, hc.chunk_id, hc.chunk_index, hc.chunk_text, hc.chunk_source, hc.raw_score * ${sourceFactorCaseOnSlug} AS score, CASE WHEN hc.updated_at < ( diff --git a/src/core/postgres-engine.ts b/src/core/postgres-engine.ts index cdc12c29c..c7538849e 100644 --- a/src/core/postgres-engine.ts +++ b/src/core/postgres-engine.ts @@ -865,6 +865,7 @@ export class PostgresEngine implements BrainEngine { WITH ranked_chunks AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, + p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score FROM content_chunks cc @@ -895,6 +896,7 @@ export class PostgresEngine implements BrainEngine { ORDER BY slug, score DESC ) SELECT slug, page_id, title, type, source_id, + effective_date, effective_date_source, chunk_id, chunk_index, chunk_text, chunk_source, score, false AS stale FROM best_per_page @@ -1005,6 +1007,7 @@ export class PostgresEngine implements BrainEngine { const rawQuery = ` SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, + p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score, false AS stale @@ -1138,6 +1141,7 @@ export class PostgresEngine implements BrainEngine { WITH hnsw_candidates AS ( SELECT p.slug, p.id as page_id, p.title, p.type, p.source_id, + p.effective_date, p.effective_date_source, cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source, 1 - (cc.${col} <=> $1::vector) AS raw_score FROM content_chunks cc @@ -1160,6 +1164,7 @@ export class PostgresEngine implements BrainEngine { ) SELECT slug, page_id, title, type, source_id, + effective_date, effective_date_source, chunk_id, chunk_index, chunk_text, chunk_source, raw_score * ${sourceFactorCaseOnSlug} AS score, false AS stale diff --git a/src/core/types.ts b/src/core/types.ts index 2bd3706af..58dc28b33 100644 --- a/src/core/types.ts +++ b/src/core/types.ts @@ -410,6 +410,15 @@ export interface SearchResult { * 'default' for pre-v0.17 rows that lacked the column. */ source_id?: string; + /** + * v0.34 — page-level effective_date (and its source) carried through from + * the pages join. Format: YYYY-MM-DD (ISO date-only). Consumers (currently + * the contradiction probe's date-aware judge prompt + date pre-filter) + * treat null and undefined the same: "no temporal anchor for this chunk." + * Pre-v0.34 engines that don't project these columns leave both undefined. + */ + effective_date?: string | null; + effective_date_source?: string | null; } export interface SearchOpts { diff --git a/src/core/utils.ts b/src/core/utils.ts index d6d04abc1..59e786f9b 100644 --- a/src/core/utils.ts +++ b/src/core/utils.ts @@ -243,6 +243,34 @@ export function rowToSearchResult(row: Record): SearchResult { if (typeof row.source_id === 'string') { result.source_id = row.source_id; } + // v0.34: effective_date / effective_date_source carried through from the + // pages join. Same three-state read as readOptionalDate elsewhere: the + // field is left UNTOUCHED when the column isn't in the projection (so + // legacy callers see undefined), set to null when the column was selected + // but the page row has no date, and to YYYY-MM-DD when populated. Postgres + // returns Date objects via postgres.js; PGLite returns strings. Normalize + // to date-only ISO so downstream prompt-builders don't see noise from + // midnight-UTC timestamps. + if ('effective_date' in row) { + const raw = row.effective_date; + if (raw === null) { + result.effective_date = null; + } else if (raw instanceof Date) { + result.effective_date = raw.toISOString().slice(0, 10); + } else if (typeof raw === 'string' && raw) { + // Postgres TIMESTAMPTZ already serializes as "YYYY-MM-DD ..." — slice + // the date portion. PGLite returns the same shape via its parser. + result.effective_date = raw.slice(0, 10); + } + } + if ('effective_date_source' in row) { + const raw = row.effective_date_source; + if (raw === null) { + result.effective_date_source = null; + } else if (typeof raw === 'string' && raw) { + result.effective_date_source = raw; + } + } return result; } diff --git a/test/e2e/eval-contradictions-postgres.test.ts b/test/e2e/eval-contradictions-postgres.test.ts index afe098d05..a8c8c0163 100644 --- a/test/e2e/eval-contradictions-postgres.test.ts +++ b/test/e2e/eval-contradictions-postgres.test.ts @@ -60,7 +60,16 @@ function mkReport(opts: Partial = {}): ProbeReport { sampling: 'deterministic', queries_evaluated: 50, queries_with_contradiction: 12, + queries_with_any_finding: 18, total_contradictions_flagged: 18, + verdict_breakdown: { + no_contradiction: 282, + contradiction: 12, + temporal_supersession: 4, + temporal_regression: 1, + temporal_evolution: 1, + negation_artifact: 0, + }, calibration: { queries_total: 50, queries_judged_clean: 38, @@ -160,11 +169,11 @@ describe('E2E: P2 persistent cache with real now()', () => { const cache = new JudgeCache({ engine, modelId: 'haiku-pg-test' }); expect(await cache.lookup('text-a', 'text-b')).toBeNull(); await cache.store('text-a', 'text-b', { - contradicts: true, severity: 'high', axis: 'pg-test', confidence: 0.9, resolution_kind: 'dream_synthesize', + verdict: 'contradiction', severity: 'high', axis: 'pg-test', confidence: 0.9, resolution_kind: 'dream_synthesize', }); const hit = await cache.lookup('text-a', 'text-b'); expect(hit).not.toBeNull(); - expect(hit?.contradicts).toBe(true); + expect(hit?.verdict).toBe('contradiction'); expect(hit?.severity).toBe('high'); }); @@ -172,7 +181,7 @@ describe('E2E: P2 persistent cache with real now()', () => { if (!engine) return; const cache = new JudgeCache({ engine, modelId: 'haiku-pg-test', ttlSeconds: 60 }); await cache.store('expire-me-a', 'expire-me-b', { - contradicts: false, severity: 'low', axis: '', confidence: 0.3, resolution_kind: null, + verdict: 'no_contradiction', severity: 'info', axis: '', confidence: 0.3, resolution_kind: null, }); // Backdate expires_at by 1 second. const key = buildCacheKey({ textA: 'expire-me-a', textB: 'expire-me-b', modelId: 'haiku-pg-test' }); @@ -191,7 +200,7 @@ describe('E2E: P2 persistent cache with real now()', () => { if (!engine) return; const cache1 = new JudgeCache({ engine, modelId: 'haiku-pg-test' }); await cache1.store('shared-a', 'shared-b', { - contradicts: true, severity: 'medium', axis: '', confidence: 0.85, resolution_kind: 'manual_review', + verdict: 'contradiction', severity: 'medium', axis: '', confidence: 0.85, resolution_kind: 'manual_review', }); // Direct engine call with a different prompt_version should miss. const wrong = await engine.getContradictionCacheEntry({ @@ -262,9 +271,10 @@ describe('E2E: find_contradictions MCP op on Postgres', () => { contradictions: [ { kind: 'cross_slug_chunks', - a: { slug: 'companies/acme-example', chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' }, - b: { slug: 'openclaw/chat/x', chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b' }, + a: { slug: 'companies/acme-example', chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a', effective_date: null, effective_date_source: null }, + b: { slug: 'openclaw/chat/x', chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b', effective_date: null, effective_date_source: null }, combined_score: 1.5, + verdict: 'contradiction', severity: 'high', axis: 'MRR figure', confidence: 0.9, @@ -273,9 +283,10 @@ describe('E2E: find_contradictions MCP op on Postgres', () => { }, { kind: 'cross_slug_chunks', - a: { slug: 'people/alice-example', chunk_id: 3, take_id: null, source_tier: 'curated', holder: null, text: 'c' }, - b: { slug: 'people/alice-smith-example', chunk_id: 4, take_id: null, source_tier: 'curated', holder: null, text: 'd' }, + a: { slug: 'people/alice-example', chunk_id: 3, take_id: null, source_tier: 'curated', holder: null, text: 'c', effective_date: null, effective_date_source: null }, + b: { slug: 'people/alice-smith-example', chunk_id: 4, take_id: null, source_tier: 'curated', holder: null, text: 'd', effective_date: null, effective_date_source: null }, combined_score: 1.2, + verdict: 'contradiction', severity: 'low', axis: 'name format', confidence: 0.75, diff --git a/test/eval-contradictions-auto-supersession.test.ts b/test/eval-contradictions-auto-supersession.test.ts index 310bf5d01..44a248998 100644 --- a/test/eval-contradictions-auto-supersession.test.ts +++ b/test/eval-contradictions-auto-supersession.test.ts @@ -17,8 +17,8 @@ import type { function mkCrossSlugPair(slugA: string, slugB: string): ContradictionPair { return { kind: 'cross_slug_chunks', - a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' }, - b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b' }, + a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a', effective_date: null, effective_date_source: null }, + b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b', effective_date: null, effective_date_source: null }, combined_score: 1, }; } @@ -26,8 +26,8 @@ function mkCrossSlugPair(slugA: string, slugB: string): ContradictionPair { function mkIntraPagePair(pageSlug: string, takeId: number): ContradictionPair { return { kind: 'intra_page_chunk_take', - a: { slug: pageSlug, chunk_id: 5, take_id: null, source_tier: 'curated', holder: null, text: 'chunk text' }, - b: { slug: pageSlug, chunk_id: null, take_id: takeId, source_tier: 'curated', holder: 'garry', text: 'take claim' }, + a: { slug: pageSlug, chunk_id: 5, take_id: null, source_tier: 'curated', holder: null, text: 'chunk text', effective_date: null, effective_date_source: null }, + b: { slug: pageSlug, chunk_id: null, take_id: takeId, source_tier: 'curated', holder: 'garry', text: 'take claim', effective_date: null, effective_date_source: null }, combined_score: 1, }; } @@ -120,13 +120,14 @@ describe('pairToFinding', () => { test('merges pair + verdict into a finding', () => { const pair = mkIntraPagePair('people/alice', 7); const verdict: JudgeVerdict = { - contradicts: true, + verdict: 'contradiction', severity: 'high', axis: 'CFO role status', confidence: 0.92, resolution_kind: 'takes_supersede', }; const finding = pairToFinding(pair, verdict); + expect(finding.verdict).toBe('contradiction'); expect(finding.severity).toBe('high'); expect(finding.axis).toBe('CFO role status'); expect(finding.confidence).toBe(0.92); diff --git a/test/eval-contradictions-cache.test.ts b/test/eval-contradictions-cache.test.ts index d56171d61..2763ffc1d 100644 --- a/test/eval-contradictions-cache.test.ts +++ b/test/eval-contradictions-cache.test.ts @@ -31,7 +31,7 @@ beforeEach(async () => { }); const verdictHit: JudgeVerdict = { - contradicts: true, + verdict: 'contradiction', severity: 'medium', axis: 'MRR vs ARR', confidence: 0.85, @@ -87,7 +87,7 @@ describe('JudgeCache wrapper', () => { await cache.store('text-a', 'text-b', verdictHit); const hit = await cache.lookup('text-a', 'text-b'); expect(hit).not.toBeNull(); - expect(hit?.contradicts).toBe(true); + expect(hit?.verdict).toBe('contradiction'); expect(hit?.severity).toBe('medium'); expect(cache.stats().hits).toBe(1); }); @@ -97,7 +97,7 @@ describe('JudgeCache wrapper', () => { await cache.store('first', 'second', verdictHit); const hit = await cache.lookup('second', 'first'); expect(hit).not.toBeNull(); - expect(hit?.contradicts).toBe(true); + expect(hit?.verdict).toBe('contradiction'); }); test('different model_id is a separate key', async () => { diff --git a/test/eval-contradictions-cost-prompt.test.ts b/test/eval-contradictions-cost-prompt.test.ts new file mode 100644 index 000000000..60b82c014 --- /dev/null +++ b/test/eval-contradictions-cost-prompt.test.ts @@ -0,0 +1,212 @@ +/** + * Lane C cost-prompt helper tests — hermetic via injected waitFn + stderrWriter. + * + * Pins the four decision branches of `maybePromptForCostBeforeProbe`: + * - --yes override skips + * - GBRAIN_NO_PROBE_PROMPT=1 env var skips + * - prompt_version unchanged from the last persisted run skips (no surprise) + * - non-TTY auto-proceeds with a stderr note (autopilot path) + * - TTY proceeds after the grace window + * - TTY aborts on Ctrl-C + * + * Plus reads the last prompt_version via a PGLite-backed eval_contradictions_runs + * table to prove the cross-process state actually round-trips. + */ + +import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test'; +import { PGLiteEngine } from '../src/core/pglite-engine.ts'; +import { resetPgliteState } from './helpers/reset-pglite.ts'; +import { maybePromptForCostBeforeProbe } from '../src/core/eval-contradictions/cost-prompt.ts'; +import { PROMPT_VERSION } from '../src/core/eval-contradictions/types.ts'; +import { writeRunRow } from '../src/core/eval-contradictions/trends.ts'; +import type { ProbeReport } from '../src/core/eval-contradictions/types.ts'; +import { withEnv } from './helpers/with-env.ts'; + +let engine: PGLiteEngine; + +beforeAll(async () => { + engine = new PGLiteEngine(); + await engine.connect({}); + await engine.initSchema(); +}); + +afterAll(async () => { + await engine.disconnect(); +}); + +beforeEach(async () => { + await resetPgliteState(engine); +}); + +function captureStderr(): { lines: string[]; write: (s: string) => void } { + const lines: string[] = []; + return { lines, write: (s: string) => { lines.push(s); } }; +} + +function mkBaseOpts(overrides: Partial[0]> = {}) { + const stderr = captureStderr(); + return { + capture: stderr, + opts: { + engine, + queryCount: 50, + topK: 5, + judgeModel: 'anthropic:claude-haiku-4-5', + stderrWriter: stderr.write, + ...overrides, + }, + }; +} + +describe('maybePromptForCostBeforeProbe', () => { + test('--yes override returns proceed without checking versions', async () => { + const { opts, capture } = mkBaseOpts({ yesOverride: true }); + const r = await maybePromptForCostBeforeProbe(opts); + expect(r.kind).toBe('proceed'); + if (r.kind === 'proceed') expect(r.reason).toBe('yes_override'); + expect(capture.lines.length).toBe(0); + }); + + test('GBRAIN_NO_PROBE_PROMPT=1 skips entirely', async () => { + await withEnv({ GBRAIN_NO_PROBE_PROMPT: '1' }, async () => { + const { opts, capture } = mkBaseOpts(); + const r = await maybePromptForCostBeforeProbe(opts); + expect(r.kind).toBe('proceed'); + if (r.kind === 'proceed') expect(r.reason).toBe('env_skip'); + expect(capture.lines.length).toBe(0); + }); + }); + + test('prompt_version unchanged from last run → skip (no surprise)', async () => { + // Seed a run with the CURRENT PROMPT_VERSION so the comparison returns equal. + const report: ProbeReport = mkSeedReport(PROMPT_VERSION); + await writeRunRow(engine, report, 100); + const { opts, capture } = mkBaseOpts(); + const r = await maybePromptForCostBeforeProbe(opts); + expect(r.kind).toBe('proceed'); + if (r.kind === 'proceed') expect(r.reason).toBe('no_version_change'); + expect(capture.lines.length).toBe(0); + }); + + test('non-TTY auto-proceeds with stderr note when version changed', async () => { + // Seed a run with an OLDER prompt_version so the comparison detects change. + const report: ProbeReport = mkSeedReport('1'); + await writeRunRow(engine, report, 100); + const { opts, capture } = mkBaseOpts({ isTtyOverride: false }); + const r = await maybePromptForCostBeforeProbe(opts); + expect(r.kind).toBe('proceed'); + if (r.kind === 'proceed') expect(r.reason).toBe('non_tty_auto'); + const out = capture.lines.join(''); + expect(out).toContain('PROMPT_VERSION changed'); + expect(out).toContain('Non-TTY'); + }); + + test('TTY proceeds after the grace window (waitFn returns proceed)', async () => { + const report: ProbeReport = mkSeedReport('1'); + await writeRunRow(engine, report, 100); + let waitedSeconds = -1; + const { opts, capture } = mkBaseOpts({ + isTtyOverride: true, + waitFn: async (s) => { waitedSeconds = s; return 'proceed'; }, + }); + const r = await maybePromptForCostBeforeProbe(opts); + expect(r.kind).toBe('proceed'); + if (r.kind === 'proceed') expect(r.reason).toBe('tty_proceed'); + expect(waitedSeconds).toBeGreaterThan(0); // default 10s + const out = capture.lines.join(''); + expect(out).toContain('Press Ctrl-C'); + }); + + test('TTY aborts on Ctrl-C (waitFn returns abort)', async () => { + const report: ProbeReport = mkSeedReport('1'); + await writeRunRow(engine, report, 100); + const { opts, capture } = mkBaseOpts({ + isTtyOverride: true, + waitFn: async () => 'abort', + }); + const r = await maybePromptForCostBeforeProbe(opts); + expect(r.kind).toBe('abort'); + if (r.kind === 'abort') expect(r.reason).toBe('tty_ctrl_c'); + const out = capture.lines.join(''); + expect(out).toContain('aborted by Ctrl-C'); + }); + + test('fresh brain (no prior runs) fires the prompt on first run', async () => { + // No seed — readLastPromptVersion returns null. + const { opts, capture } = mkBaseOpts({ + isTtyOverride: true, + waitFn: async () => 'proceed', + }); + const r = await maybePromptForCostBeforeProbe(opts); + expect(r.kind).toBe('proceed'); + if (r.kind === 'proceed') expect(r.reason).toBe('tty_proceed'); + const out = capture.lines.join(''); + expect(out).toContain('PROMPT_VERSION changed (none →'); + }); + + test('GBRAIN_PROBE_PROMPT_GRACE_SECONDS env overrides the 10s default', async () => { + await withEnv({ GBRAIN_PROBE_PROMPT_GRACE_SECONDS: '0' }, async () => { + let waitedSeconds = -1; + const report: ProbeReport = mkSeedReport('1'); + await writeRunRow(engine, report, 100); + const { opts } = mkBaseOpts({ + isTtyOverride: true, + waitFn: async (s) => { waitedSeconds = s; return 'proceed'; }, + }); + await maybePromptForCostBeforeProbe(opts); + expect(waitedSeconds).toBe(0); + }); + }); + + test('estimate scales with query count + judge model in the banner text', async () => { + const report: ProbeReport = mkSeedReport('1'); + await writeRunRow(engine, report, 100); + const { opts, capture } = mkBaseOpts({ + isTtyOverride: false, + queryCount: 500, + judgeModel: 'anthropic:claude-sonnet-4-6', + }); + await maybePromptForCostBeforeProbe(opts); + const out = capture.lines.join(''); + expect(out).toContain('500 queries'); + expect(out).toContain('anthropic:claude-sonnet-4-6'); + expect(out).toMatch(/\$\d+\.\d{2}/); + }); +}); + +function mkSeedReport(promptVersion: string): ProbeReport { + return { + schema_version: 1, + run_id: `seed-${promptVersion}`, + judge_model: 'anthropic:claude-haiku-4-5', + prompt_version: promptVersion, + truncation_policy: '1500-chars-utf8-safe', + top_k: 5, + sampling: 'deterministic', + queries_evaluated: 50, + queries_with_contradiction: 12, + queries_with_any_finding: 12, + total_contradictions_flagged: 12, + verdict_breakdown: { + no_contradiction: 100, + contradiction: 12, + temporal_supersession: 0, + temporal_regression: 0, + temporal_evolution: 0, + negation_artifact: 0, + }, + calibration: { + queries_total: 50, + queries_judged_clean: 38, + queries_with_contradiction: 12, + wilson_ci_95: { point: 0.24, lower: 0.14, upper: 0.37 }, + }, + judge_errors: { parse_fail: 0, refusal: 0, timeout: 0, http_5xx: 0, unknown: 0, total: 0, note: '' }, + cost_usd: { judge: 1.0, embedding: 0.005, total: 1.005, estimate_note: '' }, + cache: { hits: 0, misses: 0, hit_rate: 0 }, + duration_ms: 45000, + source_tier_breakdown: { curated_vs_curated: 0, curated_vs_bulk: 0, bulk_vs_bulk: 0, other: 0 }, + per_query: [], + hot_pages: [], + }; +} diff --git a/test/eval-contradictions-cross-source.test.ts b/test/eval-contradictions-cross-source.test.ts index 6fcef751b..85980b01b 100644 --- a/test/eval-contradictions-cross-source.test.ts +++ b/test/eval-contradictions-cross-source.test.ts @@ -12,8 +12,8 @@ import type { ContradictionPair } from '../src/core/eval-contradictions/types.ts function mkPair(slugA: string, slugB: string): ContradictionPair { return { kind: 'cross_slug_chunks', - a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' }, - b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'curated', holder: null, text: 'b' }, + a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a', effective_date: null, effective_date_source: null }, + b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'curated', holder: null, text: 'b', effective_date: null, effective_date_source: null }, combined_score: 1, }; } diff --git a/test/eval-contradictions-date-filter.test.ts b/test/eval-contradictions-date-filter.test.ts index 1735b6de9..2422c8ad3 100644 --- a/test/eval-contradictions-date-filter.test.ts +++ b/test/eval-contradictions-date-filter.test.ts @@ -139,4 +139,85 @@ describe('shouldSkipForDateMismatch', () => { expect(a.length).toBe(b.length); expect(a.length).toBe(3); }); + + // ---- Lane B: page-level effective_date relaxes rule 3 ---- + // R1 IRON RULE regression suite. When BOTH sides have a non-null + // page-level effective_date, the judge will see them via the + // (from: YYYY-MM-DD) prompt tag and can classify temporal supersession. + // The v1 >30d skip would silently kill these cases — exactly the ones the + // new verdict taxonomy exists to surface. The other two rules stay intact. + + test('Lane B: both sides have effective_date → do NOT skip (judge classifies)', () => { + const d = shouldSkipForDateMismatch({ + textA: 'Sriram Krishnan: Partner — AI/ML/Eng', + textB: 'Sriram Krishnan: Senior White House AI Policy Advisor', + effectiveDateA: '2017-03-28', + effectiveDateB: '2025-01-15', + }); + expect(d.skip).toBe(false); + expect(d.reason).toBe('both_have_effective_date'); + }); + + test('Lane B: both sides effective_date overrides >30d chunk-text rule', () => { + // Without effective_dates this would be rule-3 SKIP. With them the judge + // gets to classify temporal_supersession. + const d = shouldSkipForDateMismatch({ + textA: 'Acme MRR was $50K (2024-08-01)', + textB: 'Acme MRR was $2M (2026-03-15)', + effectiveDateA: '2024-08-01', + effectiveDateB: '2026-03-15', + }); + expect(d.skip).toBe(false); + expect(d.reason).toBe('both_have_effective_date'); + }); + + test('Lane B regression: rule 1 (same-paragraph dual-date) still wins over effective_date', () => { + // Same-paragraph dual-date is checked BEFORE the effective_date branch + // so flip-flops in the chunk text always reach the judge regardless of + // page-level dates. + const d = shouldSkipForDateMismatch({ + textA: 'In Jan 2024 I said X. In Mar 2024 I reversed to not-X.', + textB: 'In 2026 I still hold not-X.', + effectiveDateA: '2024-01-15', + effectiveDateB: '2026-06-01', + }); + expect(d.skip).toBe(false); + expect(d.reason).toBe('same_paragraph_dual_date'); + }); + + test('Lane B regression: rule 2 (one side missing chunk dates) still applies when effective_dates partially present', () => { + // Only ONE side has effective_date — the relaxation doesn't fire. v1 + // rules 1 + 2 still govern. + const d = shouldSkipForDateMismatch({ + textA: 'Acme is profitable', + textB: 'Acme is unprofitable as of 2026-03-01', + effectiveDateA: null, + effectiveDateB: '2026-03-01', + }); + expect(d.skip).toBe(false); + expect(d.reason).toBe('one_or_both_missing_dates'); + }); + + test('Lane B regression: undefined effective_dates fall through to v1 behavior', () => { + // Callers that don't supply effective_dates (legacy code paths) keep the + // v1 skip behavior. Important for back-compat. + const d = shouldSkipForDateMismatch({ + textA: 'Acme MRR was $50K (2024-08-01)', + textB: 'Acme MRR was $2M (2026-03-15)', + }); + expect(d.skip).toBe(true); + expect(d.reason).toBe('both_explicit_separated'); + }); + + test('Lane B regression: empty-string effective_date is treated as missing', () => { + // An empty string is falsy; only a real date enables the relaxation. + const d = shouldSkipForDateMismatch({ + textA: 'Acme MRR was $50K (2024-08-01)', + textB: 'Acme MRR was $2M (2026-03-15)', + effectiveDateA: '', + effectiveDateB: '2026-03-15', + }); + expect(d.skip).toBe(true); + expect(d.reason).toBe('both_explicit_separated'); + }); }); diff --git a/test/eval-contradictions-engine.test.ts b/test/eval-contradictions-engine.test.ts index b6a1bf11b..7daff85d9 100644 --- a/test/eval-contradictions-engine.test.ts +++ b/test/eval-contradictions-engine.test.ts @@ -175,7 +175,7 @@ describe('contradiction cache (P2)', () => { truncation_policy: '1500-chars-utf8-safe', }; const verdict = { - contradicts: true, + verdict: 'contradiction', severity: 'medium', axis: 'MRR vs ARR', confidence: 0.85, @@ -191,12 +191,13 @@ describe('contradiction cache (P2)', () => { await engine.putContradictionCacheEntry({ ...baseKey, verdict }); const hit = await engine.getContradictionCacheEntry(baseKey); expect(hit).not.toBeNull(); - expect((hit as Record).contradicts).toBe(true); + expect((hit as Record).verdict).toBe('contradiction'); expect((hit as Record).severity).toBe('medium'); }); test('different prompt_version is a different cache key (Codex fix)', async () => { await engine.putContradictionCacheEntry({ ...baseKey, verdict }); + // baseKey has prompt_version='1'; v0.34 lookups under '2' must miss. const wrong = await engine.getContradictionCacheEntry({ ...baseKey, prompt_version: '2' }); expect(wrong).toBeNull(); }); @@ -211,11 +212,11 @@ describe('contradiction cache (P2)', () => { await engine.putContradictionCacheEntry({ ...baseKey, verdict }); await engine.putContradictionCacheEntry({ ...baseKey, - verdict: { ...verdict, contradicts: false, severity: 'low' }, + verdict: { ...verdict, verdict: 'no_contradiction', severity: 'info' }, }); const hit = await engine.getContradictionCacheEntry(baseKey); - expect((hit as Record).contradicts).toBe(false); - expect((hit as Record).severity).toBe('low'); + expect((hit as Record).verdict).toBe('no_contradiction'); + expect((hit as Record).severity).toBe('info'); }); test('expired entries are not returned by get', async () => { diff --git a/test/eval-contradictions-integrations.test.ts b/test/eval-contradictions-integrations.test.ts index 827e12fc3..c00b4cc72 100644 --- a/test/eval-contradictions-integrations.test.ts +++ b/test/eval-contradictions-integrations.test.ts @@ -55,7 +55,16 @@ function mkReport(opts: Partial & { sampling: 'deterministic', queries_evaluated: 50, queries_with_contradiction: findings.length > 0 ? Math.max(1, findings.length) : 0, + queries_with_any_finding: findings.length > 0 ? Math.max(1, findings.length) : 0, total_contradictions_flagged: findings.length, + verdict_breakdown: { + no_contradiction: 50 - findings.length, + contradiction: findings.length, + temporal_supersession: 0, + temporal_regression: 0, + temporal_evolution: 0, + negation_artifact: 0, + }, calibration: { queries_total: 50, queries_judged_clean: 50 - findings.length, @@ -76,9 +85,10 @@ function mkReport(opts: Partial & { pairs_judged: findings.length, contradictions: findings.map((f, i) => ({ kind: 'cross_slug_chunks' as const, - a: { slug: f.slugA, chunk_id: i + 1, take_id: null, source_tier: 'curated' as const, holder: null, text: 'a' }, - b: { slug: f.slugB, chunk_id: i + 100, take_id: null, source_tier: 'bulk' as const, holder: null, text: 'b' }, + a: { slug: f.slugA, chunk_id: i + 1, take_id: null, source_tier: 'curated' as const, holder: null, text: 'a', effective_date: null, effective_date_source: null }, + b: { slug: f.slugB, chunk_id: i + 100, take_id: null, source_tier: 'bulk' as const, holder: null, text: 'b', effective_date: null, effective_date_source: null }, combined_score: 1.0, + verdict: 'contradiction' as const, severity: f.severity, axis: f.axis, confidence: 0.85, diff --git a/test/eval-contradictions-judge.test.ts b/test/eval-contradictions-judge.test.ts index 1c5704f71..104102541 100644 --- a/test/eval-contradictions-judge.test.ts +++ b/test/eval-contradictions-judge.test.ts @@ -109,27 +109,58 @@ describe('buildJudgePrompt', () => { }); describe('normalizeVerdict', () => { - test('valid input passes through', () => { + test('valid contradiction verdict passes through', () => { const v = normalizeVerdict({ - contradicts: true, + verdict: 'contradiction', severity: 'medium', axis: 'MRR figure', confidence: 0.85, resolution_kind: 'dream_synthesize', }); - expect(v.contradicts).toBe(true); + expect(v.verdict).toBe('contradiction'); expect(v.severity).toBe('medium'); expect(v.confidence).toBe(0.85); expect(v.resolution_kind).toBe('dream_synthesize'); }); - test('throws on missing contradicts field', () => { + test('valid temporal_supersession verdict passes through (Lane A2)', () => { + const v = normalizeVerdict({ + verdict: 'temporal_supersession', + severity: 'info', + axis: 'role over time', + confidence: 0.9, + resolution_kind: 'temporal_supersede', + }); + expect(v.verdict).toBe('temporal_supersession'); + expect(v.severity).toBe('info'); + expect(v.resolution_kind).toBe('temporal_supersede'); + }); + + test('all 6 verdict members accepted by parser', () => { + for (const verdict of [ + 'no_contradiction', + 'contradiction', + 'temporal_supersession', + 'temporal_regression', + 'temporal_evolution', + 'negation_artifact', + ] as const) { + const v = normalizeVerdict({ verdict, severity: 'medium', confidence: 0.8 }); + expect(v.verdict).toBe(verdict); + } + }); + + test('throws on missing verdict field', () => { expect(() => normalizeVerdict({ confidence: 0.9 })).toThrow(); }); + test('throws on invalid verdict string', () => { + expect(() => normalizeVerdict({ verdict: 'maybe', confidence: 0.9 })).toThrow(); + }); + test('throws on invalid confidence', () => { - expect(() => normalizeVerdict({ contradicts: true, confidence: 'high' })).toThrow(); - expect(() => normalizeVerdict({ contradicts: true, confidence: NaN })).toThrow(); + expect(() => normalizeVerdict({ verdict: 'contradiction', confidence: 'high' })).toThrow(); + expect(() => normalizeVerdict({ verdict: 'contradiction', confidence: NaN })).toThrow(); }); test('throws on missing or non-object input', () => { @@ -138,49 +169,68 @@ describe('normalizeVerdict', () => { expect(() => normalizeVerdict('json string')).toThrow(); }); - test('C1 double-enforce: contradicts:true + confidence<0.7 downgrades to false', () => { + test('C1 double-enforce: verdict:contradiction + confidence<0.7 downgrades to no_contradiction', () => { const v = normalizeVerdict({ - contradicts: true, + verdict: 'contradiction', severity: 'high', axis: 'something', confidence: 0.6, resolution_kind: 'takes_supersede', }); - expect(v.contradicts).toBe(false); + expect(v.verdict).toBe('no_contradiction'); expect(v.axis).toBe(''); expect(v.resolution_kind).toBeNull(); }); - test('C1 boundary: confidence exactly 0.7 stays as contradicts:true', () => { + test('C1 boundary: confidence exactly 0.7 stays as verdict:contradiction', () => { const v = normalizeVerdict({ - contradicts: true, + verdict: 'contradiction', severity: 'medium', axis: 'something', confidence: 0.7, }); - expect(v.contradicts).toBe(true); + expect(v.verdict).toBe('contradiction'); expect(v.confidence).toBe(0.7); }); + test('C1 floor does NOT downgrade other verdicts on low confidence (Lane A2)', () => { + const v = normalizeVerdict({ + verdict: 'temporal_supersession', + severity: 'info', + confidence: 0.4, + }); + // Non-contradiction verdicts have no confidence floor; they survive at 0.4. + expect(v.verdict).toBe('temporal_supersession'); + }); + test('clamps confidence into [0, 1]', () => { - const v1 = normalizeVerdict({ contradicts: false, severity: 'low', confidence: -0.5 }); + const v1 = normalizeVerdict({ verdict: 'no_contradiction', severity: 'low', confidence: -0.5 }); expect(v1.confidence).toBe(0); - const v2 = normalizeVerdict({ contradicts: false, severity: 'low', confidence: 1.5 }); + const v2 = normalizeVerdict({ verdict: 'no_contradiction', severity: 'low', confidence: 1.5 }); expect(v2.confidence).toBe(1); }); - test('garbage severity defaults to low', () => { + test('garbage severity falls back to defaultSeverityForVerdict (no_contradiction → info)', () => { const v = normalizeVerdict({ - contradicts: false, + verdict: 'no_contradiction', severity: 'critical', confidence: 0.5, }); - expect(v.severity).toBe('low'); + expect(v.severity).toBe('info'); }); - test('unknown resolution_kind on contradicts:true falls back to manual_review', () => { + test('garbage severity falls back to defaultSeverityForVerdict (contradiction → medium)', () => { const v = normalizeVerdict({ - contradicts: true, + verdict: 'contradiction', + severity: 'critical', + confidence: 0.85, + }); + expect(v.severity).toBe('medium'); + }); + + test('unknown resolution_kind on contradiction falls back to manual_review (legacy)', () => { + const v = normalizeVerdict({ + verdict: 'contradiction', severity: 'medium', axis: 'X', confidence: 0.85, @@ -189,9 +239,9 @@ describe('normalizeVerdict', () => { expect(v.resolution_kind).toBe('manual_review'); }); - test('axis cleared when contradicts:false', () => { + test('axis cleared when verdict:no_contradiction', () => { const v = normalizeVerdict({ - contradicts: false, + verdict: 'no_contradiction', severity: 'low', axis: 'some axis', confidence: 0.4, @@ -212,14 +262,14 @@ describe('judgeContradiction', () => { const out = await judgeContradiction({ ...baseInput, chatFn: stubChat(mkResult(JSON.stringify({ - contradicts: true, + verdict: 'contradiction', severity: 'medium', axis: 'MRR figure', confidence: 0.85, resolution_kind: 'dream_synthesize', }))), }); - expect(out.verdict.contradicts).toBe(true); + expect(out.verdict.verdict).toBe('contradiction'); expect(out.verdict.severity).toBe('medium'); expect(out.usage.inputTokens).toBe(100); expect(out.usage.outputTokens).toBe(50); @@ -227,15 +277,15 @@ describe('judgeContradiction', () => { test('fence-wrapped JSON: parseModelJSON 4-strategy fallback', async () => { const fenced = '```json\n' + JSON.stringify({ - contradicts: false, - severity: 'low', + verdict: 'no_contradiction', + severity: 'info', confidence: 0.3, }) + '\n```'; const out = await judgeContradiction({ ...baseInput, chatFn: stubChat(mkResult(fenced)), }); - expect(out.verdict.contradicts).toBe(false); + expect(out.verdict.verdict).toBe('no_contradiction'); }); test('throws on parse failure (counted in judge_errors)', async () => { @@ -276,7 +326,7 @@ describe('judgeContradiction', () => { const userMsg = opts.messages.find((m) => m.role === 'user'); capturedPrompt = typeof userMsg?.content === 'string' ? userMsg.content : ''; return mkResult(JSON.stringify({ - contradicts: false, severity: 'low', confidence: 0.5, + verdict: 'no_contradiction', severity: 'info', confidence: 0.5, })); }), }); @@ -287,17 +337,17 @@ describe('judgeContradiction', () => { expect(DEFAULT_MAX_PAIR_CHARS).toBe(1500); }); - test('C1 enforcement reaches the verdict (low-confidence true → false)', async () => { + test('C1 enforcement reaches the verdict (low-confidence contradiction → no_contradiction)', async () => { const out = await judgeContradiction({ ...baseInput, chatFn: stubChat(mkResult(JSON.stringify({ - contradicts: true, + verdict: 'contradiction', severity: 'high', axis: 'something', confidence: 0.5, }))), }); - expect(out.verdict.contradicts).toBe(false); + expect(out.verdict.verdict).toBe('no_contradiction'); }); test('query appears in the rendered prompt (Codex fix)', async () => { @@ -308,7 +358,7 @@ describe('judgeContradiction', () => { chatFn: stubChat(async (opts) => { const m = opts.messages[0]?.content; capturedQuery = typeof m === 'string' ? m : ''; - return mkResult(JSON.stringify({ contradicts: false, severity: 'low', confidence: 0.4 })); + return mkResult(JSON.stringify({ verdict: 'no_contradiction', severity: 'info', confidence: 0.4 })); }), }); expect(capturedQuery).toContain('distinctive-query-marker-12345'); diff --git a/test/eval-contradictions-runner.test.ts b/test/eval-contradictions-runner.test.ts index 8d2f0d6c5..11c4c0e1d 100644 --- a/test/eval-contradictions-runner.test.ts +++ b/test/eval-contradictions-runner.test.ts @@ -62,8 +62,15 @@ function mkResult(slug: string, page_id: number, chunk_id: number, text: string, /** Stubbed judge that returns a fixed verdict pattern. */ function stubJudge(opts: { + /** + * v0.34 / Lane A2: `verdict` replaces `contradicts`. Legacy `contradicts: + * true` maps to verdict='contradiction'; `contradicts: false` to + * verdict='no_contradiction'. Callers can pass either field; verdict wins + * when both are set. + */ + verdict?: 'no_contradiction' | 'contradiction' | 'temporal_supersession' | 'temporal_regression' | 'temporal_evolution' | 'negation_artifact'; contradicts?: boolean; - severity?: 'low' | 'medium' | 'high'; + severity?: 'info' | 'low' | 'medium' | 'high'; confidence?: number; inputTokens?: number; outputTokens?: number; @@ -75,9 +82,11 @@ function stubJudge(opts: { if (opts.throwOn && opts.throwOn(idx)) { throw new Error('stub: simulated transient 503'); } + const resolvedVerdict = + opts.verdict ?? (opts.contradicts === false ? 'no_contradiction' : 'contradiction'); return { verdict: { - contradicts: opts.contradicts ?? true, + verdict: resolvedVerdict, severity: opts.severity ?? 'medium', axis: 'stub axis', confidence: opts.confidence ?? 0.85, @@ -374,7 +383,7 @@ describe('runContradictionProbe', () => { // Yield to let the abort fire. await new Promise((r) => setTimeout(r, 1)); return { - verdict: { contradicts: false, severity: 'low', axis: '', confidence: 0.3, resolution_kind: null }, + verdict: { verdict: 'no_contradiction', severity: 'info', axis: '', confidence: 0.3, resolution_kind: null }, usage: { inputTokens: 1, outputTokens: 1 }, }; }, @@ -418,7 +427,7 @@ describe('runContradictionProbe', () => { const recordOrder = (out: string[]): JudgeFn => async (input) => { out.push(`${input.a.slug}|${input.b.slug}`); return { - verdict: { contradicts: false, severity: 'low', axis: '', confidence: 0.4, resolution_kind: null }, + verdict: { verdict: 'no_contradiction', severity: 'info', axis: '', confidence: 0.4, resolution_kind: null }, usage: { inputTokens: 1, outputTokens: 1 }, }; }; @@ -432,4 +441,87 @@ describe('runContradictionProbe', () => { }); expect(order1).toEqual(order2); }); + + // ---- Lane D: R4 regression — runner emits findings for every non-no_contradiction verdict ---- + // Without the runner.ts:307-ish emit-rule change from Lane A2, the new + // verdicts (temporal_supersession, temporal_regression, temporal_evolution, + // negation_artifact) would silently vanish from the report. This test + // pins the broadened emit predicate one verdict at a time. + + for (const verdict of [ + 'contradiction', + 'temporal_supersession', + 'temporal_regression', + 'temporal_evolution', + 'negation_artifact', + ] as const) { + test(`R4 regression: verdict='${verdict}' surfaces as a finding`, async () => { + const idA = await seedPage(`a/${verdict}`, 'A'); + const idB = await seedPage(`b/${verdict}`, 'B'); + const out = await runContradictionProbe({ + engine, + queries: [`q-${verdict}`], + judgeFn: stubJudge({ verdict, severity: 'medium', confidence: 0.85 }), + searchFn: async () => [ + mkResult(`a/${verdict}`, idA, 1, `text-a-${verdict}`, 1.0), + mkResult(`b/${verdict}`, idB, 2, `text-b-${verdict}`, 0.5), + ], + budgetUsd: 5, + noCache: true, + }); + expect(out.report.total_contradictions_flagged).toBe(1); + expect(out.report.per_query[0].contradictions.length).toBe(1); + expect(out.report.per_query[0].contradictions[0].verdict).toBe(verdict); + // Only verdict='contradiction' counts toward the strict + // queries_with_contradiction metric (Wilson-CI denominator). + if (verdict === 'contradiction') { + expect(out.report.queries_with_contradiction).toBe(1); + } else { + expect(out.report.queries_with_contradiction).toBe(0); + } + // Every non-no_contradiction verdict counts toward queries_with_any_finding. + expect(out.report.queries_with_any_finding).toBe(1); + // Per-verdict breakdown tallies correctly. + expect(out.report.verdict_breakdown[verdict]).toBe(1); + }); + } + + test(`R4 regression: verdict='no_contradiction' produces ZERO findings`, async () => { + const idA = await seedPage('a/noc', 'A'); + const idB = await seedPage('b/noc', 'B'); + const out = await runContradictionProbe({ + engine, + queries: ['q-noc'], + judgeFn: stubJudge({ verdict: 'no_contradiction', severity: 'info', confidence: 0.85 }), + searchFn: async () => [ + mkResult('a/noc', idA, 1, 'no contradiction here', 1.0), + mkResult('b/noc', idB, 2, 'also no contradiction', 0.5), + ], + budgetUsd: 5, + noCache: true, + }); + expect(out.report.total_contradictions_flagged).toBe(0); + expect(out.report.per_query[0].contradictions.length).toBe(0); + expect(out.report.queries_with_contradiction).toBe(0); + expect(out.report.queries_with_any_finding).toBe(0); + expect(out.report.verdict_breakdown.no_contradiction).toBe(1); + }); + + // ---- Lane D: R5 regression — cache key tuple shape stays 5 fields ---- + // Lane A1 bumped PROMPT_VERSION 1→2 (invalidates cache) but kept the key + // shape unchanged. Adding a 6th field would silently break every operator's + // brain (no migration path). + + test('R5 regression: cache key tuple shape is exactly 5 fields', async () => { + const { buildCacheKey } = await import('../src/core/eval-contradictions/cache.ts'); + const key = buildCacheKey({ textA: 'a', textB: 'b', modelId: 'haiku' }); + const keys = Object.keys(key).sort(); + expect(keys).toEqual([ + 'chunk_a_hash', + 'chunk_b_hash', + 'model_id', + 'prompt_version', + 'truncation_policy', + ]); + }); }); diff --git a/test/eval-contradictions-severity.test.ts b/test/eval-contradictions-severity.test.ts index 0ed9de4ec..d6c37d495 100644 --- a/test/eval-contradictions-severity.test.ts +++ b/test/eval-contradictions-severity.test.ts @@ -25,6 +25,8 @@ function mkFinding(opts: { source_tier: 'curated', holder: null, text: 'A', + effective_date: null, + effective_date_source: null, }, b: { slug: opts.slugB, @@ -33,8 +35,11 @@ function mkFinding(opts: { source_tier: 'bulk', holder: null, text: 'B', + effective_date: null, + effective_date_source: null, }, combined_score: 1, + verdict: 'contradiction', severity: opts.severity, axis: 'test', confidence: 0.9, @@ -155,3 +160,56 @@ describe('buildHotPages', () => { expect(buildHotPages(findings, 5).length).toBe(5); }); }); + +// ---- Lane D: R6 regression — contradiction severity unchanged ---- +// The v1 `verdict === 'contradiction'` semantics include severity coming from +// the judge with a 'low' fallback (legacy). v2 must preserve this for the +// contradiction verdict specifically: garbage severity → 'medium' (the new +// per-verdict default for contradiction, NOT 'low' which would imply the +// contradiction is naming-cosmetic). + +describe('R6 regression: contradiction verdict severity preserved', () => { + test('judge-set severity wins over defaultSeverityForVerdict', async () => { + const { normalizeVerdict } = await import('../src/core/eval-contradictions/judge.ts'); + // Judge explicitly says 'high'. The v2 contract: that wins, even though + // defaultSeverityForVerdict('contradiction') is 'medium'. + const v = normalizeVerdict({ + verdict: 'contradiction', + severity: 'high', + axis: 'CFO role', + confidence: 0.85, + }); + expect(v.severity).toBe('high'); + }); + + test('judge-set severity also wins when judge picks low (legacy behavior preserved)', async () => { + const { normalizeVerdict } = await import('../src/core/eval-contradictions/judge.ts'); + const v = normalizeVerdict({ + verdict: 'contradiction', + severity: 'low', + axis: 'name format', + confidence: 0.85, + }); + expect(v.severity).toBe('low'); + }); + + test('garbage severity on contradiction falls back to medium (NOT low — that would mask conflicts)', async () => { + const { normalizeVerdict } = await import('../src/core/eval-contradictions/judge.ts'); + const v = normalizeVerdict({ + verdict: 'contradiction', + severity: 'critical', + confidence: 0.85, + }); + expect(v.severity).toBe('medium'); + }); + + test('garbage severity on temporal_regression falls back to high (real signal)', async () => { + const { normalizeVerdict } = await import('../src/core/eval-contradictions/judge.ts'); + const v = normalizeVerdict({ + verdict: 'temporal_regression', + severity: 'critical', + confidence: 0.85, + }); + expect(v.severity).toBe('high'); + }); +}); diff --git a/test/eval-contradictions-trends.test.ts b/test/eval-contradictions-trends.test.ts index 30870cd40..3a7aab6a6 100644 --- a/test/eval-contradictions-trends.test.ts +++ b/test/eval-contradictions-trends.test.ts @@ -33,13 +33,22 @@ function mkReport(runId: string, overrides: Partial = {}): ProbeRep schema_version: 1, run_id: runId, judge_model: 'anthropic:claude-haiku-4-5', - prompt_version: '1', + prompt_version: '2', truncation_policy: '1500-chars-utf8-safe', top_k: 5, sampling: 'deterministic', queries_evaluated: 50, queries_with_contradiction: 12, + queries_with_any_finding: 18, total_contradictions_flagged: 18, + verdict_breakdown: { + no_contradiction: 200, + contradiction: 12, + temporal_supersession: 4, + temporal_regression: 1, + temporal_evolution: 1, + negation_artifact: 0, + }, calibration: { queries_total: 50, queries_judged_clean: 38, diff --git a/test/scripts/check-proposal-pii.test.ts b/test/scripts/check-proposal-pii.test.ts new file mode 100644 index 000000000..2810f2a4d --- /dev/null +++ b/test/scripts/check-proposal-pii.test.ts @@ -0,0 +1,152 @@ +/** + * Tests for scripts/check-proposal-pii.sh — the privacy guard for + * docs/proposals/*.md. + * + * Strategy: each test case builds a tiny git-initialized scratch repo + * with a docs/proposals/*.md file containing the case input, invokes + * the script (which lives back in the real repo), and asserts the exit + * code + stderr signal. The script reads from `git rev-parse + * --show-toplevel` so each scratch repo behaves like a real project. + */ + +import { describe, test, expect } from 'bun:test'; +import { spawnSync } from 'node:child_process'; +import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; + +const SCRIPT_PATH = join(import.meta.dir, '..', '..', 'scripts', 'check-proposal-pii.sh'); + +function runLintIn(content: string | null): { exitCode: number; stderr: string } { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-pii-lint-')); + try { + // Initialize as git repo so `git rev-parse --show-toplevel` resolves. + spawnSync('git', ['init', '-q'], { cwd: dir }); + if (content !== null) { + mkdirSync(join(dir, 'docs', 'proposals'), { recursive: true }); + writeFileSync(join(dir, 'docs', 'proposals', 'test.md'), content); + } + const r = spawnSync('bash', [SCRIPT_PATH], { cwd: dir, encoding: 'utf8' }); + return { exitCode: r.status ?? -1, stderr: r.stderr ?? '' }; + } finally { + rmSync(dir, { recursive: true, force: true }); + } +} + +describe('check-proposal-pii.sh', () => { + test('clean proposal passes (exit 0)', () => { + const r = runLintIn(`# RFC + +This is a clean technical proposal using only generic placeholders: +alice-example, acme-corp, fund-a. Two roles, one decision. +`); + expect(r.exitCode).toBe(0); + expect(r.stderr).toBe(''); + }); + + test('garrytan/brain private repo reference is flagged', () => { + const r = runLintIn(`Context: findings resolved in garrytan/brain.\n`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain('garrytan/brain'); + }); + + test('trial separation phrase is flagged', () => { + const r = runLintIn(`A: trial separation\nB: confirmed status\n`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain('trial separation'); + }); + + test('permanent separation phrase is flagged', () => { + const r = runLintIn(`status: permanent separation, confirmed\n`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain('permanent separation'); + }); + + test('couples session phrase is flagged', () => { + const r = runLintIn(`per the couples session on 2026-05-07\n`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain('couples session'); + }); + + test('divorce attorney phrase is flagged', () => { + const r = runLintIn(`consultation with divorce attorney scheduled\n`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain('divorce attorney'); + }); + + test("grandmother's funeral phrase is flagged", () => { + const r = runLintIn(`Traveled for the grandmother's funeral last week.\n`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain("grandmother's funeral"); + }); + + test("aunt's funeral phrase is flagged", () => { + const r = runLintIn(`Traveled to the aunt's funeral in Toronto.\n`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain("aunt's funeral"); + }); + + test('wintermute is flagged (consistent with check-privacy.sh)', () => { + const r = runLintIn(`Author: Wintermute (via the user)\n`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain('wintermute'); + }); + + test('case-insensitive match: WINTERMUTE all-caps also flagged', () => { + const r = runLintIn(`See the WINTERMUTE deployment.\n`); + expect(r.exitCode).toBe(1); + }); + + test('benign separation usage stays clean (separation-of-concerns)', () => { + // "separation" alone is fine; only "trial separation" / "permanent + // separation" are flagged. Software vocabulary survives. + const r = runLintIn(`# Architecture + +Strict separation of concerns between the runner and the judge module. +The trial run uses a separate test database. +`); + expect(r.exitCode).toBe(0); + }); + + test('benign funeral metaphor stays clean (bare "funeral" not banned)', () => { + // Only combined personal-context phrases are banned. Bare "funeral" + // in a software metaphor survives the lint. + const r = runLintIn(`Killing the deprecated endpoint feels like a funeral.\n`); + expect(r.exitCode).toBe(0); + }); + + test('multiple PII patterns reported together with exit 1', () => { + const r = runLintIn(` +Context: findings from garrytan/brain. + +A: trial separation +B: permanent separation per the couples session +`); + expect(r.exitCode).toBe(1); + expect(r.stderr).toContain('garrytan/brain'); + expect(r.stderr).toContain('trial separation'); + expect(r.stderr).toContain('permanent separation'); + expect(r.stderr).toContain('couples session'); + // Summary count. + expect(r.stderr).toContain('4 PII pattern hit(s)'); + }); + + test('no proposals dir → exits 0 (not a failure)', () => { + const r = runLintIn(null); + expect(r.exitCode).toBe(0); + }); + + test('--help exits 1 with usage text', () => { + const dir = mkdtempSync(join(tmpdir(), 'gbrain-pii-lint-help-')); + try { + spawnSync('git', ['init', '-q'], { cwd: dir }); + const r = spawnSync('bash', [SCRIPT_PATH, '--help'], { cwd: dir, encoding: 'utf8' }); + expect(r.status).toBe(1); + const out = (r.stdout ?? '') + (r.stderr ?? ''); + expect(out).toContain('check-proposal-pii.sh'); + expect(out).toContain('docs/proposals'); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }); +});