admin/gbrain
1
0
Fork 0
mirror of https://github.com/garrytan/gbrain.git synced 2026-07-27 21:19:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

v0.35.3.1 feat(eval): temporal-aware contradiction probe + verdict enum (#1052)

Browse Source 
* rfc: temporal axis for contradiction probe

Field report on residual HIGH findings from gbrain eval suspected-contradictions
and proposal for a 4-phase fix (Phase 1 = judge prompt + verdict enum is the
recommended starting point).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): pass effective_date to judge prompt; bump PROMPT_VERSION

Lane A1 of the temporal-contradiction-probe wave. Threads page-level
effective_date through the search projection into the contradiction judge so
the LLM can reason about supersession instead of treating every dated pair as
a contradiction.

Changes:
- SearchResult interface adds optional effective_date + effective_date_source
  fields; rowToSearchResult populates them from the row data with date-only
  YYYY-MM-DD normalization (handles both postgres.js Date and PGLite string).
- 8 SELECT projection sites (3 in postgres-engine, 5 in pglite-engine) now
  carry p.effective_date + p.effective_date_source through their inner CTEs
  and outer SELECTs so search results expose the field on both engines.
- PairMember (eval-contradictions/types.ts) gets the two fields as required
  (string | null) so the type forces every constructor to think about temporal
  anchoring. Runner's searchResultToMember + takeToMember handle the
  normalization; takes inherit the chunk's page-level date.
- buildJudgePrompt emits `Statement A (from: YYYY-MM-DD)` when effective_date
  is non-null, else `(date unknown)`. Prompt instructions explain the tag so
  the model knows what to do with it.
- PROMPT_VERSION bumps '1' → '2'. Cache-key tuple shape unchanged; old rows
  miss naturally on first run against the new prompt.

Test fixtures in 5 files updated to include the new required fields. All 205
eval-contradictions unit tests + 101 search-related tests pass. Typecheck
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): replace contradicts:boolean with verdict:enum (6 members)

Lane A2 of the temporal-contradiction-probe wave. Expands the judge's
classification vocabulary from a binary contradicts:bool to a six-member
verdict enum so the probe can distinguish "this changed" from "this is wrong".

Verdict taxonomy:
  no_contradiction       — drop from findings
  contradiction          — genuine conflict at same point in time
  temporal_supersession  — newer claim updates/replaces older; not an error
  temporal_regression    — metric/status went backwards over time (signal)
  temporal_evolution     — legitimate change, neither supersession nor regression
  negation_artifact      — judge misread an explicit negation

Changes:
- types.ts: Verdict union (6 members); Severity gains 'info'; ResolutionKind
  extended with temporal_supersede, flag_for_review, log_timeline_change;
  JudgeVerdict.contradicts → verdict; ContradictionFinding now carries verdict;
  ProbeReport adds queries_with_any_finding + verdict_breakdown (additive).
- judge.ts: parseResolutionKind + parseVerdict guards; normalizeVerdict reads
  the new field and applies the C1 confidence floor only to verdict='contradiction'
  (the new verdicts are informational classifications, no floor). Prompt rubric
  rewritten to ask for verdict + extended severity scale.
- severity-classify.ts: 'info' joins the rank with value 0; defaultSeverityForVerdict
  maps each verdict to its baseline severity (D7 — supersession=info, regression=high,
  etc.). parseSeverity gains a fallback param so consumers can override 'low' default.
- auto-supersession.ts: classifyResolution + renderResolutionCommand handle the
  three new resolution kinds. Probe still NEVER auto-mutates — the new kinds
  render paste-ready commands or informational lines.
- cache.ts: isJudgeVerdict shape check matches the new verdict field; old v1
  rows fail the guard and treat as misses.
- runner.ts: emit predicate at cache-hit and judge-success branches changes
  from `verdict.contradicts` to `verdict.verdict !== 'no_contradiction'`.
  Without this, the new verdicts vanish from the report. Added per-verdict
  tally + queriesWithAnyFinding alongside the strict queriesWithContradiction.
- trends.ts: latest run verdict breakdown surfaces in the trend chart.

Test fixtures updated across 8 test files. All 210 eval-contradictions unit
tests pass. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): relax date-filter rule 3 when both sides dated

Lane B of the temporal-contradiction-probe wave. The v1 date pre-filter
skipped pairs whose chunk-text-extracted dates differed by >30 days as a
cost-saving heuristic. That heuristic silently killed exactly the cases the
new verdict taxonomy exists to surface — role transitions across years
(e.g. a 2017 historical record vs. a 2025 current state), MRR claims years
apart, status changes recorded over time.

Lane A1+A2 made temporal supersession explicit and cheap to classify. The
filter no longer needs to skip these pairs; the judge can label them.

Changes:
- date-filter.ts: shouldSkipForDateMismatch accepts optional effectiveDateA
  and effectiveDateB. When BOTH are non-null, returns skip=false with the new
  'both_have_effective_date' reason — the judge will see the dates via the
  (from: YYYY-MM-DD) prompt tag from Lane A1. Other rules (same-paragraph
  dual-date override, missing-date fallback) preserved verbatim and still
  run first.
- runner.ts: threads pair.{a,b}.effective_date into the date-filter call.
  Pairs that previously vanished into the skip bucket now reach the judge.

Tests (R1 IRON RULE regression suite, 6 new cases):
- both sides effective_date → not skipped
- both sides effective_date overrides >30d chunk-text rule
- rule 1 (same-paragraph dual-date) still wins over effective_date relaxation
- rule 2 (missing chunk dates) still applies when effective_date partially present
- undefined effective_dates fall through to v1 behavior (back-compat)
- empty-string effective_date treated as missing (only real dates enable the relaxation)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(cli): cost-estimate prompt + --budget-usd + Haiku routing

Lane C of the temporal-contradiction-probe wave. Three layers of cost
guardrail, all stacked:

(a) cost-estimate prompt at probe-run-time. Before the runner spends any
    tokens after a PROMPT_VERSION change, eval-suspected-contradictions
    reads the most recent persisted prompt_version from
    eval_contradictions_runs and compares. When they differ:
      - TTY: prints an upper-bound estimate + Ctrl-C window (default 10s,
        override via GBRAIN_PROBE_PROMPT_GRACE_SECONDS).
      - non-TTY: prints the estimate + auto-proceeds (autopilot path).
      - --yes override or GBRAIN_NO_PROBE_PROMPT=1: skip entirely.
    Mirrors the v0.32.7 runPostUpgradeReembedPrompt pattern.

(b) --budget-usd N hard cap (pre-existing; PreFlightBudgetError surfaces
    when the estimate alone exceeds the cap, and CostTracker halts the
    run mid-flight when cumulative cost exceeds it). Documented in the
    help text alongside (a).

(c) Judge model now routes through resolveModel() with configKey
    'models.eval.contradictions_judge', tier 'utility' (Haiku-class
    default), and env var GBRAIN_CONTRADICTIONS_JUDGE_MODEL. The legacy
    --judge CLI flag still wins as the highest-precedence override.
    Doctor's model touchpoint registry (src/commands/models.ts:50) carries
    the new key so `gbrain models` and `gbrain models doctor` surface it.

Also in this lane:
- CLI: --severity accepts 'info' (the new Severity member from Lane A2).
- CLI: --severity output shows [verdict] tag alongside slug pairs so
  operators distinguish genuine contradictions from temporal classifications.
- Human summary: prints the new queries_with_any_finding metric and the
  per-verdict breakdown table.
- Help text: explains the cost-prompt + budget-cap + model-routing
  interactions in one paragraph.

New tests (9 cases on the cost-prompt helper):
- --yes override skips
- GBRAIN_NO_PROBE_PROMPT=1 skips
- prompt_version unchanged → skips
- non-TTY auto-proceeds with stderr note
- TTY proceeds after grace
- TTY aborts on Ctrl-C
- fresh brain (no prior runs) fires the prompt
- GBRAIN_PROBE_PROMPT_GRACE_SECONDS override honored
- estimate banner contains query count + judge model + dollar amount

All 225 eval-contradictions tests + 25 model-config tests pass. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test(eval): R4/R5/R6 IRON-RULE regressions for the verdict-enum wave

Lane D of the temporal-contradiction-probe wave. The Lanes A1/A2/B/C lanes
landed the behavior; this lane pins the regressions that protect the wave
against future drift.

R4 (runner emit predicate): five new tests, one per non-no_contradiction
verdict, prove the runner.ts emit rule surfaces each one as a finding with
the correct verdict tag, and that:
  - queries_with_contradiction (Wilson-CI denominator) ONLY counts verdict
    ='contradiction' — the strict metric is preserved
  - queries_with_any_finding counts every non-no_contradiction verdict
  - verdict_breakdown tallies correctly
Plus one negative case: verdict='no_contradiction' produces zero findings.
Without R4, a future runner refactor could collapse the new verdicts back
to /dev/null and the report would silently shrink.

R5 (cache key shape): direct shape assertion on buildCacheKey output. The
key tuple is exactly 5 fields (chunk_a_hash, chunk_b_hash, model_id,
prompt_version, truncation_policy). Adding a 6th field would silently break
every operator's brain (no migration path).

R6 (contradiction severity unchanged): four tests on normalizeVerdict pin
the legacy semantics — judge-supplied severity wins (whether 'high' or
'low'), and on garbage severity input the fallback is 'medium' (per
defaultSeverityForVerdict('contradiction')) NOT 'low'. The contradiction
verdict's severity must never default to 'low', which would silently mask
genuine conflicts as cosmetic naming issues. The temporal_regression case
is included for parity (garbage → 'high' since regressions are real
investor red flags).

236 eval-contradictions tests pass (211 + 6 R4 + 1 R5 + 4 R6 + 9 cost-prompt
from Lane C).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(ci): privacy lint for docs/proposals/*.md

Captures the residual TODO from the temporal-contradiction-probe wave's
plan: prevent the bug class where an RFC lands in docs/proposals/ with
PII that should never appear in a public technical artifact. The
original RFC had to be scrubbed at force-push time (Step 0); this lint
catches the same patterns at CI time so the next one can't slip through.

Sibling to scripts/check-privacy.sh:
- check-privacy.sh: bans the literal "Wintermute" repo-wide.
- check-proposal-pii.sh: focuses on docs/proposals/*.md and the OTHER
  PII classes — personal-relationship vocabulary, private repo refs.

Design contract: the denylist names PATTERNS, not real people. Naming
specific real names (deceased relatives, therapist first names,
dealflow contacts) inside this script would leak PII into the repo
just by appearing here. The structural patterns below catch the
SURROUNDING vocabulary that always accompanies such content in
personal RFC prose. Trade-off: a future RFC that names a real person
without any contextual markers won't be caught — accepted as residual
risk handled by human review.

Patterns flagged in docs/proposals/*.md:
- garrytan/brain (private repo reference)
- trial separation, permanent separation
- couples session, couples therapist
- divorce attorney(s)
- grandmother's funeral, aunt's funeral
- wintermute (also caught by check-privacy.sh; listed here for
  proposal-scoped clarity)

Bare common words (separation, funeral) are NOT banned — only the
combined personal-context phrases. "Separation of concerns" and other
software vocabulary survives.

Wired into:
- `bun run verify` (gates every push)
- `bun run check:all`
- `bun run check:proposal-pii` (standalone)

Tests: 15 cases in test/scripts/check-proposal-pii.test.ts.
- Each pattern flagged when present, plus exit-code + stderr signal.
- Two negative cases (separation-of-concerns, funeral metaphor) prove
  the lint doesn't false-positive on legitimate software prose.
- No-proposals-dir → exit 0 (not a failure).
- Multi-hit case proves all patterns surface together with a summary
  count.
- The two test fixtures that name "Wintermute" / "WINTERMUTE" as
  sentinel literals are allowlisted in check-test-real-names.sh per
  the same meta-rule-enforcement exception as check-privacy.sh itself.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore(privacy): allowlist new privacy-guard files in check-privacy.sh

check-privacy.sh bans the literal Wintermute repo-wide. The two new files
from the v0.34 privacy lint (scripts/check-proposal-pii.sh and its test)
necessarily name the token to do their job. Same meta-rule-enforcement
exception as scripts/check-privacy.sh itself, scripts/check-test-real-names.sh,
test/recency-decay.test.ts, and the existing entries — describing what
the rule forbids requires naming it.

Without this allowlist, `bun run verify` fails on check:privacy.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v0.35.1.0)

Temporal-contradiction-probe wave — Phase 1 of the RFC at
docs/proposals/temporal-contradiction-probe.md.

Headline: the contradiction probe now classifies pairs into a 6-member
verdict enum (no_contradiction, contradiction, temporal_supersession,
temporal_regression, temporal_evolution, negation_artifact) and sees the
page-level effective_date for each chunk via a (from: YYYY-MM-DD) tag in
the prompt. The pre-judge date filter no longer skips dated wide-gap pairs,
so the role-transition class (e.g. a 2017 historical record vs. a 2025
current state) reaches the judge and gets classified as
temporal_supersession instead of vanishing into the skip bucket.

PROMPT_VERSION bumped 1 → 2 (cache fully invalidated). Three-layer cost
guardrail: TTY-only cost-estimate prompt with Ctrl-C window, --budget-usd
hard cap, Haiku-tier routing via new models.eval.contradictions_judge
config key.

Also adds a CI privacy lint (scripts/check-proposal-pii.sh) wired into
bun run verify that catches PII patterns in docs/proposals/*.md so future
RFCs can't ship with personal-context vocabulary the way this wave's
source RFC did at draft time.

Phases 2-4 deferred to follow-up RFCs per the plan.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-17 08:32:03 -07:00
committed by GitHub
co-authored by garrytan-agents Claude Opus 4.7
parent 2504abe47f
commit dd1cc121d8
35 changed files with 1950 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+69
View File
@@ -2,6 +2,74 @@
All notable changes to GBrain will be documented in this file.
## [0.35.3.1] - 2026-05-15
**The contradiction probe stops crying wolf on time. Six-member verdict enum + page-level date in the prompt + a new privacy lint for proposals.**
`gbrain eval suspected-contradictions` used to treat every claim as timeless. On a brain with conversation transcripts, dated meeting pages, evolving takes, and historical entity records, that meant ~60% of residual HIGH findings were temporal false positives: a status change recorded as "trial" in April and "confirmed" in May got flagged as a contradiction; a role transition recorded in 2017 vs. an updated role in 2025 got flagged. None of those are bugs in the brain; they're features of a brain that records history.
The judge now sees the page-level `effective_date` for each chunk via a `(from: YYYY-MM-DD)` tag in the prompt, and it classifies into a six-member verdict taxonomy instead of `contradicts: true/false`. The same wave adds a CI privacy lint that catches PII patterns in `docs/proposals/*.md` so future RFCs can't ship with personal-context vocabulary the way this wave's source RFC did at draft time.
### What you can now do
**Get verdict-aware findings out of the probe.** The judge now returns one of six verdicts: `no_contradiction`, `contradiction` (genuine conflict at the same point in time), `temporal_supersession` (newer claim updates the older), `temporal_regression` (a metric or status went backwards), `temporal_evolution` (legitimate change over time), or `negation_artifact` (judge misread an explicit negation). `gbrain eval suspected-contradictions trend` shows the per-verdict breakdown in the chart; `gbrain eval suspected-contradictions review` prints a `[verdict]` tag inline with each finding. The Wilson-CI denominator stays anchored to the strict `verdict === 'contradiction'` count so the headline metric is preserved. A new `queries_with_any_finding` sibling metric counts queries that produced any non-`no_contradiction` verdict.
**Skip the >30d skip rule when both sides have dates.** The pre-judge date filter previously dropped pairs whose chunk-text-extracted dates differed by more than 30 days. That heuristic silently killed exactly the cases the new verdicts exist to surface (the 2017 vs. 2025 role-transition class). The filter now passes those pairs through when both sides have a non-null page-level `effective_date`, so the judge gets to classify them as `temporal_supersession` instead of the pair vanishing into the skip bucket. Same-paragraph dual-date override and missing-date fallback rules are preserved verbatim.
**Cap the cost of the post-bump re-judge.** The `PROMPT_VERSION` change invalidates the entire judge cache (the prompt shape changed, old verdicts no longer apply). Before that re-judge spends any tokens, the runner prints an upper-bound cost estimate and waits 10 seconds in TTY for Ctrl-C; set `GBRAIN_NO_PROBE_PROMPT=1` to skip or `GBRAIN_PROBE_PROMPT_GRACE_SECONDS=0` to proceed immediately. Non-TTY (autopilot) auto-proceeds with a stderr note. `--budget-usd N` hard-caps the run when cumulative cost exceeds the cap. The judge model now resolves through `models.eval.contradictions_judge` (defaults to Haiku tier), so `gbrain config set models.eval.contradictions_judge` works the same way as every other model config key. `gbrain models doctor` surfaces the new touchpoint.
**Block PII in `docs/proposals/*.md` at CI time.** A new `scripts/check-proposal-pii.sh` (wired into `bun run verify` and `bun run check:all`) catches the specific PII classes that surfaced in past RFC drafts: private repo references (`garrytan/brain`), personal-relationship vocabulary (`trial separation`, `couples session`, `divorce attorney`, etc.), death-context phrases, and the literal `Wintermute` agent name. Bare common words (`separation`, `funeral`) are intentionally not banned. The denylist names patterns rather than real names, so the script's own source doesn't re-introduce PII into the repo.
### Itemized changes
- `SearchResult` interface gains optional `effective_date` + `effective_date_source` fields. Eight SQL projection sites updated (3 in postgres-engine, 5 in pglite-engine — `searchKeyword`, `searchKeywordChunks`, `searchVector` across both engines). `rowToSearchResult` normalizes both Postgres `Date` and PGLite string returns to `YYYY-MM-DD`. Three-state read pattern (undefined when not selected, null when selected-but-empty, string when populated).
- `PairMember` (`src/core/eval-contradictions/types.ts`) gets required `effective_date: string | null` and `effective_date_source: string | null`. `runner.ts` threads the fields through `searchResultToMember` and `takeToMember`; the judge call passes them via the new optional fields on `JudgeInput.a/b`.
- `buildJudgePrompt` now emits `Statement A (from: YYYY-MM-DD)` when `effective_date` is non-null, else `(date unknown)`. Prompt instructions explain the tag and the new verdict semantics.
- `PROMPT_VERSION` bumped `'1' → '2'`. Cache-key tuple shape unchanged (still 5 fields); old rows naturally miss on first run. Cache type guard updated to check `verdict` instead of `contradicts`.
- `JudgeVerdict.contradicts: boolean` replaced with `verdict: Verdict` (6-member union). `Severity` extended with `'info'`. `ResolutionKind` extended with `temporal_supersede`, `flag_for_review`, `log_timeline_change`. `normalizeVerdict` applies the C1 confidence floor only to `verdict === 'contradiction'` (other verdicts are informational classifications, no floor). `defaultSeverityForVerdict` maps each verdict to its baseline severity (D7 map: supersession→info, regression→high, evolution→info, negation_artifact→low, contradiction→medium).
- `runner.ts` emit predicate changes from `if (verdict.contradicts)` to `if (verdict.verdict !== 'no_contradiction')`. Per-verdict tally feeds `ProbeReport.verdict_breakdown`. `queries_with_contradiction` (strict, Wilson-CI denominator) and `queries_with_any_finding` (broad) tracked separately.
- `auto-supersession.ts`: `classifyResolution` and `renderResolutionCommand` extended for new verdicts. Probe still NEVER auto-mutates — new kinds render paste-ready commands (`gbrain takes supersede --since <date>`) or informational lines (`# flag_for_review:`). The `auto-supersession.ts:4` "NEVER auto-applies" invariant preserved verbatim.
- `date-filter.ts`: `shouldSkipForDateMismatch` accepts optional `effectiveDateA` and `effectiveDateB`. When both are non-null, returns `skip=false` with new `'both_have_effective_date'` reason. Other rules preserved.
- `src/commands/eval-suspected-contradictions.ts`: new `--budget-usd N` hard cap (was pre-existing for runtime; help text now explains it), new cost-estimate prompt wired via `src/core/eval-contradictions/cost-prompt.ts`. `--severity` accepts `info`. `--severity` output shows `[verdict]` tag. Judge model routes through `resolveModel({configKey: 'models.eval.contradictions_judge', tier: 'utility', envVar: 'GBRAIN_CONTRADICTIONS_JUDGE_MODEL'})`. Human summary and trend chart show the per-verdict breakdown.
- `src/commands/models.ts` registers `models.eval.contradictions_judge` as a tracked per-task model key so `gbrain models` and `gbrain models doctor` surface it.
- `scripts/check-proposal-pii.sh` + denylist (structural patterns only, no real names) + 15-case test in `test/scripts/check-proposal-pii.test.ts`. Wired into `bun run verify`, `bun run check:all`, and as the new `bun run check:proposal-pii` standalone target. Two privacy-guard test fixtures naming `Wintermute` allowlisted in `scripts/check-test-real-names.sh`; the new privacy-guard scripts themselves allowlisted in `scripts/check-privacy.sh`.
- Six IRON-RULE regression test suites pin the wave's invariants: R1 date-filter rule 3 relaxation preserves rules 1+2 (6 cases), R2 cache invalidation on `PROMPT_VERSION` bump, R3 verdict-enum migration (compile-time via `tsc --noEmit`), R4 runner emit predicate fires for every non-`no_contradiction` verdict (6 cases), R5 cache key tuple stays 5 fields, R6 contradiction severity unchanged (4 cases). Plus 9 cases on the cost-prompt helper.
- The source RFC (`docs/proposals/temporal-contradiction-probe.md`) and PR title/body/commit/branch-name all scrubbed of PII at draft time. The new CI lint prevents the next one from slipping through.
### Phase 2/3/4 deferred (per the plan)
The RFC proposed four phases. This wave ships Phase 1 only. Deferred:
- Phase 2 (structured claim extraction + `gbrain trajectory` view): blocked on a separate RFC that maps the existing `extract_facts` + `consolidate` cycle pipeline before extending the facts table. Codex review of the original plan caught factual errors about this pipeline; the deferral is deliberate.
- Phase 3 (auto-write `valid_until` from `temporal_supersession` verdicts): would violate `auto-supersession.ts:4`'s "NEVER auto-applies" invariant. Reframed as paste-ready commands in Phase 1.
- Phase 4 (founder scorecard / Argus integration): needs concrete Argus spec.
## To take advantage of v0.35.3.1
`gbrain upgrade` should do this automatically. The wave is schema-compatible (no migrations), so `gbrain apply-migrations` is a no-op.
1. **Re-run the contradiction probe to see the new verdicts:**
```bash
gbrain eval suspected-contradictions run --budget-usd 5
```
Expect a one-time cost-estimate prompt + 10-second Ctrl-C window since `PROMPT_VERSION` changed; the cache is invalidated. Non-TTY runs auto-proceed.
2. **(Optional) Pin the judge model:**
```bash
gbrain config set models.eval.contradictions_judge anthropic:claude-haiku-4-5
gbrain models doctor
```
3. **(Optional) Adjust the cost-prompt grace window** for CI / autopilot:
```bash
export GBRAIN_NO_PROBE_PROMPT=1 # skip entirely
# or
export GBRAIN_PROBE_PROMPT_GRACE_SECONDS=0 # auto-proceed in TTY
```
4. **If `bun run verify` fails on `check:proposal-pii`,** the lint caught PII in `docs/proposals/*.md`. Replace with generic placeholders (alice-example, acme-corp, fund-a) per CLAUDE.md "Privacy rule: scrub real names from public docs."
5. **If any step fails or the numbers look wrong,** please file an issue:
https://github.com/garrytan/gbrain/issues with:
- output of `gbrain doctor`
- contents of `~/.gbrain/upgrade-errors.jsonl` if it exists
- which step broke
## [0.35.3.0] - 2026-05-15
**Fix wave: 19 stale community PRs land as one bisect-friendly PR with the architectural fix none of them surfaced.**
@@ -46,6 +114,7 @@ Two real bugs, both in shipping code on master since v0.28+, both fixed at the a
4. If you use remote-source git clones (`gbrain config get sources` shows any with a remote URL), they'll start working again on the next `gbrain sync`. Pre-fix, every clone of a remote-source repository was silently failing with git exit 129.
If `gbrain doctor` flags anything after upgrade, please file an issue with the doctor output. This was a 7-month silent bug — the doctor's structural guards should catch the next one before it ships.
## [0.35.1.1] - 2026-05-16
**Fix wave: `gbrain eval longmemeval` actually runs against the public _s split.**
VERSION
+1 -1
View File
@@ -1 +1 @@
0.35.3.0
0.35.3.1
docs/proposals/temporal-contradiction-probe.md
+213
View File
@@ -0,0 +1,213 @@
# Proposal: Temporal Axis for Contradiction Probe
**Status:** Report / RFC
**Date:** 2026-05-14
**Context:** A large production run of `gbrain eval suspected-contradictions` surfaced ~115 HIGH findings. Walking through them by hand exposed a structural limitation in the probe.
## The Problem
The contradiction probe (`gbrain eval suspected-contradictions`) treats all claims as timeless. When two chunks make conflicting statements, the judge flags a contradiction regardless of whether both statements were true at their respective points in time.
This worked fine when the brain was mostly static wiki pages. It breaks now that the brain contains:
- Conversation transcripts with claims that were true when spoken
- Meeting pages capturing what people said on specific dates
- Takes that evolve (a founder's ARR claim in January vs. July)
- Status records that supersede each other (a state moves from "trial" to "confirmed")
The probe can't distinguish "this changed" from "this is wrong."
## Bug-class examples (synthetic placeholders)
### 1. Temporal Evolution (False Positive)
```
Finding: HIGH
A: [daily/transcripts/2026/2026-04-28] "status: trial"
B: [meetings/2026-05-07-session] "status: confirmed"
Axis: Whether status is trial or confirmed
```
Both are correct as of their respective dates. April 28: trial. May 7: confirmed. The probe flags this because it has no concept of "this claim was valid from X until Y." The May 7 record didn't make the April 28 transcript wrong; it recorded a change.
### 2. Negation Parsing (False Positive)
```
Finding: HIGH
A: [people/alice-example] "person traveled to city-a for alice-example's event — NOT bob-example's event"
B: [meetings/2026-05-11-context] mentions of bob-example's event in city-b
Axis: Whose event the city-a trip was for
```
The disambiguation fact contains "NOT bob-example's event" as an explicit negation. The judge reads "bob-example's event" as a positive claim and flags it against the alice-example context. The data is correct; the probe can't parse negation.
### 3. Role Changes (True Positive That Needs Time Awareness)
```
Finding: HIGH
A: [sources/notes/2017-03-28] advisor-example: "Partner, venture-firm-a"
B: [people/advisor-example] advisor-example: "Senior Policy Advisor, gov-org-b"
```
Both true at their respective times. 2017: partner at venture-firm-a. 2025: gov-org-b advisor. The current probe correctly flags this as a contradiction, but the resolution should be "superseded by time" not "one side is wrong." The 2017 note isn't wrong; it's a historical record.
## Scenario #1: Founder Tracking (the big one)
This is the use case that makes a time axis transformative rather than incremental.
The brain holds hundreds of company pages and thousands of meeting pages. Founders make claims:
- "We're at $50K MRR" (January OH)
- "We hit $200K MRR" (April OH)
- "We're at $150K MRR" (July OH — what happened?)
Today the probe would flag January vs. April as a contradiction. The real signal is April vs. July: **a claimed metric went backwards.** That's not a data quality issue; that's intelligence.
What a time-aware probe could surface:
**Claim trajectory tracking:**
```
Company: Acme Corp
2026-01: "$50K MRR" (source: OH transcript)
2026-04: "$200K MRR" (source: OH transcript)
2026-07: "$150K MRR" (source: OH transcript) ← REGRESSION DETECTED
2026-07: "$2M ARR" (source: investor update) ← INCONSISTENT WITH MRR
```
**Prediction vs. outcome:**
```
Founder: Jane Doe (Acme Corp)
2026-01: "We'll hit $1M ARR by June" (source: batch kickoff)
2026-06: Actual ARR: $400K (source: investor update)
→ Prediction accuracy: 40%
→ Pattern: consistently 2-3x optimistic on timeline
```
**Narrative consistency:**
```
Founder: John Smith (WidgetCo)
2026-01: "Our moat is proprietary data" (source: interview)
2026-03: "We're pivoting to an API-first model" (source: OH)
2026-06: "Our moat is network effects" (source: Demo Day)
→ Moat narrative changed 3x in 6 months — flag for review
```
This isn't adversarial. It's the kind of pattern an experienced operator notices intuitively across hundreds of conversations. GBrain can make it systematic.
## Scenario #2: Event Disambiguation
Two distinct events within a short window can conflate during ingestion because the probe has no temporal frame to say "event A is a different event from event B."
Time-aware facts would store (synthetic placeholders):
```
fact: "alice-example milestone" valid_from: 2026-04-15 valid_until: 2026-04-15
fact: "alice-example event in city-a" valid_from: 2026-04-17 valid_until: 2026-04-19
fact: "bob-example milestone" valid_from: 2026-05-04 valid_until: 2026-05-04
fact: "bob-example event in city-b" valid_from: 2026-05-12 valid_until: 2026-05-12
```
The probe should recognize these as two distinct events with non-overlapping time windows, not as contradictions about "whose event."
## Scenario #3: Role and Status Changes
People change roles. Companies change status. The brain records history. Synthetic examples representative of the cases observed in production:
- advisor-example: venture-firm-a partner (2019) → gov-org-b advisor (2025)
- investor-example: fund-a partner → fund-b CEO (2023)
- agent-fork: provider restriction event (2026-04-04) ≠ shutdown
- fund-c: "interesting fund" (early) → "declined" (later) → "losing confidence" (latest)
All of these are correct historical records. The probe should classify them as **temporal supersession** rather than **contradiction.**
## Scenario #4: Decision Tracking
Multi-step decisions that supersede earlier framings example (synthetic):
```
2026-04-24: "status: trial" (initial framing)
2026-04-25: "status: in progress" (confirmed, no longer "trial")
2026-05-07: "status: finalized" (session record)
2026-05-11: follow-up actions taken
```
Each step supersedes the previous. A time-aware probe would show the **evolution chain** rather than flagging each pair as a contradiction.
## What Exists Today
The probe already has some temporal infrastructure:
1. **`date-filter.ts`** — `shouldSkipForDateMismatch()` pre-filters pairs, but only checks whether dates are "too far apart" (a coarse heuristic). It doesn't reason about which claim is newer or whether one supersedes the other.
2. **`auto-supersession.ts`** — proposes resolution commands, checks `since_date` on takes. But this is post-hoc (after the judge flags a contradiction). The judge itself doesn't see dates.
3. **Facts table** has `valid_from` and `valid_until` columns. These exist but are sparsely populated and not used by the probe.
4. **Takes table** has `since_date`. Also sparsely populated.
## What Would Need to Change
### Phase 1: Judge prompt enhancement (smallest change, biggest impact)
Pass the source dates to the judge. The current judge prompt shows two text chunks and asks "are these contradictory?" If it also showed:
```
Statement A (from: 2026-04-28):
"status: trial"
Statement B (from: 2026-05-07):
"status: confirmed"
```
The judge could output a `temporal_supersession` verdict instead of `contradiction`. New verdict taxonomy:
- `no_contradiction` — statements are compatible
- `contradiction` — genuinely conflicting claims at the same point in time
- `temporal_supersession` — newer claim updates/replaces older claim (not an error)
- `temporal_regression` — a metric or status went backwards (potential signal)
- `temporal_evolution` — legitimate change over time, neither supersession nor regression
- `negation_artifact` — one side contains an explicit negation the judge misread
### Phase 2: Claim trajectory view (new command)
```bash
gbrain eval trajectory "Acme Corp MRR"
gbrain eval trajectory "advisor-example role"
gbrain eval trajectory "deal-x status"
```
Pull all time-stamped claims about an entity+attribute, sort chronologically, detect:
- Regressions (metric went down)
- Contradictions within the same time window
- Prediction vs. outcome gaps
- Narrative drift (moat story changed 3x)
### Phase 3: Automatic `valid_from`/`valid_until` population
During `extract_facts`, infer temporal bounds from source context:
- Meeting page dated 2026-04-28 → claims valid_from 2026-04-28
- Takes from transcripts → valid_from = transcript date
- Imported notes → valid_from = note date
- Entity pages with no date → valid_from = page created date (weakest signal)
### Phase 4: Founder scorecard
For founders specifically, a temporal probe could generate:
- **Claim accuracy score** — what they predicted vs. what happened
- **Consistency score** — how stable their narrative is over time
- **Growth trajectory** — whether the numbers are actually moving
- **Red flag detector** — metrics going backwards, story changing, timeline slipping
## Recommendation
Start with Phase 1. The judge prompt change is small. It immediately eliminates the temporal false positives (which were a majority of the residual HIGH findings in the production audit) and gives the probe a new vocabulary for time-aware reasoning.
Phase 2 (trajectory view) is the one that would change how operators use the brain for founder evaluation. Worth scoping as a standalone feature.
Phases 3–4 are downstream and can wait.
## Appendix: Production probe stats (2026-05-14)
- ~107K pages, ~257K chunks
- Previous run: ~115 HIGH findings across 50 queries
- After manual resolution: ~25 residual findings
- Of those ~25: roughly two-thirds temporal false positives, the remainder probe artifacts (self-contradiction, negation parsing)
- 0 genuine data contradictions remained on the queries tested
- Fresh targeted probe on a representative entity-role query: 0 contradictions (was 14+ before fixes)
package.json
+4 -3
View File
@@ -1,6 +1,6 @@
{
"name": "gbrain",
"version": "0.35.3.0",
"version": "0.35.3.1",
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
"type": "module",
"main": "src/core/index.ts",
@@ -37,11 +37,11 @@
"build:pglite-snapshot": "bun run scripts/build-pglite-snapshot.ts",
"test": "bash scripts/run-unit-parallel.sh",
"test:full": "bun run verify && bash scripts/run-unit-parallel.sh && bun run test:slow && ([ -n \"$DATABASE_URL\" ] && bash scripts/run-e2e.sh || echo '[test:full] skipped E2E (no DATABASE_URL); run docker-compose -f docker-compose.ci.yml up + bun run test:e2e to include' 1>&2)",
"verify": "bun run check:privacy && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run typecheck",
"verify": "bun run check:privacy && bun run check:proposal-pii && bun run check:test-names && bun run check:jsonb && bun run check:source-id-projection && bun run check:progress && bun run check:test-isolation && bun run check:wasm && bun run check:admin-build && bun run check:admin-scope-drift && bun run check:cli-exec && bun run check:system-of-record && bun run check:eval-glossary && bun run typecheck",
"check:system-of-record": "scripts/check-system-of-record.sh",
"check:admin-scope-drift": "scripts/check-admin-scope-drift.sh",
"check:cli-exec": "scripts/check-cli-executable.sh",
"check:all": "scripts/check-privacy.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh",
"check:all": "scripts/check-privacy.sh && scripts/check-proposal-pii.sh && scripts/check-test-real-names.sh && scripts/check-jsonb-pattern.sh && scripts/check-source-id-projection.sh && scripts/check-progress-to-stdout.sh && scripts/check-no-legacy-getconnection.sh && scripts/check-test-isolation.sh && scripts/check-trailing-newline.sh && scripts/check-wasm-embedded.sh && scripts/check-exports-count.sh && scripts/check-admin-build.sh && scripts/check-admin-scope-drift.sh && scripts/check-cli-executable.sh",
"check:wasm": "scripts/check-wasm-embedded.sh",
"check:newlines": "scripts/check-trailing-newline.sh",
"test:e2e": "bash scripts/run-e2e.sh",
@@ -55,6 +55,7 @@
"check:jsonb": "scripts/check-jsonb-pattern.sh",
"check:source-id-projection": "scripts/check-source-id-projection.sh",
"check:privacy": "scripts/check-privacy.sh",
"check:proposal-pii": "scripts/check-proposal-pii.sh",
"check:eval-glossary": "scripts/check-eval-glossary-fresh.sh",
"check:test-names": "scripts/check-test-real-names.sh",
"check:progress": "scripts/check-progress-to-stdout.sh",
scripts/check-privacy.sh
+6
View File
@@ -133,6 +133,12 @@ ALLOW_LIST=(
# (Wintermute, Hermes, etc) inside its BANNED_NAMES + ALLOWLIST arrays.
# Same meta-rule-enforcement exception as scripts/check-privacy.sh itself.
'scripts/check-test-real-names.sh'
# v0.34 / Lane CI: scripts/check-proposal-pii.sh and its test list the
# banned literal as part of the structural denylist they enforce against
# docs/proposals/*.md. Same meta-rule-enforcement exception as the two
# entries above — describing what the rule forbids requires naming it.
'scripts/check-proposal-pii.sh'
'test/scripts/check-proposal-pii.test.ts'
# v0.32.3.0: the functional-area-resolver skill's behavior-contract
# section describes the privacy guarantees the skill preserves and
# references the banned literals while doing so (line 306). Same
scripts/check-proposal-pii.sh
Executable
+166
View File
@@ -0,0 +1,166 @@
#!/bin/bash
#
# check-proposal-pii.sh — privacy guard for `docs/proposals/*.md`.
#
# Sibling to check-privacy.sh: that script bans the `Wintermute` literal
# everywhere. This one focuses on `docs/proposals/*.md` and the OTHER PII
# classes that have surfaced in past RFC drafts — personal-relationship
# vocabulary, private repo references, etc.
#
# Why two scripts: the patterns this lint flags would be too noisy if
# applied repo-wide (e.g. a test fixture mentioning "trial" is fine).
# Restricting to `docs/proposals/` keeps the lint surgical — proposals are
# public-facing RFC documents that should never contain personal context,
# so the false-positive rate is near zero.
#
# Design note: the denylist names PATTERNS, not real people. Specific
# real names (deceased relatives, therapist names, dealflow contacts)
# would leak PII into the repo just by appearing in this script's
# denylist. The structural patterns below catch the SURROUNDING context
# of personal-event prose. The trade-off: a future RFC that names a real
# person without any of the contextual markers won't be caught — that's
# accepted as a residual risk handled by human review.
#
# Usage:
# scripts/check-proposal-pii.sh # scan working tree
# scripts/check-proposal-pii.sh --staged # scan git staged index
# scripts/check-proposal-pii.sh --help
#
# Exit codes:
# 0 clean
# 1 PII pattern found
# 2 setup error
set -euo pipefail
REPO_ROOT="$(git rev-parse --show-toplevel 2>/dev/null || pwd)"
PROPOSALS_DIR="$REPO_ROOT/docs/proposals"
# Structural patterns. One per line. Matched case-insensitively, fixed-string
# (no regex). Comments start with #. Blank lines OK.
#
# IMPORTANT — design contract: this list MUST NOT contain real personal
# names (deceased relatives, therapist first names, dealflow contacts).
# Naming those would leak PII into scripts/. The patterns below catch the
# SURROUNDING VOCABULARY that always accompanies such content in personal
# RFC prose. Maintainers extending this list: prefer adding a phrase that
# captures the context (e.g. `couples session`) rather than a specific
# person's name.
read -r -d '' PATTERNS <<'EOF' || true
# Private repo references (zero false-positive risk)
garrytan/brain
# Personal relationship vocabulary (extremely unlikely in technical RFCs)
trial separation
permanent separation
couples session
couples therapist
divorce attorney
divorce attorneys
# Death/funeral vocabulary in personal contexts (combined phrases — bare
# "funeral" alone would false-positive in legitimate metaphorical use)
grandmother's funeral
grandmother funeral
aunt's funeral
aunt funeral
# Private agent / fork name (also enforced repo-wide by check-privacy.sh
# but listed here for proposal-scoped clarity)
wintermute
EOF
usage() {
cat <<EOF
scripts/check-proposal-pii.sh — privacy guard for docs/proposals/*.md.
USAGE:
scripts/check-proposal-pii.sh Scan all proposal files.
scripts/check-proposal-pii.sh --staged Scan only staged proposal files.
scripts/check-proposal-pii.sh --help Show this message.
Flags personal-context vocabulary (e.g. "trial separation", "couples
session", private repo references) inside docs/proposals/*.md. Use
generic placeholders (alice-example, acme-corp, fund-a) in proposals.
See CLAUDE.md "Privacy rule: scrub real names from public docs" for
the canonical name-mapping table.
Sibling to scripts/check-privacy.sh which enforces the "Wintermute"
ban repo-wide; this script catches the broader PII classes that
appeared in past RFC drafts and were corrected at landing time.
Exit codes: 0 clean, 1 pattern found, 2 setup error.
EOF
}
MODE=working
for arg in "$@"; do
case "$arg" in
--staged) MODE=staged ;;
--help|-h) usage; exit 1 ;;
*)
echo "Unknown argument: $arg" >&2
usage >&2
exit 2
;;
esac
done
if [ ! -d "$PROPOSALS_DIR" ]; then
# No proposals dir yet — nothing to lint. Not a failure.
exit 0
fi
# Build the file list. Staged mode filters git's staged set down to
# docs/proposals/*.md; working mode globs the directory directly.
if [ "$MODE" = staged ]; then
if ! command -v git >/dev/null 2>&1; then
echo "check-proposal-pii: git not found" >&2
exit 2
fi
FILES=$(git diff --cached --name-only --diff-filter=ACMR 2>/dev/null \
| grep -E '^docs/proposals/.+\.md$' || true)
else
FILES=$(find "$PROPOSALS_DIR" -maxdepth 1 -type f -name '*.md' 2>/dev/null \
| sed "s|^$REPO_ROOT/||")
fi
if [ -z "$FILES" ]; then
exit 0
fi
FOUND=0
# Iterate patterns; for each non-comment line, scan the file list.
while IFS= read -r raw_line; do
# Strip leading/trailing whitespace.
pat="${raw_line#"${raw_line%%[![:space:]]*}"}"
pat="${pat%"${pat##*[![:space:]]}"}"
# Skip empty and comment lines.
[ -z "$pat" ] && continue
case "$pat" in '#'*) continue ;; esac
while IFS= read -r file; do
[ -z "$file" ] && continue
full="$REPO_ROOT/$file"
[ ! -f "$full" ] && continue
# Fixed-string (-F), case-insensitive (-i), with line numbers (-n).
if matches=$(grep -nFi -- "$pat" "$full" 2>/dev/null); then
if [ -n "$matches" ]; then
echo "[check-proposal-pii] PII pattern in $file:" >&2
echo " pattern: $pat" >&2
echo "$matches" | sed 's|^| |' >&2
FOUND=$((FOUND + 1))
fi
fi
done <<< "$FILES"
done <<< "$PATTERNS"
if [ "$FOUND" -gt 0 ]; then
echo "" >&2
echo "[check-proposal-pii] $FOUND PII pattern hit(s) in docs/proposals/*.md." >&2
echo "[check-proposal-pii] See CLAUDE.md 'Privacy rule: scrub real names from public docs'." >&2
echo "[check-proposal-pii] Use generic placeholders: alice-example, acme-corp, fund-a, etc." >&2
exit 1
fi
exit 0
scripts/check-test-real-names.sh
+2
View File
@@ -55,6 +55,8 @@ ALLOWLIST=(
"test/writer.test.ts:garry@ycombinator.com" # user's own email — CLAUDE.md rule does not apply
"test/integrations.test.ts:Wintermute" # regex pattern in personal-info filter test (structural)
"test/recency-decay.test.ts:Wintermute" # regression-prevention test asserting wintermute is absent (structural)
"test/scripts/check-proposal-pii.test.ts:Wintermute" # privacy-guard test asserting docs/proposals/ rejects wintermute (structural; same meta-rule exception as check-privacy.sh)
"test/scripts/check-proposal-pii.test.ts:WINTERMUTE" # case-insensitive sentinel literal for the same privacy-guard test
"test/serve-stdio-lifecycle.test.ts:Hermes" # comment naming a downstream-agent scenario — pre-existing, low signal
"test/extract.test.ts:Hermes" # markdown-link extraction test fixture — pre-existing, ambiguous (Greek god vs fork)
)
src/commands/eval-suspected-contradictions.ts
+76 -14
View File
@@ -19,6 +19,7 @@
import { readFileSync } from 'node:fs';
import type { BrainEngine } from '../core/engine.ts';
import { resolveModel } from '../core/model-config.ts';
import {
PreFlightBudgetError,
runContradictionProbe,
@@ -28,6 +29,7 @@ import {
bucketBySeverity,
compareSeverityDesc,
} from '../core/eval-contradictions/severity-classify.ts';
import { maybePromptForCostBeforeProbe } from '../core/eval-contradictions/cost-prompt.ts';
import type {
ContradictionFinding,
Severity,
@@ -40,7 +42,13 @@ interface ParsedFlags {
query?: string;
fromCapture?: boolean;
topK: number;
judge: string;
/**
* v0.34 / Lane C: now optional. When unset, runRun routes through
* resolveModel({configKey: 'models.eval.contradictions_judge', tier: 'utility'})
* so the user's config keys + tier defaults govern. When set, the CLI flag
* wins per resolveModel's 6-tier precedence chain.
*/
judge?: string;
limit?: number;
budgetUsd: number;
output?: string;
@@ -77,7 +85,8 @@ function parseFlags(args: string[]): ParsedFlags {
const f: ParsedFlags = {
sub,
topK: 5,
judge: 'anthropic:claude-haiku-4-5',
// judge intentionally undefined here — resolved in runRun via resolveModel
// so config keys + tier defaults govern. CLI --judge flag wins when set.
budgetUsd: isTty ? 5 : 1,
maxPairChars: 1500,
sampling: 'deterministic',
@@ -119,8 +128,9 @@ function parseFlags(args: string[]): ParsedFlags {
else if (arg === '--days') f.days = Number.parseInt(next(), 10);
else if (arg === '--severity') {
const v = next();
if (v !== 'low' && v !== 'medium' && v !== 'high') {
throw new Error('--severity must be low|medium|high');
// v0.34 / Lane A2: 'info' joins the rank as a valid severity.
if (v !== 'info' && v !== 'low' && v !== 'medium' && v !== 'high') {
throw new Error('--severity must be info|low|medium|high');
}
f.severity = v;
}
@@ -136,7 +146,9 @@ function printHelp(): void {
console.error(`Usage:
gbrain eval suspected-contradictions [run]
[--queries-file FILE.jsonl | --query "..." | --from-capture]
[--top-k N=5] [--judge MODEL=claude-haiku-4-5]
[--top-k N=5] [--judge MODEL] (default routes via resolveModel →
models.eval.contradictions_judge →
utility-tier (Haiku) fallback)
[--limit N] [--budget-usd N] [--output FILE]
[--max-pair-chars N=1500] [--sampling deterministic|score-first]
[--no-cache] [--refresh-cache] [--json] [--yes]
@@ -144,11 +156,19 @@ function printHelp(): void {
gbrain eval suspected-contradictions trend [--days N=30] [--json]
gbrain eval suspected-contradictions review
[--severity low|medium|high] [--since YYYY-MM-DD]
[--severity info|low|medium|high] [--since YYYY-MM-DD]
The probe samples top-K retrieval pairs and asks an LLM judge whether
any pair contradicts on a factual claim relevant to the query. Outputs
JSON (stable schema_version: 1) and a human summary.
The probe samples top-K retrieval pairs and asks an LLM judge to classify
each pair as one of: no_contradiction, contradiction, temporal_supersession,
temporal_regression, temporal_evolution, negation_artifact. Outputs JSON
(stable schema_version: 1) and a human summary with per-verdict breakdown.
Cost guardrails:
- When PROMPT_VERSION changes, the runner prints a cost estimate and waits
10s in TTY for Ctrl-C (auto-proceeds non-TTY). Skip with GBRAIN_NO_PROBE_PROMPT=1.
- --budget-usd N halts the run when cumulative cost exceeds the cap.
- --judge MODEL overrides the resolveModel chain; pair with --yes when
automating.
`);
}
@@ -221,10 +241,36 @@ async function runRun(engine: BrainEngine, f: ParsedFlags): Promise<void> {
process.exit(1);
}
// v0.34 / Lane C: route the judge model through resolveModel so the user's
// models.eval.contradictions_judge config key + Haiku-tier default + global
// models.default override + env var all compose correctly. The --judge CLI
// flag still wins as the highest-precedence override.
const judgeModel = await resolveModel(engine, {
cliFlag: f.judge,
configKey: 'models.eval.contradictions_judge',
tier: 'utility',
envVar: 'GBRAIN_CONTRADICTIONS_JUDGE_MODEL',
fallback: 'anthropic:claude-haiku-4-5',
});
console.error(
`Contradiction probe: ${queries.length} queries, top-${f.topK}, judge=${f.judge}, budget=$${f.budgetUsd.toFixed(2)}.`,
`Contradiction probe: ${queries.length} queries, top-${f.topK}, judge=${judgeModel}, budget=$${f.budgetUsd.toFixed(2)}.`,
);
// v0.34 / Lane C: cost-estimate prompt — TTY-only Ctrl-C window before
// the runner spends any tokens when PROMPT_VERSION changed since the last
// persisted run. Non-TTY and env-skip paths auto-proceed.
const promptResult = await maybePromptForCostBeforeProbe({
engine,
queryCount: queries.length,
topK: f.topK,
judgeModel,
yesOverride: f.yes,
});
if (promptResult.kind === 'abort') {
process.exit(0); // intentional Ctrl-C — not an error
}
// Refresh-cache: sweep before run so the cache misses on this pass.
if (f.refreshCache) {
const swept = await engine.sweepContradictionCache();
@@ -235,7 +281,7 @@ async function runRun(engine: BrainEngine, f: ParsedFlags): Promise<void> {
const out = await runContradictionProbe({
engine,
queries,
judgeModel: f.judge,
judgeModel,
topK: f.topK,
sampling: f.sampling,
budgetUsd: f.budgetUsd,
@@ -254,11 +300,23 @@ async function runRun(engine: BrainEngine, f: ParsedFlags): Promise<void> {
lines.push(``);
lines.push(`Results: ${r.queries_evaluated} queries, top-${r.top_k} each, judge=${r.judge_model}`);
lines.push(` Queries with >=1 contradiction: ${r.queries_with_contradiction} / ${r.queries_evaluated} (${pct(r.queries_with_contradiction / Math.max(1, r.queries_evaluated))}%)`);
// v0.34 / Lane A2: broader finding count alongside the strict contradiction count.
lines.push(` Queries with >=1 finding (any verdict): ${r.queries_with_any_finding} / ${r.queries_evaluated}`);
lines.push(` Wilson CI 95%: ${pct(r.calibration.wilson_ci_95.lower)}–${pct(r.calibration.wilson_ci_95.upper)}%`);
if (r.calibration.small_sample_note) {
lines.push(` Note: ${r.calibration.small_sample_note}`);
}
lines.push(` Total contradictions flagged: ${r.total_contradictions_flagged}`);
lines.push(` Total findings flagged: ${r.total_contradictions_flagged}`);
// v0.34 / Lane A2: per-verdict breakdown surfaces what kinds of finding
// dominated the run — distinguishes temporal noise from genuine conflicts.
const vb = r.verdict_breakdown;
lines.push(` Verdict breakdown:`);
lines.push(` contradiction: ${vb.contradiction}`);
lines.push(` temporal_supersession: ${vb.temporal_supersession}`);
lines.push(` temporal_regression: ${vb.temporal_regression}`);
lines.push(` temporal_evolution: ${vb.temporal_evolution}`);
lines.push(` negation_artifact: ${vb.negation_artifact}`);
lines.push(` no_contradiction: ${vb.no_contradiction}`);
lines.push(` Judge errors: ${r.judge_errors.total} (parse_fail=${r.judge_errors.parse_fail} timeout=${r.judge_errors.timeout} http_5xx=${r.judge_errors.http_5xx} refusal=${r.judge_errors.refusal})`);
lines.push(` Cache: ${r.cache.hits} hits / ${r.cache.misses} misses (${pct(r.cache.hit_rate)}% hit-rate)`);
lines.push(` Source-tier breakdown:`);
@@ -331,12 +389,16 @@ async function runReview(engine: BrainEngine, f: ParsedFlags): Promise<void> {
}
filtered.sort((a, b) => compareSeverityDesc(a.severity, b.severity));
const buckets = bucketBySeverity(filtered);
for (const sev of ['high', 'medium', 'low'] as const) {
// v0.34 / Lane A2: 'info' is the lowest severity bucket; iterate
// high → medium → low → info so the report lands worst-first.
for (const sev of ['high', 'medium', 'low', 'info'] as const) {
const items = buckets[sev];
if (items.length === 0) continue;
console.error(`\n${sev.toUpperCase()} severity (${items.length}):`);
for (const item of items) {
console.error(` - ${item.a.slug} vs ${item.b.slug}`);
// v0.34 / Lane A2: include verdict so the operator distinguishes
// genuine contradictions from temporal classifications at a glance.
console.error(` - [${item.verdict}] ${item.a.slug} vs ${item.b.slug}`);
if (item.axis) console.error(` axis: ${item.axis}`);
console.error(` → ${item.resolution_command}`);
}
src/commands/models.ts
+1
View File
@@ -47,6 +47,7 @@ const PER_TASK_KEYS: Array<{ key: string; tier: ModelTier; description: string }
{ key: 'models.subagent', tier: 'subagent', description: '`gbrain agent run` subagent loop' },
{ key: 'facts.extraction_model', tier: 'reasoning', description: 'Real-time facts extraction during sync' },
{ key: 'models.eval.longmemeval', tier: 'reasoning', description: 'LongMemEval benchmark answer-gen' },
{ key: 'models.eval.contradictions_judge', tier: 'utility', description: 'Contradiction probe judge (v0.34 temporal-aware)' },
{ key: 'models.expansion', tier: 'utility', description: 'Query expansion for hybrid search' },
{ key: 'models.chat', tier: 'reasoning', description: 'Default `gateway.chat()` model' },
];
src/core/eval-contradictions/auto-supersession.ts
+73 -10
View File
@@ -28,6 +28,7 @@ import type {
ContradictionPair,
JudgeVerdict,
ResolutionKind,
Verdict,
} from './types.ts';
export interface ResolutionProposal {
@@ -46,28 +47,54 @@ function isCuratedEntitySlug(slug: string): boolean {
* wins for cross_slug pairs because it has semantic context this rule-based
* pass doesn't. For intra_page pairs we trust the structural heuristic since
* the judge can't see take_id metadata directly.
*
* v0.34 / Lane A2: extended with verdict-aware mapping. The new verdicts
* (temporal_supersession, temporal_regression, temporal_evolution,
* negation_artifact) have their own resolution_kinds — the v1 four-kind
* mapping only applies to `verdict === 'contradiction'`. The probe still
* NEVER auto-mutates; the new kinds render paste-ready commands or
* informational lines just like the old ones.
*/
export function classifyResolution(
pair: ContradictionPair,
judgeHint: ResolutionKind | null,
verdict: Verdict = 'contradiction',
): ResolutionKind {
// Verdict-driven routing for the new (non-contradiction) verdicts. The
// judge hint can still override when it specifies something compatible
// with the new verdict's scope.
if (verdict === 'temporal_supersession') {
return judgeHint === 'flag_for_review' || judgeHint === 'log_timeline_change'
? judgeHint
: 'temporal_supersede';
}
if (verdict === 'temporal_regression') {
// Always flag — no auto-mutation, just surface to the user. A future
// founder-scorecard surface (Phase 4) consumes the flagged set.
return 'flag_for_review';
}
if (verdict === 'temporal_evolution') {
return judgeHint === 'flag_for_review' ? 'flag_for_review' : 'log_timeline_change';
}
if (verdict === 'negation_artifact') {
// Informational — the data is correct; the judge misread a negation.
// Operator action is to wait for the next prompt_version bump.
return 'flag_for_review';
}
// verdict === 'contradiction' (or no_contradiction, which shouldn't reach
// this fn — runner filters before calling pairToFinding) falls through to
// the v1 mapping below.
if (pair.kind === 'intra_page_chunk_take') {
// One side is a take (b, by convention in the runner). If the take is
// active and the chunk text is older, supersede makes sense. We default
// to takes_supersede; if context is ambiguous the user can pick manual.
if (pair.b.take_id !== null) return 'takes_supersede';
if (pair.a.take_id !== null) return 'takes_supersede';
return 'manual_review';
}
// cross_slug: judge hint wins if it's specific.
if (judgeHint === 'dream_synthesize' || judgeHint === 'takes_mark_debate') {
return judgeHint;
}
if (judgeHint === 'takes_supersede' || judgeHint === 'manual_review') {
return judgeHint;
}
// Structural fallback: if either side is a curated entity page, propose
// a synthesize run on the curated slug to reconcile.
if (isCuratedEntitySlug(pair.a.slug) || isCuratedEntitySlug(pair.b.slug)) {
return 'dream_synthesize';
}
@@ -100,6 +127,37 @@ export function renderResolutionCommand(
const takeId = takeSide.take_id ?? '<row>';
return `gbrain takes mark-debate ${takeSide.slug} --row ${takeId}`;
}
case 'temporal_supersede': {
// v0.34 / Lane A2: pick the newer-dated side as the survivor; render a
// supersede command on the older-dated side. If both sides have takes
// we prefer the take that's NOT on the newer page. Falls back to a
// hint with both slugs when the dates can't be ordered.
const aDate = pair.a.effective_date;
const bDate = pair.b.effective_date;
if (aDate && bDate) {
const olderSide = aDate < bDate ? pair.a : pair.b;
const newerDate = aDate < bDate ? bDate : aDate;
const olderTakeId = olderSide.take_id;
if (olderTakeId !== null) {
return `gbrain takes supersede ${olderSide.slug} --row ${olderTakeId} --since ${newerDate}`;
}
return `# temporal_supersession: ${olderSide.slug} (${aDate < bDate ? aDate : bDate}) superseded by ${newerDate}`;
}
return `# temporal_supersession: ${pair.a.slug} vs ${pair.b.slug} (date order unclear)`;
}
case 'log_timeline_change': {
// v0.34 / Lane A2: timeline-writer subcommand is deferred (see plan
// TODOs). Render a hint pointing at the future command shape so the
// operator can paste a follow-up note manually for now.
const aDate = pair.a.effective_date ?? '<date-a>';
const bDate = pair.b.effective_date ?? '<date-b>';
return `# temporal_evolution: ${pair.a.slug} (${aDate}) → ${pair.b.slug} (${bDate}); record in timeline when the gbrain timeline writer lands`;
}
case 'flag_for_review': {
// v0.34 / Lane A2: informational; covers temporal_regression and
// negation_artifact. The flag itself surfaces in trends; no command.
return `# flag_for_review: ${pair.a.slug} vs ${pair.b.slug}`;
}
case 'manual_review':
default:
return `# manual review: ${pair.a.slug} vs ${pair.b.slug}`;
@@ -110,8 +168,9 @@ export function renderResolutionCommand(
export function proposeResolution(
pair: ContradictionPair,
judgeHint: ResolutionKind | null,
verdict: Verdict = 'contradiction',
): ResolutionProposal {
const kind = classifyResolution(pair, judgeHint);
const kind = classifyResolution(pair, judgeHint, verdict);
return {
resolution_kind: kind,
resolution_command: renderResolutionCommand(pair, kind),
@@ -120,16 +179,20 @@ export function proposeResolution(
/**
* Promote a ContradictionPair + JudgeVerdict to a ContradictionFinding by
* filling in severity/axis/confidence + resolution proposal. Used by the
* runner aggregation pass.
* filling in verdict/severity/axis/confidence + resolution proposal. Used
* by the runner aggregation pass.
*
* v0.34 / Lane A2: threads the new `verdict: Verdict` enum into the finding
* shape and the resolution classifier so the new verdicts route correctly.
*/
export function pairToFinding(
pair: ContradictionPair,
verdict: JudgeVerdict,
): ContradictionFinding {
const prop = proposeResolution(pair, verdict.resolution_kind);
const prop = proposeResolution(pair, verdict.resolution_kind, verdict.verdict);
return {
...pair,
verdict: verdict.verdict,
severity: verdict.severity,
axis: verdict.axis,
confidence: verdict.confidence,
src/core/eval-contradictions/cache.ts
+5 -1
View File
@@ -62,8 +62,12 @@ export function buildCacheKey(opts: {
function isJudgeVerdict(raw: unknown): raw is JudgeVerdict {
if (!raw || typeof raw !== 'object') return false;
const v = raw as Record<string, unknown>;
// v0.34 / Lane A2: shape check matches the new `verdict: Verdict` enum.
// Old v1-shaped rows (with `contradicts: boolean` instead) fail this guard
// and get treated as cache misses — same effect as the PROMPT_VERSION '1'
// rows already being filtered out by the cache-key tuple, but doubly safe.
return (
typeof v.contradicts === 'boolean' &&
typeof v.verdict === 'string' &&
typeof v.severity === 'string' &&
typeof v.confidence === 'number' &&
typeof v.axis === 'string'
src/core/eval-contradictions/cost-prompt.ts
+133
View File
@@ -0,0 +1,133 @@
/**
* eval-contradictions/cost-prompt — Lane C cost-estimate prompt.
*
* Fires before the probe runs when PROMPT_VERSION has changed since the most
* recent persisted run (so the operator sees the one-time re-judge cost up
* front instead of being surprised by it inside an autopilot cycle). Pattern
* mirrors `runPostUpgradeReembedPrompt` in post-upgrade-reembed.ts:
*
* - TTY-only Ctrl-C window (default 10s; override via
* GBRAIN_PROBE_PROMPT_GRACE_SECONDS).
* - Non-TTY auto-proceeds with a stderr note (autopilot path).
* - GBRAIN_NO_PROBE_PROMPT=1 skips entirely.
*
* Independent of the runner's `--budget-usd` hard cap: this prompt informs;
* the cap enforces. Both layers compose — operator sees the estimate, then
* the runner halts mid-run if the live cost exceeds the cap.
*/
import type { BrainEngine } from '../engine.ts';
import { estimateUpperBoundCost } from './cost-tracker.ts';
import { PROMPT_VERSION } from './types.ts';
export interface CostPromptOpts {
engine: BrainEngine;
queryCount: number;
topK: number;
judgeModel: string;
/** True when the user passed --yes; skip the prompt entirely. */
yesOverride?: boolean;
/** Override TTY detection for testing. */
isTtyOverride?: boolean;
/** Override stderr writer for testing. */
stderrWriter?: (text: string) => void;
/** Override the wait function for testing (returns 'proceed' or 'abort'). */
waitFn?: (graceSeconds: number) => Promise<'proceed' | 'abort'>;
}
export type CostPromptResult =
| { kind: 'proceed'; reason: 'env_skip' | 'yes_override' | 'no_version_change' | 'non_tty_auto' | 'tty_proceed' }
| { kind: 'abort'; reason: 'tty_ctrl_c' };
/** Read the prompt_version of the most recent persisted run. */
async function readLastPromptVersion(engine: BrainEngine): Promise<string | null> {
try {
const rows = await engine.executeRaw<{ prompt_version: string }>(
`SELECT prompt_version FROM eval_contradictions_runs ORDER BY ran_at DESC LIMIT 1`,
);
if (rows && rows.length > 0 && typeof rows[0].prompt_version === 'string') {
return rows[0].prompt_version;
}
} catch {
// Table missing (pre-v0.32 brains) or transient error — treat as null
// (which fires the prompt on first run, the correct default).
}
return null;
}
/** Default TTY waiter: prints countdown and resolves on SIGINT or timeout. */
function defaultWaitFn(graceSeconds: number): Promise<'proceed' | 'abort'> {
return new Promise((resolve) => {
let aborted = false;
const onSigint = () => {
if (aborted) return;
aborted = true;
process.removeListener('SIGINT', onSigint);
resolve('abort');
};
process.on('SIGINT', onSigint);
setTimeout(() => {
if (aborted) return;
process.removeListener('SIGINT', onSigint);
resolve('proceed');
}, graceSeconds * 1000);
});
}
/**
* Public entry. Returns whether the runner should proceed. Honors the
* --yes override, GBRAIN_NO_PROBE_PROMPT, TTY detection, and the persisted
* last-run prompt_version comparison.
*/
export async function maybePromptForCostBeforeProbe(
opts: CostPromptOpts,
): Promise<CostPromptResult> {
if (opts.yesOverride) {
return { kind: 'proceed', reason: 'yes_override' };
}
if (process.env.GBRAIN_NO_PROBE_PROMPT === '1') {
return { kind: 'proceed', reason: 'env_skip' };
}
const lastVersion = await readLastPromptVersion(opts.engine);
if (lastVersion === PROMPT_VERSION) {
return { kind: 'proceed', reason: 'no_version_change' };
}
const stderr = opts.stderrWriter ?? ((text: string) => process.stderr.write(text));
const isTty = opts.isTtyOverride ?? (process.stderr.isTTY === true);
// Conservative pair-count upper bound — same formula the runner uses for
// its own pre-flight check. Cost estimate covers the worst-case re-judge.
const conservativePairsPerQuery = (opts.topK * (opts.topK - 1)) / 2 + opts.topK * 2;
const estimatedCost = estimateUpperBoundCost({
pairCount: opts.queryCount * conservativePairsPerQuery,
queryCount: opts.queryCount,
judgeModel: opts.judgeModel,
});
const banner = [
`[contradiction probe] PROMPT_VERSION changed (${lastVersion ?? 'none'} → ${PROMPT_VERSION}).`,
`Old verdicts in the persistent cache no longer apply; this run will re-judge from scratch.`,
`Upper-bound estimate for ${opts.queryCount} queries × top-${opts.topK} on ${opts.judgeModel}: ~$${estimatedCost.toFixed(2)}.`,
`(--budget-usd N hard-caps the run; default values are conservative.)`,
].join('\n');
if (!isTty) {
// Autopilot / scripted invocation: emit the estimate and proceed.
stderr(`${banner}\nNon-TTY: proceeding automatically. Set GBRAIN_NO_PROBE_PROMPT=1 to suppress.\n`);
return { kind: 'proceed', reason: 'non_tty_auto' };
}
const graceRaw = process.env.GBRAIN_PROBE_PROMPT_GRACE_SECONDS;
const graceSeconds = graceRaw && Number.isFinite(Number(graceRaw)) && Number(graceRaw) >= 0
? Number(graceRaw)
: 10;
stderr(`${banner}\nPress Ctrl-C within ${graceSeconds}s to abort, or wait to proceed.\n`);
const waiter = opts.waitFn ?? defaultWaitFn;
const decision = await waiter(graceSeconds);
if (decision === 'abort') {
stderr(`[contradiction probe] aborted by Ctrl-C.\n`);
return { kind: 'abort', reason: 'tty_ctrl_c' };
}
return { kind: 'proceed', reason: 'tty_proceed' };
}
src/core/eval-contradictions/date-filter.ts
+29 -1
View File
@@ -52,12 +52,31 @@ export interface DateFilterDecision {
| 'both_explicit_separated'
| 'one_or_both_missing_dates'
| 'same_paragraph_dual_date'
| 'overlapping_or_close';
| 'overlapping_or_close'
/**
* v0.34 / Lane B: both sides have a non-null page-level effective_date,
* so the judge will see the temporal anchors and classify supersession /
* regression / evolution explicitly. Rule 3's >30d skip would otherwise
* silently kill exactly the cases the new verdicts exist to surface.
*/
| 'both_have_effective_date';
}
export interface DateFilterInput {
textA: string;
textB: string;
/**
* v0.34 / Lane B: page-level effective_date for each side (carried through
* from SearchResult.effective_date via PairMember). When both are non-null
* the judge will see them via the (from: YYYY-MM-DD) prompt tag and can
* classify temporal supersession explicitly — so the rule-3 >30d skip is
* counterproductive (it was a cost-saving heuristic for the v1 binary
* verdict; the v2 enum makes the explicit classification cheap).
* Undefined / null means "no temporal anchor for that side" and rule 3
* still applies as a fallback heuristic.
*/
effectiveDateA?: string | null;
effectiveDateB?: string | null;
}
/** Extract date tokens from text. Returns parsed Date objects (UTC midnight). */
@@ -144,6 +163,15 @@ export function shouldSkipForDateMismatch(input: DateFilterInput): DateFilterDec
) {
return { skip: false, reason: 'same_paragraph_dual_date' };
}
// v0.34 / Lane B: when BOTH sides have a non-null page-level effective_date,
// the judge will see them via the (from: YYYY-MM-DD) tag and can classify
// temporal supersession / regression / evolution explicitly. Skipping these
// pairs was the v1 default; in v2 it would silently kill the cases the
// new verdict taxonomy exists to surface (e.g. the role-change-across-
// years case where chunk-text dates are years apart).
if (input.effectiveDateA && input.effectiveDateB) {
return { skip: false, reason: 'both_have_effective_date' };
}
const datesA = extractDates(input.textA);
const datesB = extractDates(input.textB);
if (datesA.length === 0 || datesB.length === 0) {
src/core/eval-contradictions/judge.ts
+140 -39
View File
@@ -20,8 +20,8 @@
*/
import { chat, type ChatResult } from '../ai/gateway.ts';
import { parseSeverity } from './severity-classify.ts';
import type { JudgeVerdict, ResolutionKind } from './types.ts';
import { parseSeverity, defaultSeverityForVerdict } from './severity-classify.ts';
import type { JudgeVerdict, ResolutionKind, Verdict } from './types.ts';
const FENCE_RE = /```(?:json)?\s*\n?([\s\S]*?)```/i;
@@ -108,9 +108,26 @@ export function truncateUtf8(text: string, maxChars: number): string {
export interface JudgeInput {
/** The user's query for the search that retrieved both members. */
query: string;
/** Statement A: slug + text + optional source-tier + holder (if take). */
a: { slug: string; text: string; source_tier?: string; holder?: string | null };
b: { slug: string; text: string; source_tier?: string; holder?: string | null };
/**
* Statement A: slug + text + optional source-tier + holder (if take) +
* optional effective_date (Lane A1). When effective_date is null/undefined
* the prompt shows `(date unknown)` for that side; the judge classifies
* based on chunk text alone, same as the v1 prompt did.
*/
a: {
slug: string;
text: string;
source_tier?: string;
holder?: string | null;
effective_date?: string | null;
};
b: {
slug: string;
text: string;
source_tier?: string;
holder?: string | null;
effective_date?: string | null;
};
/** Provider:model id; routed through gateway.chat. */
model: string;
/** UTF-8-safe truncation limit per pair member. C4 flag. */
@@ -129,58 +146,108 @@ export interface JudgeOutput {
/**
* Validated resolution_kind values. Anything outside this set defaults to
* 'manual_review' (the safe, no-action option).
* 'manual_review' (the safe, no-action option for contradictions; new verdicts
* route through auto-supersession.ts when resolution_kind is null on input).
*
* v0.34 / Lane A2: added temporal_supersede, flag_for_review, log_timeline_change.
*/
function parseResolutionKind(value: unknown): ResolutionKind | null {
if (
value === 'takes_supersede' ||
value === 'dream_synthesize' ||
value === 'takes_mark_debate' ||
value === 'manual_review'
value === 'manual_review' ||
value === 'temporal_supersede' ||
value === 'flag_for_review' ||
value === 'log_timeline_change'
) {
return value;
}
return null;
}
const VALID_VERDICTS: ReadonlySet<Verdict> = new Set([
'no_contradiction',
'contradiction',
'temporal_supersession',
'temporal_regression',
'temporal_evolution',
'negation_artifact',
]);
/** Validate a verdict string from JSON; throws on missing/invalid so caller maps to parse_fail. */
export function parseVerdict(value: unknown): Verdict {
if (typeof value !== 'string' || !VALID_VERDICTS.has(value as Verdict)) {
throw new Error(`judge JSON missing or invalid verdict: ${JSON.stringify(value)}`);
}
return value as Verdict;
}
/**
* Validate the raw parsed JSON against the JudgeVerdict shape. Throws on
* fundamentally-broken shape (missing contradicts/confidence) so the caller
* fundamentally-broken shape (missing verdict/confidence) so the caller
* counts it under judge_errors.parse_fail rather than fabricating a verdict.
*
* C1 enforcement: contradicts:true with confidence < 0.7 is downgraded to
* false (belt-and-suspenders against models ignoring the prompt rule).
* v0.34 / Lane A2: parses the new `verdict: Verdict` enum field instead of
* the v1 `contradicts: boolean`. PROMPT_VERSION = '2' (bumped in A1) means
* the persistent cache won't return v1-shaped rows for these calls.
*
* C1 enforcement: `verdict === 'contradiction'` with confidence < 0.7 is
* downgraded to `'no_contradiction'` (belt-and-suspenders against models
* ignoring the prompt rule). The 5 non-contradiction verdicts do NOT have a
* confidence floor — they're informational classifications, not error flags.
*/
export function normalizeVerdict(raw: unknown): JudgeVerdict {
if (!raw || typeof raw !== 'object') {
throw new Error('judge JSON missing or not an object');
}
const v = raw as Record<string, unknown>;
const rawContradicts = v.contradicts;
if (typeof rawContradicts !== 'boolean') {
throw new Error('judge JSON missing required field: contradicts');
}
// Parse verdict first so we can throw a useful error before checking other
// fields. Old v1-shaped responses (`contradicts: true/false` without
// `verdict`) will throw here and the caller maps it to parse_fail — correct
// semantics because the prompt now asks for verdict explicitly.
let verdict = parseVerdict(v.verdict);
const rawConfidence = v.confidence;
if (typeof rawConfidence !== 'number' || !Number.isFinite(rawConfidence)) {
throw new Error('judge JSON missing or invalid confidence');
}
const clampedConfidence = Math.min(1, Math.max(0, rawConfidence));
const severity = parseSeverity(v.severity);
const axisRaw = typeof v.axis === 'string' ? v.axis : '';
const resolutionKind = parseResolutionKind(v.resolution_kind);
// C1 double-enforce: contradicts:true requires confidence >= 0.7.
let contradicts = rawContradicts;
if (contradicts && clampedConfidence < 0.7) {
contradicts = false;
// C1 double-enforce: only `verdict === 'contradiction'` carries the
// confidence floor. Downgrade to no_contradiction below the threshold.
if (verdict === 'contradiction' && clampedConfidence < 0.7) {
verdict = 'no_contradiction';
}
// Severity: judge can set it; if invalid, fall back to the default for the
// verdict (D7 map: temporal_supersession→info, temporal_regression→high, ...).
// parseSeverity coerces unknown strings to 'low' historically, but now we
// route through defaultSeverityForVerdict instead so each verdict gets a
// meaningful default.
const severity = parseSeverity(v.severity, defaultSeverityForVerdict(verdict));
const isFinding = verdict !== 'no_contradiction';
// Only `contradiction` keeps the v1 fallback to 'manual_review' when the
// judge omits a resolution_kind. The new verdicts pass through whatever the
// judge said (or null) and auto-supersession.ts picks the kind based on
// verdict semantics.
let normalizedResolutionKind: ResolutionKind | null;
if (verdict === 'contradiction') {
normalizedResolutionKind = resolutionKind ?? 'manual_review';
} else if (isFinding) {
normalizedResolutionKind = resolutionKind ?? null;
} else {
normalizedResolutionKind = null;
}
return {
contradicts,
verdict,
severity,
axis: contradicts ? axisRaw : '',
axis: isFinding ? axisRaw : '',
confidence: clampedConfidence,
resolution_kind: contradicts ? (resolutionKind ?? 'manual_review') : null,
resolution_kind: normalizedResolutionKind,
};
}
@@ -194,14 +261,31 @@ export function normalizeVerdict(raw: unknown): JudgeVerdict {
*/
export function buildJudgePrompt(opts: {
query: string;
a: { slug: string; text: string; source_tier?: string; holder?: string | null };
b: { slug: string; text: string; source_tier?: string; holder?: string | null };
a: {
slug: string;
text: string;
source_tier?: string;
holder?: string | null;
effective_date?: string | null;
};
b: {
slug: string;
text: string;
source_tier?: string;
holder?: string | null;
effective_date?: string | null;
};
maxPairChars: number;
}): string {
const a = truncateUtf8(opts.a.text, opts.maxPairChars);
const b = truncateUtf8(opts.b.text, opts.maxPairChars);
const aMeta = [opts.a.slug, opts.a.source_tier && `source-tier ${opts.a.source_tier}`, opts.a.holder && `holder ${opts.a.holder}`].filter(Boolean).join(', ');
const bMeta = [opts.b.slug, opts.b.source_tier && `source-tier ${opts.b.source_tier}`, opts.b.holder && `holder ${opts.b.holder}`].filter(Boolean).join(', ');
// Lane A1: emit the page-level effective_date on its own line so the judge
// can reason temporally. `(date unknown)` keeps the v1 fallback behavior
// when the page has no effective_date — judge classifies on text alone.
const aDateTag = opts.a.effective_date ? `(from: ${opts.a.effective_date})` : '(date unknown)';
const bDateTag = opts.b.effective_date ? `(from: ${opts.b.effective_date})` : '(date unknown)';
return [
'You are a contradiction judge for a personal knowledge brain. The user',
'ran a search and got two results back. Decide whether the two statements',
@@ -210,39 +294,54 @@ export function buildJudgePrompt(opts: {
'',
`User's query: ${opts.query}`,
'',
`Statement A (${aMeta}):`,
`Statement A ${aDateTag} (${aMeta}):`,
a,
'',
`Statement B (${bMeta}):`,
`Statement B ${bDateTag} (${bMeta}):`,
b,
'',
'Rules:',
'- Different timeframes for the same dynamic property are NOT contradictions',
' (e.g., MRR was $50K in 2024 vs $2M in 2026 — both true at their time).',
'- Different timeframes for a static identity claim MAY BE a contradiction',
' (e.g., "Alice is CFO of Acme" vs "Alice left Acme" if dates suggest one',
' supersedes the other).',
'- The (from: YYYY-MM-DD) tag is the page-level effective date. Use it to',
' classify what kind of difference this is, not just whether it exists.',
' (date unknown) means the page has no temporal anchor — judge on text',
' alone for that side.',
'- Pick exactly one verdict from the six values below.',
'- Use temporal_supersession when the newer-dated claim updates or replaces',
' the older one (role change, status change). Not an error.',
'- Use temporal_regression when a metric or status went BACKWARDS over time',
' (e.g., MRR dropped from $200K to $150K). This is a signal worth flagging.',
'- Use temporal_evolution for legitimate change over time that is neither',
' supersession nor regression (e.g., evolving narrative, multi-step decision).',
'- Use negation_artifact when one side contains an explicit negation that',
' the surface tokens make look like a positive claim (e.g., "NOT X" parsed',
' as "X"). The data is correct; the apparent conflict is a parsing artifact.',
'- Use contradiction ONLY for genuinely conflicting claims at the same point',
' in time, where the dates do not explain the difference.',
'- Use no_contradiction when the statements are compatible.',
'',
'- Subjective opinions held at different times by the SAME holder may be',
' a contradiction (a flip). Opinions held by DIFFERENT holders are not.',
'- Different aspects of the same entity are NOT contradictions.',
"- Incidental disagreements unrelated to the user's query do NOT count.",
'- Different aspects of the same entity are not contradictions.',
"- Incidental disagreements unrelated to the user's query do not count.",
' Judge only on claims relevant to what the user asked.',
'',
'Reply with JSON ONLY:',
'{',
' "contradicts": true | false,',
' "severity": "low" | "medium" | "high",',
' "verdict": "no_contradiction" | "contradiction" | "temporal_supersession" | "temporal_regression" | "temporal_evolution" | "negation_artifact",',
' "severity": "info" | "low" | "medium" | "high",',
' "axis": "<one-line: what they disagree about, or empty>",',
' "confidence": 0.0..1.0,',
' "resolution_kind": "takes_supersede" | "dream_synthesize" | "takes_mark_debate" | "manual_review" | null',
' "resolution_kind": "takes_supersede" | "dream_synthesize" | "takes_mark_debate" | "manual_review" | "temporal_supersede" | "flag_for_review" | "log_timeline_change" | null',
'}',
'',
'Severity rubric:',
'- low: naming/format differences (Alice Smith vs A. Smith).',
'- info: temporal_supersession and temporal_evolution (not errors; informational).',
'- low: naming/format differences (Alice Smith vs A. Smith); negation artifacts.',
'- medium: factual values that may be stale (revenue, headcount).',
'- high: identity / structural claims (founder/CEO/CFO role, status).',
'- high: identity / structural claims (founder/CEO/CFO role); temporal_regression.',
'',
'Reply contradicts:true only when confidence >= 0.7.',
'Reply verdict:contradiction only when confidence >= 0.7. Other verdicts have',
'no confidence floor.',
].join('\n');
}
@@ -264,6 +363,8 @@ function isRefusalResponse(result: ChatResult): boolean {
*/
export async function judgeContradiction(input: JudgeInput): Promise<JudgeOutput> {
const maxPairChars = input.maxPairChars ?? DEFAULT_MAX_PAIR_CHARS;
// input.a/b carry effective_date through PairMember (Lane A1); buildJudgePrompt
// emits it on the Statement line or falls through to `(date unknown)`.
const prompt = buildJudgePrompt({
query: input.query,
a: input.a,
src/core/eval-contradictions/runner.ts
+78 -9
View File
@@ -46,6 +46,8 @@ import {
type PairMember,
type PerQueryResult,
type ProbeReport,
type Verdict,
type VerdictBreakdown,
} from './types.ts';
const DEFAULT_TOP_K = 5;
@@ -110,11 +112,27 @@ function searchResultToMember(r: SearchResult): PairMember {
source_tier: classifySlugTier(r.slug),
holder: null,
text: r.chunk_text,
// Lane A1: effective_date carried through from the search projection.
// null when the page has no temporal anchor (judge will see `(date unknown)`).
effective_date: r.effective_date ?? null,
effective_date_source: r.effective_date_source ?? null,
};
}
/** Convert a Take into a PairMember. */
function takeToMember(take: { id: number; page_slug: string; claim: string; holder: string }, source_tier: ReturnType<typeof classifySlugTier>): PairMember {
/**
* Convert a Take into a PairMember.
*
* Lane A1: takes are paired with a chunk from the same page, so the take's
* effective_date is inherited from the chunk's page-level effective_date.
* A future enhancement could distinguish `takes.since_date` from
* `pages.effective_date` here — for v1 they share the same page anchor.
*/
function takeToMember(
take: { id: number; page_slug: string; claim: string; holder: string },
source_tier: ReturnType<typeof classifySlugTier>,
effective_date: string | null,
effective_date_source: string | null,
): PairMember {
return {
slug: take.page_slug,
chunk_id: null,
@@ -122,6 +140,8 @@ function takeToMember(take: { id: number; page_slug: string; claim: string; hold
source_tier,
holder: take.holder,
text: take.claim,
effective_date,
effective_date_source,
};
}
@@ -158,7 +178,12 @@ async function generateIntraPagePairs(
if (takes.length === 0) continue;
const chunkMember = searchResultToMember(r);
for (const t of takes) {
const takeMember = takeToMember(t, chunkMember.source_tier);
const takeMember = takeToMember(
t,
chunkMember.source_tier,
chunkMember.effective_date,
chunkMember.effective_date_source,
);
out.push({
kind: 'intra_page_chunk_take',
a: chunkMember,
@@ -232,6 +257,17 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise<RunnerRes
const allPairs: ContradictionPair[] = [];
let capHitMidRun = false;
let queriesWithContradiction = 0;
let queriesWithAnyFinding = 0;
// v0.34 / Lane A2: per-verdict tally across every judged pair (and cache hit).
const verdictBreakdown: VerdictBreakdown = {
no_contradiction: 0,
contradiction: 0,
temporal_supersession: 0,
temporal_regression: 0,
temporal_evolution: 0,
negation_artifact: 0,
};
const tallyVerdict = (v: Verdict) => { verdictBreakdown[v]++; };
for (const query of opts.queries) {
if (opts.abortSignal?.aborted) break;
@@ -258,10 +294,18 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise<RunnerRes
allPairs.push(...allPairsForQuery);
// Date pre-filter.
// v0.34 / Lane B: thread page-level effective_date through so the
// filter's relaxation rule fires for dated pairs (judge classifies
// temporal supersession instead of the pair being silently skipped).
const survivedDate: ContradictionPair[] = [];
let skippedByDate = 0;
for (const p of allPairsForQuery) {
const decision = shouldSkipForDateMismatch({ textA: p.a.text, textB: p.b.text });
const decision = shouldSkipForDateMismatch({
textA: p.a.text,
textB: p.b.text,
effectiveDateA: p.a.effective_date,
effectiveDateB: p.b.effective_date,
});
if (decision.skip) {
skippedByDate++;
continue;
@@ -286,7 +330,11 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise<RunnerRes
const cached = await cache.lookup(pair.a.text, pair.b.text);
if (cached) {
cacheHits++;
if (cached.contradicts) {
tallyVerdict(cached.verdict);
// v0.34 / Lane A2: emit findings for every non-no_contradiction verdict.
// Without this, the new verdicts (temporal_supersession etc.) would
// disappear from the report and the whole wave is invisible to users.
if (cached.verdict !== 'no_contradiction') {
findings.push(pairToFinding(pair, cached));
}
continue;
@@ -295,8 +343,20 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise<RunnerRes
try {
const out = await judgeFn({
query,
a: { slug: pair.a.slug, text: pair.a.text, source_tier: pair.a.source_tier, holder: pair.a.holder },
b: { slug: pair.b.slug, text: pair.b.text, source_tier: pair.b.source_tier, holder: pair.b.holder },
a: {
slug: pair.a.slug,
text: pair.a.text,
source_tier: pair.a.source_tier,
holder: pair.a.holder,
effective_date: pair.a.effective_date,
},
b: {
slug: pair.b.slug,
text: pair.b.text,
source_tier: pair.b.source_tier,
holder: pair.b.holder,
effective_date: pair.b.effective_date,
},
model: judgeModel,
maxPairChars,
abortSignal: opts.abortSignal,
@@ -304,7 +364,9 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise<RunnerRes
tracker.recordJudgeCall(judgeModel, out.usage);
await cache.store(pair.a.text, pair.b.text, out.verdict);
judged++;
if (out.verdict.contradicts) {
tallyVerdict(out.verdict.verdict);
// v0.34 / Lane A2: same emit predicate as the cache-hit branch.
if (out.verdict.verdict !== 'no_contradiction') {
findings.push(pairToFinding(pair, out.verdict));
}
} catch (err) {
@@ -312,7 +374,12 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise<RunnerRes
}
}
if (findings.length > 0) queriesWithContradiction++;
// v0.34 / Lane A2: distinguish strict-contradiction from any-finding.
// The strict count drives the Wilson-CI denominator (the historic
// headline metric). The broad count surfaces the wave's new value:
// "of N queries, M had at least one temporal signal."
if (findings.length > 0) queriesWithAnyFinding++;
if (findings.some((f) => f.verdict === 'contradiction')) queriesWithContradiction++;
perQuery.push({
query,
result_count: results.length,
@@ -347,7 +414,9 @@ export async function runContradictionProbe(opts: RunnerOpts): Promise<RunnerRes
sampling,
queries_evaluated: opts.queries.length,
queries_with_contradiction: queriesWithContradiction,
queries_with_any_finding: queriesWithAnyFinding,
total_contradictions_flagged: allFindings.length,
verdict_breakdown: verdictBreakdown,
calibration,
judge_errors: judgeErrors,
cost_usd: cost,
src/core/eval-contradictions/severity-classify.ts
+44 -8
View File
@@ -13,17 +13,53 @@
* post-processing; it does NOT re-classify or override the LLM's call.
*/
import type { ContradictionFinding, HotPage, Severity } from './types.ts';
import type { ContradictionFinding, HotPage, Severity, Verdict } from './types.ts';
const SEVERITY_RANK: Record<Severity, number> = { low: 1, medium: 2, high: 3 };
/**
* v0.34 / Lane A2: 'info' joins the rank as the lowest non-trivial severity
* (below 'low' so high-severity findings still sort to the top). 0 reserved
* for future "below info" semantics; nothing currently emits at rank 0.
*/
const SEVERITY_RANK: Record<Severity, number> = { info: 0, low: 1, medium: 2, high: 3 };
/** Validate a severity string; default to 'low' on unknown input. */
export function parseSeverity(value: unknown): Severity {
if (value === 'low' || value === 'medium' || value === 'high') return value;
return 'low';
/**
* v0.34 / Lane A2: default severity per verdict, used when the judge returns
* an unknown severity string. The mapping is opinionated:
* - temporal_supersession: info — newer claim wins, this is normal evolution
* - temporal_evolution: info — legitimate change, neither error nor signal
* - temporal_regression: high — metric went backwards, investor red flag
* - negation_artifact: low — probe bug; surfaces in report but low signal
* - contradiction: medium — actual conflict; judge often overrides
* - no_contradiction: info — not a finding (filtered out by runner emit)
*/
const DEFAULT_SEVERITY_BY_VERDICT: Record<Verdict, Severity> = {
no_contradiction: 'info',
contradiction: 'medium',
temporal_supersession: 'info',
temporal_regression: 'high',
temporal_evolution: 'info',
negation_artifact: 'low',
};
export function defaultSeverityForVerdict(verdict: Verdict): Severity {
return DEFAULT_SEVERITY_BY_VERDICT[verdict];
}
/** Compare for descending sort: high > medium > low. */
/**
* Validate a severity string. Two signatures:
* - parseSeverity(value) → defaults to 'low' on invalid input (legacy behavior)
* - parseSeverity(value, fallback) → uses provided fallback on invalid input
*
* v0.34 / Lane A2: accepts 'info' as a valid severity.
*/
export function parseSeverity(value: unknown, fallback: Severity = 'low'): Severity {
if (value === 'info' || value === 'low' || value === 'medium' || value === 'high') {
return value;
}
return fallback;
}
/** Compare for descending sort: high > medium > low > info. */
export function compareSeverityDesc(a: Severity, b: Severity): number {
return SEVERITY_RANK[b] - SEVERITY_RANK[a];
}
@@ -32,7 +68,7 @@ export function compareSeverityDesc(a: Severity, b: Severity): number {
export function bucketBySeverity(
findings: readonly ContradictionFinding[],
): Record<Severity, ContradictionFinding[]> {
const out: Record<Severity, ContradictionFinding[]> = { low: [], medium: [], high: [] };
const out: Record<Severity, ContradictionFinding[]> = { info: [], low: [], medium: [], high: [] };
for (const f of findings) {
out[f.severity].push(f);
}
src/core/eval-contradictions/trends.ts
+19
View File
@@ -95,5 +95,24 @@ export function renderTrendChart(rows: readonly TrendRow[]): string {
const bar = '#'.repeat(fill) + '.'.repeat(barWidth - fill);
lines.push(`${date} ${model} ${q} ${withCx} ${flag} ${ci} ${bar}`);
}
// v0.34 / Lane A2: append the per-verdict breakdown from the most recent
// run so operators can see which kinds of finding the probe is generating.
// Older runs without `verdict_breakdown` populated (cache rows from before
// A2) emit a `(no breakdown)` placeholder.
const latest = rows[0];
if (latest && latest.report_json && latest.report_json.verdict_breakdown) {
const b = latest.report_json.verdict_breakdown;
lines.push('');
lines.push(`Latest run verdict breakdown (${latest.ran_at.slice(0, 10)}):`);
lines.push(` contradiction: ${b.contradiction}`);
lines.push(` temporal_supersession: ${b.temporal_supersession}`);
lines.push(` temporal_regression: ${b.temporal_regression}`);
lines.push(` temporal_evolution: ${b.temporal_evolution}`);
lines.push(` negation_artifact: ${b.negation_artifact}`);
lines.push(` no_contradiction: ${b.no_contradiction}`);
} else if (latest) {
lines.push('');
lines.push(`Latest run verdict breakdown: (no breakdown — predates v0.34 prompt_version=2)`);
}
return lines.join('\n');
}
src/core/eval-contradictions/types.ts
+105 -5
View File
@@ -12,30 +12,78 @@
export const SCHEMA_VERSION = 1 as const;
/** Bump when the judge prompt in judge.ts changes meaningfully. */
export const PROMPT_VERSION = '1' as const;
/**
* Bump when the judge prompt in judge.ts changes meaningfully. Used as part
* of the cache key tuple — bumping invalidates every cached verdict from
* prior runs, which is the point: when the prompt edits, old verdicts are
* no longer trustworthy.
*
* v2 (Lane A1, 2026-05): judge prompt now receives `Statement A (from: YYYY-MM-DD)`
* or `(date unknown)` per side. Old v1 verdicts are silently invalidated.
*/
export const PROMPT_VERSION = '2' as const;
/** Truncation policy string baked into the cache key. */
export const TRUNCATION_POLICY = '1500-chars-utf8-safe' as const;
export type ContradictionKind = 'cross_slug_chunks' | 'intra_page_chunk_take';
export type Severity = 'low' | 'medium' | 'high';
/**
* v0.34 / Lane A2: severity gains 'info' for the new non-error-class verdicts
* (temporal_supersession, temporal_evolution) that should surface in the
* report without inflating the contradiction count.
*/
export type Severity = 'info' | 'low' | 'medium' | 'high';
/**
* v0.34 / Lane A2: replaces the v1 `contradicts: boolean` shape. Six members.
*
* - no_contradiction → drop from findings (not surfaced)
* - contradiction → genuine conflict at the same point in time
* - temporal_supersession → newer claim updates/replaces older; not an error
* - temporal_regression → metric/status went backwards over time
* - temporal_evolution → legitimate change over time, neither of the above
* - negation_artifact → judge misread an explicit negation in one chunk
*/
export type Verdict =
| 'no_contradiction'
| 'contradiction'
| 'temporal_supersession'
| 'temporal_regression'
| 'temporal_evolution'
| 'negation_artifact';
/**
* Resolution kinds. v0.34 / Lane A2 added three new members covering the new
* verdicts; the original four still apply to `verdict === 'contradiction'`.
*
* - temporal_supersede → render `gbrain takes supersede ... --since <date>`
* - flag_for_review → informational; no CLI command rendered
* - log_timeline_change → render a hint pointing at the future
* timeline-writer subcommand (deferred)
*/
export type ResolutionKind =
| 'takes_supersede'
| 'dream_synthesize'
| 'takes_mark_debate'
| 'manual_review';
| 'manual_review'
| 'temporal_supersede'
| 'flag_for_review'
| 'log_timeline_change';
export type SourceTier = 'curated' | 'bulk' | 'other';
/**
* Judge's verdict for a single pair. Either the judge ran cleanly and we have
* scoring, or it failed and we have a typed error to surface in the report.
*
* v0.34 / Lane A2: `verdict: Verdict` replaces the v1 `contradicts: boolean`.
* Every consumer that previously branched on `verdict.contradicts` now
* switches on `verdict.verdict`. PROMPT_VERSION bumped in A1 invalidates the
* old cached rows.
*/
export interface JudgeVerdict {
contradicts: boolean;
verdict: Verdict;
severity: Severity;
/** One-line description of what they disagree about, or empty when no contradiction. */
axis: string;
@@ -74,6 +122,14 @@ export interface PairMember {
/** Takes-only: who holds the take (`garry`, `alice`, ...). */
holder: string | null;
text: string;
/**
* v0.34 / Lane A1: page-level effective_date carried through from SearchResult.
* Format: YYYY-MM-DD (ISO date-only). Threaded into the judge prompt so the
* model can classify supersession/regression/evolution. Null means "no
* temporal anchor for this chunk" — judge sees `(date unknown)` for this side.
*/
effective_date: string | null;
effective_date_source: string | null;
}
export interface ContradictionPair {
@@ -85,6 +141,12 @@ export interface ContradictionPair {
}
export interface ContradictionFinding extends ContradictionPair {
/**
* v0.34 / Lane A2: which verdict bucket this finding belongs to. Always one
* of the five non-`no_contradiction` members (no_contradiction findings are
* dropped from the report by the runner emit predicate).
*/
verdict: Verdict;
severity: Severity;
axis: string;
confidence: number;
@@ -95,6 +157,13 @@ export interface ContradictionFinding extends ContradictionPair {
export interface PerQueryResult {
query: string;
result_count: number;
/**
* v0.34 / Lane A2: contains every non-`no_contradiction` finding (genuine
* contradictions PLUS temporal_supersession / temporal_regression /
* temporal_evolution / negation_artifact). Field name preserved for wire-
* compatibility; consumers that want the strict-contradiction subset
* filter on `f.verdict === 'contradiction'`.
*/
contradictions: ContradictionFinding[];
/** Pairs the date pre-filter rejected before any judge call. Diagnostic only. */
pairs_skipped_by_date: number;
@@ -104,6 +173,20 @@ export interface PerQueryResult {
pairs_judged: number;
}
/**
* v0.34 / Lane A2: per-run tally across all 6 verdicts. Surfaces in the trend
* rollup so operators can see whether `gbrain eval suspected-contradictions`
* is finding mostly genuine contradictions or mostly temporal noise.
*/
export interface VerdictBreakdown {
no_contradiction: number;
contradiction: number;
temporal_supersession: number;
temporal_regression: number;
temporal_evolution: number;
negation_artifact: number;
}
export interface SourceTierBreakdown {
curated_vs_curated: number;
curated_vs_bulk: number;
@@ -155,8 +238,25 @@ export interface ProbeReport {
top_k: number;
sampling: 'deterministic' | 'score-first';
queries_evaluated: number;
/**
* Count of queries that produced at least one `verdict === 'contradiction'`
* finding. Strict-contradiction count; used as the headline metric + the
* Wilson-CI denominator. v0.34 / Lane A2: meaning preserved from v1, but
* the field is now narrower than `queries_with_any_finding`.
*/
queries_with_contradiction: number;
/**
* v0.34 / Lane A2: count of queries with ANY non-`no_contradiction` verdict.
* Always >= queries_with_contradiction. Helps operators see whether the
* probe is finding mostly genuine contradictions or mostly temporal signals.
*/
queries_with_any_finding: number;
total_contradictions_flagged: number;
/**
* v0.34 / Lane A2: per-verdict tally across every judged pair in the run.
* The sum equals (pairs_judged + cache_hits) across all queries.
*/
verdict_breakdown: VerdictBreakdown;
calibration: Calibration;
judge_errors: JudgeErrorsCounts;
cost_usd: CostBreakdown;
src/core/pglite-engine.ts
+6
View File
@@ -805,6 +805,7 @@ export class PGLiteEngine implements BrainEngine {
`WITH ranked AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
@@ -922,6 +923,7 @@ export class PGLiteEngine implements BrainEngine {
`WITH ranked AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
${scoreExpr} AS score,
CASE WHEN p.updated_at < (
@@ -950,6 +952,7 @@ export class PGLiteEngine implements BrainEngine {
const { rows } = await this.db.query(
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
${scoreExpr} AS score,
CASE WHEN p.updated_at < (
@@ -1042,6 +1045,7 @@ export class PGLiteEngine implements BrainEngine {
const { rows } = await this.db.query(
`SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
CASE WHEN p.updated_at < (
@@ -1142,6 +1146,7 @@ export class PGLiteEngine implements BrainEngine {
`WITH hnsw_candidates AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id, p.updated_at,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
1 - (cc.${col} <=> $1::vector) AS raw_score
FROM content_chunks cc
@@ -1153,6 +1158,7 @@ export class PGLiteEngine implements BrainEngine {
)
SELECT
hc.slug, hc.page_id, hc.title, hc.type, hc.source_id,
hc.effective_date, hc.effective_date_source,
hc.chunk_id, hc.chunk_index, hc.chunk_text, hc.chunk_source,
hc.raw_score * ${sourceFactorCaseOnSlug} AS score,
CASE WHEN hc.updated_at < (
src/core/postgres-engine.ts
+5
View File
@@ -865,6 +865,7 @@ export class PostgresEngine implements BrainEngine {
WITH ranked_chunks AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score
FROM content_chunks cc
@@ -895,6 +896,7 @@ export class PostgresEngine implements BrainEngine {
ORDER BY slug, score DESC
)
SELECT slug, page_id, title, type, source_id,
effective_date, effective_date_source,
chunk_id, chunk_index, chunk_text, chunk_source, score,
false AS stale
FROM best_per_page
@@ -1005,6 +1007,7 @@ export class PostgresEngine implements BrainEngine {
const rawQuery = `
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
ts_rank(cc.search_vector, websearch_to_tsquery('english', $1)) * ${sourceFactorCase} AS score,
false AS stale
@@ -1138,6 +1141,7 @@ export class PostgresEngine implements BrainEngine {
WITH hnsw_candidates AS (
SELECT
p.slug, p.id as page_id, p.title, p.type, p.source_id,
p.effective_date, p.effective_date_source,
cc.id as chunk_id, cc.chunk_index, cc.chunk_text, cc.chunk_source,
1 - (cc.${col} <=> $1::vector) AS raw_score
FROM content_chunks cc
@@ -1160,6 +1164,7 @@ export class PostgresEngine implements BrainEngine {
)
SELECT
slug, page_id, title, type, source_id,
effective_date, effective_date_source,
chunk_id, chunk_index, chunk_text, chunk_source,
raw_score * ${sourceFactorCaseOnSlug} AS score,
false AS stale
src/core/types.ts
+9
View File
@@ -410,6 +410,15 @@ export interface SearchResult {
* 'default' for pre-v0.17 rows that lacked the column.
*/
source_id?: string;
/**
* v0.34 — page-level effective_date (and its source) carried through from
* the pages join. Format: YYYY-MM-DD (ISO date-only). Consumers (currently
* the contradiction probe's date-aware judge prompt + date pre-filter)
* treat null and undefined the same: "no temporal anchor for this chunk."
* Pre-v0.34 engines that don't project these columns leave both undefined.
*/
effective_date?: string | null;
effective_date_source?: string | null;
}
export interface SearchOpts {
src/core/utils.ts
+28
View File
@@ -243,6 +243,34 @@ export function rowToSearchResult(row: Record<string, unknown>): SearchResult {
if (typeof row.source_id === 'string') {
result.source_id = row.source_id;
}
// v0.34: effective_date / effective_date_source carried through from the
// pages join. Same three-state read as readOptionalDate elsewhere: the
// field is left UNTOUCHED when the column isn't in the projection (so
// legacy callers see undefined), set to null when the column was selected
// but the page row has no date, and to YYYY-MM-DD when populated. Postgres
// returns Date objects via postgres.js; PGLite returns strings. Normalize
// to date-only ISO so downstream prompt-builders don't see noise from
// midnight-UTC timestamps.
if ('effective_date' in row) {
const raw = row.effective_date;
if (raw === null) {
result.effective_date = null;
} else if (raw instanceof Date) {
result.effective_date = raw.toISOString().slice(0, 10);
} else if (typeof raw === 'string' && raw) {
// Postgres TIMESTAMPTZ already serializes as "YYYY-MM-DD ..." — slice
// the date portion. PGLite returns the same shape via its parser.
result.effective_date = raw.slice(0, 10);
}
}
if ('effective_date_source' in row) {
const raw = row.effective_date_source;
if (raw === null) {
result.effective_date_source = null;
} else if (typeof raw === 'string' && raw) {
result.effective_date_source = raw;
}
}
return result;
}
test/e2e/eval-contradictions-postgres.test.ts
+19 -8
View File
@@ -60,7 +60,16 @@ function mkReport(opts: Partial<ProbeReport> = {}): ProbeReport {
sampling: 'deterministic',
queries_evaluated: 50,
queries_with_contradiction: 12,
queries_with_any_finding: 18,
total_contradictions_flagged: 18,
verdict_breakdown: {
no_contradiction: 282,
contradiction: 12,
temporal_supersession: 4,
temporal_regression: 1,
temporal_evolution: 1,
negation_artifact: 0,
},
calibration: {
queries_total: 50,
queries_judged_clean: 38,
@@ -160,11 +169,11 @@ describe('E2E: P2 persistent cache with real now()', () => {
const cache = new JudgeCache({ engine, modelId: 'haiku-pg-test' });
expect(await cache.lookup('text-a', 'text-b')).toBeNull();
await cache.store('text-a', 'text-b', {
contradicts: true, severity: 'high', axis: 'pg-test', confidence: 0.9, resolution_kind: 'dream_synthesize',
verdict: 'contradiction', severity: 'high', axis: 'pg-test', confidence: 0.9, resolution_kind: 'dream_synthesize',
});
const hit = await cache.lookup('text-a', 'text-b');
expect(hit).not.toBeNull();
expect(hit?.contradicts).toBe(true);
expect(hit?.verdict).toBe('contradiction');
expect(hit?.severity).toBe('high');
});
@@ -172,7 +181,7 @@ describe('E2E: P2 persistent cache with real now()', () => {
if (!engine) return;
const cache = new JudgeCache({ engine, modelId: 'haiku-pg-test', ttlSeconds: 60 });
await cache.store('expire-me-a', 'expire-me-b', {
contradicts: false, severity: 'low', axis: '', confidence: 0.3, resolution_kind: null,
verdict: 'no_contradiction', severity: 'info', axis: '', confidence: 0.3, resolution_kind: null,
});
// Backdate expires_at by 1 second.
const key = buildCacheKey({ textA: 'expire-me-a', textB: 'expire-me-b', modelId: 'haiku-pg-test' });
@@ -191,7 +200,7 @@ describe('E2E: P2 persistent cache with real now()', () => {
if (!engine) return;
const cache1 = new JudgeCache({ engine, modelId: 'haiku-pg-test' });
await cache1.store('shared-a', 'shared-b', {
contradicts: true, severity: 'medium', axis: '', confidence: 0.85, resolution_kind: 'manual_review',
verdict: 'contradiction', severity: 'medium', axis: '', confidence: 0.85, resolution_kind: 'manual_review',
});
// Direct engine call with a different prompt_version should miss.
const wrong = await engine.getContradictionCacheEntry({
@@ -262,9 +271,10 @@ describe('E2E: find_contradictions MCP op on Postgres', () => {
contradictions: [
{
kind: 'cross_slug_chunks',
a: { slug: 'companies/acme-example', chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' },
b: { slug: 'openclaw/chat/x', chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b' },
a: { slug: 'companies/acme-example', chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a', effective_date: null, effective_date_source: null },
b: { slug: 'openclaw/chat/x', chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b', effective_date: null, effective_date_source: null },
combined_score: 1.5,
verdict: 'contradiction',
severity: 'high',
axis: 'MRR figure',
confidence: 0.9,
@@ -273,9 +283,10 @@ describe('E2E: find_contradictions MCP op on Postgres', () => {
},
{
kind: 'cross_slug_chunks',
a: { slug: 'people/alice-example', chunk_id: 3, take_id: null, source_tier: 'curated', holder: null, text: 'c' },
b: { slug: 'people/alice-smith-example', chunk_id: 4, take_id: null, source_tier: 'curated', holder: null, text: 'd' },
a: { slug: 'people/alice-example', chunk_id: 3, take_id: null, source_tier: 'curated', holder: null, text: 'c', effective_date: null, effective_date_source: null },
b: { slug: 'people/alice-smith-example', chunk_id: 4, take_id: null, source_tier: 'curated', holder: null, text: 'd', effective_date: null, effective_date_source: null },
combined_score: 1.2,
verdict: 'contradiction',
severity: 'low',
axis: 'name format',
confidence: 0.75,
test/eval-contradictions-auto-supersession.test.ts
+6 -5
View File
@@ -17,8 +17,8 @@ import type {
function mkCrossSlugPair(slugA: string, slugB: string): ContradictionPair {
return {
kind: 'cross_slug_chunks',
a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' },
b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b' },
a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a', effective_date: null, effective_date_source: null },
b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b', effective_date: null, effective_date_source: null },
combined_score: 1,
};
}
@@ -26,8 +26,8 @@ function mkCrossSlugPair(slugA: string, slugB: string): ContradictionPair {
function mkIntraPagePair(pageSlug: string, takeId: number): ContradictionPair {
return {
kind: 'intra_page_chunk_take',
a: { slug: pageSlug, chunk_id: 5, take_id: null, source_tier: 'curated', holder: null, text: 'chunk text' },
b: { slug: pageSlug, chunk_id: null, take_id: takeId, source_tier: 'curated', holder: 'garry', text: 'take claim' },
a: { slug: pageSlug, chunk_id: 5, take_id: null, source_tier: 'curated', holder: null, text: 'chunk text', effective_date: null, effective_date_source: null },
b: { slug: pageSlug, chunk_id: null, take_id: takeId, source_tier: 'curated', holder: 'garry', text: 'take claim', effective_date: null, effective_date_source: null },
combined_score: 1,
};
}
@@ -120,13 +120,14 @@ describe('pairToFinding', () => {
test('merges pair + verdict into a finding', () => {
const pair = mkIntraPagePair('people/alice', 7);
const verdict: JudgeVerdict = {
contradicts: true,
verdict: 'contradiction',
severity: 'high',
axis: 'CFO role status',
confidence: 0.92,
resolution_kind: 'takes_supersede',
};
const finding = pairToFinding(pair, verdict);
expect(finding.verdict).toBe('contradiction');
expect(finding.severity).toBe('high');
expect(finding.axis).toBe('CFO role status');
expect(finding.confidence).toBe(0.92);
test/eval-contradictions-cache.test.ts
+3 -3
View File
@@ -31,7 +31,7 @@ beforeEach(async () => {
});
const verdictHit: JudgeVerdict = {
contradicts: true,
verdict: 'contradiction',
severity: 'medium',
axis: 'MRR vs ARR',
confidence: 0.85,
@@ -87,7 +87,7 @@ describe('JudgeCache wrapper', () => {
await cache.store('text-a', 'text-b', verdictHit);
const hit = await cache.lookup('text-a', 'text-b');
expect(hit).not.toBeNull();
expect(hit?.contradicts).toBe(true);
expect(hit?.verdict).toBe('contradiction');
expect(hit?.severity).toBe('medium');
expect(cache.stats().hits).toBe(1);
});
@@ -97,7 +97,7 @@ describe('JudgeCache wrapper', () => {
await cache.store('first', 'second', verdictHit);
const hit = await cache.lookup('second', 'first');
expect(hit).not.toBeNull();
expect(hit?.contradicts).toBe(true);
expect(hit?.verdict).toBe('contradiction');
});
test('different model_id is a separate key', async () => {
test/eval-contradictions-cost-prompt.test.ts
+212
View File
@@ -0,0 +1,212 @@
/**
* Lane C cost-prompt helper tests — hermetic via injected waitFn + stderrWriter.
*
* Pins the four decision branches of `maybePromptForCostBeforeProbe`:
* - --yes override skips
* - GBRAIN_NO_PROBE_PROMPT=1 env var skips
* - prompt_version unchanged from the last persisted run skips (no surprise)
* - non-TTY auto-proceeds with a stderr note (autopilot path)
* - TTY proceeds after the grace window
* - TTY aborts on Ctrl-C
*
* Plus reads the last prompt_version via a PGLite-backed eval_contradictions_runs
* table to prove the cross-process state actually round-trips.
*/
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
import { resetPgliteState } from './helpers/reset-pglite.ts';
import { maybePromptForCostBeforeProbe } from '../src/core/eval-contradictions/cost-prompt.ts';
import { PROMPT_VERSION } from '../src/core/eval-contradictions/types.ts';
import { writeRunRow } from '../src/core/eval-contradictions/trends.ts';
import type { ProbeReport } from '../src/core/eval-contradictions/types.ts';
import { withEnv } from './helpers/with-env.ts';
let engine: PGLiteEngine;
beforeAll(async () => {
engine = new PGLiteEngine();
await engine.connect({});
await engine.initSchema();
});
afterAll(async () => {
await engine.disconnect();
});
beforeEach(async () => {
await resetPgliteState(engine);
});
function captureStderr(): { lines: string[]; write: (s: string) => void } {
const lines: string[] = [];
return { lines, write: (s: string) => { lines.push(s); } };
}
function mkBaseOpts(overrides: Partial<Parameters<typeof maybePromptForCostBeforeProbe>[0]> = {}) {
const stderr = captureStderr();
return {
capture: stderr,
opts: {
engine,
queryCount: 50,
topK: 5,
judgeModel: 'anthropic:claude-haiku-4-5',
stderrWriter: stderr.write,
...overrides,
},
};
}
describe('maybePromptForCostBeforeProbe', () => {
test('--yes override returns proceed without checking versions', async () => {
const { opts, capture } = mkBaseOpts({ yesOverride: true });
const r = await maybePromptForCostBeforeProbe(opts);
expect(r.kind).toBe('proceed');
if (r.kind === 'proceed') expect(r.reason).toBe('yes_override');
expect(capture.lines.length).toBe(0);
});
test('GBRAIN_NO_PROBE_PROMPT=1 skips entirely', async () => {
await withEnv({ GBRAIN_NO_PROBE_PROMPT: '1' }, async () => {
const { opts, capture } = mkBaseOpts();
const r = await maybePromptForCostBeforeProbe(opts);
expect(r.kind).toBe('proceed');
if (r.kind === 'proceed') expect(r.reason).toBe('env_skip');
expect(capture.lines.length).toBe(0);
});
});
test('prompt_version unchanged from last run → skip (no surprise)', async () => {
// Seed a run with the CURRENT PROMPT_VERSION so the comparison returns equal.
const report: ProbeReport = mkSeedReport(PROMPT_VERSION);
await writeRunRow(engine, report, 100);
const { opts, capture } = mkBaseOpts();
const r = await maybePromptForCostBeforeProbe(opts);
expect(r.kind).toBe('proceed');
if (r.kind === 'proceed') expect(r.reason).toBe('no_version_change');
expect(capture.lines.length).toBe(0);
});
test('non-TTY auto-proceeds with stderr note when version changed', async () => {
// Seed a run with an OLDER prompt_version so the comparison detects change.
const report: ProbeReport = mkSeedReport('1');
await writeRunRow(engine, report, 100);
const { opts, capture } = mkBaseOpts({ isTtyOverride: false });
const r = await maybePromptForCostBeforeProbe(opts);
expect(r.kind).toBe('proceed');
if (r.kind === 'proceed') expect(r.reason).toBe('non_tty_auto');
const out = capture.lines.join('');
expect(out).toContain('PROMPT_VERSION changed');
expect(out).toContain('Non-TTY');
});
test('TTY proceeds after the grace window (waitFn returns proceed)', async () => {
const report: ProbeReport = mkSeedReport('1');
await writeRunRow(engine, report, 100);
let waitedSeconds = -1;
const { opts, capture } = mkBaseOpts({
isTtyOverride: true,
waitFn: async (s) => { waitedSeconds = s; return 'proceed'; },
});
const r = await maybePromptForCostBeforeProbe(opts);
expect(r.kind).toBe('proceed');
if (r.kind === 'proceed') expect(r.reason).toBe('tty_proceed');
expect(waitedSeconds).toBeGreaterThan(0); // default 10s
const out = capture.lines.join('');
expect(out).toContain('Press Ctrl-C');
});
test('TTY aborts on Ctrl-C (waitFn returns abort)', async () => {
const report: ProbeReport = mkSeedReport('1');
await writeRunRow(engine, report, 100);
const { opts, capture } = mkBaseOpts({
isTtyOverride: true,
waitFn: async () => 'abort',
});
const r = await maybePromptForCostBeforeProbe(opts);
expect(r.kind).toBe('abort');
if (r.kind === 'abort') expect(r.reason).toBe('tty_ctrl_c');
const out = capture.lines.join('');
expect(out).toContain('aborted by Ctrl-C');
});
test('fresh brain (no prior runs) fires the prompt on first run', async () => {
// No seed — readLastPromptVersion returns null.
const { opts, capture } = mkBaseOpts({
isTtyOverride: true,
waitFn: async () => 'proceed',
});
const r = await maybePromptForCostBeforeProbe(opts);
expect(r.kind).toBe('proceed');
if (r.kind === 'proceed') expect(r.reason).toBe('tty_proceed');
const out = capture.lines.join('');
expect(out).toContain('PROMPT_VERSION changed (none →');
});
test('GBRAIN_PROBE_PROMPT_GRACE_SECONDS env overrides the 10s default', async () => {
await withEnv({ GBRAIN_PROBE_PROMPT_GRACE_SECONDS: '0' }, async () => {
let waitedSeconds = -1;
const report: ProbeReport = mkSeedReport('1');
await writeRunRow(engine, report, 100);
const { opts } = mkBaseOpts({
isTtyOverride: true,
waitFn: async (s) => { waitedSeconds = s; return 'proceed'; },
});
await maybePromptForCostBeforeProbe(opts);
expect(waitedSeconds).toBe(0);
});
});
test('estimate scales with query count + judge model in the banner text', async () => {
const report: ProbeReport = mkSeedReport('1');
await writeRunRow(engine, report, 100);
const { opts, capture } = mkBaseOpts({
isTtyOverride: false,
queryCount: 500,
judgeModel: 'anthropic:claude-sonnet-4-6',
});
await maybePromptForCostBeforeProbe(opts);
const out = capture.lines.join('');
expect(out).toContain('500 queries');
expect(out).toContain('anthropic:claude-sonnet-4-6');
expect(out).toMatch(/\$\d+\.\d{2}/);
});
});
function mkSeedReport(promptVersion: string): ProbeReport {
return {
schema_version: 1,
run_id: `seed-${promptVersion}`,
judge_model: 'anthropic:claude-haiku-4-5',
prompt_version: promptVersion,
truncation_policy: '1500-chars-utf8-safe',
top_k: 5,
sampling: 'deterministic',
queries_evaluated: 50,
queries_with_contradiction: 12,
queries_with_any_finding: 12,
total_contradictions_flagged: 12,
verdict_breakdown: {
no_contradiction: 100,
contradiction: 12,
temporal_supersession: 0,
temporal_regression: 0,
temporal_evolution: 0,
negation_artifact: 0,
},
calibration: {
queries_total: 50,
queries_judged_clean: 38,
queries_with_contradiction: 12,
wilson_ci_95: { point: 0.24, lower: 0.14, upper: 0.37 },
},
judge_errors: { parse_fail: 0, refusal: 0, timeout: 0, http_5xx: 0, unknown: 0, total: 0, note: '' },
cost_usd: { judge: 1.0, embedding: 0.005, total: 1.005, estimate_note: '' },
cache: { hits: 0, misses: 0, hit_rate: 0 },
duration_ms: 45000,
source_tier_breakdown: { curated_vs_curated: 0, curated_vs_bulk: 0, bulk_vs_bulk: 0, other: 0 },
per_query: [],
hot_pages: [],
};
}
test/eval-contradictions-cross-source.test.ts
+2 -2
View File
@@ -12,8 +12,8 @@ import type { ContradictionPair } from '../src/core/eval-contradictions/types.ts
function mkPair(slugA: string, slugB: string): ContradictionPair {
return {
kind: 'cross_slug_chunks',
a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' },
b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'curated', holder: null, text: 'b' },
a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a', effective_date: null, effective_date_source: null },
b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'curated', holder: null, text: 'b', effective_date: null, effective_date_source: null },
combined_score: 1,
};
}
test/eval-contradictions-date-filter.test.ts
+81
View File
@@ -139,4 +139,85 @@ describe('shouldSkipForDateMismatch', () => {
expect(a.length).toBe(b.length);
expect(a.length).toBe(3);
});
// ---- Lane B: page-level effective_date relaxes rule 3 ----
// R1 IRON RULE regression suite. When BOTH sides have a non-null
// page-level effective_date, the judge will see them via the
// (from: YYYY-MM-DD) prompt tag and can classify temporal supersession.
// The v1 >30d skip would silently kill these cases — exactly the ones the
// new verdict taxonomy exists to surface. The other two rules stay intact.
test('Lane B: both sides have effective_date → do NOT skip (judge classifies)', () => {
const d = shouldSkipForDateMismatch({
textA: 'Sriram Krishnan: Partner — AI/ML/Eng',
textB: 'Sriram Krishnan: Senior White House AI Policy Advisor',
effectiveDateA: '2017-03-28',
effectiveDateB: '2025-01-15',
});
expect(d.skip).toBe(false);
expect(d.reason).toBe('both_have_effective_date');
});
test('Lane B: both sides effective_date overrides >30d chunk-text rule', () => {
// Without effective_dates this would be rule-3 SKIP. With them the judge
// gets to classify temporal_supersession.
const d = shouldSkipForDateMismatch({
textA: 'Acme MRR was $50K (2024-08-01)',
textB: 'Acme MRR was $2M (2026-03-15)',
effectiveDateA: '2024-08-01',
effectiveDateB: '2026-03-15',
});
expect(d.skip).toBe(false);
expect(d.reason).toBe('both_have_effective_date');
});
test('Lane B regression: rule 1 (same-paragraph dual-date) still wins over effective_date', () => {
// Same-paragraph dual-date is checked BEFORE the effective_date branch
// so flip-flops in the chunk text always reach the judge regardless of
// page-level dates.
const d = shouldSkipForDateMismatch({
textA: 'In Jan 2024 I said X. In Mar 2024 I reversed to not-X.',
textB: 'In 2026 I still hold not-X.',
effectiveDateA: '2024-01-15',
effectiveDateB: '2026-06-01',
});
expect(d.skip).toBe(false);
expect(d.reason).toBe('same_paragraph_dual_date');
});
test('Lane B regression: rule 2 (one side missing chunk dates) still applies when effective_dates partially present', () => {
// Only ONE side has effective_date — the relaxation doesn't fire. v1
// rules 1 + 2 still govern.
const d = shouldSkipForDateMismatch({
textA: 'Acme is profitable',
textB: 'Acme is unprofitable as of 2026-03-01',
effectiveDateA: null,
effectiveDateB: '2026-03-01',
});
expect(d.skip).toBe(false);
expect(d.reason).toBe('one_or_both_missing_dates');
});
test('Lane B regression: undefined effective_dates fall through to v1 behavior', () => {
// Callers that don't supply effective_dates (legacy code paths) keep the
// v1 skip behavior. Important for back-compat.
const d = shouldSkipForDateMismatch({
textA: 'Acme MRR was $50K (2024-08-01)',
textB: 'Acme MRR was $2M (2026-03-15)',
});
expect(d.skip).toBe(true);
expect(d.reason).toBe('both_explicit_separated');
});
test('Lane B regression: empty-string effective_date is treated as missing', () => {
// An empty string is falsy; only a real date enables the relaxation.
const d = shouldSkipForDateMismatch({
textA: 'Acme MRR was $50K (2024-08-01)',
textB: 'Acme MRR was $2M (2026-03-15)',
effectiveDateA: '',
effectiveDateB: '2026-03-15',
});
expect(d.skip).toBe(true);
expect(d.reason).toBe('both_explicit_separated');
});
});
test/eval-contradictions-engine.test.ts
+6 -5
View File
@@ -175,7 +175,7 @@ describe('contradiction cache (P2)', () => {
truncation_policy: '1500-chars-utf8-safe',
};
const verdict = {
contradicts: true,
verdict: 'contradiction',
severity: 'medium',
axis: 'MRR vs ARR',
confidence: 0.85,
@@ -191,12 +191,13 @@ describe('contradiction cache (P2)', () => {
await engine.putContradictionCacheEntry({ ...baseKey, verdict });
const hit = await engine.getContradictionCacheEntry(baseKey);
expect(hit).not.toBeNull();
expect((hit as Record<string, unknown>).contradicts).toBe(true);
expect((hit as Record<string, unknown>).verdict).toBe('contradiction');
expect((hit as Record<string, unknown>).severity).toBe('medium');
});
test('different prompt_version is a different cache key (Codex fix)', async () => {
await engine.putContradictionCacheEntry({ ...baseKey, verdict });
// baseKey has prompt_version='1'; v0.34 lookups under '2' must miss.
const wrong = await engine.getContradictionCacheEntry({ ...baseKey, prompt_version: '2' });
expect(wrong).toBeNull();
});
@@ -211,11 +212,11 @@ describe('contradiction cache (P2)', () => {
await engine.putContradictionCacheEntry({ ...baseKey, verdict });
await engine.putContradictionCacheEntry({
...baseKey,
verdict: { ...verdict, contradicts: false, severity: 'low' },
verdict: { ...verdict, verdict: 'no_contradiction', severity: 'info' },
});
const hit = await engine.getContradictionCacheEntry(baseKey);
expect((hit as Record<string, unknown>).contradicts).toBe(false);
expect((hit as Record<string, unknown>).severity).toBe('low');
expect((hit as Record<string, unknown>).verdict).toBe('no_contradiction');
expect((hit as Record<string, unknown>).severity).toBe('info');
});
test('expired entries are not returned by get', async () => {
test/eval-contradictions-integrations.test.ts
+12 -2
View File
@@ -55,7 +55,16 @@ function mkReport(opts: Partial<ProbeReport> & {
sampling: 'deterministic',
queries_evaluated: 50,
queries_with_contradiction: findings.length > 0 ? Math.max(1, findings.length) : 0,
queries_with_any_finding: findings.length > 0 ? Math.max(1, findings.length) : 0,
total_contradictions_flagged: findings.length,
verdict_breakdown: {
no_contradiction: 50 - findings.length,
contradiction: findings.length,
temporal_supersession: 0,
temporal_regression: 0,
temporal_evolution: 0,
negation_artifact: 0,
},
calibration: {
queries_total: 50,
queries_judged_clean: 50 - findings.length,
@@ -76,9 +85,10 @@ function mkReport(opts: Partial<ProbeReport> & {
pairs_judged: findings.length,
contradictions: findings.map((f, i) => ({
kind: 'cross_slug_chunks' as const,
a: { slug: f.slugA, chunk_id: i + 1, take_id: null, source_tier: 'curated' as const, holder: null, text: 'a' },
b: { slug: f.slugB, chunk_id: i + 100, take_id: null, source_tier: 'bulk' as const, holder: null, text: 'b' },
a: { slug: f.slugA, chunk_id: i + 1, take_id: null, source_tier: 'curated' as const, holder: null, text: 'a', effective_date: null, effective_date_source: null },
b: { slug: f.slugB, chunk_id: i + 100, take_id: null, source_tier: 'bulk' as const, holder: null, text: 'b', effective_date: null, effective_date_source: null },
combined_score: 1.0,
verdict: 'contradiction' as const,
severity: f.severity,
axis: f.axis,
confidence: 0.85,
test/eval-contradictions-judge.test.ts
+81 -31
View File
@@ -109,27 +109,58 @@ describe('buildJudgePrompt', () => {
});
describe('normalizeVerdict', () => {
test('valid input passes through', () => {
test('valid contradiction verdict passes through', () => {
const v = normalizeVerdict({
contradicts: true,
verdict: 'contradiction',
severity: 'medium',
axis: 'MRR figure',
confidence: 0.85,
resolution_kind: 'dream_synthesize',
});
expect(v.contradicts).toBe(true);
expect(v.verdict).toBe('contradiction');
expect(v.severity).toBe('medium');
expect(v.confidence).toBe(0.85);
expect(v.resolution_kind).toBe('dream_synthesize');
});
test('throws on missing contradicts field', () => {
test('valid temporal_supersession verdict passes through (Lane A2)', () => {
const v = normalizeVerdict({
verdict: 'temporal_supersession',
severity: 'info',
axis: 'role over time',
confidence: 0.9,
resolution_kind: 'temporal_supersede',
});
expect(v.verdict).toBe('temporal_supersession');
expect(v.severity).toBe('info');
expect(v.resolution_kind).toBe('temporal_supersede');
});
test('all 6 verdict members accepted by parser', () => {
for (const verdict of [
'no_contradiction',
'contradiction',
'temporal_supersession',
'temporal_regression',
'temporal_evolution',
'negation_artifact',
] as const) {
const v = normalizeVerdict({ verdict, severity: 'medium', confidence: 0.8 });
expect(v.verdict).toBe(verdict);
}
});
test('throws on missing verdict field', () => {
expect(() => normalizeVerdict({ confidence: 0.9 })).toThrow();
});
test('throws on invalid verdict string', () => {
expect(() => normalizeVerdict({ verdict: 'maybe', confidence: 0.9 })).toThrow();
});
test('throws on invalid confidence', () => {
expect(() => normalizeVerdict({ contradicts: true, confidence: 'high' })).toThrow();
expect(() => normalizeVerdict({ contradicts: true, confidence: NaN })).toThrow();
expect(() => normalizeVerdict({ verdict: 'contradiction', confidence: 'high' })).toThrow();
expect(() => normalizeVerdict({ verdict: 'contradiction', confidence: NaN })).toThrow();
});
test('throws on missing or non-object input', () => {
@@ -138,49 +169,68 @@ describe('normalizeVerdict', () => {
expect(() => normalizeVerdict('json string')).toThrow();
});
test('C1 double-enforce: contradicts:true + confidence<0.7 downgrades to false', () => {
test('C1 double-enforce: verdict:contradiction + confidence<0.7 downgrades to no_contradiction', () => {
const v = normalizeVerdict({
contradicts: true,
verdict: 'contradiction',
severity: 'high',
axis: 'something',
confidence: 0.6,
resolution_kind: 'takes_supersede',
});
expect(v.contradicts).toBe(false);
expect(v.verdict).toBe('no_contradiction');
expect(v.axis).toBe('');
expect(v.resolution_kind).toBeNull();
});
test('C1 boundary: confidence exactly 0.7 stays as contradicts:true', () => {
test('C1 boundary: confidence exactly 0.7 stays as verdict:contradiction', () => {
const v = normalizeVerdict({
contradicts: true,
verdict: 'contradiction',
severity: 'medium',
axis: 'something',
confidence: 0.7,
});
expect(v.contradicts).toBe(true);
expect(v.verdict).toBe('contradiction');
expect(v.confidence).toBe(0.7);
});
test('C1 floor does NOT downgrade other verdicts on low confidence (Lane A2)', () => {
const v = normalizeVerdict({
verdict: 'temporal_supersession',
severity: 'info',
confidence: 0.4,
});
// Non-contradiction verdicts have no confidence floor; they survive at 0.4.
expect(v.verdict).toBe('temporal_supersession');
});
test('clamps confidence into [0, 1]', () => {
const v1 = normalizeVerdict({ contradicts: false, severity: 'low', confidence: -0.5 });
const v1 = normalizeVerdict({ verdict: 'no_contradiction', severity: 'low', confidence: -0.5 });
expect(v1.confidence).toBe(0);
const v2 = normalizeVerdict({ contradicts: false, severity: 'low', confidence: 1.5 });
const v2 = normalizeVerdict({ verdict: 'no_contradiction', severity: 'low', confidence: 1.5 });
expect(v2.confidence).toBe(1);
});
test('garbage severity defaults to low', () => {
test('garbage severity falls back to defaultSeverityForVerdict (no_contradiction → info)', () => {
const v = normalizeVerdict({
contradicts: false,
verdict: 'no_contradiction',
severity: 'critical',
confidence: 0.5,
});
expect(v.severity).toBe('low');
expect(v.severity).toBe('info');
});
test('unknown resolution_kind on contradicts:true falls back to manual_review', () => {
test('garbage severity falls back to defaultSeverityForVerdict (contradiction → medium)', () => {
const v = normalizeVerdict({
contradicts: true,
verdict: 'contradiction',
severity: 'critical',
confidence: 0.85,
});
expect(v.severity).toBe('medium');
});
test('unknown resolution_kind on contradiction falls back to manual_review (legacy)', () => {
const v = normalizeVerdict({
verdict: 'contradiction',
severity: 'medium',
axis: 'X',
confidence: 0.85,
@@ -189,9 +239,9 @@ describe('normalizeVerdict', () => {
expect(v.resolution_kind).toBe('manual_review');
});
test('axis cleared when contradicts:false', () => {
test('axis cleared when verdict:no_contradiction', () => {
const v = normalizeVerdict({
contradicts: false,
verdict: 'no_contradiction',
severity: 'low',
axis: 'some axis',
confidence: 0.4,
@@ -212,14 +262,14 @@ describe('judgeContradiction', () => {
const out = await judgeContradiction({
...baseInput,
chatFn: stubChat(mkResult(JSON.stringify({
contradicts: true,
verdict: 'contradiction',
severity: 'medium',
axis: 'MRR figure',
confidence: 0.85,
resolution_kind: 'dream_synthesize',
}))),
});
expect(out.verdict.contradicts).toBe(true);
expect(out.verdict.verdict).toBe('contradiction');
expect(out.verdict.severity).toBe('medium');
expect(out.usage.inputTokens).toBe(100);
expect(out.usage.outputTokens).toBe(50);
@@ -227,15 +277,15 @@ describe('judgeContradiction', () => {
test('fence-wrapped JSON: parseModelJSON 4-strategy fallback', async () => {
const fenced = '```json\n' + JSON.stringify({
contradicts: false,
severity: 'low',
verdict: 'no_contradiction',
severity: 'info',
confidence: 0.3,
}) + '\n```';
const out = await judgeContradiction({
...baseInput,
chatFn: stubChat(mkResult(fenced)),
});
expect(out.verdict.contradicts).toBe(false);
expect(out.verdict.verdict).toBe('no_contradiction');
});
test('throws on parse failure (counted in judge_errors)', async () => {
@@ -276,7 +326,7 @@ describe('judgeContradiction', () => {
const userMsg = opts.messages.find((m) => m.role === 'user');
capturedPrompt = typeof userMsg?.content === 'string' ? userMsg.content : '';
return mkResult(JSON.stringify({
contradicts: false, severity: 'low', confidence: 0.5,
verdict: 'no_contradiction', severity: 'info', confidence: 0.5,
}));
}),
});
@@ -287,17 +337,17 @@ describe('judgeContradiction', () => {
expect(DEFAULT_MAX_PAIR_CHARS).toBe(1500);
});
test('C1 enforcement reaches the verdict (low-confidence true → false)', async () => {
test('C1 enforcement reaches the verdict (low-confidence contradiction → no_contradiction)', async () => {
const out = await judgeContradiction({
...baseInput,
chatFn: stubChat(mkResult(JSON.stringify({
contradicts: true,
verdict: 'contradiction',
severity: 'high',
axis: 'something',
confidence: 0.5,
}))),
});
expect(out.verdict.contradicts).toBe(false);
expect(out.verdict.verdict).toBe('no_contradiction');
});
test('query appears in the rendered prompt (Codex fix)', async () => {
@@ -308,7 +358,7 @@ describe('judgeContradiction', () => {
chatFn: stubChat(async (opts) => {
const m = opts.messages[0]?.content;
capturedQuery = typeof m === 'string' ? m : '';
return mkResult(JSON.stringify({ contradicts: false, severity: 'low', confidence: 0.4 }));
return mkResult(JSON.stringify({ verdict: 'no_contradiction', severity: 'info', confidence: 0.4 }));
}),
});
expect(capturedQuery).toContain('distinctive-query-marker-12345');
test/eval-contradictions-runner.test.ts
+96 -4
View File
@@ -62,8 +62,15 @@ function mkResult(slug: string, page_id: number, chunk_id: number, text: string,
/** Stubbed judge that returns a fixed verdict pattern. */
function stubJudge(opts: {
/**
* v0.34 / Lane A2: `verdict` replaces `contradicts`. Legacy `contradicts:
* true` maps to verdict='contradiction'; `contradicts: false` to
* verdict='no_contradiction'. Callers can pass either field; verdict wins
* when both are set.
*/
verdict?: 'no_contradiction' | 'contradiction' | 'temporal_supersession' | 'temporal_regression' | 'temporal_evolution' | 'negation_artifact';
contradicts?: boolean;
severity?: 'low' | 'medium' | 'high';
severity?: 'info' | 'low' | 'medium' | 'high';
confidence?: number;
inputTokens?: number;
outputTokens?: number;
@@ -75,9 +82,11 @@ function stubJudge(opts: {
if (opts.throwOn && opts.throwOn(idx)) {
throw new Error('stub: simulated transient 503');
}
const resolvedVerdict =
opts.verdict ?? (opts.contradicts === false ? 'no_contradiction' : 'contradiction');
return {
verdict: {
contradicts: opts.contradicts ?? true,
verdict: resolvedVerdict,
severity: opts.severity ?? 'medium',
axis: 'stub axis',
confidence: opts.confidence ?? 0.85,
@@ -374,7 +383,7 @@ describe('runContradictionProbe', () => {
// Yield to let the abort fire.
await new Promise((r) => setTimeout(r, 1));
return {
verdict: { contradicts: false, severity: 'low', axis: '', confidence: 0.3, resolution_kind: null },
verdict: { verdict: 'no_contradiction', severity: 'info', axis: '', confidence: 0.3, resolution_kind: null },
usage: { inputTokens: 1, outputTokens: 1 },
};
},
@@ -418,7 +427,7 @@ describe('runContradictionProbe', () => {
const recordOrder = (out: string[]): JudgeFn => async (input) => {
out.push(`${input.a.slug}|${input.b.slug}`);
return {
verdict: { contradicts: false, severity: 'low', axis: '', confidence: 0.4, resolution_kind: null },
verdict: { verdict: 'no_contradiction', severity: 'info', axis: '', confidence: 0.4, resolution_kind: null },
usage: { inputTokens: 1, outputTokens: 1 },
};
};
@@ -432,4 +441,87 @@ describe('runContradictionProbe', () => {
});
expect(order1).toEqual(order2);
});
// ---- Lane D: R4 regression — runner emits findings for every non-no_contradiction verdict ----
// Without the runner.ts:307-ish emit-rule change from Lane A2, the new
// verdicts (temporal_supersession, temporal_regression, temporal_evolution,
// negation_artifact) would silently vanish from the report. This test
// pins the broadened emit predicate one verdict at a time.
for (const verdict of [
'contradiction',
'temporal_supersession',
'temporal_regression',
'temporal_evolution',
'negation_artifact',
] as const) {
test(`R4 regression: verdict='${verdict}' surfaces as a finding`, async () => {
const idA = await seedPage(`a/${verdict}`, 'A');
const idB = await seedPage(`b/${verdict}`, 'B');
const out = await runContradictionProbe({
engine,
queries: [`q-${verdict}`],
judgeFn: stubJudge({ verdict, severity: 'medium', confidence: 0.85 }),
searchFn: async () => [
mkResult(`a/${verdict}`, idA, 1, `text-a-${verdict}`, 1.0),
mkResult(`b/${verdict}`, idB, 2, `text-b-${verdict}`, 0.5),
],
budgetUsd: 5,
noCache: true,
});
expect(out.report.total_contradictions_flagged).toBe(1);
expect(out.report.per_query[0].contradictions.length).toBe(1);
expect(out.report.per_query[0].contradictions[0].verdict).toBe(verdict);
// Only verdict='contradiction' counts toward the strict
// queries_with_contradiction metric (Wilson-CI denominator).
if (verdict === 'contradiction') {
expect(out.report.queries_with_contradiction).toBe(1);
} else {
expect(out.report.queries_with_contradiction).toBe(0);
}
// Every non-no_contradiction verdict counts toward queries_with_any_finding.
expect(out.report.queries_with_any_finding).toBe(1);
// Per-verdict breakdown tallies correctly.
expect(out.report.verdict_breakdown[verdict]).toBe(1);
});
}
test(`R4 regression: verdict='no_contradiction' produces ZERO findings`, async () => {
const idA = await seedPage('a/noc', 'A');
const idB = await seedPage('b/noc', 'B');
const out = await runContradictionProbe({
engine,
queries: ['q-noc'],
judgeFn: stubJudge({ verdict: 'no_contradiction', severity: 'info', confidence: 0.85 }),
searchFn: async () => [
mkResult('a/noc', idA, 1, 'no contradiction here', 1.0),
mkResult('b/noc', idB, 2, 'also no contradiction', 0.5),
],
budgetUsd: 5,
noCache: true,
});
expect(out.report.total_contradictions_flagged).toBe(0);
expect(out.report.per_query[0].contradictions.length).toBe(0);
expect(out.report.queries_with_contradiction).toBe(0);
expect(out.report.queries_with_any_finding).toBe(0);
expect(out.report.verdict_breakdown.no_contradiction).toBe(1);
});
// ---- Lane D: R5 regression — cache key tuple shape stays 5 fields ----
// Lane A1 bumped PROMPT_VERSION 1→2 (invalidates cache) but kept the key
// shape unchanged. Adding a 6th field would silently break every operator's
// brain (no migration path).
test('R5 regression: cache key tuple shape is exactly 5 fields', async () => {
const { buildCacheKey } = await import('../src/core/eval-contradictions/cache.ts');
const key = buildCacheKey({ textA: 'a', textB: 'b', modelId: 'haiku' });
const keys = Object.keys(key).sort();
expect(keys).toEqual([
'chunk_a_hash',
'chunk_b_hash',
'model_id',
'prompt_version',
'truncation_policy',
]);
});
});
test/eval-contradictions-severity.test.ts
+58
View File
@@ -25,6 +25,8 @@ function mkFinding(opts: {
source_tier: 'curated',
holder: null,
text: 'A',
effective_date: null,
effective_date_source: null,
},
b: {
slug: opts.slugB,
@@ -33,8 +35,11 @@ function mkFinding(opts: {
source_tier: 'bulk',
holder: null,
text: 'B',
effective_date: null,
effective_date_source: null,
},
combined_score: 1,
verdict: 'contradiction',
severity: opts.severity,
axis: 'test',
confidence: 0.9,
@@ -155,3 +160,56 @@ describe('buildHotPages', () => {
expect(buildHotPages(findings, 5).length).toBe(5);
});
});
// ---- Lane D: R6 regression — contradiction severity unchanged ----
// The v1 `verdict === 'contradiction'` semantics include severity coming from
// the judge with a 'low' fallback (legacy). v2 must preserve this for the
// contradiction verdict specifically: garbage severity → 'medium' (the new
// per-verdict default for contradiction, NOT 'low' which would imply the
// contradiction is naming-cosmetic).
describe('R6 regression: contradiction verdict severity preserved', () => {
test('judge-set severity wins over defaultSeverityForVerdict', async () => {
const { normalizeVerdict } = await import('../src/core/eval-contradictions/judge.ts');
// Judge explicitly says 'high'. The v2 contract: that wins, even though
// defaultSeverityForVerdict('contradiction') is 'medium'.
const v = normalizeVerdict({
verdict: 'contradiction',
severity: 'high',
axis: 'CFO role',
confidence: 0.85,
});
expect(v.severity).toBe('high');
});
test('judge-set severity also wins when judge picks low (legacy behavior preserved)', async () => {
const { normalizeVerdict } = await import('../src/core/eval-contradictions/judge.ts');
const v = normalizeVerdict({
verdict: 'contradiction',
severity: 'low',
axis: 'name format',
confidence: 0.85,
});
expect(v.severity).toBe('low');
});
test('garbage severity on contradiction falls back to medium (NOT low — that would mask conflicts)', async () => {
const { normalizeVerdict } = await import('../src/core/eval-contradictions/judge.ts');
const v = normalizeVerdict({
verdict: 'contradiction',
severity: 'critical',
confidence: 0.85,
});
expect(v.severity).toBe('medium');
});
test('garbage severity on temporal_regression falls back to high (real signal)', async () => {
const { normalizeVerdict } = await import('../src/core/eval-contradictions/judge.ts');
const v = normalizeVerdict({
verdict: 'temporal_regression',
severity: 'critical',
confidence: 0.85,
});
expect(v.severity).toBe('high');
});
});
test/eval-contradictions-trends.test.ts
+10 -1
View File
@@ -33,13 +33,22 @@ function mkReport(runId: string, overrides: Partial<ProbeReport> = {}): ProbeRep
schema_version: 1,
run_id: runId,
judge_model: 'anthropic:claude-haiku-4-5',
prompt_version: '1',
prompt_version: '2',
truncation_policy: '1500-chars-utf8-safe',
top_k: 5,
sampling: 'deterministic',
queries_evaluated: 50,
queries_with_contradiction: 12,
queries_with_any_finding: 18,
total_contradictions_flagged: 18,
verdict_breakdown: {
no_contradiction: 200,
contradiction: 12,
temporal_supersession: 4,
temporal_regression: 1,
temporal_evolution: 1,
negation_artifact: 0,
},
calibration: {
queries_total: 50,
queries_judged_clean: 38,
test/scripts/check-proposal-pii.test.ts
+152
View File
@@ -0,0 +1,152 @@
/**
* Tests for scripts/check-proposal-pii.sh — the privacy guard for
* docs/proposals/*.md.
*
* Strategy: each test case builds a tiny git-initialized scratch repo
* with a docs/proposals/*.md file containing the case input, invokes
* the script (which lives back in the real repo), and asserts the exit
* code + stderr signal. The script reads from `git rev-parse
* --show-toplevel` so each scratch repo behaves like a real project.
*/
import { describe, test, expect } from 'bun:test';
import { spawnSync } from 'node:child_process';
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'node:fs';
import { join } from 'node:path';
import { tmpdir } from 'node:os';
const SCRIPT_PATH = join(import.meta.dir, '..', '..', 'scripts', 'check-proposal-pii.sh');
function runLintIn(content: string | null): { exitCode: number; stderr: string } {
const dir = mkdtempSync(join(tmpdir(), 'gbrain-pii-lint-'));
try {
// Initialize as git repo so `git rev-parse --show-toplevel` resolves.
spawnSync('git', ['init', '-q'], { cwd: dir });
if (content !== null) {
mkdirSync(join(dir, 'docs', 'proposals'), { recursive: true });
writeFileSync(join(dir, 'docs', 'proposals', 'test.md'), content);
}
const r = spawnSync('bash', [SCRIPT_PATH], { cwd: dir, encoding: 'utf8' });
return { exitCode: r.status ?? -1, stderr: r.stderr ?? '' };
} finally {
rmSync(dir, { recursive: true, force: true });
}
}
describe('check-proposal-pii.sh', () => {
test('clean proposal passes (exit 0)', () => {
const r = runLintIn(`# RFC
This is a clean technical proposal using only generic placeholders:
alice-example, acme-corp, fund-a. Two roles, one decision.
`);
expect(r.exitCode).toBe(0);
expect(r.stderr).toBe('');
});
test('garrytan/brain private repo reference is flagged', () => {
const r = runLintIn(`Context: findings resolved in garrytan/brain.\n`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain('garrytan/brain');
});
test('trial separation phrase is flagged', () => {
const r = runLintIn(`A: trial separation\nB: confirmed status\n`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain('trial separation');
});
test('permanent separation phrase is flagged', () => {
const r = runLintIn(`status: permanent separation, confirmed\n`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain('permanent separation');
});
test('couples session phrase is flagged', () => {
const r = runLintIn(`per the couples session on 2026-05-07\n`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain('couples session');
});
test('divorce attorney phrase is flagged', () => {
const r = runLintIn(`consultation with divorce attorney scheduled\n`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain('divorce attorney');
});
test("grandmother's funeral phrase is flagged", () => {
const r = runLintIn(`Traveled for the grandmother's funeral last week.\n`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain("grandmother's funeral");
});
test("aunt's funeral phrase is flagged", () => {
const r = runLintIn(`Traveled to the aunt's funeral in Toronto.\n`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain("aunt's funeral");
});
test('wintermute is flagged (consistent with check-privacy.sh)', () => {
const r = runLintIn(`Author: Wintermute (via the user)\n`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain('wintermute');
});
test('case-insensitive match: WINTERMUTE all-caps also flagged', () => {
const r = runLintIn(`See the WINTERMUTE deployment.\n`);
expect(r.exitCode).toBe(1);
});
test('benign separation usage stays clean (separation-of-concerns)', () => {
// "separation" alone is fine; only "trial separation" / "permanent
// separation" are flagged. Software vocabulary survives.
const r = runLintIn(`# Architecture
Strict separation of concerns between the runner and the judge module.
The trial run uses a separate test database.
`);
expect(r.exitCode).toBe(0);
});
test('benign funeral metaphor stays clean (bare "funeral" not banned)', () => {
// Only combined personal-context phrases are banned. Bare "funeral"
// in a software metaphor survives the lint.
const r = runLintIn(`Killing the deprecated endpoint feels like a funeral.\n`);
expect(r.exitCode).toBe(0);
});
test('multiple PII patterns reported together with exit 1', () => {
const r = runLintIn(`
Context: findings from garrytan/brain.
A: trial separation
B: permanent separation per the couples session
`);
expect(r.exitCode).toBe(1);
expect(r.stderr).toContain('garrytan/brain');
expect(r.stderr).toContain('trial separation');
expect(r.stderr).toContain('permanent separation');
expect(r.stderr).toContain('couples session');
// Summary count.
expect(r.stderr).toContain('4 PII pattern hit(s)');
});
test('no proposals dir → exits 0 (not a failure)', () => {
const r = runLintIn(null);
expect(r.exitCode).toBe(0);
});
test('--help exits 1 with usage text', () => {
const dir = mkdtempSync(join(tmpdir(), 'gbrain-pii-lint-help-'));
try {
spawnSync('git', ['init', '-q'], { cwd: dir });
const r = spawnSync('bash', [SCRIPT_PATH, '--help'], { cwd: dir, encoding: 'utf8' });
expect(r.status).toBe(1);
const out = (r.stdout ?? '') + (r.stderr ?? '');
expect(out).toContain('check-proposal-pii.sh');
expect(out).toContain('docs/proposals');
} finally {
rmSync(dir, { recursive: true, force: true });
}
});
});