mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* rfc: temporal axis for contradiction probe Field report on residual HIGH findings from gbrain eval suspected-contradictions and proposal for a 4-phase fix (Phase 1 = judge prompt + verdict enum is the recommended starting point). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): pass effective_date to judge prompt; bump PROMPT_VERSION Lane A1 of the temporal-contradiction-probe wave. Threads page-level effective_date through the search projection into the contradiction judge so the LLM can reason about supersession instead of treating every dated pair as a contradiction. Changes: - SearchResult interface adds optional effective_date + effective_date_source fields; rowToSearchResult populates them from the row data with date-only YYYY-MM-DD normalization (handles both postgres.js Date and PGLite string). - 8 SELECT projection sites (3 in postgres-engine, 5 in pglite-engine) now carry p.effective_date + p.effective_date_source through their inner CTEs and outer SELECTs so search results expose the field on both engines. - PairMember (eval-contradictions/types.ts) gets the two fields as required (string | null) so the type forces every constructor to think about temporal anchoring. Runner's searchResultToMember + takeToMember handle the normalization; takes inherit the chunk's page-level date. - buildJudgePrompt emits `Statement A (from: YYYY-MM-DD)` when effective_date is non-null, else `(date unknown)`. Prompt instructions explain the tag so the model knows what to do with it. - PROMPT_VERSION bumps '1' → '2'. Cache-key tuple shape unchanged; old rows miss naturally on first run against the new prompt. Test fixtures in 5 files updated to include the new required fields. All 205 eval-contradictions unit tests + 101 search-related tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): replace contradicts:boolean with verdict:enum (6 members) Lane A2 of the temporal-contradiction-probe wave. Expands the judge's classification vocabulary from a binary contradicts:bool to a six-member verdict enum so the probe can distinguish "this changed" from "this is wrong". Verdict taxonomy: no_contradiction — drop from findings contradiction — genuine conflict at same point in time temporal_supersession — newer claim updates/replaces older; not an error temporal_regression — metric/status went backwards over time (signal) temporal_evolution — legitimate change, neither supersession nor regression negation_artifact — judge misread an explicit negation Changes: - types.ts: Verdict union (6 members); Severity gains 'info'; ResolutionKind extended with temporal_supersede, flag_for_review, log_timeline_change; JudgeVerdict.contradicts → verdict; ContradictionFinding now carries verdict; ProbeReport adds queries_with_any_finding + verdict_breakdown (additive). - judge.ts: parseResolutionKind + parseVerdict guards; normalizeVerdict reads the new field and applies the C1 confidence floor only to verdict='contradiction' (the new verdicts are informational classifications, no floor). Prompt rubric rewritten to ask for verdict + extended severity scale. - severity-classify.ts: 'info' joins the rank with value 0; defaultSeverityForVerdict maps each verdict to its baseline severity (D7 — supersession=info, regression=high, etc.). parseSeverity gains a fallback param so consumers can override 'low' default. - auto-supersession.ts: classifyResolution + renderResolutionCommand handle the three new resolution kinds. Probe still NEVER auto-mutates — the new kinds render paste-ready commands or informational lines. - cache.ts: isJudgeVerdict shape check matches the new verdict field; old v1 rows fail the guard and treat as misses. - runner.ts: emit predicate at cache-hit and judge-success branches changes from `verdict.contradicts` to `verdict.verdict !== 'no_contradiction'`. Without this, the new verdicts vanish from the report. Added per-verdict tally + queriesWithAnyFinding alongside the strict queriesWithContradiction. - trends.ts: latest run verdict breakdown surfaces in the trend chart. Test fixtures updated across 8 test files. All 210 eval-contradictions unit tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(eval): relax date-filter rule 3 when both sides dated Lane B of the temporal-contradiction-probe wave. The v1 date pre-filter skipped pairs whose chunk-text-extracted dates differed by >30 days as a cost-saving heuristic. That heuristic silently killed exactly the cases the new verdict taxonomy exists to surface — role transitions across years (e.g. a 2017 historical record vs. a 2025 current state), MRR claims years apart, status changes recorded over time. Lane A1+A2 made temporal supersession explicit and cheap to classify. The filter no longer needs to skip these pairs; the judge can label them. Changes: - date-filter.ts: shouldSkipForDateMismatch accepts optional effectiveDateA and effectiveDateB. When BOTH are non-null, returns skip=false with the new 'both_have_effective_date' reason — the judge will see the dates via the (from: YYYY-MM-DD) prompt tag from Lane A1. Other rules (same-paragraph dual-date override, missing-date fallback) preserved verbatim and still run first. - runner.ts: threads pair.{a,b}.effective_date into the date-filter call. Pairs that previously vanished into the skip bucket now reach the judge. Tests (R1 IRON RULE regression suite, 6 new cases): - both sides effective_date → not skipped - both sides effective_date overrides >30d chunk-text rule - rule 1 (same-paragraph dual-date) still wins over effective_date relaxation - rule 2 (missing chunk dates) still applies when effective_date partially present - undefined effective_dates fall through to v1 behavior (back-compat) - empty-string effective_date treated as missing (only real dates enable the relaxation) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): cost-estimate prompt + --budget-usd + Haiku routing Lane C of the temporal-contradiction-probe wave. Three layers of cost guardrail, all stacked: (a) cost-estimate prompt at probe-run-time. Before the runner spends any tokens after a PROMPT_VERSION change, eval-suspected-contradictions reads the most recent persisted prompt_version from eval_contradictions_runs and compares. When they differ: - TTY: prints an upper-bound estimate + Ctrl-C window (default 10s, override via GBRAIN_PROBE_PROMPT_GRACE_SECONDS). - non-TTY: prints the estimate + auto-proceeds (autopilot path). - --yes override or GBRAIN_NO_PROBE_PROMPT=1: skip entirely. Mirrors the v0.32.7 runPostUpgradeReembedPrompt pattern. (b) --budget-usd N hard cap (pre-existing; PreFlightBudgetError surfaces when the estimate alone exceeds the cap, and CostTracker halts the run mid-flight when cumulative cost exceeds it). Documented in the help text alongside (a). (c) Judge model now routes through resolveModel() with configKey 'models.eval.contradictions_judge', tier 'utility' (Haiku-class default), and env var GBRAIN_CONTRADICTIONS_JUDGE_MODEL. The legacy --judge CLI flag still wins as the highest-precedence override. Doctor's model touchpoint registry (src/commands/models.ts:50) carries the new key so `gbrain models` and `gbrain models doctor` surface it. Also in this lane: - CLI: --severity accepts 'info' (the new Severity member from Lane A2). - CLI: --severity output shows [verdict] tag alongside slug pairs so operators distinguish genuine contradictions from temporal classifications. - Human summary: prints the new queries_with_any_finding metric and the per-verdict breakdown table. - Help text: explains the cost-prompt + budget-cap + model-routing interactions in one paragraph. New tests (9 cases on the cost-prompt helper): - --yes override skips - GBRAIN_NO_PROBE_PROMPT=1 skips - prompt_version unchanged → skips - non-TTY auto-proceeds with stderr note - TTY proceeds after grace - TTY aborts on Ctrl-C - fresh brain (no prior runs) fires the prompt - GBRAIN_PROBE_PROMPT_GRACE_SECONDS override honored - estimate banner contains query count + judge model + dollar amount All 225 eval-contradictions tests + 25 model-config tests pass. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(eval): R4/R5/R6 IRON-RULE regressions for the verdict-enum wave Lane D of the temporal-contradiction-probe wave. The Lanes A1/A2/B/C lanes landed the behavior; this lane pins the regressions that protect the wave against future drift. R4 (runner emit predicate): five new tests, one per non-no_contradiction verdict, prove the runner.ts emit rule surfaces each one as a finding with the correct verdict tag, and that: - queries_with_contradiction (Wilson-CI denominator) ONLY counts verdict ='contradiction' — the strict metric is preserved - queries_with_any_finding counts every non-no_contradiction verdict - verdict_breakdown tallies correctly Plus one negative case: verdict='no_contradiction' produces zero findings. Without R4, a future runner refactor could collapse the new verdicts back to /dev/null and the report would silently shrink. R5 (cache key shape): direct shape assertion on buildCacheKey output. The key tuple is exactly 5 fields (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy). Adding a 6th field would silently break every operator's brain (no migration path). R6 (contradiction severity unchanged): four tests on normalizeVerdict pin the legacy semantics — judge-supplied severity wins (whether 'high' or 'low'), and on garbage severity input the fallback is 'medium' (per defaultSeverityForVerdict('contradiction')) NOT 'low'. The contradiction verdict's severity must never default to 'low', which would silently mask genuine conflicts as cosmetic naming issues. The temporal_regression case is included for parity (garbage → 'high' since regressions are real investor red flags). 236 eval-contradictions tests pass (211 + 6 R4 + 1 R5 + 4 R6 + 9 cost-prompt from Lane C). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(ci): privacy lint for docs/proposals/*.md Captures the residual TODO from the temporal-contradiction-probe wave's plan: prevent the bug class where an RFC lands in docs/proposals/ with PII that should never appear in a public technical artifact. The original RFC had to be scrubbed at force-push time (Step 0); this lint catches the same patterns at CI time so the next one can't slip through. Sibling to scripts/check-privacy.sh: - check-privacy.sh: bans the literal "Wintermute" repo-wide. - check-proposal-pii.sh: focuses on docs/proposals/*.md and the OTHER PII classes — personal-relationship vocabulary, private repo refs. Design contract: the denylist names PATTERNS, not real people. Naming specific real names (deceased relatives, therapist first names, dealflow contacts) inside this script would leak PII into the repo just by appearing here. The structural patterns below catch the SURROUNDING vocabulary that always accompanies such content in personal RFC prose. Trade-off: a future RFC that names a real person without any contextual markers won't be caught — accepted as residual risk handled by human review. Patterns flagged in docs/proposals/*.md: - garrytan/brain (private repo reference) - trial separation, permanent separation - couples session, couples therapist - divorce attorney(s) - grandmother's funeral, aunt's funeral - wintermute (also caught by check-privacy.sh; listed here for proposal-scoped clarity) Bare common words (separation, funeral) are NOT banned — only the combined personal-context phrases. "Separation of concerns" and other software vocabulary survives. Wired into: - `bun run verify` (gates every push) - `bun run check:all` - `bun run check:proposal-pii` (standalone) Tests: 15 cases in test/scripts/check-proposal-pii.test.ts. - Each pattern flagged when present, plus exit-code + stderr signal. - Two negative cases (separation-of-concerns, funeral metaphor) prove the lint doesn't false-positive on legitimate software prose. - No-proposals-dir → exit 0 (not a failure). - Multi-hit case proves all patterns surface together with a summary count. - The two test fixtures that name "Wintermute" / "WINTERMUTE" as sentinel literals are allowlisted in check-test-real-names.sh per the same meta-rule-enforcement exception as check-privacy.sh itself. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(privacy): allowlist new privacy-guard files in check-privacy.sh check-privacy.sh bans the literal Wintermute repo-wide. The two new files from the v0.34 privacy lint (scripts/check-proposal-pii.sh and its test) necessarily name the token to do their job. Same meta-rule-enforcement exception as scripts/check-privacy.sh itself, scripts/check-test-real-names.sh, test/recency-decay.test.ts, and the existing entries — describing what the rule forbids requires naming it. Without this allowlist, `bun run verify` fails on check:privacy. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: bump version and changelog (v0.35.1.0) Temporal-contradiction-probe wave — Phase 1 of the RFC at docs/proposals/temporal-contradiction-probe.md. Headline: the contradiction probe now classifies pairs into a 6-member verdict enum (no_contradiction, contradiction, temporal_supersession, temporal_regression, temporal_evolution, negation_artifact) and sees the page-level effective_date for each chunk via a (from: YYYY-MM-DD) tag in the prompt. The pre-judge date filter no longer skips dated wide-gap pairs, so the role-transition class (e.g. a 2017 historical record vs. a 2025 current state) reaches the judge and gets classified as temporal_supersession instead of vanishing into the skip bucket. PROMPT_VERSION bumped 1 → 2 (cache fully invalidated). Three-layer cost guardrail: TTY-only cost-estimate prompt with Ctrl-C window, --budget-usd hard cap, Haiku-tier routing via new models.eval.contradictions_judge config key. Also adds a CI privacy lint (scripts/check-proposal-pii.sh) wired into bun run verify that catches PII patterns in docs/proposals/*.md so future RFCs can't ship with personal-context vocabulary the way this wave's source RFC did at draft time. Phases 2-4 deferred to follow-up RFCs per the plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: garrytan-agents <garrytan-agents@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
153 lines
5.5 KiB
TypeScript
153 lines
5.5 KiB
TypeScript
/**
|
|
* Tests for scripts/check-proposal-pii.sh — the privacy guard for
|
|
* docs/proposals/*.md.
|
|
*
|
|
* Strategy: each test case builds a tiny git-initialized scratch repo
|
|
* with a docs/proposals/*.md file containing the case input, invokes
|
|
* the script (which lives back in the real repo), and asserts the exit
|
|
* code + stderr signal. The script reads from `git rev-parse
|
|
* --show-toplevel` so each scratch repo behaves like a real project.
|
|
*/
|
|
|
|
import { describe, test, expect } from 'bun:test';
|
|
import { spawnSync } from 'node:child_process';
|
|
import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'node:fs';
|
|
import { join } from 'node:path';
|
|
import { tmpdir } from 'node:os';
|
|
|
|
const SCRIPT_PATH = join(import.meta.dir, '..', '..', 'scripts', 'check-proposal-pii.sh');
|
|
|
|
function runLintIn(content: string | null): { exitCode: number; stderr: string } {
|
|
const dir = mkdtempSync(join(tmpdir(), 'gbrain-pii-lint-'));
|
|
try {
|
|
// Initialize as git repo so `git rev-parse --show-toplevel` resolves.
|
|
spawnSync('git', ['init', '-q'], { cwd: dir });
|
|
if (content !== null) {
|
|
mkdirSync(join(dir, 'docs', 'proposals'), { recursive: true });
|
|
writeFileSync(join(dir, 'docs', 'proposals', 'test.md'), content);
|
|
}
|
|
const r = spawnSync('bash', [SCRIPT_PATH], { cwd: dir, encoding: 'utf8' });
|
|
return { exitCode: r.status ?? -1, stderr: r.stderr ?? '' };
|
|
} finally {
|
|
rmSync(dir, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
describe('check-proposal-pii.sh', () => {
|
|
test('clean proposal passes (exit 0)', () => {
|
|
const r = runLintIn(`# RFC
|
|
|
|
This is a clean technical proposal using only generic placeholders:
|
|
alice-example, acme-corp, fund-a. Two roles, one decision.
|
|
`);
|
|
expect(r.exitCode).toBe(0);
|
|
expect(r.stderr).toBe('');
|
|
});
|
|
|
|
test('garrytan/brain private repo reference is flagged', () => {
|
|
const r = runLintIn(`Context: findings resolved in garrytan/brain.\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain('garrytan/brain');
|
|
});
|
|
|
|
test('trial separation phrase is flagged', () => {
|
|
const r = runLintIn(`A: trial separation\nB: confirmed status\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain('trial separation');
|
|
});
|
|
|
|
test('permanent separation phrase is flagged', () => {
|
|
const r = runLintIn(`status: permanent separation, confirmed\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain('permanent separation');
|
|
});
|
|
|
|
test('couples session phrase is flagged', () => {
|
|
const r = runLintIn(`per the couples session on 2026-05-07\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain('couples session');
|
|
});
|
|
|
|
test('divorce attorney phrase is flagged', () => {
|
|
const r = runLintIn(`consultation with divorce attorney scheduled\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain('divorce attorney');
|
|
});
|
|
|
|
test("grandmother's funeral phrase is flagged", () => {
|
|
const r = runLintIn(`Traveled for the grandmother's funeral last week.\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain("grandmother's funeral");
|
|
});
|
|
|
|
test("aunt's funeral phrase is flagged", () => {
|
|
const r = runLintIn(`Traveled to the aunt's funeral in Toronto.\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain("aunt's funeral");
|
|
});
|
|
|
|
test('wintermute is flagged (consistent with check-privacy.sh)', () => {
|
|
const r = runLintIn(`Author: Wintermute (via the user)\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain('wintermute');
|
|
});
|
|
|
|
test('case-insensitive match: WINTERMUTE all-caps also flagged', () => {
|
|
const r = runLintIn(`See the WINTERMUTE deployment.\n`);
|
|
expect(r.exitCode).toBe(1);
|
|
});
|
|
|
|
test('benign separation usage stays clean (separation-of-concerns)', () => {
|
|
// "separation" alone is fine; only "trial separation" / "permanent
|
|
// separation" are flagged. Software vocabulary survives.
|
|
const r = runLintIn(`# Architecture
|
|
|
|
Strict separation of concerns between the runner and the judge module.
|
|
The trial run uses a separate test database.
|
|
`);
|
|
expect(r.exitCode).toBe(0);
|
|
});
|
|
|
|
test('benign funeral metaphor stays clean (bare "funeral" not banned)', () => {
|
|
// Only combined personal-context phrases are banned. Bare "funeral"
|
|
// in a software metaphor survives the lint.
|
|
const r = runLintIn(`Killing the deprecated endpoint feels like a funeral.\n`);
|
|
expect(r.exitCode).toBe(0);
|
|
});
|
|
|
|
test('multiple PII patterns reported together with exit 1', () => {
|
|
const r = runLintIn(`
|
|
Context: findings from garrytan/brain.
|
|
|
|
A: trial separation
|
|
B: permanent separation per the couples session
|
|
`);
|
|
expect(r.exitCode).toBe(1);
|
|
expect(r.stderr).toContain('garrytan/brain');
|
|
expect(r.stderr).toContain('trial separation');
|
|
expect(r.stderr).toContain('permanent separation');
|
|
expect(r.stderr).toContain('couples session');
|
|
// Summary count.
|
|
expect(r.stderr).toContain('4 PII pattern hit(s)');
|
|
});
|
|
|
|
test('no proposals dir → exits 0 (not a failure)', () => {
|
|
const r = runLintIn(null);
|
|
expect(r.exitCode).toBe(0);
|
|
});
|
|
|
|
test('--help exits 1 with usage text', () => {
|
|
const dir = mkdtempSync(join(tmpdir(), 'gbrain-pii-lint-help-'));
|
|
try {
|
|
spawnSync('git', ['init', '-q'], { cwd: dir });
|
|
const r = spawnSync('bash', [SCRIPT_PATH, '--help'], { cwd: dir, encoding: 'utf8' });
|
|
expect(r.status).toBe(1);
|
|
const out = (r.stdout ?? '') + (r.stderr ?? '');
|
|
expect(out).toContain('check-proposal-pii.sh');
|
|
expect(out).toContain('docs/proposals');
|
|
} finally {
|
|
rmSync(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
});
|