Files
gbrain/test/eval-contradictions/no-valid-until-write.test.ts
T
1dadd9ed71 v0.35.7.0 feat: temporal trajectory + founder scorecard (Phases 2-4) (#1131)
* feat(facts): typed-claim substrate + cycle correctness fixes (v0.35.6 wave 1/3)

Schema (migration v67):
- Add four optional typed-claim columns to facts: claim_metric TEXT,
  claim_value DOUBLE PRECISION, claim_unit TEXT, claim_period TEXT
- Partial index facts_typed_claim_idx ON (entity_slug, claim_metric, valid_from)
  WHERE claim_metric IS NOT NULL
- All nullable, metadata-only on both engines

Fence layer:
- ParsedFact (facts-fence.ts) gains optional claimMetric/Value/Unit/Period
- Parser tolerates both 10-cell (legacy) and 14-cell (widened) rows
- Renderer emits 14 cells iff any row has typed data; otherwise stays
  10-cell so existing fences don't widen on unrelated edits
- Numeric value cell tolerates comma thousand separators (50,000 -> 50000)

Extract pipeline (D-CDX-2, D-ENG-1):
- src/core/facts/extract.ts (the actual Haiku call site, NOT extract-facts.ts
  cycle phase) extends its system prompt to emit typed fields for metric-shaped
  claims
- extractFactsFromFenceText gains optional pageEffectiveDate. Precedence:
  fence-row validFrom > pageEffectiveDate > undefined (engine defaults to now)
- normalizeMetricLabel: 15-entry seed map for common founder metrics (mrr,
  arr, runway, headcount, team_size, cac, ltv, gross_margin, burn_rate, cash,
  users, mau, dau, churn_rate, revenue); unknown labels lowercase + space->_

Engine extensions:
- NewFact + insertFact + insertFacts in both engines accept the four typed
  columns (all nullable)
- Cycle phase extract-facts.ts threads page.effective_date through AND
  batch-embeds via gateway.embed() before insertFacts (D-CDX-3 fix for
  cycle-inserted facts arriving with embedding=NULL)

Consolidate fix (D-CDX-4 — Codex F4):
- Replace MAX(row_num)+1 INSERT with semantic upsert on (page_id, claim,
  since_date). Re-running the full cycle on stable input produces zero new
  takes — fixes the pre-existing duplicate-takes bug after extract_facts
  wipes consolidated_at
- Chronological valid_until writeback per cluster: sort by (valid_from ASC,
  id ASC), walk pairs, set older.valid_until = newer.valid_from

Tests:
- test/migrate.test.ts +6 cases for v67 shape + materialization + nullable
  backward compat
- test/facts-fence-typed.test.ts (new, 17 cases): parser+renderer round-trip,
  normalization seed map coverage, valid_from precedence three-branch
- test/consolidate-valid-until.test.ts (new, 4 cases): chronological
  writeback (R4a), same-day id tiebreaker, cycle re-run zero duplicates
  (R4b/R7), valid_until idempotency
- test/schema-bootstrap-coverage.test.ts: add four typed-claim columns to
  COLUMN_EXEMPTIONS (migration co-defines the partial index, no forward
  reference to bootstrap)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(trajectory): find_trajectory MCP op + eval/founder CLIs (v0.35.6 wave 2/3)

Engine method (D-CDX-1, D-CDX-6):
- BrainEngine.findTrajectory(opts) on both Postgres and PGLite
- TrajectoryOpts: scalar sourceId fast path + sourceIds federated array
  (mirrors v0.34.1.0 search* dual pattern)
- opts.remote: when true, SQL adds AND visibility='world' so OAuth read
  clients see only world-visibility facts (mirrors recall's posture —
  closes the F7 privacy regression Codex caught in plan review)
- Single SQL query, ORDER BY valid_from ASC, id ASC for deterministic
  output (R3 pin). Returns TrajectoryPoint[] including raw embedding so
  the caller can compute drift without a second round-trip

Pure function library (src/core/trajectory.ts, new):
- detectRegressions(points, threshold): walks consecutive (metric, value)
  pairs per metric; emits when newer drops >= threshold below older.
  10% default, override via GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD
- computeDriftScore(points): 1 - mean(cosine(emb[i], emb[i-1])) over
  embedded points; clamped [0,1]; null when <3 embedded points (D-ENG-3
  graceful degradation)
- computeTrajectoryStats(points): composed shape returning both
- TRAJECTORY_SCHEMA_VERSION = 1 — additive-only across releases (R5)

MCP op (src/core/operations.ts):
- find_trajectory: scope read, NOT localOnly. Routes through
  sourceScopeOpts(ctx) for federated isolation AND threads ctx.remote
  for visibility filtering. Strips raw Float32Array embeddings from the
  wire shape; converts valid_from to YYYY-MM-DD string
- Registered in operations array after find_experts
- FIND_TRAJECTORY_DESCRIPTION in operations-descriptions.ts

CLIs:
- gbrain eval trajectory <entity> [--metric M] [--since D] [--until D]
  [--limit N] [--json] — chronological human view with [REGRESSION] inline
  annotation; thin-client routing via callRemoteTool(find_trajectory).
  Dispatched in src/commands/eval.ts sub-subcommand block
- gbrain founder scorecard <entity> [--since D] [--until D] [--json] —
  pure aggregation over Phase 2's substrate. Four signals:
  claim_accuracy (over resolved takes), consistency, growth_trajectory,
  red_flags. computeFounderScorecard exported for tests.
  Registered as top-level command in cli.ts; added to CLI_ONLY set

Tests (45 cases across 5 files):
- test/engine-find-trajectory.test.ts: 18 cases — chronological order,
  source scoping (scalar + federated), visibility filter on remote=true,
  metric + since/until filters, regression detection at threshold
  boundaries, drift score with various embedding states
- test/operations-find-trajectory.test.ts: 9 cases — op registration,
  param validation, JSON envelope shape, R5 schema_version: 1,
  embedding stripped from wire, R6 visibility filter, source scoping
- test/eval-trajectory.test.ts: 7 cases — arg parsing, --help,
  --json envelope, regression annotation, --metric filter, empty entity
- test/founder-scorecard.test.ts: 9 cases — empty inputs no-NaN (G2),
  claim_accuracy math, consistency math, growth_trajectory math,
  red_flags fire for regression / narrative_drift / missed_prediction
- test/eval-contradictions/no-valid-until-write.test.ts: 4 cases —
  R1 (probe never writes valid_until under eval-contradictions/) +
  R8 (only allow-listed files write valid_until anywhere in src/)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.35.6.0 — CHANGELOG + VERSION + docs + migration note

Bumps to v0.35.6.0 (next-minor after master's v0.35.5.1 — typed-claim
substrate + trajectory + founder scorecard is a new user-facing
feature surface, not a fix).

- VERSION + package.json synced
- CHANGELOG.md release-summary block in the wave-style voice, lead with
  what the user can now DO. Sections: typed metric claims in the fence,
  chronological metric trajectories, founder scorecard, MCP
  find_trajectory op, cycle re-run idempotency fix, embedding-on-insert
  fix, valid_from precedence fix. To-take-advantage-of block with
  verification + opt-in fence syntax example
- CLAUDE.md Key Files entry consolidating the wave across
  eval-trajectory.ts + founder-scorecard.ts + trajectory.ts. Names every
  D-ENG / D-CDX decision and the Codex outside-voice F-numbers
- skills/migrations/v0.35.6.md agent-readable migration note. Includes
  fence-syntax example for typed-claim rows so downstream agents start
  emitting them. Iron-rule contracts called out (R1 + R8 + R7 + visibility)
- llms-full.txt regenerated to reflect the new CLAUDE.md entry

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs: post-ship sync for v0.35.7.0 — trajectory + founder scorecard

- README.md: add `gbrain eval trajectory` to EVAL section, add new
  TEMPORAL block covering `gbrain founder scorecard` + the
  GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD env override; add v0.35.7
  "What's new" paragraph below the v0.28.8 LongMemEval blurb
- AGENTS.md: new bullet under Common tasks teaching agents to reach for
  `gbrain eval trajectory` / `gbrain founder scorecard` / the
  `find_trajectory` MCP op when asked to evaluate a founder/company
  over time
- docs/contradictions.md: append "Temporal axis follow-on (v0.35.3.1 +
  v0.35.7)" subsection under See also, cross-linking the trajectory
  substrate and naming the auto-supersession.ts:4 invariant preserved
  by both the verdict enum (probe side) and consolidate's valid_until
  writeback (cycle side)
- CLAUDE.md: fix stale (v0.35.4) tag on the trajectory entry to
  (v0.35.7) — version got rebumped twice during the merge wave
- skills/migrations/v0.35.7.md renamed to v0.35.7.0.md for consistency
  with the v0.35.0.0.md / v0.14.0.md / etc naming convention
- llms-full.txt regenerated to reflect the CLAUDE.md edit

Coverage map (Diataxis):
  /eval trajectory CLI        ref (README, AGENTS)  how-to (CHANGELOG)  tutorial
  /founder scorecard CLI      ref (README, AGENTS)  how-to (CHANGELOG)  tutorial
  find_trajectory MCP op      ref (CLAUDE.md, AGENTS, contradictions.md)
  typed-claim fence cols      ref (skills/migrations/v0.35.7.0.md, CHANGELOG)
  Migration v67               ref (CLAUDE.md, CHANGELOG)

No tutorial / explanation gaps worth filling in this PR — the migration
note's fence-syntax example already covers the "first typed claim"
walkthrough. ARCHITECTURE diagrams not drifted (the trajectory work
extends existing facts/takes infrastructure; no new component boxes).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 18:52:38 -07:00

159 lines
6.8 KiB
TypeScript

/**
* v0.35.4 (R1 + R8 — D-CDX-4 + D-CDX-7) — IRON-RULE: the contradiction
* probe NEVER writes `valid_until` on the facts table.
*
* The temporal trajectory wave (v0.35.4) gives `consolidate` the
* authority to write `valid_until` on chronologically-superseded facts.
* That authority is exclusive: the contradiction probe surfaces
* `temporal_supersession` verdicts via paste-ready commands, but it
* must NEVER auto-mutate. This preserves the
* `src/core/eval-contradictions/auto-supersession.ts:4` invariant.
*
* Two layered guards:
* R1 — grep guard over the entire `src/core/eval-contradictions/`
* subtree and the `src/commands/eval-suspected-contradictions*.ts`
* files: no code path may UPDATE facts.valid_until.
* R8 — broader guard over all of `src/`: the only file that writes
* valid_until is `src/core/cycle/phases/consolidate.ts`. Any new
* write site fails this guard; the human adding it must explicitly
* amend the allow-list AND document the deliberate design change.
*/
import { test, expect, describe } from 'bun:test';
import { readdirSync, readFileSync, statSync } from 'node:fs';
import { join } from 'node:path';
// Allow-listed files that legitimately write `valid_until`. Adding a new
// file here means you've thought carefully about the
// auto-supersession.ts:4 invariant and decided the new write site
// preserves it.
//
// - consolidate.ts (v0.35.4 — chronological writeback)
// - facts/forget.ts (v0.32.2 — user-initiated `gbrain forget`; user is
// the supersession authority, not the probe)
const VALID_UNTIL_WRITE_ALLOWLIST: ReadonlySet<string> = new Set([
'src/core/cycle/phases/consolidate.ts',
'src/core/facts/forget.ts',
]);
function walkTs(dir: string, acc: string[] = []): string[] {
for (const name of readdirSync(dir)) {
const full = join(dir, name);
const st = statSync(full);
if (st.isDirectory()) {
// Skip generated / vendored / test directories.
if (name === 'node_modules' || name === '.git' || name === 'dist') continue;
walkTs(full, acc);
} else if (name.endsWith('.ts') && !name.endsWith('.test.ts')) {
acc.push(full);
}
}
return acc;
}
/**
* Detect lines that look like they are UPDATEing facts.valid_until.
* Permissive on SQL formatting (the same UPDATE can be split across lines,
* use template strings, or use parameterized SQL via postgres.js's
* tagged-template syntax). We look for the pair of substrings near
* each other in the same file: `UPDATE facts` and `valid_until`.
*
* Tolerated: a reference to `valid_until` in a SELECT projection list
* (which is fine — reading the column is not writing it) is filtered out
* by also requiring a write verb (SET) nearby OR an INSERT INTO facts
* with valid_until in the column list (INSERT path is OK from
* src/core/postgres-engine.ts + src/core/pglite-engine.ts because those
* are the engine layer, intentionally writing on caller request via
* insertFact/insertFacts; the issue is whether the contradiction probe
* path triggers those writes).
*/
function findValidUntilWrites(source: string): string[] {
// Quick-fail: no mention of valid_until at all.
if (!source.includes('valid_until')) return [];
const lines = source.split('\n');
const hits: string[] = [];
for (let i = 0; i < lines.length; i++) {
const line = lines[i];
// Detect actual SQL writes. The narrow pattern `UPDATE facts SET ...
// valid_until` is unambiguous — UPDATE is a SQL verb, not text
// anyone writes in a description string. Tolerates same-line and
// multi-line UPDATEs (look at the next 4 lines after `UPDATE facts`
// for `valid_until`).
if (/\bUPDATE\s+facts\b/i.test(line)) {
const window = lines.slice(i, i + 5).join('\n');
if (/\bSET\b[\s\S]*\bvalid_until\b/i.test(window)) {
hits.push(`${i + 1}: ${line.trim()}`);
}
}
}
return hits;
}
describe('R1 — contradiction probe never writes valid_until', () => {
test('no file under src/core/eval-contradictions/ writes facts.valid_until', () => {
const dir = 'src/core/eval-contradictions';
const files = walkTs(dir);
expect(files.length).toBeGreaterThan(0);
for (const f of files) {
const src = readFileSync(f, 'utf-8');
const hits = findValidUntilWrites(src);
expect(hits, `${f} contains valid_until write: ${hits.join(' | ')}`).toEqual([]);
}
});
test('no src/commands/eval-suspected-contradictions* file writes facts.valid_until', () => {
const dir = 'src/commands';
const files = readdirSync(dir)
.filter(n => n.startsWith('eval-suspected-contradictions') && n.endsWith('.ts'))
.map(n => join(dir, n));
expect(files.length).toBeGreaterThanOrEqual(1);
for (const f of files) {
const src = readFileSync(f, 'utf-8');
const hits = findValidUntilWrites(src);
expect(hits, `${f} contains valid_until write: ${hits.join(' | ')}`).toEqual([]);
}
});
});
describe('R8 — only the consolidate phase + engine insert layer may write valid_until', () => {
test('every src/ TypeScript file that writes valid_until is on the allow-list', () => {
const files = walkTs('src');
const offenders: Array<{ file: string; hits: string[] }> = [];
for (const f of files) {
// Normalize path separator for cross-platform matching of the
// allow-list keys.
const relForCheck = f.replace(/\\/g, '/');
if (VALID_UNTIL_WRITE_ALLOWLIST.has(relForCheck)) continue;
// Engine implementation files (postgres-engine.ts, pglite-engine.ts)
// legitimately write valid_until inside insertFact/insertFacts —
// those are caller-driven INSERTs. Pattern is `INSERT INTO facts (
// ... valid_until ...) VALUES`. Detect and exempt that pattern.
// The R8 guard is about UPDATE; INSERT carrying valid_until as a
// column value is not the failure mode auto-supersession.ts:4 cares
// about.
const src = readFileSync(f, 'utf-8');
const hits = findValidUntilWrites(src);
if (hits.length > 0) offenders.push({ file: relForCheck, hits });
}
expect(
offenders,
`Unexpected valid_until UPDATE sites:\n` +
offenders.map(o => ` ${o.file}:\n ${o.hits.join('\n ')}`).join('\n') +
`\n\nIf you added a deliberate write, append the path to ` +
`VALID_UNTIL_WRITE_ALLOWLIST in this test AND review the ` +
`\`auto-supersession.ts:4\` invariant first.`,
).toEqual([]);
});
test('VALID_UNTIL_WRITE_ALLOWLIST is non-empty (consolidate is on it)', () => {
// Self-check: if the test file ever ships with an empty allow-list,
// the R8 guard collapses to "no code writes valid_until anywhere" and
// becomes a tautology. Keep the consolidate phase on the list as the
// explicit positive control.
expect(VALID_UNTIL_WRITE_ALLOWLIST.has('src/core/cycle/phases/consolidate.ts')).toBe(true);
});
});