mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(facts): typed-claim substrate + cycle correctness fixes (v0.35.6 wave 1/3) Schema (migration v67): - Add four optional typed-claim columns to facts: claim_metric TEXT, claim_value DOUBLE PRECISION, claim_unit TEXT, claim_period TEXT - Partial index facts_typed_claim_idx ON (entity_slug, claim_metric, valid_from) WHERE claim_metric IS NOT NULL - All nullable, metadata-only on both engines Fence layer: - ParsedFact (facts-fence.ts) gains optional claimMetric/Value/Unit/Period - Parser tolerates both 10-cell (legacy) and 14-cell (widened) rows - Renderer emits 14 cells iff any row has typed data; otherwise stays 10-cell so existing fences don't widen on unrelated edits - Numeric value cell tolerates comma thousand separators (50,000 -> 50000) Extract pipeline (D-CDX-2, D-ENG-1): - src/core/facts/extract.ts (the actual Haiku call site, NOT extract-facts.ts cycle phase) extends its system prompt to emit typed fields for metric-shaped claims - extractFactsFromFenceText gains optional pageEffectiveDate. Precedence: fence-row validFrom > pageEffectiveDate > undefined (engine defaults to now) - normalizeMetricLabel: 15-entry seed map for common founder metrics (mrr, arr, runway, headcount, team_size, cac, ltv, gross_margin, burn_rate, cash, users, mau, dau, churn_rate, revenue); unknown labels lowercase + space->_ Engine extensions: - NewFact + insertFact + insertFacts in both engines accept the four typed columns (all nullable) - Cycle phase extract-facts.ts threads page.effective_date through AND batch-embeds via gateway.embed() before insertFacts (D-CDX-3 fix for cycle-inserted facts arriving with embedding=NULL) Consolidate fix (D-CDX-4 — Codex F4): - Replace MAX(row_num)+1 INSERT with semantic upsert on (page_id, claim, since_date). Re-running the full cycle on stable input produces zero new takes — fixes the pre-existing duplicate-takes bug after extract_facts wipes consolidated_at - Chronological valid_until writeback per cluster: sort by (valid_from ASC, id ASC), walk pairs, set older.valid_until = newer.valid_from Tests: - test/migrate.test.ts +6 cases for v67 shape + materialization + nullable backward compat - test/facts-fence-typed.test.ts (new, 17 cases): parser+renderer round-trip, normalization seed map coverage, valid_from precedence three-branch - test/consolidate-valid-until.test.ts (new, 4 cases): chronological writeback (R4a), same-day id tiebreaker, cycle re-run zero duplicates (R4b/R7), valid_until idempotency - test/schema-bootstrap-coverage.test.ts: add four typed-claim columns to COLUMN_EXEMPTIONS (migration co-defines the partial index, no forward reference to bootstrap) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(trajectory): find_trajectory MCP op + eval/founder CLIs (v0.35.6 wave 2/3) Engine method (D-CDX-1, D-CDX-6): - BrainEngine.findTrajectory(opts) on both Postgres and PGLite - TrajectoryOpts: scalar sourceId fast path + sourceIds federated array (mirrors v0.34.1.0 search* dual pattern) - opts.remote: when true, SQL adds AND visibility='world' so OAuth read clients see only world-visibility facts (mirrors recall's posture — closes the F7 privacy regression Codex caught in plan review) - Single SQL query, ORDER BY valid_from ASC, id ASC for deterministic output (R3 pin). Returns TrajectoryPoint[] including raw embedding so the caller can compute drift without a second round-trip Pure function library (src/core/trajectory.ts, new): - detectRegressions(points, threshold): walks consecutive (metric, value) pairs per metric; emits when newer drops >= threshold below older. 10% default, override via GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD - computeDriftScore(points): 1 - mean(cosine(emb[i], emb[i-1])) over embedded points; clamped [0,1]; null when <3 embedded points (D-ENG-3 graceful degradation) - computeTrajectoryStats(points): composed shape returning both - TRAJECTORY_SCHEMA_VERSION = 1 — additive-only across releases (R5) MCP op (src/core/operations.ts): - find_trajectory: scope read, NOT localOnly. Routes through sourceScopeOpts(ctx) for federated isolation AND threads ctx.remote for visibility filtering. Strips raw Float32Array embeddings from the wire shape; converts valid_from to YYYY-MM-DD string - Registered in operations array after find_experts - FIND_TRAJECTORY_DESCRIPTION in operations-descriptions.ts CLIs: - gbrain eval trajectory <entity> [--metric M] [--since D] [--until D] [--limit N] [--json] — chronological human view with [REGRESSION] inline annotation; thin-client routing via callRemoteTool(find_trajectory). Dispatched in src/commands/eval.ts sub-subcommand block - gbrain founder scorecard <entity> [--since D] [--until D] [--json] — pure aggregation over Phase 2's substrate. Four signals: claim_accuracy (over resolved takes), consistency, growth_trajectory, red_flags. computeFounderScorecard exported for tests. Registered as top-level command in cli.ts; added to CLI_ONLY set Tests (45 cases across 5 files): - test/engine-find-trajectory.test.ts: 18 cases — chronological order, source scoping (scalar + federated), visibility filter on remote=true, metric + since/until filters, regression detection at threshold boundaries, drift score with various embedding states - test/operations-find-trajectory.test.ts: 9 cases — op registration, param validation, JSON envelope shape, R5 schema_version: 1, embedding stripped from wire, R6 visibility filter, source scoping - test/eval-trajectory.test.ts: 7 cases — arg parsing, --help, --json envelope, regression annotation, --metric filter, empty entity - test/founder-scorecard.test.ts: 9 cases — empty inputs no-NaN (G2), claim_accuracy math, consistency math, growth_trajectory math, red_flags fire for regression / narrative_drift / missed_prediction - test/eval-contradictions/no-valid-until-write.test.ts: 4 cases — R1 (probe never writes valid_until under eval-contradictions/) + R8 (only allow-listed files write valid_until anywhere in src/) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: v0.35.6.0 — CHANGELOG + VERSION + docs + migration note Bumps to v0.35.6.0 (next-minor after master's v0.35.5.1 — typed-claim substrate + trajectory + founder scorecard is a new user-facing feature surface, not a fix). - VERSION + package.json synced - CHANGELOG.md release-summary block in the wave-style voice, lead with what the user can now DO. Sections: typed metric claims in the fence, chronological metric trajectories, founder scorecard, MCP find_trajectory op, cycle re-run idempotency fix, embedding-on-insert fix, valid_from precedence fix. To-take-advantage-of block with verification + opt-in fence syntax example - CLAUDE.md Key Files entry consolidating the wave across eval-trajectory.ts + founder-scorecard.ts + trajectory.ts. Names every D-ENG / D-CDX decision and the Codex outside-voice F-numbers - skills/migrations/v0.35.6.md agent-readable migration note. Includes fence-syntax example for typed-claim rows so downstream agents start emitting them. Iron-rule contracts called out (R1 + R8 + R7 + visibility) - llms-full.txt regenerated to reflect the new CLAUDE.md entry Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: post-ship sync for v0.35.7.0 — trajectory + founder scorecard - README.md: add `gbrain eval trajectory` to EVAL section, add new TEMPORAL block covering `gbrain founder scorecard` + the GBRAIN_TRAJECTORY_REGRESSION_THRESHOLD env override; add v0.35.7 "What's new" paragraph below the v0.28.8 LongMemEval blurb - AGENTS.md: new bullet under Common tasks teaching agents to reach for `gbrain eval trajectory` / `gbrain founder scorecard` / the `find_trajectory` MCP op when asked to evaluate a founder/company over time - docs/contradictions.md: append "Temporal axis follow-on (v0.35.3.1 + v0.35.7)" subsection under See also, cross-linking the trajectory substrate and naming the auto-supersession.ts:4 invariant preserved by both the verdict enum (probe side) and consolidate's valid_until writeback (cycle side) - CLAUDE.md: fix stale (v0.35.4) tag on the trajectory entry to (v0.35.7) — version got rebumped twice during the merge wave - skills/migrations/v0.35.7.md renamed to v0.35.7.0.md for consistency with the v0.35.0.0.md / v0.14.0.md / etc naming convention - llms-full.txt regenerated to reflect the CLAUDE.md edit Coverage map (Diataxis): /eval trajectory CLI ✅ ref (README, AGENTS) ✅ how-to (CHANGELOG) ❌ tutorial /founder scorecard CLI ✅ ref (README, AGENTS) ✅ how-to (CHANGELOG) ❌ tutorial find_trajectory MCP op ✅ ref (CLAUDE.md, AGENTS, contradictions.md) typed-claim fence cols ✅ ref (skills/migrations/v0.35.7.0.md, CHANGELOG) Migration v67 ✅ ref (CLAUDE.md, CHANGELOG) No tutorial / explanation gaps worth filling in this PR — the migration note's fence-syntax example already covers the "first typed claim" walkthrough. ARCHITECTURE diagrams not drifted (the trajectory work extends existing facts/takes infrastructure; no new component boxes). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
186 lines
8.0 KiB
TypeScript
186 lines
8.0 KiB
TypeScript
/**
|
|
* v0.35.4 — find_trajectory MCP op (T5) tests.
|
|
*
|
|
* Pins:
|
|
* - Param validation: entity_slug required, non-empty.
|
|
* - Visibility filter on remote=true callers (R6 / D-CDX-1).
|
|
* - Source scoping via sourceScopeOpts (federated vs scalar).
|
|
* - Stable JSON envelope: points + regressions + drift_score + schema_version=1 (R5).
|
|
* - Engine result's raw Float32Array embedding is NOT serialized to wire.
|
|
* - Empty-result graceful shape (G1).
|
|
* - The op is registered + read-scope + not localOnly.
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
|
import { operationsByName } from '../src/core/operations.ts';
|
|
import type { OperationContext } from '../src/core/operations.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({});
|
|
await engine.initSchema();
|
|
});
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
await engine.executeRaw(`DELETE FROM facts WHERE entity_slug LIKE 'optraj-%'`);
|
|
await engine.executeRaw(`DELETE FROM sources WHERE id LIKE 'optraj-%'`);
|
|
});
|
|
|
|
function unitVec(idx: number): string {
|
|
const a = new Float32Array(1536);
|
|
a[idx % 1536] = 1.0;
|
|
return '[' + Array.from(a).join(',') + ']';
|
|
}
|
|
|
|
async function insertTyped(args: {
|
|
source_id?: string;
|
|
entity_slug: string;
|
|
metric: string;
|
|
value: number;
|
|
valid_from: Date;
|
|
visibility?: 'private' | 'world';
|
|
vecIdx?: number;
|
|
}): Promise<void> {
|
|
const sid = args.source_id ?? 'default';
|
|
await engine.executeRaw(
|
|
`INSERT INTO sources (id, name) VALUES ($1, $1) ON CONFLICT DO NOTHING`,
|
|
[sid],
|
|
);
|
|
await engine.executeRaw(
|
|
`INSERT INTO facts (source_id, entity_slug, fact, kind, source, valid_from,
|
|
claim_metric, claim_value, claim_unit, claim_period,
|
|
visibility, embedding, embedded_at)
|
|
VALUES ($1, $2, $3, 'fact', 'test', $4::timestamptz,
|
|
$5, $6, 'USD', 'monthly',
|
|
$7, $8::vector, $4::timestamptz)`,
|
|
[
|
|
sid, args.entity_slug, `${args.metric} = ${args.value}`,
|
|
args.valid_from.toISOString(), args.metric, args.value,
|
|
args.visibility ?? 'private', unitVec(args.vecIdx ?? 0),
|
|
],
|
|
);
|
|
}
|
|
|
|
function mkCtx(overrides: Partial<OperationContext> = {}): OperationContext {
|
|
return {
|
|
engine,
|
|
config: {} as any,
|
|
logger: { info: () => {}, warn: () => {}, error: () => {} } as any,
|
|
dryRun: false,
|
|
remote: false,
|
|
...overrides,
|
|
} as OperationContext;
|
|
}
|
|
|
|
describe('find_trajectory MCP op — registration + shape', () => {
|
|
test('registered with read scope, NOT localOnly', () => {
|
|
const op = operationsByName['find_trajectory'];
|
|
expect(op).toBeDefined();
|
|
expect(op.scope).toBe('read');
|
|
expect(op.localOnly).toBeUndefined();
|
|
// Description references the v0.35.4 contract.
|
|
expect(op.description).toContain('schema_version');
|
|
});
|
|
|
|
test('throws on missing entity_slug', async () => {
|
|
const op = operationsByName['find_trajectory'];
|
|
await expect(op.handler(mkCtx(), {})).rejects.toThrow(/entity_slug/);
|
|
await expect(op.handler(mkCtx(), { entity_slug: '' })).rejects.toThrow(/entity_slug/);
|
|
await expect(op.handler(mkCtx(), { entity_slug: ' ' })).rejects.toThrow(/entity_slug/);
|
|
});
|
|
|
|
test('returns stable JSON shape with schema_version: 1', async () => {
|
|
await insertTyped({ entity_slug: 'optraj-shape', metric: 'mrr', value: 50000, valid_from: new Date('2026-01-15') });
|
|
const op = operationsByName['find_trajectory'];
|
|
const result = await op.handler(mkCtx(), { entity_slug: 'optraj-shape' }) as any;
|
|
expect(result).toHaveProperty('points');
|
|
expect(result).toHaveProperty('regressions');
|
|
expect(result).toHaveProperty('drift_score');
|
|
expect(result.schema_version).toBe(1);
|
|
// Embedding NOT serialized to the wire.
|
|
expect(result.points[0]).not.toHaveProperty('embedding');
|
|
// valid_from is YYYY-MM-DD string.
|
|
expect(result.points[0].valid_from).toMatch(/^\d{4}-\d{2}-\d{2}$/);
|
|
});
|
|
|
|
test('unknown entity returns graceful empty shape (G1)', async () => {
|
|
const op = operationsByName['find_trajectory'];
|
|
const result = await op.handler(mkCtx(), { entity_slug: 'optraj-does-not-exist' }) as any;
|
|
expect(result.points).toEqual([]);
|
|
expect(result.regressions).toEqual([]);
|
|
expect(result.drift_score).toBeNull();
|
|
expect(result.schema_version).toBe(1);
|
|
});
|
|
});
|
|
|
|
describe('find_trajectory MCP op — visibility filter (R6 / D-CDX-1)', () => {
|
|
test('remote=true sees only world-visibility points', async () => {
|
|
await insertTyped({ entity_slug: 'optraj-vis', metric: 'mrr', value: 50000, visibility: 'private', valid_from: new Date('2026-01-15') });
|
|
await insertTyped({ entity_slug: 'optraj-vis', metric: 'mrr', value: 99999, visibility: 'world', valid_from: new Date('2026-04-12') });
|
|
|
|
const op = operationsByName['find_trajectory'];
|
|
const local = await op.handler(mkCtx({ remote: false }), { entity_slug: 'optraj-vis' }) as any;
|
|
expect(local.points.length).toBe(2);
|
|
|
|
const remote = await op.handler(mkCtx({ remote: true }), { entity_slug: 'optraj-vis' }) as any;
|
|
expect(remote.points.length).toBe(1);
|
|
expect(remote.points[0].value).toBe(99999);
|
|
});
|
|
});
|
|
|
|
describe('find_trajectory MCP op — source scoping (D-CDX-6)', () => {
|
|
test('federated sourceIds from auth.allowedSources narrows scope', async () => {
|
|
await insertTyped({ source_id: 'optraj-A', entity_slug: 'optraj-fed', metric: 'mrr', value: 1, valid_from: new Date('2026-01-15') });
|
|
await insertTyped({ source_id: 'optraj-B', entity_slug: 'optraj-fed', metric: 'mrr', value: 2, valid_from: new Date('2026-04-12') });
|
|
await insertTyped({ source_id: 'optraj-C', entity_slug: 'optraj-fed', metric: 'mrr', value: 3, valid_from: new Date('2026-07-08') });
|
|
|
|
const op = operationsByName['find_trajectory'];
|
|
const ctx = mkCtx({
|
|
auth: { allowedSources: ['optraj-A', 'optraj-B'] } as any,
|
|
});
|
|
const result = await op.handler(ctx, { entity_slug: 'optraj-fed' }) as any;
|
|
expect(result.points.length).toBe(2);
|
|
expect(result.points.map((p: any) => p.value)).toEqual([1, 2]);
|
|
});
|
|
|
|
test('scalar ctx.sourceId narrows to that single source', async () => {
|
|
await insertTyped({ source_id: 'optraj-X', entity_slug: 'optraj-scalar', metric: 'mrr', value: 100, valid_from: new Date('2026-01-15') });
|
|
await insertTyped({ source_id: 'optraj-Y', entity_slug: 'optraj-scalar', metric: 'mrr', value: 200, valid_from: new Date('2026-01-15') });
|
|
|
|
const op = operationsByName['find_trajectory'];
|
|
const ctx = mkCtx({ sourceId: 'optraj-X' });
|
|
const result = await op.handler(ctx, { entity_slug: 'optraj-scalar' }) as any;
|
|
expect(result.points.length).toBe(1);
|
|
expect(result.points[0].value).toBe(100);
|
|
});
|
|
});
|
|
|
|
describe('find_trajectory MCP op — regression + drift surface', () => {
|
|
test('regressions populate when newer value drops >= 10% (D-ENG-2 default)', async () => {
|
|
await insertTyped({ entity_slug: 'optraj-reg', metric: 'mrr', value: 200000, valid_from: new Date('2026-04-12'), vecIdx: 0 });
|
|
await insertTyped({ entity_slug: 'optraj-reg', metric: 'mrr', value: 150000, valid_from: new Date('2026-07-08'), vecIdx: 0 });
|
|
|
|
const op = operationsByName['find_trajectory'];
|
|
const result = await op.handler(mkCtx(), { entity_slug: 'optraj-reg' }) as any;
|
|
expect(result.regressions.length).toBe(1);
|
|
expect(result.regressions[0].metric).toBe('mrr');
|
|
expect(result.regressions[0].delta_pct).toBeCloseTo(-0.25, 3);
|
|
});
|
|
|
|
test('drift_score returns null with <3 embedded points (G3)', async () => {
|
|
await insertTyped({ entity_slug: 'optraj-drift', metric: 'mrr', value: 1, valid_from: new Date('2026-01-15') });
|
|
await insertTyped({ entity_slug: 'optraj-drift', metric: 'mrr', value: 2, valid_from: new Date('2026-04-12') });
|
|
|
|
const op = operationsByName['find_trajectory'];
|
|
const result = await op.handler(mkCtx(), { entity_slug: 'optraj-drift' }) as any;
|
|
expect(result.drift_score).toBeNull();
|
|
});
|
|
});
|