mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* feat(facts): add event_type column + trajectory-format helper (v0.40.2.0 Commit 1)
Substrate work for v0.40.2.0 Track B (trajectory routing for temporal +
knowledge_update). This commit lands the schema + the shared formatter;
think wiring + LongMemEval extractor + intent routing come in Commits 2-4.
Migration v81 (facts_event_type_column):
ALTER TABLE facts ADD COLUMN event_type TEXT (nullable, metadata-only).
Lets the v0.35.4 typed-claim substrate carry event-shaped rows
(event_type='meeting'/'job_change'/'location_change') alongside the
metric-shaped rows (claim_metric/claim_value etc) it has carried since
v67. Temporal-reasoning questions ("when did I last meet Marco") need
the event shape; the metric shape doesn't fit them.
Engine changes (pglite + postgres parity):
- TrajectoryPoint.event_type: string | null added; projection in both
findTrajectory SQL paths returns the column.
- TrajectoryOpts.kind?: 'metric' | 'event' | 'all' added (default 'all').
Defensive opt that future-proofs filtering once event rows accumulate.
- Both engines apply the new kind filter at SQL level when set.
Back-compat (codex outside-voice concern):
Existing callers (founder-scorecard, eval-trajectory) already defensively
skip metric === null rows in their per-metric math. Event-only rows
(metric=NULL, event_type='meeting') ride through invisibly to those
callers — verified by the new regression test that asserts byte-identical
computeFounderScorecard + computeTrajectoryStats output with and without
event rows in the input. Both callers now pass kind:'metric' explicitly
for call-site clarity (no behavior change).
MCP find_trajectory op:
- event_type added to the wire-shape map.
- kind param added to the op declaration (enum metric/event/all).
Shared formatter (src/core/trajectory-format.ts, new):
formatTrajectoryBlock(points, entitySlug, opts) — sibling shape to
renderTakesBlock + renderChatBlock. Groups by (metric ?? event_type).
Per-metric cap 20, total cap 100 (prompt-budget guardrail). For
knowledge_update intent, annotates value-change rows with
"(superseded prior)" — the explicit signal codex flagged was missing
from default RRF-ordered retrieval. Promoted to src/core/ so both
gbrain think (Commit 2) and the LongMemEval harness (Commit 4)
consume one source of truth.
Prompt-injection coverage (codex Problem 10):
src/core/think/sanitize.ts INJECTION_PATTERNS extended with three
new entries — close-trajectory, open-trajectory, xml-attr-inject —
so adversarial </trajectory> sequences in extracted text get
escaped before reaching the model. Parity with the existing
</take> coverage.
Tests (all hermetic, no DATABASE_URL):
- test/trajectory-format.test.ts (17 cases, all green): grouping,
caps, sanitization, supersession annotation, determinism,
provenance, text-cap, adversarial </trajectory> escape.
- test/engine-parity-event-type.test.ts (6 cases): PGLite round-trip
of the column + kind filter matrix.
- test/regressions/v0_40_2_0-trajectory-backcompat.test.ts (4 cases):
pins the byte-identical-output contract that founder-scorecard's
per-metric math ignores event rows.
- test/migrate.test.ts: v81 round-trip verified via existing
structural assertion harness.
- 209 tests across 5 impacted suites pass; bun run verify clean
(17 pre-checks including privacy, jsonb, type, fuzz purity).
Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md
GSTACK REVIEW REPORT: CEO + ENG + CODEX CLEARED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(think): trajectory injection for temporal + knowledge_update (v0.40.2.0 Commit 2)
Wires the v0.40.2.0 substrate (Commit 1's facts.event_type column +
formatTrajectoryBlock) into the production `gbrain think` surface.
Default ON; flip `think.trajectory_enabled=false` to opt out.
New pure modules (zero engine dependency):
- src/core/think/intent.ts — classifyIntent(question): regex-first
routing into 'temporal' | 'knowledge_update' | 'other'. KU wins over
temporal when both match. The 'other' fast path short-circuits with
zero SQL.
- src/core/think/entity-extract.ts — extractCandidateEntities() pulls
high-precision candidates from retrieval slugs (people/, companies/,
organizations/, deals/) and medium-precision noun phrases from the
question. Word-level tokenization + stop-word boundaries stitch
"Blue Bottle" as one candidate while splitting "I last meet Marco"
correctly. Leading-verb stripper drops "meet", "visit" etc so
"marco" surfaces cleanly. Cap of 5 per question.
Engine-touching wiring (src/core/think/index.ts):
- RunThinkOpts gains 4 fields: withTrajectory (default true),
sourceId, allowedSources, remote.
- readThinkTrajectoryEnabled() reads the config kill switch; default
true; survives missing config table on legacy brains.
- Trajectory orchestration sits between gather and prompt assembly:
intent classify → extract candidates → per-candidate
resolveEntitySlugWithSource → skip fallback_slugify → 5s timeout
Promise.race + 3-wide concurrency cap → formatTrajectoryBlock.
Any error degrades to "no block" + TRAJECTORY_INJECTION_FAILED
warning; the think call itself never crashes from trajectory.
- On success, TRAJECTORY_INJECTED_<N>_POINTS warning records the
count for downstream telemetry.
Prompt placement (src/core/think/prompt.ts) — Codex Problem 6 fix:
buildThinkUserMessage's trajectoryBlock slot honors BOTH existing
orderings — calibration mode inserts trajectory between calibration
and question; default mode inserts between retrieval and the output
instruction. NO third ordering is introduced. Empty trajectoryBlock
skips the "Known trajectory:" header entirely (don't cue the model
we tried).
Resolution-source signal (src/core/entities/resolve.ts) — Codex Problem 5:
New companion resolveEntitySlugWithSource() returns
{slug, source: 'exact_page' | 'fuzzy_match' | 'fallback_slugify'}
so trajectory routing can skip fallback-only resolutions —
querying findTrajectory on an invented slug always returns [] and
wastes a SQL round-trip. The original resolveEntitySlug keeps its
contract for pre-v0.40 callers.
MCP think op handler (src/core/operations.ts):
Extracts sourceScopeOpts(ctx) into scalar sourceId + allowedSources
+ remote, threads through to runThink. CLI callers omit (engine
default source, remote=false). Mirrors the same source-scope
discipline applied to all other read paths in v0.34.1.0.
Sanitization (Commit 1 already extended INJECTION_PATTERNS for
</trajectory> — consumed here).
Test coverage (all hermetic, no DATABASE_URL, no API keys):
- test/think-intent.test.ts (14 cases) — temporal, KU, other,
precedence (KU wins when both match), defensive non-string inputs.
- test/think-entity-extract.test.ts (10 cases) — retrieved-slug
source, noun-phrase source, stop-word stripping, leading-verb
stripping, dedup across sources, 5-candidate cap.
- test/think-trajectory-injection.test.ts (7 cases against PGLite
in-memory) — temporal intent injection happy path with superseded-
prior annotation, "other" intent short-circuit, withTrajectory:
false bypass, think.trajectory_enabled=false config bypass,
empty-trajectory skip, engine.findTrajectory throw is caught
(Promise.allSettled defense), TRAJECTORY_INJECTED warning count.
- Existing test/think-pipeline.serial.test.ts re-asserted unchanged
(10 cases — calibration mode parity, gather, sanitization,
cite-render all intact).
72 tests pass across 7 impacted suites; bun run verify clean (17 pre-
checks). Defaulted on per CEO + Eng D1; kill switch via config.
Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(longmemeval): inline Haiku claim extractor + content-hash cache (v0.40.2.0 Commit 3)
Populates the LongMemEval benchmark brain's facts table inline at
import time so Commit 4's intent routing has data to retrieve. Per the
CHANGELOG D1 decision, this is full-haystack preprocessing — disclosed
explicitly in the benchmark output's methodology_note field (Commit 4).
New module src/eval/longmemeval/extract.ts:
extractAndInsertClaims({engine, client, model, sessionSlug,
sessionId, sessionBody, sourceId, aliasMap})
- Hashes the session body (sha256) for cache lookup.
- Cache hit → reuses parsed claims (cuts a 3-iteration benchmark
run from $1.50 to $0.50 when sessions repeat across questions,
as they do in LongMemEval).
- Cache miss → one Haiku call. System prompt asks for
{entity, metric, value, unit, period, event_type, valid_from, text}[]
JSON. New parseExtractedJsonArray() helper does fence-strip + parse
(parseModelJSON from cross-modal-eval is shaped for scored objects,
not arrays — different parser needed here).
- Per-record validateClaim() drops malformed records (missing
entity, bad date) silently; the rest land in NewFact rows.
- Per-question AliasMap (Codex Problem 4 — semantics pinned):
"Marco" + "Marco Smith" + "marco" in the SAME question collapse
to one slug via first-mention-wins canonicalization. Across
questions, the harness creates a fresh map (no leak).
- Real-page-aware entity resolution via the v0.40.2.0
resolveEntitySlugWithSource (Commit 2). Slugify-fallback rows
still insert (we need the data); the resolution_source signal
is only consulted at trajectory retrieval time (Commit 4).
- Bulk insert via engine.insertFacts with the
`gbrain-allow-direct-insert` allow-list comment per the
check-system-of-record CI guard contract — benchmark brain is
ephemeral in-memory PGLite, no markdown source-of-truth applies.
- Fail-open posture: Haiku throw, malformed JSON, insert collision
all return inserted=0 without throwing. One bad session never
kills the per-question loop.
- getCacheStats() exposes hits/misses/size for the per-run stderr
telemetry Codex Problem 14 asked for (empirical hit-rate
reporting; the optimistic claim self-verifies).
Substrate plumbing (extends Commit 1):
- NewFact.event_type?: string | null added in engine.ts so the
extractor can pass event-shaped rows through to insertFacts.
- PGLite engine + Postgres engine insertFacts() now persist
event_type. Param-positional dispatch extended to 20/21 placeholders
(null-embedding vs embedding-present); tx.unsafe vector cast on
Postgres path unchanged.
Test coverage (test/longmemeval-extract.test.ts, 13 cases, hermetic):
- Happy path: typed-claim + event rows both insert with correct
kind (event_type='meeting' → kind='event'; claim_metric='mrr'
→ kind='fact').
- Alias map: per-session collapsing ("Marco" + "Marco Smith"),
cross-session persistence within one question, fresh map per
question (caller-clears semantics pinned).
- Content-hash cache: identical body → cache hit, only ONE Haiku
call across two sessions; different bodies miss; getCacheStats
reports hits/misses/size.
- Fail-open: malformed JSON, Haiku throw, empty array output,
invalid records (missing entity, bad date) — none crash; 0
inserted in each case.
55 tests pass across 4 impacted suites; bun run verify clean.
Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(longmemeval): trajectory intent routing + prompt splice + methodology disclosure (v0.40.2.0 Commit 4)
The final wiring: per-question intent classification + trajectory call
+ block splice into the answer-gen prompt. Plus the methodology
disclosure stamps that close out the Codex D1 contract.
New module src/eval/longmemeval/intent.ts:
classifyIntent(q): prefers q.question_type from the dataset
(LongMemEval ships labels like 'temporal-reasoning',
'knowledge-update', 'single-session-user') before falling back to
the SHARED regex set imported from src/core/think/intent.ts.
Single source of truth for the regex — think and longmemeval
cannot drift.
Harness wiring in src/commands/eval-longmemeval.ts:
- runEvalLongMemEval() spawns an extractor model via resolveModel
(tier:'utility' → haiku) when trajectory routing is enabled.
Calls resetExtractorState() once per benchmark run so the
content-hash cache + counters start clean.
- runOneQuestion() creates a FRESH per-question AliasMap (Codex
Problem 4 — first-mention-wins canonicalization stays scoped to
one question, never leaks across).
- Per session: after importFromContent lands, extractAndInsertClaims
populates the facts table. Fail-open if the Haiku call errors;
next session keeps going.
- After hybridSearch returns: classifyIntent(q) routes
temporal/knowledge_update through extractCandidateEntities (the
SHARED helper from Commit 2's think/entity-extract) → per-candidate
findTrajectory with 5s Promise.race timeout → formatTrajectoryBlock.
First candidate with a non-empty trajectory wins.
- generateAnswer() splices the trajectory block BEFORE the
Retrieved sessions block. Empty block (no entity match / no
points) → no "Known trajectory:" header (don't cue the model
we tried).
- JSON envelope gains 5 fields per question when trajectory routing
is on: intent, trajectory_points, entity_resolved,
resolution_source, methodology_note. methodology_note also
written to stderr at run completion.
Resolution-source gate DIVERGES from think (intentional):
In the think production path, fallback_slugify results are skipped
because querying invented slugs wastes SQL — production brains have
canonical pages. In the LongMemEval benchmark, there ARE no
canonical pages; both the extractor and the lookup go through
slugify-fallback on the same free-form name, so they cohere on the
same slug. Applying the think-path gate here would permanently
block trajectory injection on the benchmark. Comment in
runOneQuestion documents the divergence.
New CLI flag --no-trajectory:
Bypasses BOTH the Haiku extractor AND the per-question intent
routing. Used by the measurement protocol to baseline default-on vs
no-trajectory across 3 seeds per condition with paired-bootstrap
CI. Documented in the help text.
New RunOpts fields:
- extractorClient?: ThinkLLMClient — separate stub from the
answer-gen client so tests can isolate the two surfaces.
- extractorModel?: string — model override for the Haiku call.
methodology_note = 'extractor=haiku-preprocess-full-haystack-v1'
stamped on:
- Every per-question JSON envelope row.
- Stderr summary at run completion.
This is the Codex D1 contract: the temporal-reasoning delta we
publish is "gbrain + Haiku-preprocess pipeline" vs "gbrain alone",
not directly comparable to LongMemEval's published baselines
without that disclosure.
Extractor cache hit-rate stderr summary (Codex Problem 14):
'[longmemeval] extractor.cache_hits: 412 / 489 sessions (84.2%,
cached_bodies=412)' — empirical verification of the optimistic
hit-rate claim. The optimistic number self-verifies per run.
Test coverage (all hermetic, no API keys):
- test/longmemeval-intent.test.ts (9 cases) — dataset
question_type → Intent mapping for all six LongMemEval labels;
dataset label trumps question-text signal; unknown labels fall
through to the regex classifier.
- test/longmemeval-trajectory-routing.test.ts (4 cases) —
end-to-end through runEvalLongMemEval with both clients stubbed:
trajectory block lands in answer-gen prompt for temporal
intent + absent for 'other'; --no-trajectory bypasses extractor
AND injection AND omits envelope fields; methodology_note
stamped on every routed row; perf gate preserved (< 10s for
2-question fixture).
118 tests pass across 11 impacted suites; bun run verify clean.
Wave complete. CHANGELOG draft + measurement plan live in the plan
file. v0.40.2.0 ready for /ship after a real-LLM spot-check run.
Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump version and changelog (v0.40.2.0)
v0.40.2.0 trajectory routing wave — gbrain think now grounds answers
about temporal/knowledge-update questions in the typed-claim timeline
the brain has been quietly building via the extract_facts cycle phase.
Default ON; flip think.trajectory_enabled=false to opt out.
LongMemEval-side wiring lands the same plumbing in the benchmark
harness with explicit methodology disclosure (extractor=haiku-preprocess-
full-haystack-v1) in the JSON envelope and stderr summary — the published
temporal-reasoning number is "gbrain + Haiku-preprocess" vs "gbrain alone",
not directly comparable to LongMemEval's published baselines without that
disclosure.
Plan: ~/.claude/plans/system-instruction-you-are-working-crystalline-owl.md
3 review passes: CEO + ENG + CODEX all CLEARED.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: sync project docs for v0.40.2.0 trajectory routing
CLAUDE.md, README.md, AGENTS.md extended with the v0.40.2.0 trajectory
routing surface: gbrain think integration (default ON via
think.trajectory_enabled config key), facts.event_type schema column +
TrajectoryPoint.event_type + TrajectoryOpts.kind filter, shared
formatTrajectoryBlock helper in src/core/trajectory-format.ts,
LongMemEval extractor + intent routing + methodology disclosure,
migration v82.
llms-full.txt regenerated to match CLAUDE.md edits (CI test/build-llms
gate).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: fill v0.40.2.0 unit + e2e gaps (71 new tests across 7 gap areas)
Audit of the trajectory-routing wave's test surface vs the shipped
code surfaced 7 gaps. All filled, all green. Total: 343 tests across
17 impacted suites (was 272 pre-fill).
Gap 1 — Migration v86 structural tests (11 new in test/migrate.test.ts):
- v86 entry exists with documented name + idempotent
- exactly one event_type column add to facts
- IF NOT EXISTS guard
- column is nullable (no NOT NULL, no DEFAULT regression guard)
- does NOT create any index (event_type is selectivity-poor)
- does NOT touch any other table (blast-radius pin)
- does NOT carry a sqlFor override (engine-shared SQL contract)
- PGLite round-trip: column exists with right type + nullable
- event_type INSERT/SELECT round-trip
- NULL round-trip for legacy + metric-only rows
- LATEST_VERSION >= 86 contract pin
Gap 2 — resolveEntitySlugWithSource branch coverage (12 new in
test/entity-resolve.test.ts):
- exact_page branch (full slug, slug-shape match)
- fuzzy_match branch (Title-cased display name, bare first name via
prefix expansion)
- fallback_slugify branch (unseeded name, multi-word non-match
phrase, accented input)
- null tail (empty + whitespace)
- back-compat parity with resolveEntitySlug for both exact_page and
fallback_slugify branches
Gap 3 — INJECTION_PATTERNS dedicated coverage for new entries (18 new
in test/think-sanitize-trajectory.test.ts):
- close-trajectory entry registered + matches canonical and
whitespace/case variations
- open-trajectory entry registered + matches both no-attr and
with-attrs forms
- xml-attr-inject strips entity=/metric=/event_type=/kind=
- does NOT strip non-trajectory attribute names (class/id/title)
- combined multi-vector attack: all three patterns fire
- formatTrajectoryBlock end-to-end with adversarial extractor text:
one live </trajectory> (the wrapper, not the injection); one
entity= attribute (the wrapper, not the injection)
- pattern ordering invariant: new entries land after close-take
Gap 4 — runThink calibration-mode placement contract (3 new in
test/think-trajectory-injection.test.ts):
- default mode: question → pages → takes → trajectory → instruction
- calibration mode: pages → takes → calibration → trajectory →
question → instruction (Codex P6 — no third ordering invented)
- empty trajectory in calibration mode preserves the existing
calibration shape (no false-positive cue)
Gap 5 — runThink resolution_source != fallback_slugify gate (1 new
in test/think-trajectory-injection.test.ts):
- candidate that only matches via fallback_slugify is NOT queried
(think-path divergence from longmemeval-path which accepts it)
Gap 6 — E2E for runThink trajectory injection (7 new in
test/e2e/think-trajectory-pglite.test.ts):
- full pipeline lands <trajectory> block in answer-gen prompt
- knowledge_update intent annotates value-change rows with
(superseded prior)
- 'other' intent short-circuits (no block, no SQL)
- think.trajectory_enabled=false config bypasses entire path
- empty brain → graceful no-op (no crash, no block)
- multi-entity deterministic ordering
- adversarial </trajectory> in seeded fact text is escaped before
reaching the LLM (end-to-end sanitization gate)
Gap 7 — longmemeval extractor stress + persistence pins (6 new in
test/longmemeval-extract.test.ts):
- alias map cross-session stress with 12 sessions in one question;
all 12 rows collapse under ONE entity_slug
- different entities stay separate across many sessions
- embedding + embedded_at both NULL on benchmark-inserted rows
(regression guard against accidental embed-on-write)
- row_num sequential + source_markdown_slug stamped per session
(v0.32.2 partial UNIQUE index contract)
- source field stamped "longmemeval:extractor" (audit-tag pin)
- cache key invariance: same body hash hits cache across different
sessionId/slug
bun run verify clean (17 pre-checks). No regressions in any of the
14 non-new impacted suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: exempt facts.event_type from schema-bootstrap-coverage CI guard
The schema-bootstrap-coverage CI guard (test/schema-bootstrap-coverage.test.ts)
enforces that every ALTER TABLE ADD COLUMN in MIGRATIONS is covered by
applyForwardReferenceBootstrap OR by PGLITE_SCHEMA_SQL's CREATE TABLE
bodies OR by COLUMN_EXEMPTIONS.
v0.40.2.0's migration v87 adds facts.event_type but deliberately ships
without a bootstrap probe because:
- No CREATE INDEX in PGLITE_SCHEMA_SQL references event_type
- No FK references event_type
- All existing callers (founder-scorecard, eval-trajectory, gbrain
think trajectory injection) defensively skip NULL-metric rows in
per-metric math, so event_type=NULL on pre-v87 brains is invisible
- Pre-v87 brains land event_type=NULL via the migration ALTER
Exactly mirrors the precedent set by facts.claim_metric / claim_value /
claim_unit / claim_period exemptions (v67 typed-claim columns) which
are exempted for the same structural reason: column-only migration,
no forward-reference index, no downstream filter breaks on old brains.
Adding facts.event_type to COLUMN_EXEMPTIONS with a brief rationale
comment matching the existing v0.35.6 entry shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
616 lines
25 KiB
TypeScript
616 lines
25 KiB
TypeScript
/**
|
|
* v0.40.2.0 — LongMemEval inline Haiku extractor.
|
|
*
|
|
* Hermetic — uses a stubbed ThinkLLMClient + in-memory PGLite. No API
|
|
* keys, no DATABASE_URL.
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
|
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
|
import {
|
|
extractAndInsertClaims,
|
|
makeAliasMap,
|
|
resetExtractorState,
|
|
getCacheStats,
|
|
type ExtractedClaim,
|
|
} from '../src/eval/longmemeval/extract.ts';
|
|
import type { ThinkLLMClient } from '../src/core/think/index.ts';
|
|
|
|
let engine: PGLiteEngine;
|
|
|
|
beforeAll(async () => {
|
|
engine = new PGLiteEngine();
|
|
await engine.connect({ database_url: '' });
|
|
await engine.initSchema();
|
|
}, 60_000);
|
|
|
|
afterAll(async () => {
|
|
await engine.disconnect();
|
|
});
|
|
|
|
beforeEach(async () => {
|
|
await engine.executeRaw(`DELETE FROM facts`);
|
|
resetExtractorState();
|
|
});
|
|
|
|
function stubClient(claimsBySession: Map<string, ExtractedClaim[]>): {
|
|
client: ThinkLLMClient;
|
|
calls: number;
|
|
} {
|
|
const calls = { count: 0 };
|
|
const client: ThinkLLMClient = {
|
|
create: async (params) => {
|
|
calls.count++;
|
|
const userMsg = params.messages[0]?.content;
|
|
const userText = typeof userMsg === 'string' ? userMsg : '';
|
|
// Stub looks for the session-id marker we embed in body keys to
|
|
// return per-session claim sets.
|
|
let claims: ExtractedClaim[] = [];
|
|
for (const [key, value] of claimsBySession.entries()) {
|
|
if (userText.includes(key)) {
|
|
claims = value;
|
|
break;
|
|
}
|
|
}
|
|
return {
|
|
id: 'stub',
|
|
type: 'message',
|
|
role: 'assistant',
|
|
model: 'stub',
|
|
stop_reason: 'end_turn',
|
|
stop_sequence: null,
|
|
usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0, server_tool_use: null, service_tier: null },
|
|
content: [{ type: 'text', text: JSON.stringify(claims) }],
|
|
} as never;
|
|
},
|
|
};
|
|
return { client, get calls() { return calls.count; } } as { client: ThinkLLMClient; calls: number };
|
|
}
|
|
|
|
describe('extractAndInsertClaims — happy path', () => {
|
|
test('inserts validated typed-claim + event rows', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['sess-1', [
|
|
{
|
|
entity: 'Marco',
|
|
metric: 'role',
|
|
value: 1,
|
|
unit: null,
|
|
period: null,
|
|
event_type: null,
|
|
valid_from: '2026-01-01',
|
|
text: 'Marco is engineer at acme',
|
|
},
|
|
{
|
|
entity: 'Marco',
|
|
metric: null,
|
|
value: null,
|
|
unit: null,
|
|
period: null,
|
|
event_type: 'meeting',
|
|
valid_from: '2026-02-15',
|
|
text: 'coffee with Marco at Blue Bottle',
|
|
},
|
|
]]
|
|
]));
|
|
const result = await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/sess-1',
|
|
sessionId: 'sess-1',
|
|
sessionBody: 'session sess-1 content here',
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
expect(result.inserted).toBe(2);
|
|
expect(result.parsed).toBe(2);
|
|
expect(result.cacheHit).toBe(false);
|
|
|
|
const rows = await engine.executeRaw<{
|
|
entity_slug: string; claim_metric: string | null; event_type: string | null;
|
|
}>(`SELECT entity_slug, claim_metric, event_type FROM facts ORDER BY id`);
|
|
expect(rows.length).toBe(2);
|
|
expect(rows[0].claim_metric).toBe('role');
|
|
expect(rows[1].event_type).toBe('meeting');
|
|
// Both rows should share the same entity slug (canonicalized).
|
|
expect(rows[0].entity_slug).toBe(rows[1].entity_slug);
|
|
});
|
|
});
|
|
|
|
describe('extractAndInsertClaims — alias map', () => {
|
|
test('per-question scope: "Marco" + "Marco Smith" collapse to one slug within a session', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['session-with-both', [
|
|
{ entity: 'Marco', metric: 'role', value: 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'engineer' },
|
|
{ entity: 'Marco Smith', metric: 'role', value: 2, unit: null, period: null, event_type: null, valid_from: '2026-04-01', text: 'VP' },
|
|
]]
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/session-with-both',
|
|
sessionId: 'sess-1',
|
|
sessionBody: 'body with marker session-with-both',
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
const rows = await engine.executeRaw<{ entity_slug: string }>(`SELECT entity_slug FROM facts ORDER BY id`);
|
|
expect(rows.length).toBe(2);
|
|
// Both rows MUST share one slug — alias map collapsed them.
|
|
expect(rows[0].entity_slug).toBe(rows[1].entity_slug);
|
|
});
|
|
|
|
test('per-question scope: aliases persist across sessions within ONE question', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['session-A', [
|
|
{ entity: 'Marco', metric: 'role', value: 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'engineer' },
|
|
]],
|
|
['session-B', [
|
|
{ entity: 'Marco Smith', metric: 'role', value: 2, unit: null, period: null, event_type: null, valid_from: '2026-04-01', text: 'VP' },
|
|
]],
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/session-A',
|
|
sessionId: 'sess-A',
|
|
sessionBody: 'session-A body',
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/session-B',
|
|
sessionId: 'sess-B',
|
|
sessionBody: 'session-B body',
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
const rows = await engine.executeRaw<{ entity_slug: string }>(`SELECT entity_slug FROM facts ORDER BY id`);
|
|
expect(rows.length).toBe(2);
|
|
// Same person across sessions → same slug.
|
|
expect(rows[0].entity_slug).toBe(rows[1].entity_slug);
|
|
});
|
|
|
|
test('per-question scope: fresh map per question keeps aliases independent', async () => {
|
|
// First question's map: Marco resolves to alias-slug-A.
|
|
const aliasMap1 = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['q1-session', [
|
|
{ entity: 'Marco', metric: 'role', value: 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'X' },
|
|
]],
|
|
['q2-session', [
|
|
{ entity: 'Marco', metric: 'role', value: 2, unit: null, period: null, event_type: null, valid_from: '2026-04-01', text: 'Y' },
|
|
]],
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/q1-session',
|
|
sessionId: 'q1', sessionBody: 'q1-session', sourceId: 'default', aliasMap: aliasMap1,
|
|
});
|
|
// TRUNCATE between questions (harness contract).
|
|
await engine.executeRaw('DELETE FROM facts');
|
|
// Second question gets a FRESH alias map.
|
|
const aliasMap2 = makeAliasMap();
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/q2-session',
|
|
sessionId: 'q2', sessionBody: 'q2-session', sourceId: 'default', aliasMap: aliasMap2,
|
|
});
|
|
// Both aliasMaps should have resolved "marco" — but they're separate.
|
|
expect(aliasMap1.has('marco')).toBe(true);
|
|
expect(aliasMap2.has('marco')).toBe(true);
|
|
// (We can't easily assert independence because both resolve to the
|
|
// same slugify-fallback. The KEY assertion is that aliasMap1 and
|
|
// aliasMap2 are separate Map instances — the caller cleared between
|
|
// questions, not us. This test pins the contract that the function
|
|
// doesn't reach into shared module state.)
|
|
});
|
|
});
|
|
|
|
describe('extractAndInsertClaims — content-hash cache', () => {
|
|
test('second call with identical body hits cache (no extra LLM call)', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const callCounter = { count: 0 };
|
|
const client: ThinkLLMClient = {
|
|
create: async () => {
|
|
callCounter.count++;
|
|
return {
|
|
id: 'x', type: 'message', role: 'assistant', model: 's',
|
|
stop_reason: 'end_turn', stop_sequence: null,
|
|
usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0, server_tool_use: null, service_tier: null },
|
|
content: [{ type: 'text', text: JSON.stringify([
|
|
{ entity: 'X', metric: 'role', value: 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'role X' },
|
|
]) }],
|
|
} as never;
|
|
},
|
|
};
|
|
const body = 'identical session body text';
|
|
const r1 = await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/a', sessionId: 'a', sessionBody: body,
|
|
sourceId: 'default', aliasMap,
|
|
});
|
|
const r2 = await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/b', sessionId: 'b', sessionBody: body,
|
|
sourceId: 'default', aliasMap,
|
|
});
|
|
expect(r1.cacheHit).toBe(false);
|
|
expect(r2.cacheHit).toBe(true);
|
|
expect(callCounter.count).toBe(1); // Only ONE Haiku call across two sessions
|
|
const stats = getCacheStats();
|
|
expect(stats.hits).toBe(1);
|
|
expect(stats.misses).toBe(1);
|
|
});
|
|
|
|
test('different bodies miss cache', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const callCounter = { count: 0 };
|
|
const client: ThinkLLMClient = {
|
|
create: async () => {
|
|
callCounter.count++;
|
|
return {
|
|
id: 'x', type: 'message', role: 'assistant', model: 's',
|
|
stop_reason: 'end_turn', stop_sequence: null,
|
|
usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0, server_tool_use: null, service_tier: null },
|
|
content: [{ type: 'text', text: '[]' }],
|
|
} as never;
|
|
},
|
|
};
|
|
await extractAndInsertClaims({ engine, client, model: 'stub', sessionSlug: 'chat/a', sessionId: 'a', sessionBody: 'body A', sourceId: 'default', aliasMap });
|
|
await extractAndInsertClaims({ engine, client, model: 'stub', sessionSlug: 'chat/b', sessionId: 'b', sessionBody: 'body B', sourceId: 'default', aliasMap });
|
|
expect(callCounter.count).toBe(2);
|
|
});
|
|
});
|
|
|
|
describe('extractAndInsertClaims — fail-open paths', () => {
|
|
test('malformed JSON output → 0 inserted, no throw', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const client: ThinkLLMClient = {
|
|
create: async () => ({
|
|
id: 'x', type: 'message', role: 'assistant', model: 's',
|
|
stop_reason: 'end_turn', stop_sequence: null,
|
|
usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0, server_tool_use: null, service_tier: null },
|
|
content: [{ type: 'text', text: 'this is not JSON {{{' }],
|
|
} as never),
|
|
};
|
|
const result = await extractAndInsertClaims({
|
|
engine, client, model: 'stub', sessionSlug: 'chat/a', sessionId: 'a',
|
|
sessionBody: 'body', sourceId: 'default', aliasMap,
|
|
});
|
|
expect(result.inserted).toBe(0);
|
|
});
|
|
|
|
test('Haiku call throws → 0 inserted, no throw', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const client: ThinkLLMClient = {
|
|
create: async () => { throw new Error('synthetic API failure'); },
|
|
};
|
|
const result = await extractAndInsertClaims({
|
|
engine, client, model: 'stub', sessionSlug: 'chat/a', sessionId: 'a',
|
|
sessionBody: 'body', sourceId: 'default', aliasMap,
|
|
});
|
|
expect(result.inserted).toBe(0);
|
|
});
|
|
|
|
test('empty array output → 0 inserted, no throw', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const client: ThinkLLMClient = {
|
|
create: async () => ({
|
|
id: 'x', type: 'message', role: 'assistant', model: 's',
|
|
stop_reason: 'end_turn', stop_sequence: null,
|
|
usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0, server_tool_use: null, service_tier: null },
|
|
content: [{ type: 'text', text: '[]' }],
|
|
} as never),
|
|
};
|
|
const result = await extractAndInsertClaims({
|
|
engine, client, model: 'stub', sessionSlug: 'chat/a', sessionId: 'a',
|
|
sessionBody: 'body', sourceId: 'default', aliasMap,
|
|
});
|
|
expect(result.inserted).toBe(0);
|
|
expect(result.parsed).toBe(0);
|
|
});
|
|
|
|
test('invalid records (missing entity, bad date) are dropped silently', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const client: ThinkLLMClient = {
|
|
create: async () => ({
|
|
id: 'x', type: 'message', role: 'assistant', model: 's',
|
|
stop_reason: 'end_turn', stop_sequence: null,
|
|
usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0, server_tool_use: null, service_tier: null },
|
|
content: [{ type: 'text', text: JSON.stringify([
|
|
{ metric: 'mrr', value: 100, text: 'missing entity', valid_from: '2026-01-01' }, // no entity
|
|
{ entity: 'X', metric: 'mrr', value: 100, text: 'bad date', valid_from: 'not-a-date' },
|
|
{ entity: 'Valid', metric: 'mrr', value: 50, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'ok row' }, // ok
|
|
]) }],
|
|
} as never),
|
|
};
|
|
const result = await extractAndInsertClaims({
|
|
engine, client, model: 'stub', sessionSlug: 'chat/a', sessionId: 'a',
|
|
sessionBody: 'body', sourceId: 'default', aliasMap,
|
|
});
|
|
expect(result.parsed).toBe(1); // Only the valid row passed validation
|
|
expect(result.inserted).toBe(1);
|
|
});
|
|
});
|
|
|
|
describe('extractAndInsertClaims — event vs metric kind', () => {
|
|
test('event rows are inserted with kind="event"', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['sess-1', [
|
|
{ entity: 'X', metric: null, value: null, unit: null, period: null, event_type: 'meeting', valid_from: '2026-01-01', text: 'met at coffee' },
|
|
]]
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub', sessionSlug: 'chat/sess-1', sessionId: 'sess-1',
|
|
sessionBody: 'sess-1', sourceId: 'default', aliasMap,
|
|
});
|
|
const rows = await engine.executeRaw<{ kind: string }>(`SELECT kind FROM facts`);
|
|
expect(rows[0].kind).toBe('event');
|
|
});
|
|
|
|
test('metric rows are inserted with kind="fact"', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['sess-1', [
|
|
{ entity: 'X', metric: 'mrr', value: 100, unit: 'USD', period: 'monthly', event_type: null, valid_from: '2026-01-01', text: 'MRR' },
|
|
]]
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub', sessionSlug: 'chat/sess-1', sessionId: 'sess-1',
|
|
sessionBody: 'sess-1', sourceId: 'default', aliasMap,
|
|
});
|
|
const rows = await engine.executeRaw<{ kind: string }>(`SELECT kind FROM facts`);
|
|
expect(rows[0].kind).toBe('fact');
|
|
});
|
|
});
|
|
|
|
describe('extractAndInsertClaims — cache stats reporting', () => {
|
|
test('getCacheStats returns hits/misses/size', async () => {
|
|
resetExtractorState();
|
|
expect(getCacheStats()).toEqual({ hits: 0, misses: 0, size: 0 });
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([['x', [
|
|
{ entity: 'X', metric: 'mrr', value: 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'X' },
|
|
]]]));
|
|
await extractAndInsertClaims({ engine, client, model: 'stub', sessionSlug: 'chat/a', sessionId: 'a', sessionBody: 'body-x', sourceId: 'default', aliasMap });
|
|
await extractAndInsertClaims({ engine, client, model: 'stub', sessionSlug: 'chat/b', sessionId: 'b', sessionBody: 'body-x', sourceId: 'default', aliasMap });
|
|
const stats = getCacheStats();
|
|
expect(stats.misses).toBe(1);
|
|
expect(stats.hits).toBe(1);
|
|
expect(stats.size).toBe(1);
|
|
});
|
|
});
|
|
|
|
// ─────────────────────────────────────────────────────────────────────
|
|
// v0.40.2.0 — extractor stress + persistence shape pins
|
|
// ─────────────────────────────────────────────────────────────────────
|
|
|
|
describe('extractAndInsertClaims — alias map cross-session stress (10+ sessions)', () => {
|
|
test('one canonical slug per name across 12 sessions in the same question', async () => {
|
|
// Tests the core LongMemEval contract: the user mentions a person
|
|
// by varying name forms ("Marco", "Marco Smith", "marco") across
|
|
// many haystack sessions in ONE question. The alias map collapses
|
|
// them all under one slug. If it didn't, the trajectory router
|
|
// would later split the entity across multiple slugs and fragment
|
|
// the timeline.
|
|
const aliasMap = makeAliasMap();
|
|
|
|
// Build 12 sessions, each contributing 1 claim about "Marco" in
|
|
// varying name forms.
|
|
const nameForms = [
|
|
'Marco', 'marco', 'Marco Smith', 'marco smith',
|
|
'MARCO', 'Marco', 'Marco Smith Jr', 'Marco S.',
|
|
'marco', 'Marco', 'Marco Smith', 'marco',
|
|
];
|
|
|
|
for (let i = 0; i < nameForms.length; i++) {
|
|
const name = nameForms[i];
|
|
const sessionId = `sess-${i + 1}`;
|
|
const { client } = stubClient(new Map([
|
|
[`marker-${sessionId}`, [
|
|
{
|
|
entity: name,
|
|
metric: 'role',
|
|
value: i + 1,
|
|
unit: null,
|
|
period: null,
|
|
event_type: null,
|
|
valid_from: `2026-${String((i % 12) + 1).padStart(2, '0')}-01`,
|
|
text: `claim ${i + 1} about ${name}`,
|
|
},
|
|
]]
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine,
|
|
client,
|
|
model: 'stub',
|
|
sessionSlug: `chat/${sessionId}`,
|
|
sessionId,
|
|
sessionBody: `body marker-${sessionId}`,
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
}
|
|
|
|
// Pin: all 12 rows landed under ONE entity_slug (the alias map
|
|
// collapsed every name form to the first-mention canonical).
|
|
const rows = await engine.executeRaw<{ entity_slug: string }>(
|
|
`SELECT DISTINCT entity_slug FROM facts ORDER BY entity_slug`,
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
|
|
const total = await engine.executeRaw<{ count: string }>(
|
|
`SELECT COUNT(*)::text AS count FROM facts`,
|
|
);
|
|
expect(Number(total[0].count)).toBe(12);
|
|
});
|
|
|
|
test('different entities stay separate across many sessions', async () => {
|
|
const aliasMap = makeAliasMap();
|
|
// 6 sessions mixing two entities (Marco + Alice). Each session
|
|
// mentions only ONE of them. Pin: 2 distinct entity_slugs after
|
|
// all sessions land.
|
|
const interleaved = ['Marco', 'Alice', 'Marco', 'Alice', 'Marco Smith', 'Alice Example'];
|
|
for (let i = 0; i < interleaved.length; i++) {
|
|
const name = interleaved[i];
|
|
const { client } = stubClient(new Map([
|
|
[`pair-${i}`, [
|
|
{ entity: name, metric: 'role', value: i + 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: `claim ${i}` },
|
|
]],
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: `chat/pair-${i}`,
|
|
sessionId: `pair-${i}`,
|
|
sessionBody: `body pair-${i}`,
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
}
|
|
const rows = await engine.executeRaw<{ entity_slug: string }>(
|
|
`SELECT DISTINCT entity_slug FROM facts ORDER BY entity_slug`,
|
|
);
|
|
expect(rows.length).toBe(2);
|
|
});
|
|
});
|
|
|
|
describe('extractAndInsertClaims — persistence shape pins', () => {
|
|
test('embedding column is NULL on every benchmark-inserted row', async () => {
|
|
// The extractor passes `embedding: null` because the benchmark
|
|
// doesn't need drift_score. Pin: every inserted row has NULL in
|
|
// both embedding AND embedded_at. If a future refactor adds an
|
|
// embed-on-write path, this test catches it.
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['sess-emb', [
|
|
{ entity: 'X', metric: 'mrr', value: 100, unit: 'USD', period: 'monthly', event_type: null, valid_from: '2026-01-01', text: 'mrr' },
|
|
{ entity: 'X', metric: null, value: null, unit: null, period: null, event_type: 'meeting', valid_from: '2026-02-01', text: 'met X' },
|
|
]],
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/sess-emb',
|
|
sessionId: 'sess-emb',
|
|
sessionBody: 'body sess-emb',
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
|
|
const rows = await engine.executeRaw<{
|
|
embedding: string | null;
|
|
embedded_at: Date | string | null;
|
|
}>(`SELECT embedding::text AS embedding, embedded_at FROM facts`);
|
|
expect(rows.length).toBe(2);
|
|
for (const r of rows) {
|
|
expect(r.embedding).toBeNull();
|
|
expect(r.embedded_at).toBeNull();
|
|
}
|
|
});
|
|
|
|
test('row_num + source_markdown_slug populated correctly across multi-claim sessions', async () => {
|
|
// The extractor assigns sequential row_num (1, 2, 3, ...) and
|
|
// stamps source_markdown_slug to the session slug. Pin both for
|
|
// the v0.32.2 partial UNIQUE index that requires
|
|
// (source_id, source_markdown_slug, row_num) uniqueness.
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['multi', [
|
|
{ entity: 'A', metric: 'mrr', value: 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'a' },
|
|
{ entity: 'B', metric: 'mrr', value: 2, unit: null, period: null, event_type: null, valid_from: '2026-02-01', text: 'b' },
|
|
{ entity: 'C', metric: 'mrr', value: 3, unit: null, period: null, event_type: null, valid_from: '2026-03-01', text: 'c' },
|
|
]],
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/multi-rownum',
|
|
sessionId: 'multi-rownum',
|
|
sessionBody: 'body multi',
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
|
|
const rows = await engine.executeRaw<{
|
|
row_num: number;
|
|
source_markdown_slug: string;
|
|
}>(`SELECT row_num, source_markdown_slug FROM facts ORDER BY row_num`);
|
|
expect(rows.length).toBe(3);
|
|
expect(rows.map(r => r.row_num)).toEqual([1, 2, 3]);
|
|
for (const r of rows) {
|
|
expect(r.source_markdown_slug).toBe('chat/multi-rownum');
|
|
}
|
|
});
|
|
|
|
test('source field is stamped "longmemeval:extractor" for audit', async () => {
|
|
// Pins the source-tag that distinguishes benchmark-extracted facts
|
|
// from production facts (the autoplan path uses `cli:think`,
|
|
// extract_facts cycle uses `cycle:extract_facts`, etc).
|
|
const aliasMap = makeAliasMap();
|
|
const { client } = stubClient(new Map([
|
|
['tag', [
|
|
{ entity: 'X', metric: 'mrr', value: 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'x' },
|
|
]],
|
|
]));
|
|
await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/tag',
|
|
sessionId: 'tag',
|
|
sessionBody: 'body tag',
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
const rows = await engine.executeRaw<{ source: string; source_session: string }>(
|
|
`SELECT source, source_session FROM facts`,
|
|
);
|
|
expect(rows.length).toBe(1);
|
|
expect(rows[0].source).toBe('longmemeval:extractor');
|
|
expect(rows[0].source_session).toBe('tag');
|
|
});
|
|
});
|
|
|
|
describe('extractAndInsertClaims — cache key invariance', () => {
|
|
test('same body text → same hash → second call hits cache regardless of sessionId/slug', async () => {
|
|
resetExtractorState();
|
|
const aliasMap = makeAliasMap();
|
|
const callCounter = { count: 0 };
|
|
const client: ThinkLLMClient = {
|
|
create: async () => {
|
|
callCounter.count++;
|
|
return {
|
|
id: 'k', type: 'message', role: 'assistant', model: 's',
|
|
stop_reason: 'end_turn', stop_sequence: null,
|
|
usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0, server_tool_use: null, service_tier: null },
|
|
content: [{ type: 'text', text: JSON.stringify([
|
|
{ entity: 'X', metric: 'mrr', value: 1, unit: null, period: null, event_type: null, valid_from: '2026-01-01', text: 'x' },
|
|
]) }],
|
|
} as never;
|
|
},
|
|
};
|
|
const body = 'identical body shared across two completely different sessions';
|
|
|
|
const r1 = await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/alpha',
|
|
sessionId: 'alpha',
|
|
sessionBody: body,
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
const r2 = await extractAndInsertClaims({
|
|
engine, client, model: 'stub',
|
|
sessionSlug: 'chat/beta-different-slug',
|
|
sessionId: 'beta-completely-different-id',
|
|
sessionBody: body,
|
|
sourceId: 'default',
|
|
aliasMap,
|
|
});
|
|
// Cache hit on r2 because sessionBody hash matches; sessionId and
|
|
// sessionSlug are NOT in the hash key (cache is body-content scoped).
|
|
expect(r1.cacheHit).toBe(false);
|
|
expect(r2.cacheHit).toBe(true);
|
|
expect(callCounter.count).toBe(1);
|
|
});
|
|
});
|