mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* v0.31 feat(migrate): facts hot memory schema (migration v40) Phase 1 of v0.31 hot-memory. - New facts table with source_id (TEXT FK to sources, per-source isolation), kind CHECK (event/preference/commitment/belief/fact), visibility CHECK (private/world for takes-style ACL parity), valid_from/valid_until/ expired_at/superseded_by for temporal + supersession audit, and consolidated_at/consolidated_into pointing at takes(id) for the dream- cycle hot→cold bridge. - Embedding column dim resolved at migration time from config.embedding_dimensions so non-OpenAI brains (Voyage etc) work out-of-the-box. HALFVEC where pgvector >= 0.7; falls back to VECTOR with stderr warn on older versions. Matching opclass per column type (halfvec_cosine_ops vs vector_cosine_ops). - 5 partial indexes leading on source_id so every read uses the trust boundary as part of the index, not a callback. HNSW partial index excludes expired/null rows so footprint stays proportional to active fact count. - RLS DO-block matches takes pattern (Postgres BYPASSRLS gate; PGLite no-op). - v0_31_0.ts orchestrator follows v0_28_0.ts pattern — phase A asserts schema version >= 40 + facts table presence; runner owns ledger. All 87 existing migrate.test.ts cases pass. PGLite smoke test confirms table + indexes + CHECK constraints + ON DELETE CASCADE all behave. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 chore(version): bump VERSION + package.json to 0.31.0 Phase 1 closer. CHANGELOG entry written when Phase 7 lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(engine): facts hot memory engine API (Phase 2) Phase 2 of v0.31 hot-memory. Adds 8 facts methods to BrainEngine implemented on both PGLite and Postgres engines: - insertFact(input, ctx) — INSERT with optional supersedeId; expires the named row in the same transaction. Per-entity advisory lock on Postgres (`pg_advisory_xact_lock(hashtextextended(source_id::text || ':' || entity_slug, 0))`) for the dedup window. PGLite is single-process so the lock is a no-op. - expireFact(id, opts) — sets expired_at + optional superseded_by. Idempotent-as-false (already-expired returns false). - listFactsByEntity / listFactsSince / listFactsBySession — list surfaces with FactListOpts filters (activeOnly, kinds, visibility, limit/offset). Every query starts WHERE source_id = $X so the trust boundary is part of the index path. - listSupersessions — audit log; activeOnly:false + expired_at IS NOT NULL + superseded_by IS NOT NULL. - findCandidateDuplicates(source_id, entity_slug, factText, k) — entity-prefiltered (mandatory), k=5 default, hard cap 20. Embedding- cosine ordering when caller supplies an embedding, recency fallback otherwise. Bounds the contradiction-classifier blast radius. - consolidateFact(id, takeId) — sets consolidated_at + consolidated_into. Never DELETE; facts stay as audit trail for the resulting take. - getFactsHealth(source_id) — per-source counters consumed by `gbrain doctor` facts_health check. Public types in engine.ts: FactKind (5-value union), FactVisibility, FactInsertStatus, FactRow, NewFact, FactListOpts, FactsHealth. PGLite + Postgres helpers: rowToFact / rowToFactPg parse the text-format pgvector embedding back into Float32Array; toPgVectorLiteral encodes for the supersede-path INSERT (postgres-js can't bind Float32Array directly to a vector column without an explicit literal cast). Smoke test confirms every method end-to-end on PGLite. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(facts): extraction code path (Phase 3) Phase 3 of v0.31 hot-memory. Five new modules under src/core/facts/ + src/core/entities/: - src/core/facts/decay.ts — pure helper. effectiveConfidence(fact, now) applies confidence × exp(-age/halflife) with per-kind halflife table (event 7d, commitment 90d, preference 90d, belief 365d, fact 365d). Returns 0 for expired or past-valid_until rows. Single source of truth consumed by recall, supersession audit, facts_health, and the MCP _meta injector (eD8 DRY). - src/core/facts/queue.ts — bounded in-memory queue. Cap 100 default, drop-oldest on overflow with counter. Per-session in-flight=1 serializes burst chat. AbortSignal threading from server SIGTERM (mirrors minion worker pattern per eD7): 5s grace for in-flight, then drop pending with counter. getFactsQueue() process-singleton; __resetFactsQueueForTests for hermetic tests. - src/core/facts/classify.ts — contradiction classifier with cosine fast-path (D13: ≥0.95 → duplicate, skip LLM) and classifier-failure fallback (D12: cosine ≥0.92 → duplicate, else INSERT). Pure cosine helper exported. JSON-strict output with 4-strategy parse fallback; refusal stop-reason maps to fallback path. Caller-provided abort signal propagated to the gateway chat call. - src/core/facts/extract.ts — Haiku turn-extractor. Reuses INJECTION_PATTERNS from src/core/think/sanitize.ts on the way IN (turn_text) AND on the way OUT (each fact). Tight system prompt with 5-kind taxonomy, 0..1 confidence scoring, entity slug or display name. Anti-loop check on isDreamGenerated (reuses v0.23.2 marker semantics). Synchronous embedOne() per fact via the gateway so classifier paths have embeddings available; AbortError re-thrown explicitly so SIGTERM during embed never writes a NULL-embedding row meant to be cancelled (eE8 distinction). - src/core/entities/resolve.ts — slug canonicalization shared by signal-detector AND facts. Resolution order: exact slug match → pg_trgm fuzzy match (similarity ≥0.4) → deterministic slugify fallback. slugify exported standalone for tests + callers that want the floor. Smoke tests confirm decay table, cosine math, slugify rules, queue drop-oldest under overflow, and shutdown grace + drop-pending semantics. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(mcp+cli): MCP ops + recall CLI + _meta + transport refactor (Phase 4) Phase 4 of v0.31 hot-memory. Three new MCP ops on the contract-first surface: - `extract_facts` (write scope, localOnly:false): extracts facts from a conversation turn via the Haiku extractor, runs the cosine fast-path dedup, INSERTs into per-source hot memory. Returns counts + fact_ids[]. Skips on is_dream_generated:true (anti-loop). - `recall` (read scope): query the per-source hot memory by entity / since / session / supersessions / grep filter. Visibility- aware: remote callers see visibility='world' rows only (takes-style ACL parity, eD21). Returns most-recent first; pagination via limit. - `forget_fact` (write scope): expireFact wrapper. Idempotent-as-error on unknown id; uses the new 'fact_not_found' ErrorCode. ErrorCode union opened (eD6 / eE7): TS forward-compat via the `(string & {})` autocomplete-friendly hack so downstream consumers (gbrain-evals etc) don't break their typecheck on every new code. Three new codes: 'rate_limited', 'extraction_failed', 'fact_not_found'. OperationContext gains source_id?:string (eD4 / eE2 — TEXT not INTEGER per schema reality). Resolved once in buildOperationContext from DispatchOpts.sourceId. Stdio MCP defaults to GBRAIN_SOURCE env or 'default'; HTTP MCP reads it from the per-token sources scope (eE3). ToolResult gains _meta?: Record<string, unknown> (eD3). Dispatcher calls a configurable metaHook AFTER op.handler succeeds, wrapped in its own try/catch so a DB blip degrades to no-_meta rather than flipping the whole tool call to error (eE4). New module src/core/facts/meta-hook.ts: - getBrainHotMemoryMeta(name, ctx) builds the _meta.brain_hot_memory payload. Cache key (source_id, session_id, hash(takesHoldersAllowList sorted)) (eD10 / eE5). 30s TTL per session. Visibility filter applies: remote → world only; local → all. Top-K=10 ranked by effective confidence (decay). Skips injection on recall/extract_facts/forget_fact themselves. bumpHotMemoryCache() invalidates per (source_id, session_id) on extraction event. D12 (eE1) accepted: serve-http.ts:801 inlined dispatch path REFACTORED to call dispatchToolCall. HTTP MCP now inherits source_id, _meta injection, error envelope unification, and OperationContext shape from the same code path stdio uses. Scope check + mcp_request_log + SSE broadcast stay in serve-http.ts (HTTP-specific concerns); the dispatcher returns ToolResult and the HTTP handler reads isError + content + _meta to fan into the audit + broadcast paths. put_page compliance backstop (D23): when a conversation-shape page is written (note/meeting/slack/email/calendar-event/source/writing) with a substantive body (>=80 chars) on a non-subagent slug AND no dream_generated:true marker, fire-and-forget enqueue an extraction job into the bounded queue. Never blocks the put_page response. Skipped reasons (no_parsed_page / subagent_namespace / dream_generated / kind:* / too_short / queue_shutdown / backstop_error) are stable strings consumed by tests. `gbrain recall` + `gbrain forget` CLI commands (src/commands/recall.ts): - recall <entity> | --since DUR | --session ID | --today (markdown with kind icons 📅🎯🤝💭📌) | --grep TEXT | --supersessions | --include-expired | --as-context (prompt-injection-ready) | --json - forget <fact-id> shorthand for expireFact Wired into src/cli.ts dispatch table next to takes / think. Smoke tests confirm: dispatch surfaces (extract_facts → ops → listFactsByEntity), forget_fact + idempotent re-call, _meta visibility filter (remote sees world only, local sees all), CLI markdown render with kind icons + age strings + decayed confidence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(cycle): consolidate phase — facts → takes promotion (Phase 5) Phase 5 of v0.31 hot-memory. New 10th cycle phase `consolidate` between `patterns` and `embed`: - src/core/cycle.ts: * CyclePhase union extended with 'consolidate' * ALL_PHASES gets 'consolidate' between patterns and embed (graph-fresh after patterns; embed runs after so the new takes get embedded same-cycle) * NEEDS_LOCK_PHASES gets 'consolidate' (writes takes + UPDATEs facts) * CycleReport.totals gains facts_consolidated + consolidate_takes_written * runCycle dispatches the new phase via dynamic import - src/core/cycle/phases/consolidate.ts (new): * Scans (source_id, entity_slug) buckets where COUNT(unconsolidated facts) >= 3 (uses idx_facts_unconsolidated partial index) * Skips buckets where the OLDEST fact is < 24h old (gives signal time to settle before locking it into cold memory) * Greedy cosine clustering at threshold 0.85; head-element centroid keeps it deterministic + cheap. Singletons (no embedding) stay unconsolidated this cycle. * For each cluster size >= 2: picks the highest-confidence fact's text as the take claim (v0.31 deterministic; v0.32 swaps to Sonnet synthesis pass). avg confidence → take weight, earliest valid_from → take since_date, concatenated source_sessions → take.source. * Resolves entity_slug → page_id via pages.slug (per source). Skips cluster if page is missing in this source — no auto-page-creation in v0.31. * INSERT into takes(kind='fact', holder='self') with row_num = MAX(existing) + 1. * UPDATE contributing facts: consolidated_at = now() + consolidated_into = takes.id. NEVER DELETE — facts are the audit trail for the resulting take. * dryRun honored: pretends the writes happened; counters still tick so operators can preview load before the first real run. * yieldDuringPhase keepalive between buckets so the Minions worker job lock + cycle-lock TTL don't drift on long runs. Smoke test on PGLite confirms: 4 unconsolidated facts → clustered (cosine 1.0 since same vector) → 1 take row created → all 4 facts marked consolidated_into. runCycle({phases:['consolidate']}) wires through to the report totals. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: 18 facts test files (Phase 6) Phase 6 of v0.31 hot-memory: comprehensive coverage across the new substrate. 110 unit tests pass; 5 E2E test files added (skip gracefully without DATABASE_URL). Unit tests (PGLite in-memory, no DATABASE_URL): - test/facts-decay.test.ts (12 cases) — HALFLIFE_DAYS pinned per kind, effectiveConfidence math: age=0 / age=halflife (~1/e) / age=2×halflife (~1/e²) / expired returns 0 / valid_until past returns 0 / preference-vs-event slower decay / belief-vs-commitment crossover. - test/facts-queue.test.ts (10 cases) — FIFO within session, drop-oldest on overflow, per-session in-flight=1 serializes, different sessions parallelize, failed jobs counter, shutdown grace + drop_pending + external AbortController triggers shutdown. - test/facts-classify.test.ts (8 cases) — cosineSimilarity edge cases, empty candidates → independent, cheap fast-path ≥0.95 → duplicate no LLM, threshold-configurable cosine_fallback path. - test/facts-engine.test.ts (13 cases) — every BrainEngine fact method end-to-end: insertFact (insert/supersede), expireFact idempotency, list*, findCandidateDuplicates entity-prefiltered + k cap + cosine ordering, consolidateFact never DELETE, getFactsHealth shape + total_today ⊆ total_week. - test/facts-multi-tenant.test.ts (6 cases) — cross-source isolation on every list method + CASCADE delete on sources. - test/facts-visibility.test.ts (6 cases) — visibility column private/ world; remote=true filters to world-only via dispatchToolCall; remote=false sees all. - test/facts-canonicality.test.ts (10 cases) — slugify rules including NFKD diacritic strip ("Crème Brûlée" → "creme-brulee"), exact slug match, fallback to slugify when no fuzzy match. - test/facts-extract.test.ts (4 cases) — empty turn returns [], dream- generated short-circuit, graceful no-API-key return. - test/facts-backstop-gating.test.ts (5 cases) — put_page backstop: too_short, subagent_namespace, dream_generated, eligible note path, non-eligible kind:guide. - test/facts-anti-loop.test.ts (4 cases) — extractor + put_page both respect dream_generated:true marker. - test/facts-doctor-shape.test.ts (4 cases) — facts_health JSON shape pinned for downstream consumers. - test/facts-mcp-allowlist.serial.test.ts (5 cases) — extract_facts write-scope, recall read-scope, forget_fact write-scope, forget_fact fact_not_found error code, extract_facts no-API-key zero counts. - test/facts-context-injection.serial.test.ts (6 cases) — _meta injection on success, world-only filter under remote=true, anti-loop on facts ops themselves, best-effort degrade on hook error, cache-key includes allow-list hash. - test/facts-separation-pglite.test.ts (2 cases) — Garry's Separation Test as primary ship gate, plus expired hidden-by-default contract. - test/facts-recall-render.test.ts (3 cases) — --today markdown render with all 5 kind icons, --json shape with effective_confidence, --as-context emits comment-wrapped block. - test/facts-migration-dim.test.ts (4 cases) — embedding column type is HALFVEC/VECTOR (not arbitrary), dim matches gateway-configured embedding_dimensions, HNSW opclass agrees with column type, idempotent re-init. - test/cycle-consolidate.test.ts (5 cases) — below-count + below-age thresholds skip, happy path 4 facts → 1 take + all consolidated never DELETE, dryRun honored, missing page → bucket skipped. E2E tests (skip gracefully on DATABASE_URL unset; required gates by CLAUDE.md test policy): - test/e2e/facts-separation-postgres.test.ts — Postgres parity for the ship gate. - test/e2e/facts-cross-source-isolation.test.ts — cross-source ACL on PG + CASCADE delete. - test/e2e/facts-forget.test.ts — full forget_fact MCP roundtrip. - test/e2e/facts-context-injection-postgres.test.ts — _meta injection end-to-end on PG. - test/e2e/facts-recall-render.test.ts — recall --today markdown on PG. - test/e2e/serve-http-meta.test.ts — eE1 regression: HTTP MCP transport inherits _meta + sourceId + scope correctness via dispatchToolCall. Side-effect: src/core/entities/resolve.ts NFKD post-decompose strips combining marks (U+0300..U+036F) before hyphenating non-alphanumerics, so "Crème" → "creme", not "cre-me-". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(operational): kill switch + doctor check + CHANGELOG + README (Phase 7) Phase 7 of v0.31 hot-memory. - src/core/facts/extract.ts: new isFactsExtractionEnabled(engine) helper reads `facts.extraction_enabled` config row. Defaults to TRUE; flip to 'false'/'0'/'no'/'off' (case-insensitive) via `gbrain config set facts.extraction_enabled false` to kill extraction across the brain without binary downgrade. - extract_facts MCP op short-circuits with zero-counts envelope + a 'skipped: extraction_disabled' field when the flag is off (clean success, not permission_denied). - put_page facts backstop respects the same flag — eligibility check now returns 'extraction_disabled' as the skipped reason. - src/commands/doctor.ts: new facts_health check (runs after queue_health, before index_audit). Probes for the facts table existence (post-v40 guard), then surfaces total_active / total_today / total_week / total_consolidated + top-3 entities for the default source. Pre-v0.31 brains report "facts table not present (pre-v0.31 brain or migration pending)". - CHANGELOG.md: full v0.31.0 entry in the GStack release-summary voice. Headline + numbers-table + what-it-ships + itemized changes + "To take advantage of v0.31" upgrade block + out-of-scope. Honest about the HALFVEC + serve-http refactor + ErrorCode-open-union complications. - README.md: cycle phase list updated 8 → 10 (consolidate + purge). New "v0.31 Hot Memory" command block under Commands with recall + forget variants, kind icons, --as-context surface for headless agents. Test gates: 28 facts unit tests pass after the kill-switch wiring + doctor check ride-along. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(migrate): add facts→sources FK explicitly via ALTER TABLE The inline column-level FK declaration on facts.source_id worked on PGLite but silently got dropped on Postgres in the v0.31 e2e run — the migration handler ran via postgres-js's `unsafe()` multi-statement path and the resulting facts table came back without the `facts_source_id_fkey` constraint. Same psql input run directly against the same database produced the FK; the difference was the unsafe() pipeline, not the SQL itself. Splitting the FK into a separate ALTER TABLE inside a DO block makes the constraint declaration explicit and idempotent: the named constraint either exists or it doesn't, the ALTER is a no-op on re-runs, and the failure mode is loud rather than silently leaving a CASCADE-less foreign key behind. Without this fix, deleting a source row leaves orphaned facts rows (test/e2e/facts-cross-source-isolation.test.ts CASCADE-on-sources- delete case caught it). With this fix the constraint is in place, the cascade fires, and both PG + PGLite e2e suites stay green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: update phase-count assertions for the new consolidate phase Three e2e/unit tests pinned the cycle phase count or order, all now updated to reflect v0.31's 10-phase cycle: - test/e2e/dream-cycle-eight-phase-pglite.test.ts: describe rename "8-phase cycle" → "10-phase cycle"; ALL_PHASES expectation extended to include 'consolidate' (between patterns + embed) and 'purge' (the v0.26.5 addition that was already in ALL_PHASES but missing from the test's assertion list). totals match adds the new facts_consolidated + consolidate_takes_written fields plus the pre-existing purged_sources_count + purged_pages_count that should have been added when v0.26.5 landed. - test/e2e/cycle.test.ts: dry-run full cycle now expects report.phases.length === 10 (was 9). - test/core/cycle.serial.test.ts: yieldBetweenPhases hook count + full cycle phases.length both updated 9 → 10. Comments call out the v0.31 addition lineage so the next person to add a phase sees the precedent. These are mechanical assertion bumps. The tests pass against the updated assertions on PGLite and Postgres. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(test): truncate facts table between e2e describe blocks setupDB() truncates ALL_TABLES between every describe block's beforeAll() hook. The list missed the new v0.31 facts table, so facts seeded by an earlier describe block leaked into Garry's Separation Test on Postgres — listFactsByEntity('travel') returned 2 rows instead of 1 because a prior facts-context-injection test had also seeded a 'travel' fact. Adding 'facts' to the truncate list (before 'pages' to respect FK ordering) makes every describe-block start from an empty facts table. Pinned by re-running the e2e file ordering that originally caught it (facts-recall-render → cross-source-isolation → serve-http-meta → context-injection → separation-postgres → facts-forget) — 13 pass / 0 fail after the fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: meta-hook cache + Postgres consolidate phase coverage Two net-new test files filling real coverage gaps the earlier sweep missed: - test/facts-meta-cache.test.ts (5 cases) — pins the eD3/eD10 cache contract that the dispatcher relies on. 30s TTL hit path, post-bump fresh-query, scoped invalidation (bump for sess-A leaves sess-B cache warm — closes the cross-source leak risk codex F5 originally surfaced on the recall payload), facts-self ops skip injection (anti-loop on recall / extract_facts / forget_fact), distinct allow-lists produce distinct cache entries. - test/e2e/cycle-consolidate-postgres.test.ts (3 cases) — Postgres parity for the dream-cycle consolidate phase. Mirrors the PGLite unit test but exercises the real postgres-engine codepaths: sql.begin transactions, advisory locks on insertFact's entity-slug dedup window, unsafe('::vector') casts on findCandidateDuplicates ordering, addTakesBatch postgres-js unnest path. Happy path (4 facts → 1 take + all consolidated_into set), age-threshold skip, dry-run no-write. All 5 unit + 3 e2e tests pass. Closes the unit-only gap on the consolidate phase (was only PGLite-tested) and pins meta-cache invariants the dispatcher depends on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix: thread auth + sourceId, JSON-shape every error envelope Three bugs surfaced during the full e2e sweep that all trace back to my v0.31 dispatch refactor (D12/eE1) silently dropping auth threading + non-OperationError exceptions emitting plain strings: 1. **HTTP MCP transport lost ctx.auth.** Refactoring serve-http.ts to call dispatchToolCall meant auth had to come through DispatchOpts, but the field didn't exist yet. Every HTTP whoami call returned `unknown_transport` because ctx.auth was undefined. Added `auth?: AuthInfo` to DispatchOpts, plumbed it through buildOperationContext, and updated serve-http.ts:816 to pass `auth: authInfo` alongside sourceId/takesHoldersAllowList. Pinned by sources-remote-mcp e2e `whoami reports oauth transport + sources_admin scope`. 2. **Non-OperationError exceptions emitted plain strings, not JSON.** The pre-v0.31 serve-http.ts always wrapped errors in JSON envelope `{error, message}`; my dispatch refactor missed the unknown-tool + uncaught-throw paths and emitted `Error: ${msg}` text content. Every caller that did `JSON.parse(content)` (sources-remote-mcp callMcp helper at line 104) crashed with `Unexpected identifier "Error"`. Both error paths in dispatchToolCall now return JSON-shaped content matching the OperationError pattern. 3. **Files→sources FK silently lost on rewound bootstrap path.** test/e2e/postgres-bootstrap.test.ts simulates a pre-v0.21 brain by `DROP TABLE IF EXISTS sources CASCADE` which removes files_source_id_fkey while leaving files.source_id intact. The v23 migration's `ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id ... REFERENCES sources(id) ON DELETE CASCADE` is a no-op when the column exists, so the FK never came back on upgrade — and any sources-remove afterward stopped cascading to files. Added a defensive `IF NOT EXISTS files_source_id_fkey ... ALTER TABLE ADD CONSTRAINT` block inside v23's handler. Pinned by `multi-source — cascade delete covers every dependent row` after running postgres-bootstrap. Plus: src/core/preferences.ts now honors GBRAIN_HOME for `~/.gbrain/migrations/completed.jsonl`. Without this, the doctor exits-0 mechanical test inherits the developer machine's stale partial-migration ledger entries (0.21.0, 0.22.4, 0.28.0, 0.29.1 prior dev work) and surfaces them as the [FAIL] minions_migration check. GBRAIN_HOME-scoped tempdir per test now isolates this state cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 chore: scrub personal references from public artifacts Per the CLAUDE.md privacy rule on `Garry's Separation Test`, replace personally-coded references in v0.31 artifacts with neutral examples: - CHANGELOG.md v0.31 entry: rename "Garry's Separation Test" header to "The cross-session test" + drop the "topic-2659/topic-1941, 7 AM/2 PM, flying to Tokyo" narrative. - src/commands/migrations/v0_31_0.ts feature pitch: same scrub. - test/facts-separation-pglite.test.ts + test/e2e/facts-separation-postgres.test.ts: rename describe blocks; replace specific topic-NNNN session ids with session-A / session-B; replace personal sample fact with "sample event Tuesday". - src/core/facts/extract.ts extractor system prompt example slugs: people/sam-altman → people/alice-example; companies/anthropic → companies/acme. - src/core/entities/resolve.ts comment: Sam Altman → Alice Example. - All v0.31 test fixtures: people/sam → people/alice-example, Sam Altman → Alice Example, sam-the-cofounder → alice-the-cofounder. Test names referencing real-world entities replaced with neutral slugs. Pre-existing references to "Garry" elsewhere in CHANGELOG (v0.17, v0.19, v0.21+ entries) are untouched — that's a separate scope from this v0.31 ship. Plus: the truncate fix for the Bun-script-induced syntax error in test/e2e/mechanical.test.ts (cliEnv arrow function had ", 30_000)" tacked onto its closing brace by the bulk-add-timeouts script — repaired to a clean function definition). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(test): bump E2E phase-count assertions for 11-phase cycle Two E2E tests still asserted the v0.31 pre-merge 10-phase shape (consolidate inserted, but recompute_emotional_weight from v0.29 not yet absorbed). With master's v0.29 work merged in, the cycle is now 11 phases: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → consolidate → embed → orphans → purge. - test/e2e/cycle.test.ts: 10 → 11 - test/e2e/dream-cycle-eight-phase-pglite.test.ts: ALL_PHASES + dry-run order Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(merge): close brace between v44 and v45 migration objects The v0.30.2 merge resolution stitched master's v40-v44 migrations onto HEAD's v45 (facts hot memory) migration but lost the closing `},` between v44 and v45. tsc caught it as TS1136 Property assignment expected at migrate.ts:2188. This is a one-line bracket fix; the rest of the merge resolution is correct and tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix: put_page cliHints + buildPlan v0.31.0 in skippedFuture Two unit-test failures surfaced after the v0.30.2 merge: 1. operations.ts: put_page had `cliHints: { name: 'put', positional: ['stdin'] }` from earlier v0.31 development. The parity test enforces that every name in `positional` is a real param. Restored master's correct shape: `{ name: 'put', positional: ['slug'], stdin: 'content' }`. 2. test/apply-migrations.test.ts: the H9 regression tests pin the exact skippedFuture list. Adding v0.31.0 to the registry meant the list grew by one. Updated both `expect(...).toEqual([...])` assertions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 docs: clarify consolidate is 11th phase + regen llms-full.txt CHANGELOG.md narrative said "new 10th phase consolidate"; with v0.29's recompute_emotional_weight already on master, consolidate is the 11th phase (between recompute and embed). Schema migration is v45, not v40, after the merge resolution renumbered it to clear master's v40-v44. llms-full.txt regenerated to reflect the README's 11-phase dream-cycle phrasing (the build-llms test enforces commit-time parity). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
234 lines
8.8 KiB
TypeScript
234 lines
8.8 KiB
TypeScript
/**
|
|
* E2E tests for src/mcp/http-transport.ts against real Postgres.
|
|
*
|
|
* Catches schema drift (column-name typos that would slip past the unit suite's
|
|
* stubbed engine.sql) and proves the F1+F2+F3 dispatch pipeline works against a
|
|
* real handler doing real DB work. Also exercises the SQL-level last_used_at
|
|
* debounce against real Postgres semantics.
|
|
*
|
|
* Run: DATABASE_URL=... bun test test/e2e/http-transport.test.ts
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { createHash, randomBytes } from 'crypto';
|
|
import { startHttpTransport } from '../../src/mcp/http-transport.ts';
|
|
import { hasDatabase, setupDB, teardownDB, getEngine, getConn } from './helpers.ts';
|
|
|
|
const skip = !hasDatabase();
|
|
const describeE2E = skip ? describe.skip : describe;
|
|
|
|
if (skip) {
|
|
console.log('Skipping E2E http-transport tests (DATABASE_URL not set)');
|
|
}
|
|
|
|
interface ServerHandle {
|
|
port: number;
|
|
stop: () => Promise<void>;
|
|
}
|
|
|
|
function generateToken(): string {
|
|
return 'gbrain_test_' + randomBytes(16).toString('hex');
|
|
}
|
|
|
|
function hashToken(token: string): string {
|
|
return createHash('sha256').update(token).digest('hex');
|
|
}
|
|
|
|
async function startServer(): Promise<ServerHandle> {
|
|
const engine = getEngine();
|
|
const server = await startHttpTransport({ port: 0, engine: engine as any });
|
|
return {
|
|
port: (server as any).port,
|
|
stop: async () => { (server as any).stop(true); },
|
|
};
|
|
}
|
|
|
|
function rpc(method: string, params?: unknown, id: number = 1) {
|
|
return JSON.stringify({ jsonrpc: '2.0', id, method, ...(params !== undefined ? { params } : {}) });
|
|
}
|
|
|
|
describeE2E('http-transport E2E (real Postgres)', () => {
|
|
let srv: ServerHandle;
|
|
let validToken: string;
|
|
let revokedToken: string;
|
|
let validTokenName: string;
|
|
|
|
beforeAll(async () => {
|
|
await setupDB();
|
|
const conn = getConn();
|
|
|
|
// Seed a valid + revoked token directly via SQL (mirrors auth.ts's create path).
|
|
validToken = generateToken();
|
|
validTokenName = 'e2e-valid-' + randomBytes(4).toString('hex');
|
|
await conn.unsafe(
|
|
'INSERT INTO access_tokens (name, token_hash) VALUES ($1, $2)',
|
|
[validTokenName, hashToken(validToken)],
|
|
);
|
|
revokedToken = generateToken();
|
|
await conn.unsafe(
|
|
'INSERT INTO access_tokens (name, token_hash, revoked_at) VALUES ($1, $2, now())',
|
|
['e2e-revoked-' + randomBytes(4).toString('hex'), hashToken(revokedToken)],
|
|
);
|
|
|
|
srv = await startServer();
|
|
}, 30_000);
|
|
|
|
afterAll(async () => {
|
|
if (srv) await srv.stop();
|
|
await teardownDB();
|
|
});
|
|
|
|
test('1. /health → 200 with expected JSON shape', async () => {
|
|
const r = await fetch(`http://localhost:${srv.port}/health`);
|
|
expect(r.status).toBe(200);
|
|
const body = await r.json();
|
|
expect(body.status).toBe('ok');
|
|
expect(body.transport).toBe('http');
|
|
expect(body.version).toBeString();
|
|
});
|
|
|
|
test('2. /mcp tools/list with valid Bearer → 200 + ops list', async () => {
|
|
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
|
|
method: 'POST',
|
|
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
|
body: rpc('tools/list'),
|
|
});
|
|
expect(r.status).toBe(200);
|
|
const body = await r.json();
|
|
expect(body.result.tools).toBeArray();
|
|
expect(body.result.tools.length).toBeGreaterThan(5);
|
|
expect(r.headers.get('content-type')).toContain('application/json');
|
|
});
|
|
|
|
test('3. /mcp tools/call (real op: list_pages) round-trips successfully — F1+F2+F3 guard', async () => {
|
|
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
|
|
method: 'POST',
|
|
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
|
body: rpc('tools/call', { name: 'list_pages', arguments: { limit: 5 } }),
|
|
});
|
|
expect(r.status).toBe(200);
|
|
const body = await r.json();
|
|
expect(body.jsonrpc).toBe('2.0');
|
|
expect(body.result.content).toBeArray();
|
|
// Should NOT be an error — handler ran successfully against the real engine.
|
|
expect(body.result.isError).toBeUndefined();
|
|
// Result text should parse as JSON (list_pages returns an object/array)
|
|
const resultText = body.result.content[0].text;
|
|
const parsed = JSON.parse(resultText);
|
|
expect(parsed).toBeDefined();
|
|
});
|
|
|
|
test('4. revoked token → 401', async () => {
|
|
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
|
|
method: 'POST',
|
|
headers: { 'Authorization': `Bearer ${revokedToken}`, 'Content-Type': 'application/json' },
|
|
body: rpc('tools/list'),
|
|
});
|
|
expect(r.status).toBe(401);
|
|
});
|
|
|
|
test('5. last_used_at debounce: two consecutive valid calls → only one UPDATE within 60s', async () => {
|
|
const conn = getConn();
|
|
|
|
// Reset last_used_at to NULL so the first call definitely updates
|
|
await conn.unsafe('UPDATE access_tokens SET last_used_at = NULL WHERE name = $1', [validTokenName]);
|
|
|
|
// First request — should update last_used_at
|
|
await fetch(`http://localhost:${srv.port}/mcp`, {
|
|
method: 'POST',
|
|
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
|
body: rpc('tools/list'),
|
|
});
|
|
// Give the fire-and-forget UPDATE a moment to land
|
|
await new Promise(r => setTimeout(r, 50));
|
|
|
|
const [row1] = await conn.unsafe(
|
|
'SELECT last_used_at FROM access_tokens WHERE name = $1',
|
|
[validTokenName],
|
|
) as { last_used_at: Date | null }[];
|
|
expect(row1.last_used_at).not.toBeNull();
|
|
const firstUpdate = row1.last_used_at;
|
|
|
|
// Second request immediately — should NOT trigger another UPDATE (debounced by SQL WHERE)
|
|
await fetch(`http://localhost:${srv.port}/mcp`, {
|
|
method: 'POST',
|
|
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
|
body: rpc('tools/list'),
|
|
});
|
|
await new Promise(r => setTimeout(r, 50));
|
|
|
|
const [row2] = await conn.unsafe(
|
|
'SELECT last_used_at FROM access_tokens WHERE name = $1',
|
|
[validTokenName],
|
|
) as { last_used_at: Date | null }[];
|
|
// Same timestamp = same UPDATE = debounce held
|
|
expect(row2.last_used_at?.getTime()).toBe(firstUpdate?.getTime());
|
|
});
|
|
|
|
test('6. last_used_at debounce: simulating 65s gap → second request DOES update', async () => {
|
|
const conn = getConn();
|
|
|
|
// Set last_used_at to 65 seconds ago — simulates the time gap without waiting in real time
|
|
await conn.unsafe(
|
|
`UPDATE access_tokens SET last_used_at = now() - interval '65 seconds' WHERE name = $1`,
|
|
[validTokenName],
|
|
);
|
|
const [before] = await conn.unsafe(
|
|
'SELECT last_used_at FROM access_tokens WHERE name = $1',
|
|
[validTokenName],
|
|
) as { last_used_at: Date | null }[];
|
|
|
|
await fetch(`http://localhost:${srv.port}/mcp`, {
|
|
method: 'POST',
|
|
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
|
body: rpc('tools/list'),
|
|
});
|
|
await new Promise(r => setTimeout(r, 50));
|
|
|
|
const [after] = await conn.unsafe(
|
|
'SELECT last_used_at FROM access_tokens WHERE name = $1',
|
|
[validTokenName],
|
|
) as { last_used_at: Date | null }[];
|
|
expect(after.last_used_at?.getTime()).toBeGreaterThan(before.last_used_at!.getTime());
|
|
});
|
|
|
|
test('7. mcp_request_log gets a row per request', async () => {
|
|
const conn = getConn();
|
|
const beforeRows = await conn.unsafe('SELECT count(*)::int AS n FROM mcp_request_log') as { n: number }[];
|
|
const beforeN = beforeRows[0].n;
|
|
|
|
await fetch(`http://localhost:${srv.port}/mcp`, {
|
|
method: 'POST',
|
|
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
|
body: rpc('tools/list'),
|
|
});
|
|
// Fire-and-forget audit insert — give it a tick
|
|
await new Promise(r => setTimeout(r, 100));
|
|
|
|
const afterRows = await conn.unsafe('SELECT count(*)::int AS n FROM mcp_request_log') as { n: number }[];
|
|
expect(afterRows[0].n).toBeGreaterThan(beforeN);
|
|
|
|
const [row] = await conn.unsafe(
|
|
`SELECT token_name, operation, status, latency_ms FROM mcp_request_log
|
|
WHERE token_name = $1 ORDER BY created_at DESC LIMIT 1`,
|
|
[validTokenName],
|
|
) as { token_name: string; operation: string; status: string; latency_ms: number }[];
|
|
expect(row.token_name).toBe(validTokenName);
|
|
expect(row.operation).toBe('tools/list');
|
|
expect(row.status).toBe('success');
|
|
expect(row.latency_ms).toBeGreaterThanOrEqual(0);
|
|
});
|
|
|
|
test('8. tools/call with malformed params → isError result with invalid_params', async () => {
|
|
const r = await fetch(`http://localhost:${srv.port}/mcp`, {
|
|
method: 'POST',
|
|
headers: { 'Authorization': `Bearer ${validToken}`, 'Content-Type': 'application/json' },
|
|
body: rpc('tools/call', { name: 'get_page', arguments: { slug: 42 } }),
|
|
});
|
|
expect(r.status).toBe(200);
|
|
const body = await r.json();
|
|
expect(body.result.isError).toBe(true);
|
|
expect(body.result.content[0].text).toContain('invalid_params');
|
|
});
|
|
});
|