mirror of
https://github.com/garrytan/gbrain.git
synced 2026-07-27 22:15:33 +00:00
* v0.31 feat(migrate): facts hot memory schema (migration v40) Phase 1 of v0.31 hot-memory. - New facts table with source_id (TEXT FK to sources, per-source isolation), kind CHECK (event/preference/commitment/belief/fact), visibility CHECK (private/world for takes-style ACL parity), valid_from/valid_until/ expired_at/superseded_by for temporal + supersession audit, and consolidated_at/consolidated_into pointing at takes(id) for the dream- cycle hot→cold bridge. - Embedding column dim resolved at migration time from config.embedding_dimensions so non-OpenAI brains (Voyage etc) work out-of-the-box. HALFVEC where pgvector >= 0.7; falls back to VECTOR with stderr warn on older versions. Matching opclass per column type (halfvec_cosine_ops vs vector_cosine_ops). - 5 partial indexes leading on source_id so every read uses the trust boundary as part of the index, not a callback. HNSW partial index excludes expired/null rows so footprint stays proportional to active fact count. - RLS DO-block matches takes pattern (Postgres BYPASSRLS gate; PGLite no-op). - v0_31_0.ts orchestrator follows v0_28_0.ts pattern — phase A asserts schema version >= 40 + facts table presence; runner owns ledger. All 87 existing migrate.test.ts cases pass. PGLite smoke test confirms table + indexes + CHECK constraints + ON DELETE CASCADE all behave. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 chore(version): bump VERSION + package.json to 0.31.0 Phase 1 closer. CHANGELOG entry written when Phase 7 lands. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(engine): facts hot memory engine API (Phase 2) Phase 2 of v0.31 hot-memory. Adds 8 facts methods to BrainEngine implemented on both PGLite and Postgres engines: - insertFact(input, ctx) — INSERT with optional supersedeId; expires the named row in the same transaction. Per-entity advisory lock on Postgres (`pg_advisory_xact_lock(hashtextextended(source_id::text || ':' || entity_slug, 0))`) for the dedup window. PGLite is single-process so the lock is a no-op. - expireFact(id, opts) — sets expired_at + optional superseded_by. Idempotent-as-false (already-expired returns false). - listFactsByEntity / listFactsSince / listFactsBySession — list surfaces with FactListOpts filters (activeOnly, kinds, visibility, limit/offset). Every query starts WHERE source_id = $X so the trust boundary is part of the index path. - listSupersessions — audit log; activeOnly:false + expired_at IS NOT NULL + superseded_by IS NOT NULL. - findCandidateDuplicates(source_id, entity_slug, factText, k) — entity-prefiltered (mandatory), k=5 default, hard cap 20. Embedding- cosine ordering when caller supplies an embedding, recency fallback otherwise. Bounds the contradiction-classifier blast radius. - consolidateFact(id, takeId) — sets consolidated_at + consolidated_into. Never DELETE; facts stay as audit trail for the resulting take. - getFactsHealth(source_id) — per-source counters consumed by `gbrain doctor` facts_health check. Public types in engine.ts: FactKind (5-value union), FactVisibility, FactInsertStatus, FactRow, NewFact, FactListOpts, FactsHealth. PGLite + Postgres helpers: rowToFact / rowToFactPg parse the text-format pgvector embedding back into Float32Array; toPgVectorLiteral encodes for the supersede-path INSERT (postgres-js can't bind Float32Array directly to a vector column without an explicit literal cast). Smoke test confirms every method end-to-end on PGLite. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(facts): extraction code path (Phase 3) Phase 3 of v0.31 hot-memory. Five new modules under src/core/facts/ + src/core/entities/: - src/core/facts/decay.ts — pure helper. effectiveConfidence(fact, now) applies confidence × exp(-age/halflife) with per-kind halflife table (event 7d, commitment 90d, preference 90d, belief 365d, fact 365d). Returns 0 for expired or past-valid_until rows. Single source of truth consumed by recall, supersession audit, facts_health, and the MCP _meta injector (eD8 DRY). - src/core/facts/queue.ts — bounded in-memory queue. Cap 100 default, drop-oldest on overflow with counter. Per-session in-flight=1 serializes burst chat. AbortSignal threading from server SIGTERM (mirrors minion worker pattern per eD7): 5s grace for in-flight, then drop pending with counter. getFactsQueue() process-singleton; __resetFactsQueueForTests for hermetic tests. - src/core/facts/classify.ts — contradiction classifier with cosine fast-path (D13: ≥0.95 → duplicate, skip LLM) and classifier-failure fallback (D12: cosine ≥0.92 → duplicate, else INSERT). Pure cosine helper exported. JSON-strict output with 4-strategy parse fallback; refusal stop-reason maps to fallback path. Caller-provided abort signal propagated to the gateway chat call. - src/core/facts/extract.ts — Haiku turn-extractor. Reuses INJECTION_PATTERNS from src/core/think/sanitize.ts on the way IN (turn_text) AND on the way OUT (each fact). Tight system prompt with 5-kind taxonomy, 0..1 confidence scoring, entity slug or display name. Anti-loop check on isDreamGenerated (reuses v0.23.2 marker semantics). Synchronous embedOne() per fact via the gateway so classifier paths have embeddings available; AbortError re-thrown explicitly so SIGTERM during embed never writes a NULL-embedding row meant to be cancelled (eE8 distinction). - src/core/entities/resolve.ts — slug canonicalization shared by signal-detector AND facts. Resolution order: exact slug match → pg_trgm fuzzy match (similarity ≥0.4) → deterministic slugify fallback. slugify exported standalone for tests + callers that want the floor. Smoke tests confirm decay table, cosine math, slugify rules, queue drop-oldest under overflow, and shutdown grace + drop-pending semantics. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(mcp+cli): MCP ops + recall CLI + _meta + transport refactor (Phase 4) Phase 4 of v0.31 hot-memory. Three new MCP ops on the contract-first surface: - `extract_facts` (write scope, localOnly:false): extracts facts from a conversation turn via the Haiku extractor, runs the cosine fast-path dedup, INSERTs into per-source hot memory. Returns counts + fact_ids[]. Skips on is_dream_generated:true (anti-loop). - `recall` (read scope): query the per-source hot memory by entity / since / session / supersessions / grep filter. Visibility- aware: remote callers see visibility='world' rows only (takes-style ACL parity, eD21). Returns most-recent first; pagination via limit. - `forget_fact` (write scope): expireFact wrapper. Idempotent-as-error on unknown id; uses the new 'fact_not_found' ErrorCode. ErrorCode union opened (eD6 / eE7): TS forward-compat via the `(string & {})` autocomplete-friendly hack so downstream consumers (gbrain-evals etc) don't break their typecheck on every new code. Three new codes: 'rate_limited', 'extraction_failed', 'fact_not_found'. OperationContext gains source_id?:string (eD4 / eE2 — TEXT not INTEGER per schema reality). Resolved once in buildOperationContext from DispatchOpts.sourceId. Stdio MCP defaults to GBRAIN_SOURCE env or 'default'; HTTP MCP reads it from the per-token sources scope (eE3). ToolResult gains _meta?: Record<string, unknown> (eD3). Dispatcher calls a configurable metaHook AFTER op.handler succeeds, wrapped in its own try/catch so a DB blip degrades to no-_meta rather than flipping the whole tool call to error (eE4). New module src/core/facts/meta-hook.ts: - getBrainHotMemoryMeta(name, ctx) builds the _meta.brain_hot_memory payload. Cache key (source_id, session_id, hash(takesHoldersAllowList sorted)) (eD10 / eE5). 30s TTL per session. Visibility filter applies: remote → world only; local → all. Top-K=10 ranked by effective confidence (decay). Skips injection on recall/extract_facts/forget_fact themselves. bumpHotMemoryCache() invalidates per (source_id, session_id) on extraction event. D12 (eE1) accepted: serve-http.ts:801 inlined dispatch path REFACTORED to call dispatchToolCall. HTTP MCP now inherits source_id, _meta injection, error envelope unification, and OperationContext shape from the same code path stdio uses. Scope check + mcp_request_log + SSE broadcast stay in serve-http.ts (HTTP-specific concerns); the dispatcher returns ToolResult and the HTTP handler reads isError + content + _meta to fan into the audit + broadcast paths. put_page compliance backstop (D23): when a conversation-shape page is written (note/meeting/slack/email/calendar-event/source/writing) with a substantive body (>=80 chars) on a non-subagent slug AND no dream_generated:true marker, fire-and-forget enqueue an extraction job into the bounded queue. Never blocks the put_page response. Skipped reasons (no_parsed_page / subagent_namespace / dream_generated / kind:* / too_short / queue_shutdown / backstop_error) are stable strings consumed by tests. `gbrain recall` + `gbrain forget` CLI commands (src/commands/recall.ts): - recall <entity> | --since DUR | --session ID | --today (markdown with kind icons 📅🎯🤝💭📌) | --grep TEXT | --supersessions | --include-expired | --as-context (prompt-injection-ready) | --json - forget <fact-id> shorthand for expireFact Wired into src/cli.ts dispatch table next to takes / think. Smoke tests confirm: dispatch surfaces (extract_facts → ops → listFactsByEntity), forget_fact + idempotent re-call, _meta visibility filter (remote sees world only, local sees all), CLI markdown render with kind icons + age strings + decayed confidence. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(cycle): consolidate phase — facts → takes promotion (Phase 5) Phase 5 of v0.31 hot-memory. New 10th cycle phase `consolidate` between `patterns` and `embed`: - src/core/cycle.ts: * CyclePhase union extended with 'consolidate' * ALL_PHASES gets 'consolidate' between patterns and embed (graph-fresh after patterns; embed runs after so the new takes get embedded same-cycle) * NEEDS_LOCK_PHASES gets 'consolidate' (writes takes + UPDATEs facts) * CycleReport.totals gains facts_consolidated + consolidate_takes_written * runCycle dispatches the new phase via dynamic import - src/core/cycle/phases/consolidate.ts (new): * Scans (source_id, entity_slug) buckets where COUNT(unconsolidated facts) >= 3 (uses idx_facts_unconsolidated partial index) * Skips buckets where the OLDEST fact is < 24h old (gives signal time to settle before locking it into cold memory) * Greedy cosine clustering at threshold 0.85; head-element centroid keeps it deterministic + cheap. Singletons (no embedding) stay unconsolidated this cycle. * For each cluster size >= 2: picks the highest-confidence fact's text as the take claim (v0.31 deterministic; v0.32 swaps to Sonnet synthesis pass). avg confidence → take weight, earliest valid_from → take since_date, concatenated source_sessions → take.source. * Resolves entity_slug → page_id via pages.slug (per source). Skips cluster if page is missing in this source — no auto-page-creation in v0.31. * INSERT into takes(kind='fact', holder='self') with row_num = MAX(existing) + 1. * UPDATE contributing facts: consolidated_at = now() + consolidated_into = takes.id. NEVER DELETE — facts are the audit trail for the resulting take. * dryRun honored: pretends the writes happened; counters still tick so operators can preview load before the first real run. * yieldDuringPhase keepalive between buckets so the Minions worker job lock + cycle-lock TTL don't drift on long runs. Smoke test on PGLite confirms: 4 unconsolidated facts → clustered (cosine 1.0 since same vector) → 1 take row created → all 4 facts marked consolidated_into. runCycle({phases:['consolidate']}) wires through to the report totals. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: 18 facts test files (Phase 6) Phase 6 of v0.31 hot-memory: comprehensive coverage across the new substrate. 110 unit tests pass; 5 E2E test files added (skip gracefully without DATABASE_URL). Unit tests (PGLite in-memory, no DATABASE_URL): - test/facts-decay.test.ts (12 cases) — HALFLIFE_DAYS pinned per kind, effectiveConfidence math: age=0 / age=halflife (~1/e) / age=2×halflife (~1/e²) / expired returns 0 / valid_until past returns 0 / preference-vs-event slower decay / belief-vs-commitment crossover. - test/facts-queue.test.ts (10 cases) — FIFO within session, drop-oldest on overflow, per-session in-flight=1 serializes, different sessions parallelize, failed jobs counter, shutdown grace + drop_pending + external AbortController triggers shutdown. - test/facts-classify.test.ts (8 cases) — cosineSimilarity edge cases, empty candidates → independent, cheap fast-path ≥0.95 → duplicate no LLM, threshold-configurable cosine_fallback path. - test/facts-engine.test.ts (13 cases) — every BrainEngine fact method end-to-end: insertFact (insert/supersede), expireFact idempotency, list*, findCandidateDuplicates entity-prefiltered + k cap + cosine ordering, consolidateFact never DELETE, getFactsHealth shape + total_today ⊆ total_week. - test/facts-multi-tenant.test.ts (6 cases) — cross-source isolation on every list method + CASCADE delete on sources. - test/facts-visibility.test.ts (6 cases) — visibility column private/ world; remote=true filters to world-only via dispatchToolCall; remote=false sees all. - test/facts-canonicality.test.ts (10 cases) — slugify rules including NFKD diacritic strip ("Crème Brûlée" → "creme-brulee"), exact slug match, fallback to slugify when no fuzzy match. - test/facts-extract.test.ts (4 cases) — empty turn returns [], dream- generated short-circuit, graceful no-API-key return. - test/facts-backstop-gating.test.ts (5 cases) — put_page backstop: too_short, subagent_namespace, dream_generated, eligible note path, non-eligible kind:guide. - test/facts-anti-loop.test.ts (4 cases) — extractor + put_page both respect dream_generated:true marker. - test/facts-doctor-shape.test.ts (4 cases) — facts_health JSON shape pinned for downstream consumers. - test/facts-mcp-allowlist.serial.test.ts (5 cases) — extract_facts write-scope, recall read-scope, forget_fact write-scope, forget_fact fact_not_found error code, extract_facts no-API-key zero counts. - test/facts-context-injection.serial.test.ts (6 cases) — _meta injection on success, world-only filter under remote=true, anti-loop on facts ops themselves, best-effort degrade on hook error, cache-key includes allow-list hash. - test/facts-separation-pglite.test.ts (2 cases) — Garry's Separation Test as primary ship gate, plus expired hidden-by-default contract. - test/facts-recall-render.test.ts (3 cases) — --today markdown render with all 5 kind icons, --json shape with effective_confidence, --as-context emits comment-wrapped block. - test/facts-migration-dim.test.ts (4 cases) — embedding column type is HALFVEC/VECTOR (not arbitrary), dim matches gateway-configured embedding_dimensions, HNSW opclass agrees with column type, idempotent re-init. - test/cycle-consolidate.test.ts (5 cases) — below-count + below-age thresholds skip, happy path 4 facts → 1 take + all consolidated never DELETE, dryRun honored, missing page → bucket skipped. E2E tests (skip gracefully on DATABASE_URL unset; required gates by CLAUDE.md test policy): - test/e2e/facts-separation-postgres.test.ts — Postgres parity for the ship gate. - test/e2e/facts-cross-source-isolation.test.ts — cross-source ACL on PG + CASCADE delete. - test/e2e/facts-forget.test.ts — full forget_fact MCP roundtrip. - test/e2e/facts-context-injection-postgres.test.ts — _meta injection end-to-end on PG. - test/e2e/facts-recall-render.test.ts — recall --today markdown on PG. - test/e2e/serve-http-meta.test.ts — eE1 regression: HTTP MCP transport inherits _meta + sourceId + scope correctness via dispatchToolCall. Side-effect: src/core/entities/resolve.ts NFKD post-decompose strips combining marks (U+0300..U+036F) before hyphenating non-alphanumerics, so "Crème" → "creme", not "cre-me-". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 feat(operational): kill switch + doctor check + CHANGELOG + README (Phase 7) Phase 7 of v0.31 hot-memory. - src/core/facts/extract.ts: new isFactsExtractionEnabled(engine) helper reads `facts.extraction_enabled` config row. Defaults to TRUE; flip to 'false'/'0'/'no'/'off' (case-insensitive) via `gbrain config set facts.extraction_enabled false` to kill extraction across the brain without binary downgrade. - extract_facts MCP op short-circuits with zero-counts envelope + a 'skipped: extraction_disabled' field when the flag is off (clean success, not permission_denied). - put_page facts backstop respects the same flag — eligibility check now returns 'extraction_disabled' as the skipped reason. - src/commands/doctor.ts: new facts_health check (runs after queue_health, before index_audit). Probes for the facts table existence (post-v40 guard), then surfaces total_active / total_today / total_week / total_consolidated + top-3 entities for the default source. Pre-v0.31 brains report "facts table not present (pre-v0.31 brain or migration pending)". - CHANGELOG.md: full v0.31.0 entry in the GStack release-summary voice. Headline + numbers-table + what-it-ships + itemized changes + "To take advantage of v0.31" upgrade block + out-of-scope. Honest about the HALFVEC + serve-http refactor + ErrorCode-open-union complications. - README.md: cycle phase list updated 8 → 10 (consolidate + purge). New "v0.31 Hot Memory" command block under Commands with recall + forget variants, kind icons, --as-context surface for headless agents. Test gates: 28 facts unit tests pass after the kill-switch wiring + doctor check ride-along. Typecheck clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(migrate): add facts→sources FK explicitly via ALTER TABLE The inline column-level FK declaration on facts.source_id worked on PGLite but silently got dropped on Postgres in the v0.31 e2e run — the migration handler ran via postgres-js's `unsafe()` multi-statement path and the resulting facts table came back without the `facts_source_id_fkey` constraint. Same psql input run directly against the same database produced the FK; the difference was the unsafe() pipeline, not the SQL itself. Splitting the FK into a separate ALTER TABLE inside a DO block makes the constraint declaration explicit and idempotent: the named constraint either exists or it doesn't, the ALTER is a no-op on re-runs, and the failure mode is loud rather than silently leaving a CASCADE-less foreign key behind. Without this fix, deleting a source row leaves orphaned facts rows (test/e2e/facts-cross-source-isolation.test.ts CASCADE-on-sources- delete case caught it). With this fix the constraint is in place, the cascade fires, and both PG + PGLite e2e suites stay green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: update phase-count assertions for the new consolidate phase Three e2e/unit tests pinned the cycle phase count or order, all now updated to reflect v0.31's 10-phase cycle: - test/e2e/dream-cycle-eight-phase-pglite.test.ts: describe rename "8-phase cycle" → "10-phase cycle"; ALL_PHASES expectation extended to include 'consolidate' (between patterns + embed) and 'purge' (the v0.26.5 addition that was already in ALL_PHASES but missing from the test's assertion list). totals match adds the new facts_consolidated + consolidate_takes_written fields plus the pre-existing purged_sources_count + purged_pages_count that should have been added when v0.26.5 landed. - test/e2e/cycle.test.ts: dry-run full cycle now expects report.phases.length === 10 (was 9). - test/core/cycle.serial.test.ts: yieldBetweenPhases hook count + full cycle phases.length both updated 9 → 10. Comments call out the v0.31 addition lineage so the next person to add a phase sees the precedent. These are mechanical assertion bumps. The tests pass against the updated assertions on PGLite and Postgres. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(test): truncate facts table between e2e describe blocks setupDB() truncates ALL_TABLES between every describe block's beforeAll() hook. The list missed the new v0.31 facts table, so facts seeded by an earlier describe block leaked into Garry's Separation Test on Postgres — listFactsByEntity('travel') returned 2 rows instead of 1 because a prior facts-context-injection test had also seeded a 'travel' fact. Adding 'facts' to the truncate list (before 'pages' to respect FK ordering) makes every describe-block start from an empty facts table. Pinned by re-running the e2e file ordering that originally caught it (facts-recall-render → cross-source-isolation → serve-http-meta → context-injection → separation-postgres → facts-forget) — 13 pass / 0 fail after the fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 test: meta-hook cache + Postgres consolidate phase coverage Two net-new test files filling real coverage gaps the earlier sweep missed: - test/facts-meta-cache.test.ts (5 cases) — pins the eD3/eD10 cache contract that the dispatcher relies on. 30s TTL hit path, post-bump fresh-query, scoped invalidation (bump for sess-A leaves sess-B cache warm — closes the cross-source leak risk codex F5 originally surfaced on the recall payload), facts-self ops skip injection (anti-loop on recall / extract_facts / forget_fact), distinct allow-lists produce distinct cache entries. - test/e2e/cycle-consolidate-postgres.test.ts (3 cases) — Postgres parity for the dream-cycle consolidate phase. Mirrors the PGLite unit test but exercises the real postgres-engine codepaths: sql.begin transactions, advisory locks on insertFact's entity-slug dedup window, unsafe('::vector') casts on findCandidateDuplicates ordering, addTakesBatch postgres-js unnest path. Happy path (4 facts → 1 take + all consolidated_into set), age-threshold skip, dry-run no-write. All 5 unit + 3 e2e tests pass. Closes the unit-only gap on the consolidate phase (was only PGLite-tested) and pins meta-cache invariants the dispatcher depends on. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix: thread auth + sourceId, JSON-shape every error envelope Three bugs surfaced during the full e2e sweep that all trace back to my v0.31 dispatch refactor (D12/eE1) silently dropping auth threading + non-OperationError exceptions emitting plain strings: 1. **HTTP MCP transport lost ctx.auth.** Refactoring serve-http.ts to call dispatchToolCall meant auth had to come through DispatchOpts, but the field didn't exist yet. Every HTTP whoami call returned `unknown_transport` because ctx.auth was undefined. Added `auth?: AuthInfo` to DispatchOpts, plumbed it through buildOperationContext, and updated serve-http.ts:816 to pass `auth: authInfo` alongside sourceId/takesHoldersAllowList. Pinned by sources-remote-mcp e2e `whoami reports oauth transport + sources_admin scope`. 2. **Non-OperationError exceptions emitted plain strings, not JSON.** The pre-v0.31 serve-http.ts always wrapped errors in JSON envelope `{error, message}`; my dispatch refactor missed the unknown-tool + uncaught-throw paths and emitted `Error: ${msg}` text content. Every caller that did `JSON.parse(content)` (sources-remote-mcp callMcp helper at line 104) crashed with `Unexpected identifier "Error"`. Both error paths in dispatchToolCall now return JSON-shaped content matching the OperationError pattern. 3. **Files→sources FK silently lost on rewound bootstrap path.** test/e2e/postgres-bootstrap.test.ts simulates a pre-v0.21 brain by `DROP TABLE IF EXISTS sources CASCADE` which removes files_source_id_fkey while leaving files.source_id intact. The v23 migration's `ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id ... REFERENCES sources(id) ON DELETE CASCADE` is a no-op when the column exists, so the FK never came back on upgrade — and any sources-remove afterward stopped cascading to files. Added a defensive `IF NOT EXISTS files_source_id_fkey ... ALTER TABLE ADD CONSTRAINT` block inside v23's handler. Pinned by `multi-source — cascade delete covers every dependent row` after running postgres-bootstrap. Plus: src/core/preferences.ts now honors GBRAIN_HOME for `~/.gbrain/migrations/completed.jsonl`. Without this, the doctor exits-0 mechanical test inherits the developer machine's stale partial-migration ledger entries (0.21.0, 0.22.4, 0.28.0, 0.29.1 prior dev work) and surfaces them as the [FAIL] minions_migration check. GBRAIN_HOME-scoped tempdir per test now isolates this state cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 chore: scrub personal references from public artifacts Per the CLAUDE.md privacy rule on `Garry's Separation Test`, replace personally-coded references in v0.31 artifacts with neutral examples: - CHANGELOG.md v0.31 entry: rename "Garry's Separation Test" header to "The cross-session test" + drop the "topic-2659/topic-1941, 7 AM/2 PM, flying to Tokyo" narrative. - src/commands/migrations/v0_31_0.ts feature pitch: same scrub. - test/facts-separation-pglite.test.ts + test/e2e/facts-separation-postgres.test.ts: rename describe blocks; replace specific topic-NNNN session ids with session-A / session-B; replace personal sample fact with "sample event Tuesday". - src/core/facts/extract.ts extractor system prompt example slugs: people/sam-altman → people/alice-example; companies/anthropic → companies/acme. - src/core/entities/resolve.ts comment: Sam Altman → Alice Example. - All v0.31 test fixtures: people/sam → people/alice-example, Sam Altman → Alice Example, sam-the-cofounder → alice-the-cofounder. Test names referencing real-world entities replaced with neutral slugs. Pre-existing references to "Garry" elsewhere in CHANGELOG (v0.17, v0.19, v0.21+ entries) are untouched — that's a separate scope from this v0.31 ship. Plus: the truncate fix for the Bun-script-induced syntax error in test/e2e/mechanical.test.ts (cliEnv arrow function had ", 30_000)" tacked onto its closing brace by the bulk-add-timeouts script — repaired to a clean function definition). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(test): bump E2E phase-count assertions for 11-phase cycle Two E2E tests still asserted the v0.31 pre-merge 10-phase shape (consolidate inserted, but recompute_emotional_weight from v0.29 not yet absorbed). With master's v0.29 work merged in, the cycle is now 11 phases: lint → backlinks → sync → synthesize → extract → patterns → recompute_emotional_weight → consolidate → embed → orphans → purge. - test/e2e/cycle.test.ts: 10 → 11 - test/e2e/dream-cycle-eight-phase-pglite.test.ts: ALL_PHASES + dry-run order Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix(merge): close brace between v44 and v45 migration objects The v0.30.2 merge resolution stitched master's v40-v44 migrations onto HEAD's v45 (facts hot memory) migration but lost the closing `},` between v44 and v45. tsc caught it as TS1136 Property assignment expected at migrate.ts:2188. This is a one-line bracket fix; the rest of the merge resolution is correct and tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 fix: put_page cliHints + buildPlan v0.31.0 in skippedFuture Two unit-test failures surfaced after the v0.30.2 merge: 1. operations.ts: put_page had `cliHints: { name: 'put', positional: ['stdin'] }` from earlier v0.31 development. The parity test enforces that every name in `positional` is a real param. Restored master's correct shape: `{ name: 'put', positional: ['slug'], stdin: 'content' }`. 2. test/apply-migrations.test.ts: the H9 regression tests pin the exact skippedFuture list. Adding v0.31.0 to the registry meant the list grew by one. Updated both `expect(...).toEqual([...])` assertions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * v0.31 docs: clarify consolidate is 11th phase + regen llms-full.txt CHANGELOG.md narrative said "new 10th phase consolidate"; with v0.29's recompute_emotional_weight already on master, consolidate is the 11th phase (between recompute and embed). Schema migration is v45, not v40, after the merge resolution renumbered it to clear master's v40-v44. llms-full.txt regenerated to reflect the README's 11-phase dream-cycle phrasing (the build-llms test enforces commit-time parity). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
879 lines
40 KiB
TypeScript
879 lines
40 KiB
TypeScript
/**
|
|
* E2E tests for serve-http.ts OAuth 2.1 fixes (v0.26.1).
|
|
*
|
|
* Spins up a real `gbrain serve --http` against real Postgres, registers an
|
|
* OAuth client, mints tokens, and exercises the full MCP JSON-RPC pipeline
|
|
* end-to-end. Catches the three bugs fixed in v0.26.1:
|
|
*
|
|
* 1. client_credentials tokens rejected at /mcp (expiresAt string vs number)
|
|
* 2. OAuth metadata missing client_credentials grant type
|
|
* 3. Express 5 trust proxy + admin SPA wildcard
|
|
*
|
|
* Run: GBRAIN_DATABASE_URL=... bun test test/e2e/serve-http-oauth.test.ts
|
|
*/
|
|
|
|
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { hasDatabase } from './helpers.ts';
|
|
|
|
const skip = !hasDatabase();
|
|
const describeE2E = skip ? describe.skip : describe;
|
|
|
|
if (skip) {
|
|
console.log('Skipping E2E serve-http-oauth tests (DATABASE_URL not set)');
|
|
}
|
|
|
|
const PORT = 19131; // Avoid collision with production 3131
|
|
const BASE = `http://localhost:${PORT}`;
|
|
|
|
describeE2E('serve-http OAuth 2.1 E2E (v0.26.1 + v0.26.2 + v0.26.3)', () => {
|
|
let serverProcess: ReturnType<typeof import('child_process').spawn> | null = null;
|
|
let clientId: string | undefined;
|
|
let clientSecret: string | undefined;
|
|
// DCR-registered clients accumulate here so afterAll can revoke them too
|
|
// (one per test that posts to /register).
|
|
const dcrClientIds: string[] = [];
|
|
|
|
beforeAll(async () => {
|
|
const { execSync, spawn } = await import('child_process');
|
|
|
|
// Register a test OAuth client via CLI.
|
|
// env: { ...process.env } is required: bun's execSync does NOT inherit
|
|
// env mutations done via `process.env.X = ...` (only OS-level env from
|
|
// before bun started). helpers.ts loads .env.testing and sets DATABASE_URL
|
|
// via process.env mutation, which is invisible to subprocesses unless we
|
|
// explicitly re-pass process.env. Same pattern applies to every execSync
|
|
// in this file.
|
|
// v0.28.10: register with admin scope so the F7 protected-name guard
|
|
// tests can mint admin-scoped tokens that actually exercise the guard
|
|
// at operations.ts:1527. Without admin in the client's allowed scopes,
|
|
// submit_job for a protected name (`shell`, `subagent`) gets rejected
|
|
// by hasScope() in serve-http.ts BEFORE reaching the F7 guard, so the
|
|
// test was validating scope enforcement instead of the RCE protection.
|
|
// Other tests that mint specific subsets ('read', 'read write') still
|
|
// get the subset they ask for — adding admin to the client's allowed
|
|
// ceiling does not auto-grant it to every minted token.
|
|
const regOutput = execSync(
|
|
'bun run src/cli.ts auth register-client e2e-oauth-test --grant-types client_credentials --scopes "read write admin"',
|
|
{ cwd: process.cwd(), encoding: 'utf8', env: { ...process.env } }
|
|
);
|
|
const idMatch = regOutput.match(/Client ID:\s+(gbrain_cl_\S+)/);
|
|
const secretMatch = regOutput.match(/Client Secret:\s+(gbrain_cs_\S+)/);
|
|
if (!idMatch || !secretMatch) throw new Error('Failed to register test client:\n' + regOutput);
|
|
clientId = idMatch[1];
|
|
clientSecret = secretMatch[1];
|
|
|
|
// Start the HTTP server. v0.26.2 adds --enable-dcr so the /register
|
|
// endpoint is reachable for the DCR response-shape test.
|
|
serverProcess = spawn('bun', [
|
|
'run', 'src/cli.ts', 'serve', '--http',
|
|
'--port', String(PORT),
|
|
'--public-url', `http://localhost:${PORT}`,
|
|
'--enable-dcr',
|
|
], {
|
|
cwd: process.cwd(),
|
|
env: process.env,
|
|
stdio: ['ignore', 'pipe', 'pipe'],
|
|
});
|
|
|
|
// Collect stderr for debugging failures
|
|
let stderr = '';
|
|
serverProcess.stderr?.on('data', (d: Buffer) => { stderr += d.toString(); });
|
|
|
|
// Wait for server to be ready (up to 15s)
|
|
let ready = false;
|
|
for (let i = 0; i < 30; i++) {
|
|
try {
|
|
const res = await fetch(`${BASE}/health`);
|
|
if (res.ok) { ready = true; break; }
|
|
} catch {}
|
|
await new Promise(r => setTimeout(r, 500));
|
|
}
|
|
if (!ready) throw new Error('Server failed to start within 15s.\nstderr: ' + stderr.slice(-500));
|
|
}, 30_000);
|
|
|
|
afterAll(async () => {
|
|
// Kill server first so it can't issue more tokens during cleanup.
|
|
if (serverProcess) {
|
|
serverProcess.kill('SIGTERM');
|
|
await new Promise(r => setTimeout(r, 1000));
|
|
if (!serverProcess.killed) serverProcess.kill('SIGKILL');
|
|
}
|
|
// v0.26.2 cleanup contract: only revoke if registration succeeded
|
|
// (clientId guard) and surface any cleanup failure to stderr without
|
|
// throwing — a real test failure is more interesting than the cleanup
|
|
// error that follows it. Same shape applies to DCR-registered clients
|
|
// tracked in dcrClientIds.
|
|
const { execSync } = await import('child_process');
|
|
const toRevoke = [...(clientId ? [clientId] : []), ...dcrClientIds];
|
|
for (const id of toRevoke) {
|
|
try {
|
|
execSync(`bun run src/cli.ts auth revoke-client "${id}"`,
|
|
{ cwd: process.cwd(), encoding: 'utf8', env: { ...process.env } });
|
|
} catch (e: any) {
|
|
// eslint-disable-next-line no-console
|
|
console.error(`[afterAll] revoke-client cleanup failed for ${id}: ${e.message}`);
|
|
}
|
|
}
|
|
}, 30_000);
|
|
|
|
// Helper: mint a token with given scopes
|
|
async function mintToken(scope = 'read write'): Promise<{ access_token: string; expires_in: number; scope: string }> {
|
|
const res = await fetch(`${BASE}/token`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
body: `grant_type=client_credentials&client_id=${clientId}&client_secret=${clientSecret}&scope=${encodeURIComponent(scope)}`,
|
|
});
|
|
expect(res.ok).toBe(true);
|
|
return res.json() as any;
|
|
}
|
|
|
|
// Helper: call MCP JSON-RPC with a bearer token
|
|
async function mcpCall(token: string, method: string, params?: any): Promise<Response> {
|
|
return fetch(`${BASE}/mcp`, {
|
|
method: 'POST',
|
|
headers: {
|
|
'Authorization': `Bearer ${token}`,
|
|
'Content-Type': 'application/json',
|
|
'Accept': 'application/json, text/event-stream',
|
|
},
|
|
body: JSON.stringify({ jsonrpc: '2.0', id: 1, method, ...(params ? { params } : {}) }),
|
|
});
|
|
}
|
|
|
|
// =========================================================================
|
|
// Fix 1: client_credentials tokens validate at /mcp
|
|
// =========================================================================
|
|
|
|
test('mint token via client_credentials grant', async () => {
|
|
const data = await mintToken('read write');
|
|
expect(data.access_token).toMatch(/^gbrain_at_/);
|
|
expect(data.expires_in).toBe(3600);
|
|
expect(data.scope).toContain('read');
|
|
});
|
|
|
|
test('minted token is accepted at /mcp — tools/list returns tools', async () => {
|
|
const { access_token } = await mintToken('read');
|
|
const res = await mcpCall(access_token, 'tools/list');
|
|
|
|
// Before v0.26.1 fix: 401 {"error":"invalid_token","error_description":"Token has no expiration time"}
|
|
expect(res.status).not.toBe(401);
|
|
|
|
const body = await res.text();
|
|
expect(body).toContain('tools');
|
|
expect(body).toContain('search'); // search tool should be in the list
|
|
expect(body).toContain('query'); // query tool too
|
|
}, 15_000);
|
|
|
|
test('minted token works for tools/call — search executes', async () => {
|
|
const { access_token } = await mintToken('read');
|
|
const res = await mcpCall(access_token, 'tools/call', {
|
|
name: 'search',
|
|
arguments: { query: 'gbrain', limit: 1 },
|
|
});
|
|
|
|
expect(res.status).not.toBe(401);
|
|
const body = await res.text();
|
|
// Should contain search results, not an auth error
|
|
expect(body).not.toContain('invalid_token');
|
|
expect(body).toContain('result');
|
|
}, 15_000);
|
|
|
|
test('expired/invalid token is rejected at /mcp', async () => {
|
|
const res = await mcpCall('gbrain_at_totally_fake_token', 'tools/list');
|
|
// Invalid tokens should not return 200 with tool results
|
|
const body = await res.text();
|
|
expect(body).not.toContain('"tools"');
|
|
// Should be an error status (401, 403, or 500 depending on SDK error mapping)
|
|
expect(res.status).toBeGreaterThanOrEqual(400);
|
|
});
|
|
|
|
test('missing Authorization header returns 401', async () => {
|
|
const res = await fetch(`${BASE}/mcp`, {
|
|
method: 'POST',
|
|
headers: {
|
|
'Content-Type': 'application/json',
|
|
'Accept': 'application/json, text/event-stream',
|
|
},
|
|
body: JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'tools/list' }),
|
|
});
|
|
expect(res.status).toBe(401);
|
|
});
|
|
|
|
// =========================================================================
|
|
// Fix 2: OAuth metadata includes client_credentials
|
|
// =========================================================================
|
|
|
|
test('OAuth AS metadata includes all three grant types', async () => {
|
|
const res = await fetch(`${BASE}/.well-known/oauth-authorization-server`);
|
|
expect(res.ok).toBe(true);
|
|
const meta = await res.json() as any;
|
|
expect(meta.grant_types_supported).toContain('authorization_code');
|
|
expect(meta.grant_types_supported).toContain('refresh_token');
|
|
expect(meta.grant_types_supported).toContain('client_credentials');
|
|
});
|
|
|
|
test('OAuth metadata issuer matches public URL', async () => {
|
|
const res = await fetch(`${BASE}/.well-known/oauth-authorization-server`);
|
|
const meta = await res.json() as any;
|
|
expect(meta.issuer).toBe(`http://localhost:${PORT}/`);
|
|
expect(meta.token_endpoint).toContain('/token');
|
|
expect(meta.scopes_supported).toContain('read');
|
|
expect(meta.scopes_supported).toContain('write');
|
|
expect(meta.scopes_supported).toContain('admin');
|
|
});
|
|
|
|
// T2 (eng-review): scopes_supported advertises the full ALLOWED_SCOPES_LIST
|
|
// so MCP clients (Claude Desktop, ChatGPT, Perplexity) can discover the
|
|
// v0.28 sources_admin and users_admin scopes via standard discovery.
|
|
// Pre-v0.28 the list was hardcoded to ['read','write','admin'] in
|
|
// serve-http.ts:195 and this assertion would have failed.
|
|
test('OAuth metadata advertises all 5 v0.28 scopes (sources_admin + users_admin)', async () => {
|
|
const res = await fetch(`${BASE}/.well-known/oauth-authorization-server`);
|
|
const meta = await res.json() as any;
|
|
expect(meta.scopes_supported).toContain('sources_admin');
|
|
expect(meta.scopes_supported).toContain('users_admin');
|
|
expect(meta.scopes_supported).toEqual(
|
|
expect.arrayContaining(['admin', 'read', 'sources_admin', 'users_admin', 'write']),
|
|
);
|
|
});
|
|
|
|
// =========================================================================
|
|
// Fix 3: Express 5 compatibility
|
|
// =========================================================================
|
|
|
|
test('admin dashboard serves SPA index.html (not Express error)', async () => {
|
|
const res = await fetch(`${BASE}/admin/`);
|
|
const html = await res.text();
|
|
expect(html).toContain('GBrain Admin');
|
|
expect(html).not.toContain('<pre>Cannot GET');
|
|
});
|
|
|
|
test('admin sub-routes serve SPA fallback', async () => {
|
|
const res = await fetch(`${BASE}/admin/agents`);
|
|
const html = await res.text();
|
|
expect(html).toContain('GBrain Admin');
|
|
});
|
|
|
|
test('X-Forwarded-For header does not crash server', async () => {
|
|
const res = await fetch(`${BASE}/health`, {
|
|
headers: { 'X-Forwarded-For': '10.0.0.1, 172.16.0.1' },
|
|
});
|
|
expect(res.ok).toBe(true);
|
|
const data = await res.json() as any;
|
|
expect(data.status).toBe('ok');
|
|
});
|
|
|
|
// =========================================================================
|
|
// Scope enforcement
|
|
// =========================================================================
|
|
|
|
test('read-only token is rejected for write operations', async () => {
|
|
const { access_token } = await mintToken('read');
|
|
const res = await mcpCall(access_token, 'tools/call', {
|
|
name: 'put_page',
|
|
arguments: { slug: 'e2e-scope-test', content: '---\ntitle: test\n---\ntest' },
|
|
});
|
|
|
|
const body = await res.text();
|
|
// Should be rejected via scope check (403 or JSON-RPC error with scope message)
|
|
expect(res.status === 403 || body.includes('scope') || body.includes('Insufficient')).toBe(true);
|
|
}, 15_000);
|
|
|
|
test('write-scoped token can call read operations', async () => {
|
|
const { access_token } = await mintToken('read write');
|
|
const res = await mcpCall(access_token, 'tools/call', {
|
|
name: 'search',
|
|
arguments: { query: 'test', limit: 1 },
|
|
});
|
|
|
|
expect(res.status).not.toBe(401);
|
|
expect(res.status).not.toBe(403);
|
|
const body = await res.text();
|
|
// Should get a result, not an auth error
|
|
expect(body).not.toContain('invalid_token');
|
|
expect(body).not.toContain('insufficient_scope');
|
|
}, 15_000);
|
|
|
|
// =========================================================================
|
|
// Health endpoint (no auth required) — v0.28.10 made /health liveness-only;
|
|
// engine stats moved to /admin/api/full-stats behind requireAdmin so a
|
|
// saturated pool can't pin /health and trigger orchestrator restart cascades.
|
|
// =========================================================================
|
|
|
|
test('v0.28.10: /health returns liveness-only body (no engine stats)', async () => {
|
|
const res = await fetch(`${BASE}/health`);
|
|
expect(res.ok).toBe(true);
|
|
const data = await res.json() as any;
|
|
expect(data.status).toBe('ok');
|
|
expect(data.version).toBeDefined();
|
|
expect(data.engine).toBeDefined();
|
|
// Regression: pre-v0.28.10 /health spread getStats() (page_count,
|
|
// chunk_count, etc.) into the body. The whole point of the v0.28.10
|
|
// split is that /health stops touching those tables. If page_count
|
|
// ever reappears here, the heavy probe leaked back into the public
|
|
// route and the original DoS surface is back.
|
|
expect(data.page_count).toBeUndefined();
|
|
expect(data.chunk_count).toBeUndefined();
|
|
expect(data.embedded_count).toBeUndefined();
|
|
// Body shape is exactly {status, version, engine}.
|
|
expect(Object.keys(data).sort()).toEqual(['engine', 'status', 'version']);
|
|
});
|
|
|
|
test('v0.28.10: /admin/api/full-stats without admin cookie returns 401', async () => {
|
|
const res = await fetch(`${BASE}/admin/api/full-stats`);
|
|
expect(res.status).toBe(401);
|
|
const data = await res.json() as any;
|
|
expect(data.error).toBe('Admin authentication required');
|
|
});
|
|
|
|
test('v0.28.10: /admin/api/full-stats with valid admin cookie returns getStats() body', async () => {
|
|
// Same magic-link cookie dance the existing single-use test uses.
|
|
// Skip gracefully if the bootstrap token isn't extractable — the 401
|
|
// case above pins the auth gate; this test pins the happy path.
|
|
const stderrBuf = (serverProcess as any)?._stderrBuffer || '';
|
|
const tokenMatch = String(stderrBuf).match(/Admin Token[\s\S]*?([a-f0-9]{32,64})/);
|
|
if (!tokenMatch) {
|
|
console.warn('[e2e] skipped /admin/api/full-stats happy path: could not extract bootstrap token');
|
|
return;
|
|
}
|
|
const bootstrapToken = tokenMatch[1];
|
|
|
|
const issueRes = await fetch(`${BASE}/admin/api/issue-magic-link`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${bootstrapToken}` },
|
|
body: '{}',
|
|
});
|
|
expect(issueRes.ok).toBe(true);
|
|
const { url } = await issueRes.json() as any;
|
|
|
|
const click = await fetch(url, { redirect: 'manual' });
|
|
expect(click.status).toBe(302);
|
|
const setCookie = click.headers.get('set-cookie') || '';
|
|
const cookieMatch = setCookie.match(/gbrain_admin=([^;]+)/);
|
|
expect(cookieMatch).toBeTruthy();
|
|
const cookieValue = cookieMatch![1];
|
|
|
|
const statsRes = await fetch(`${BASE}/admin/api/full-stats`, {
|
|
headers: { Cookie: `gbrain_admin=${cookieValue}` },
|
|
});
|
|
expect(statsRes.ok).toBe(true);
|
|
const stats = await statsRes.json() as any;
|
|
expect(stats.status).toBe('ok');
|
|
expect(stats.version).toBeDefined();
|
|
expect(stats.engine).toBeDefined();
|
|
// The full-stats body is probeHealth's spread of getStats() — page_count
|
|
// is the canonical signal that we're hitting the heavy path here.
|
|
expect(typeof stats.page_count).toBe('number');
|
|
expect(stats.page_count).toBeGreaterThanOrEqual(0);
|
|
}, 15_000);
|
|
|
|
// =========================================================================
|
|
// Token lifecycle
|
|
// =========================================================================
|
|
|
|
test('multiple tokens can be minted and used independently', async () => {
|
|
const t1 = await mintToken('read');
|
|
const t2 = await mintToken('read write');
|
|
|
|
// Both should work
|
|
const r1 = await mcpCall(t1.access_token, 'tools/list');
|
|
const r2 = await mcpCall(t2.access_token, 'tools/list');
|
|
|
|
expect(r1.status).not.toBe(401);
|
|
expect(r2.status).not.toBe(401);
|
|
}, 15_000);
|
|
|
|
test('wrong client_secret is rejected at token endpoint', async () => {
|
|
const res = await fetch(`${BASE}/token`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
body: `grant_type=client_credentials&client_id=${clientId}&client_secret=gbrain_cs_wrong_secret&scope=read`,
|
|
});
|
|
expect(res.ok).toBe(false);
|
|
const data = await res.json() as any;
|
|
expect(data.error).toBe('invalid_grant');
|
|
});
|
|
|
|
// =========================================================================
|
|
// v0.26.2: DCR /register response shape (RFC 7591 §3.2.1 number contract)
|
|
// =========================================================================
|
|
//
|
|
// The user-visible bug v0.26.2 protects against: postgres.js with
|
|
// `prepare: false` returns BIGINT columns as strings, and an RFC-strict
|
|
// DCR client (Claude Code, Cursor) parses the /register response as JSON
|
|
// and rejects timestamps that aren't numbers. This is the HTTP-level test;
|
|
// the internal-store shape test in test/oauth.test.ts is not enough on its
|
|
// own (Codex flagged it as the wrong seam).
|
|
|
|
test('DCR /register returns numeric client_id_issued_at (RFC 7591 §3.2.1)', async () => {
|
|
const res = await fetch(`${BASE}/register`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({
|
|
client_name: 'e2e-dcr-shape',
|
|
redirect_uris: ['https://example.com/cb'],
|
|
grant_types: ['authorization_code'],
|
|
token_endpoint_auth_method: 'client_secret_basic',
|
|
scope: 'read',
|
|
}),
|
|
});
|
|
expect(res.ok).toBe(true);
|
|
const body = await res.json() as any;
|
|
|
|
// Track for cleanup before any assertion that could throw.
|
|
if (body.client_id) dcrClientIds.push(body.client_id);
|
|
|
|
// The contract: client_id_issued_at is REQUIRED to be a JSON number per
|
|
// RFC 7591. Pre-v0.26.2 with prepare:false returned this as a string
|
|
// (e.g., "1735689600") and strict clients rejected the registration.
|
|
expect(typeof body.client_id_issued_at).toBe('number');
|
|
expect(Number.isFinite(body.client_id_issued_at)).toBe(true);
|
|
expect(body.client_id_issued_at).toBeGreaterThan(0);
|
|
|
|
// client_secret_expires_at is OPTIONAL. If present, it must also be a
|
|
// number. Undefined/missing means "does not expire" per the spec.
|
|
if (body.client_secret_expires_at !== undefined) {
|
|
expect(typeof body.client_secret_expires_at).toBe('number');
|
|
expect(Number.isFinite(body.client_secret_expires_at)).toBe(true);
|
|
}
|
|
}, 15_000);
|
|
|
|
// =========================================================================
|
|
// v0.26.2: revoke-client CLI subprocess test
|
|
// =========================================================================
|
|
//
|
|
// Validates the actual CLI router in src/commands/auth.ts, not just the
|
|
// database deletion semantics. Codex flagged that a unit test in
|
|
// test/oauth.test.ts proves DB DELETE works but does NOT prove the
|
|
// subcommand exists or routes correctly.
|
|
|
|
test('auth revoke-client (CLI) deletes client + cascades to tokens', async () => {
|
|
const { execSync } = await import('child_process');
|
|
|
|
// Step 1: register a throwaway client via CLI.
|
|
// env: { ...process.env } per the bun execSync inheritance fix above.
|
|
const regOutput = execSync(
|
|
'bun run src/cli.ts auth register-client e2e-revoke-cli --grant-types client_credentials --scopes read',
|
|
{ cwd: process.cwd(), encoding: 'utf8', env: { ...process.env } }
|
|
);
|
|
const idMatch = regOutput.match(/Client ID:\s+(gbrain_cl_\S+)/);
|
|
const secretMatch = regOutput.match(/Client Secret:\s+(gbrain_cs_\S+)/);
|
|
expect(idMatch).not.toBeNull();
|
|
expect(secretMatch).not.toBeNull();
|
|
const id = idMatch![1];
|
|
const secret = secretMatch![1];
|
|
|
|
// Step 2: mint a token through the live server.
|
|
const tokenRes = await fetch(`${BASE}/token`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
body: `grant_type=client_credentials&client_id=${id}&client_secret=${secret}&scope=read`,
|
|
});
|
|
expect(tokenRes.ok).toBe(true);
|
|
const { access_token } = await tokenRes.json() as any;
|
|
|
|
// Sanity: the freshly-minted token works at /mcp.
|
|
const before = await mcpCall(access_token, 'tools/list');
|
|
expect(before.status).not.toBe(401);
|
|
|
|
// Step 3: revoke via the CLI subprocess.
|
|
const revokeOutput = execSync(
|
|
`bun run src/cli.ts auth revoke-client "${id}"`,
|
|
{ cwd: process.cwd(), encoding: 'utf8', env: { ...process.env } }
|
|
);
|
|
// The handler prints the human confirmation lines. No exit code != 0
|
|
// here since execSync would throw.
|
|
expect(revokeOutput).toMatch(/OAuth client revoked/);
|
|
expect(revokeOutput).toMatch(/cascade/i);
|
|
|
|
// Step 4: previously-minted token must now be rejected at /mcp. Cascade
|
|
// wiped the oauth_tokens row; verifyAccessToken throws "Invalid token".
|
|
// Match the existing pattern at line 156: SDK error mapping varies
|
|
// (401/403/500), so we assert non-success status + non-success body
|
|
// rather than a single status code.
|
|
const after = await mcpCall(access_token, 'tools/list');
|
|
expect(after.status).toBeGreaterThanOrEqual(400);
|
|
const afterBody = await after.text();
|
|
expect(afterBody).not.toContain('"tools":[');
|
|
|
|
// Step 5: re-running revoke-client on the now-deleted id must exit 1.
|
|
let secondRunFailed = false;
|
|
let secondRunStderr = '';
|
|
try {
|
|
execSync(`bun run src/cli.ts auth revoke-client "${id}"`,
|
|
{ cwd: process.cwd(), encoding: 'utf8', env: { ...process.env } });
|
|
} catch (e: any) {
|
|
secondRunFailed = true;
|
|
secondRunStderr = (e.stderr || '').toString() + (e.stdout || '').toString();
|
|
}
|
|
expect(secondRunFailed).toBe(true);
|
|
expect(secondRunStderr).toMatch(/No client found/);
|
|
}, 30_000);
|
|
|
|
// =========================================================================
|
|
// v0.26.3: Migration v33 round-trip — pins the 5 new columns
|
|
// =========================================================================
|
|
//
|
|
// PR #586 referenced oauth_clients.{token_ttl, deleted_at} +
|
|
// mcp_request_log.{agent_name, params, error_message} without an
|
|
// accompanying migration. v33 adds them. This test pins the round-trip:
|
|
// make a /mcp call -> assert all three new mcp_request_log columns
|
|
// persisted correctly. Without v33, the INSERT silently swallows
|
|
// column-doesn't-exist errors via the existing best-effort try/catch
|
|
// and the row never appears.
|
|
|
|
test('v0.26.3: /mcp request persists agent_name + params + error_message', async () => {
|
|
const postgres = (await import('postgres')).default;
|
|
const sql = postgres(process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL || '', { prepare: false });
|
|
try {
|
|
// Wipe any prior log rows for our test client so we can assert exact counts.
|
|
await sql`DELETE FROM mcp_request_log WHERE token_name = ${clientId!}`;
|
|
|
|
// Mint a fresh write-scoped token and make a successful tools/list call.
|
|
const tokenRes = await fetch(`${BASE}/token`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
body: `grant_type=client_credentials&client_id=${clientId!}&client_secret=${clientSecret!}&scope=read`,
|
|
});
|
|
expect(tokenRes.ok).toBe(true);
|
|
const { access_token } = await tokenRes.json() as any;
|
|
const okRes = await mcpCall(access_token, 'tools/list');
|
|
expect(okRes.status).not.toBe(401);
|
|
|
|
// Trigger an error path so the error_message column gets a value too.
|
|
// Request a tool that doesn't exist — v0.28.10 logs unknown-op attempts
|
|
// with operation = the attempted name and error_message starting with
|
|
// 'unknown_operation:'.
|
|
await mcpCall(access_token, 'tools/call', { name: 'this_tool_does_not_exist', arguments: {} });
|
|
|
|
// Allow async best-effort INSERT to flush.
|
|
await new Promise(r => setTimeout(r, 250));
|
|
|
|
const rows = await sql`
|
|
SELECT operation, status, agent_name, params, error_message
|
|
FROM mcp_request_log
|
|
WHERE token_name = ${clientId!}
|
|
ORDER BY created_at ASC
|
|
` as unknown as Array<Record<string, unknown>>;
|
|
|
|
expect(rows.length).toBeGreaterThanOrEqual(2);
|
|
|
|
// Agent name resolved from oauth_clients.client_name (the JOIN in
|
|
// verifyAccessToken or the agent_name backfill path).
|
|
for (const row of rows) {
|
|
expect(row.agent_name).toBe('e2e-oauth-test');
|
|
}
|
|
|
|
// v0.28.10: tools/list logs as operation='tools/list' (the JSON-RPC
|
|
// method name). tools/call success/error logs as operation=<inner
|
|
// tool name> (the convention preserved from pre-v0.28.10 dispatch
|
|
// logging — agents querying mcp_request_log filter by tool name, not
|
|
// by JSON-RPC method).
|
|
const listRow = rows.find(r => r.operation === 'tools/list');
|
|
expect(listRow).toBeDefined();
|
|
expect(listRow!.status).toBe('success');
|
|
|
|
// The unknown-op call shows up with operation = the attempted name.
|
|
const callRow = rows.find(r => r.operation === 'this_tool_does_not_exist');
|
|
expect(callRow).toBeDefined();
|
|
expect(callRow!.status).toBe('error');
|
|
|
|
// error_message populated on the failed call.
|
|
const errorRow = rows.find(r => r.status === 'error');
|
|
expect(errorRow).toBeDefined();
|
|
expect(errorRow!.error_message).toBeTruthy();
|
|
expect(typeof errorRow!.error_message).toBe('string');
|
|
expect(errorRow!.error_message as string).toContain('unknown_operation');
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}, 30_000);
|
|
|
|
// =========================================================================
|
|
// v0.26.3: request-log filter injection probe
|
|
// =========================================================================
|
|
//
|
|
// Pre-fix: /admin/api/requests built WHERE clauses via sql.unsafe() with
|
|
// single-quote escape (`token_name = '${agent.replace(/'/g, "''")}'`).
|
|
// Post-fix: postgres.js tagged-template fragments. This probe sends a
|
|
// payload that, under broken escaping, would short-circuit to TRUE and
|
|
// return all rows. Under correct parameterization, it matches no rows.
|
|
|
|
test("v0.26.3: request-log filter rejects injection attempt (' OR 1=1)", async () => {
|
|
// Use a plain admin session via /admin/login + bootstrap token. This
|
|
// test covers the unauthenticated SQL-injection vector via the agent
|
|
// query parameter — even though the endpoint is admin-gated, defense-
|
|
// in-depth on parameterization matters.
|
|
//
|
|
// Extract the admin bootstrap token from the spawned server's stderr.
|
|
const probe = "alice'%20OR%201%3D1";
|
|
|
|
// We don't have a clean way to pull the admin token from the spawned
|
|
// process here (commit 16 deleted the regex extraction). The injection
|
|
// probe still works WITHOUT auth — the endpoint requires it via 401.
|
|
// We assert that the 401 lands BEFORE any SQL gets built, so we don't
|
|
// crash the server with malformed SQL on the way to the auth check.
|
|
const res = await fetch(`${BASE}/admin/api/requests?agent=${probe}`, {
|
|
method: 'GET',
|
|
});
|
|
// No admin cookie — must hit 401, not 500 (no SQL crash).
|
|
expect(res.status).toBe(401);
|
|
|
|
// Server is still alive (didn't crash on the malformed input).
|
|
const health = await fetch(`${BASE}/health`);
|
|
expect(health.ok).toBe(true);
|
|
});
|
|
|
|
// =========================================================================
|
|
// v0.26.3: per-client TTL flow
|
|
// =========================================================================
|
|
//
|
|
// PR #586 added `tokenTtl` per OAuth client. exchangeClientCredentials
|
|
// reads oauth_clients.token_ttl (per-client override) and falls back to
|
|
// the server default. This test registers a client with a custom TTL,
|
|
// mints a token, and asserts the response's expires_in matches.
|
|
|
|
test('v0.26.3: per-client token_ttl is honored on token mint', async () => {
|
|
const postgres = (await import('postgres')).default;
|
|
const sql = postgres(process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL || '', { prepare: false });
|
|
try {
|
|
// Register a client + set a custom token_ttl (24 hours = 86400 seconds).
|
|
const { execSync } = await import('child_process');
|
|
const regOutput = execSync(
|
|
'bun run src/cli.ts auth register-client e2e-test-ttl --grant-types client_credentials --scopes read',
|
|
{ cwd: process.cwd(), encoding: 'utf8', env: { ...process.env } }
|
|
);
|
|
const idMatch = regOutput.match(/Client ID:\s+(gbrain_cl_\S+)/);
|
|
const secretMatch = regOutput.match(/Client Secret:\s+(gbrain_cs_\S+)/);
|
|
expect(idMatch).not.toBeNull();
|
|
expect(secretMatch).not.toBeNull();
|
|
const id = idMatch![1];
|
|
const secret = secretMatch![1];
|
|
dcrClientIds.push(id); // afterAll cleanup
|
|
|
|
// Set a 24-hour TTL.
|
|
await sql`UPDATE oauth_clients SET token_ttl = 86400 WHERE client_id = ${id}`;
|
|
|
|
// Mint a token. Response must include expires_in close to 86400.
|
|
const tokenRes = await fetch(`${BASE}/token`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
body: `grant_type=client_credentials&client_id=${id}&client_secret=${secret}&scope=read`,
|
|
});
|
|
expect(tokenRes.ok).toBe(true);
|
|
const body = await tokenRes.json() as any;
|
|
expect(body.expires_in).toBe(86400);
|
|
|
|
// Update TTL to a different value mid-test, mint again, assert new value.
|
|
await sql`UPDATE oauth_clients SET token_ttl = 7200 WHERE client_id = ${id}`;
|
|
const tokenRes2 = await fetch(`${BASE}/token`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
body: `grant_type=client_credentials&client_id=${id}&client_secret=${secret}&scope=read`,
|
|
});
|
|
expect(tokenRes2.ok).toBe(true);
|
|
const body2 = await tokenRes2.json() as any;
|
|
expect(body2.expires_in).toBe(7200);
|
|
|
|
// NULL token_ttl falls back to server default (3600 = 1 hour).
|
|
await sql`UPDATE oauth_clients SET token_ttl = NULL WHERE client_id = ${id}`;
|
|
const tokenRes3 = await fetch(`${BASE}/token`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
body: `grant_type=client_credentials&client_id=${id}&client_secret=${secret}&scope=read`,
|
|
});
|
|
expect(tokenRes3.ok).toBe(true);
|
|
const body3 = await tokenRes3.json() as any;
|
|
expect(body3.expires_in).toBe(3600);
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}, 30_000);
|
|
|
|
// =========================================================================
|
|
// v0.26.3: magic-link single-use + 401 styled error page
|
|
// =========================================================================
|
|
//
|
|
// D11=C: /admin/auth/:nonce is single-use. First click consumes the nonce,
|
|
// second click fails with the styled 401 page. No bootstrap token in URL.
|
|
//
|
|
// Also covers F6.5: server returns Content-Type: text/html on the 401
|
|
// path (Express auto-sets this for HTML body) so browsers render the
|
|
// styled page instead of treating it as plain text.
|
|
|
|
test('v0.26.3: invalid magic-link nonce returns styled 401 HTML page', async () => {
|
|
const res = await fetch(`${BASE}/admin/auth/garbage_nonce_that_does_not_exist`, { redirect: 'manual' });
|
|
expect(res.status).toBe(401);
|
|
const ct = res.headers.get('content-type') || '';
|
|
expect(ct).toContain('text/html');
|
|
const body = await res.text();
|
|
expect(body).toContain('expired');
|
|
expect(body).toContain('GBrain');
|
|
});
|
|
|
|
test('v0.26.3: magic-link nonce is single-use (second click fails)', async () => {
|
|
// Get a real bootstrap token from the spawned server's environment.
|
|
// The server prints it to stderr at startup but commit 16 removed our
|
|
// regex extractor. Use the issue-magic-link endpoint directly with the
|
|
// bootstrap token from process env — except that env var doesn't exist
|
|
// in the test fixture. The portable approach: extract from the server
|
|
// process's stderr.
|
|
|
|
// Pull the bootstrap token from server stderr by re-reading the
|
|
// spawn handle. The spawn already started so stderr has flushed.
|
|
// Skip if we can't extract — the test is best-effort coverage of the
|
|
// single-use semantic; the styled-401 test above covers the negative path.
|
|
const stderrBuf = (serverProcess as any)?._stderrBuffer || '';
|
|
const tokenMatch = String(stderrBuf).match(/Admin Token[\s\S]*?([a-f0-9]{32,64})/);
|
|
if (!tokenMatch) {
|
|
// No way to get the bootstrap token in this test fixture — skip gracefully.
|
|
// The unit-level coverage for nonce single-use is in oauth.test.ts and
|
|
// the styled-401 test above pins the consumed-nonce path.
|
|
console.warn('[e2e] skipped magic-link single-use: could not extract bootstrap token');
|
|
return;
|
|
}
|
|
const bootstrapToken = tokenMatch[1];
|
|
|
|
// Mint a one-time nonce.
|
|
const issueRes = await fetch(`${BASE}/admin/api/issue-magic-link`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${bootstrapToken}` },
|
|
body: '{}',
|
|
});
|
|
expect(issueRes.ok).toBe(true);
|
|
const { url } = await issueRes.json() as any;
|
|
expect(url).toContain('/admin/auth/');
|
|
|
|
// First click — should set cookie + redirect (302 to /admin/).
|
|
const first = await fetch(url, { redirect: 'manual' });
|
|
expect(first.status).toBe(302);
|
|
const cookie = first.headers.get('set-cookie') || '';
|
|
expect(cookie).toContain('gbrain_admin=');
|
|
|
|
// Second click on the same URL — must fail (single-use consumed).
|
|
const second = await fetch(url, { redirect: 'manual' });
|
|
expect(second.status).toBe(401);
|
|
const secondBody = await second.text();
|
|
expect(secondBody).toContain('GBrain');
|
|
}, 15_000);
|
|
|
|
// =========================================================================
|
|
// v0.26.3: agent_name backfill across oauth_clients + access_tokens
|
|
// =========================================================================
|
|
//
|
|
// Migration v33 backfills mcp_request_log.agent_name using
|
|
// COALESCE(oauth_clients.client_name, access_tokens.name, token_name)
|
|
// This test confirms the agent_name is correctly resolved across both
|
|
// auth lanes (oauth client + legacy api key).
|
|
|
|
test('v0.26.3: agent_name resolves correctly for OAuth + legacy paths', async () => {
|
|
const postgres = (await import('postgres')).default;
|
|
const sql = postgres(process.env.GBRAIN_DATABASE_URL || process.env.DATABASE_URL || '', { prepare: false });
|
|
try {
|
|
// Make an OAuth-authenticated request — agent_name should be the OAuth client_name.
|
|
const tokenRes = await fetch(`${BASE}/token`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
|
|
body: `grant_type=client_credentials&client_id=${clientId!}&client_secret=${clientSecret!}&scope=read`,
|
|
});
|
|
const { access_token } = await tokenRes.json() as any;
|
|
await mcpCall(access_token, 'tools/list');
|
|
await new Promise(r => setTimeout(r, 250));
|
|
|
|
const oauthRows = await sql`
|
|
SELECT agent_name FROM mcp_request_log
|
|
WHERE token_name = ${clientId!}
|
|
ORDER BY created_at DESC LIMIT 1
|
|
` as unknown as Array<{ agent_name: string }>;
|
|
expect(oauthRows.length).toBeGreaterThan(0);
|
|
expect(oauthRows[0].agent_name).toBe('e2e-oauth-test');
|
|
} finally {
|
|
await sql.end();
|
|
}
|
|
}, 15_000);
|
|
|
|
// =========================================================================
|
|
// v0.26.3: register-client missing-name returns 400
|
|
// =========================================================================
|
|
//
|
|
// Defense-in-depth: the admin register-client endpoint must validate
|
|
// input. Pre-fix would have crashed or returned 500.
|
|
|
|
test('v0.26.3: /admin/api/register-client without name returns 400', async () => {
|
|
// Endpoint is admin-cookie-gated. Without auth we should get 401, not 500.
|
|
// Without a name in the body (with auth) we should get 400. We test the
|
|
// 401 path here as a basic input-validation smoke; the 400 path requires
|
|
// an admin session which the test fixture doesn't easily produce.
|
|
const res = await fetch(`${BASE}/admin/api/register-client`, {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: '{}',
|
|
});
|
|
expect(res.status).toBe(401);
|
|
});
|
|
|
|
// =========================================================================
|
|
// F7 + F7b: HTTP MCP shell-job RCE regression
|
|
// =========================================================================
|
|
//
|
|
// The headline trust-boundary fix. Pre-fix, the inlined OperationContext
|
|
// literal in serve-http.ts forgot to set `remote: true`, which meant
|
|
// operations.ts:1391's protected-job-name guard (`if (ctx.remote && ...)`)
|
|
// saw a falsy undefined and skipped. An HTTP MCP caller with a write-scoped
|
|
// token could then submit `{name: "shell", params: {cmd: "id"}}` over /mcp
|
|
// and execute arbitrary commands on the gbrain host.
|
|
//
|
|
// The fix is two-layered:
|
|
// 1) F7 — serve-http.ts sets `remote: true` explicitly.
|
|
// 2) F7b — operations.ts:1391 + :1400 use `ctx.remote !== false` /
|
|
// `ctx.remote === false` so undefined fails closed even if a
|
|
// future transport bypasses the type via cast.
|
|
//
|
|
// Together they close the path even if either layer regresses alone.
|
|
|
|
test('F7: HTTP MCP cannot submit shell jobs (RCE regression)', async () => {
|
|
// v0.28.10: must mint admin scope. submit_job's required scope is
|
|
// 'admin'; without it, hasScope() rejects with insufficient_scope BEFORE
|
|
// the F7 protected-name guard at operations.ts:1527 fires. To validate
|
|
// the actual RCE protection (the protected-name guard), the token has
|
|
// to clear the scope check first.
|
|
const { access_token } = await mintToken('admin');
|
|
const res = await mcpCall(access_token, 'tools/call', {
|
|
name: 'submit_job',
|
|
arguments: { name: 'shell', data: { cmd: 'id' } },
|
|
});
|
|
|
|
const body = await res.text();
|
|
// Must reject. Either HTTP 4xx, or a JSON-RPC envelope carrying an
|
|
// OperationError with code permission_denied. The exact wire shape
|
|
// depends on SDK error mapping — assert the negative invariant
|
|
// (no command executed) and the positive invariant (rejection signal).
|
|
const rejected =
|
|
res.status >= 400 ||
|
|
body.includes('permission_denied') ||
|
|
body.includes('cannot be submitted over MCP');
|
|
expect(rejected).toBe(true);
|
|
|
|
// Negative: response must NOT contain a successful submit_job result
|
|
// (which would surface a job_id field). If a job ID came back the
|
|
// privesc landed.
|
|
expect(body).not.toMatch(/"job_id"\s*:\s*"?\d+/);
|
|
}, 15_000);
|
|
|
|
test('F7: HTTP MCP cannot submit subagent jobs (protected name)', async () => {
|
|
// Same admin-scope requirement as the shell-job sibling test above.
|
|
const { access_token } = await mintToken('admin');
|
|
const res = await mcpCall(access_token, 'tools/call', {
|
|
name: 'submit_job',
|
|
arguments: { name: 'subagent', data: { prompt: 'noop' } },
|
|
});
|
|
const body = await res.text();
|
|
const rejected =
|
|
res.status >= 400 ||
|
|
body.includes('permission_denied') ||
|
|
body.includes('cannot be submitted over MCP');
|
|
expect(rejected).toBe(true);
|
|
expect(body).not.toMatch(/"job_id"\s*:\s*"?\d+/);
|
|
}, 15_000);
|
|
});
|