Commit Graph
3 Commits
Author SHA1 Message Date
Aurora CapitalandGitHub 2c96787867 test(e2e): harden suite — kill flakes, no-op assertions, cross-test coupling (#1704)
* test(e2e): drop flaky wall-clock bounds in minions-resilience

The runaway-dead-letter and cascade-kill tests asserted tight real-clock
upper bounds (<2000ms, <3000ms) on top of already-complete terminal-state
checks. Those bounds carry no correctness signal — the dead/cancelled status
and abortedChildren==10 assertions fully prove behavior — and flake on loaded
CI runners where the stall/timeout sweep cadence varies. Removed both bounds,
kept the diagnostic values, de-promised the test titles.

* test(e2e): kill order-dependence + no-op assertions in mechanical

- traverse_graph: self-contained (re-adds its own idempotent link) and asserts
  the linked company is reachable, instead of depending on a prior it() and
  only checking array shape.
- file_list-without-slug: seeds its own >100 rows instead of relying on the
  previous test's 150 surviving in the DB; asserts the cap is exercised.
- precision@5: add a loose floor (every known-item query surfaces >=1 truth doc
  in top-5) so a 0% retrieval regression no longer passes silently.
- get_health: assert value bounds (page_count==16, embed_coverage 0..1) not just
  typeof; get_chunks: assert non-empty text, numeric non-decreasing chunk_index,
  and that the page name appears, instead of toBeTruthy on chunks[0].

* test(e2e): strengthen graph/search quality assertions + close coverage gaps

graph-quality: truncate+reseed 'config' in truncateAll (kills config leak where a
setConfig test throwing before its finally bleeds into later tests); replace
toBeGreaterThan(0) link/timeline floors with fixture-derived minimums; assert exact
attendee slugs are 'attended' instead of a vacuous .every; pin autoLinks.created to
the provable 2 (Alice+Acme); add direction out/both + depth:2 multi-hop and a
cycle-safety (A->B->A terminates) test.
search-quality: fix the vacuous detail=low vector test; assert pedro returns >=2
chunks; assert detail=high includes the timeline chunk; add empty-query and
zero-vector no-throw edge tests.

* test(e2e): self-contain multi-source sync test + assert ledger cascade

Break the sequential dependency where 'performSync no sourceId' relied on a prior
test writing sync.repo_path — it now sets its own config. Add the missing
file_migration_ledger COUNT(*)==0 cascade assertion. Tighten the source_id default
check from toContain('default') to exact "'default'::text".

* test(e2e): make migration-flow HOME/PATH swap throw-safe

The suite repoints process.env.HOME/PATH to a temp dir and only restored them in
afterAll, so a mid-test throw left HOME dead for the rest of the bun process and
silently broke sibling suites. Wrap each test body in try/finally restore + a
defensive restore at the top of beforeEach.

* test(e2e): loud-skip jsonb-roundtrip + doctor-progress

Both skipped silently with no DATABASE_URL, giving zero signal the regression guard
never ran. Add the console.log skip line matching the sibling e2e files.

* test(e2e): robust check-update contract + find_orphans tool coverage

upgrade: the 'no-releases' test hard-asserted update_available===false, which flips
to failing the moment the repo has a real release. Assert the JSON contract shape
(boolean update_available, current_version===VERSION, typed optional fields) instead.
mcp: add find_orphans to the asserted generated tool names.
2026-07-22 18:46:44 -07:00
89ae720959 v0.31.0 feat: hot memory — facts hook + recall CLI + MCP _meta + consolidate phase (#785)
* v0.31 feat(migrate): facts hot memory schema (migration v40)

Phase 1 of v0.31 hot-memory.

- New facts table with source_id (TEXT FK to sources, per-source isolation),
  kind CHECK (event/preference/commitment/belief/fact), visibility CHECK
  (private/world for takes-style ACL parity), valid_from/valid_until/
  expired_at/superseded_by for temporal + supersession audit, and
  consolidated_at/consolidated_into pointing at takes(id) for the dream-
  cycle hot→cold bridge.
- Embedding column dim resolved at migration time from
  config.embedding_dimensions so non-OpenAI brains (Voyage etc) work
  out-of-the-box. HALFVEC where pgvector >= 0.7; falls back to VECTOR
  with stderr warn on older versions. Matching opclass per column type
  (halfvec_cosine_ops vs vector_cosine_ops).
- 5 partial indexes leading on source_id so every read uses the trust
  boundary as part of the index, not a callback. HNSW partial index
  excludes expired/null rows so footprint stays proportional to active
  fact count.
- RLS DO-block matches takes pattern (Postgres BYPASSRLS gate; PGLite
  no-op).
- v0_31_0.ts orchestrator follows v0_28_0.ts pattern — phase A asserts
  schema version >= 40 + facts table presence; runner owns ledger.

All 87 existing migrate.test.ts cases pass. PGLite smoke test confirms
table + indexes + CHECK constraints + ON DELETE CASCADE all behave.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 chore(version): bump VERSION + package.json to 0.31.0

Phase 1 closer. CHANGELOG entry written when Phase 7 lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(engine): facts hot memory engine API (Phase 2)

Phase 2 of v0.31 hot-memory.

Adds 8 facts methods to BrainEngine implemented on both PGLite and
Postgres engines:

- insertFact(input, ctx) — INSERT with optional supersedeId; expires the
  named row in the same transaction. Per-entity advisory lock on Postgres
  (`pg_advisory_xact_lock(hashtextextended(source_id::text || ':' ||
  entity_slug, 0))`) for the dedup window. PGLite is single-process so
  the lock is a no-op.
- expireFact(id, opts) — sets expired_at + optional superseded_by.
  Idempotent-as-false (already-expired returns false).
- listFactsByEntity / listFactsSince / listFactsBySession — list surfaces
  with FactListOpts filters (activeOnly, kinds, visibility, limit/offset).
  Every query starts WHERE source_id = $X so the trust boundary is part
  of the index path.
- listSupersessions — audit log; activeOnly:false + expired_at IS NOT NULL
  + superseded_by IS NOT NULL.
- findCandidateDuplicates(source_id, entity_slug, factText, k) —
  entity-prefiltered (mandatory), k=5 default, hard cap 20. Embedding-
  cosine ordering when caller supplies an embedding, recency fallback
  otherwise. Bounds the contradiction-classifier blast radius.
- consolidateFact(id, takeId) — sets consolidated_at + consolidated_into.
  Never DELETE; facts stay as audit trail for the resulting take.
- getFactsHealth(source_id) — per-source counters consumed by `gbrain
  doctor` facts_health check.

Public types in engine.ts: FactKind (5-value union), FactVisibility,
FactInsertStatus, FactRow, NewFact, FactListOpts, FactsHealth.

PGLite + Postgres helpers: rowToFact / rowToFactPg parse the
text-format pgvector embedding back into Float32Array; toPgVectorLiteral
encodes for the supersede-path INSERT (postgres-js can't bind Float32Array
directly to a vector column without an explicit literal cast).

Smoke test confirms every method end-to-end on PGLite. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(facts): extraction code path (Phase 3)

Phase 3 of v0.31 hot-memory.

Five new modules under src/core/facts/ + src/core/entities/:

- src/core/facts/decay.ts — pure helper. effectiveConfidence(fact, now)
  applies confidence × exp(-age/halflife) with per-kind halflife table
  (event 7d, commitment 90d, preference 90d, belief 365d, fact 365d).
  Returns 0 for expired or past-valid_until rows. Single source of truth
  consumed by recall, supersession audit, facts_health, and the MCP _meta
  injector (eD8 DRY).

- src/core/facts/queue.ts — bounded in-memory queue. Cap 100 default,
  drop-oldest on overflow with counter. Per-session in-flight=1 serializes
  burst chat. AbortSignal threading from server SIGTERM (mirrors minion
  worker pattern per eD7): 5s grace for in-flight, then drop pending with
  counter. getFactsQueue() process-singleton; __resetFactsQueueForTests
  for hermetic tests.

- src/core/facts/classify.ts — contradiction classifier with cosine
  fast-path (D13: ≥0.95 → duplicate, skip LLM) and classifier-failure
  fallback (D12: cosine ≥0.92 → duplicate, else INSERT). Pure cosine
  helper exported. JSON-strict output with 4-strategy parse fallback;
  refusal stop-reason maps to fallback path. Caller-provided abort
  signal propagated to the gateway chat call.

- src/core/facts/extract.ts — Haiku turn-extractor. Reuses
  INJECTION_PATTERNS from src/core/think/sanitize.ts on the way IN
  (turn_text) AND on the way OUT (each fact). Tight system prompt with
  5-kind taxonomy, 0..1 confidence scoring, entity slug or display name.
  Anti-loop check on isDreamGenerated (reuses v0.23.2 marker semantics).
  Synchronous embedOne() per fact via the gateway so classifier paths
  have embeddings available; AbortError re-thrown explicitly so SIGTERM
  during embed never writes a NULL-embedding row meant to be cancelled
  (eE8 distinction).

- src/core/entities/resolve.ts — slug canonicalization shared by
  signal-detector AND facts. Resolution order: exact slug match →
  pg_trgm fuzzy match (similarity ≥0.4) → deterministic slugify
  fallback. slugify exported standalone for tests + callers that want
  the floor.

Smoke tests confirm decay table, cosine math, slugify rules, queue
drop-oldest under overflow, and shutdown grace + drop-pending semantics.
Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(mcp+cli): MCP ops + recall CLI + _meta + transport refactor (Phase 4)

Phase 4 of v0.31 hot-memory.

Three new MCP ops on the contract-first surface:

- `extract_facts` (write scope, localOnly:false): extracts facts from a
  conversation turn via the Haiku extractor, runs the cosine fast-path
  dedup, INSERTs into per-source hot memory. Returns counts +
  fact_ids[]. Skips on is_dream_generated:true (anti-loop).
- `recall` (read scope): query the per-source hot memory by
  entity / since / session / supersessions / grep filter. Visibility-
  aware: remote callers see visibility='world' rows only (takes-style
  ACL parity, eD21). Returns most-recent first; pagination via limit.
- `forget_fact` (write scope): expireFact wrapper. Idempotent-as-error
  on unknown id; uses the new 'fact_not_found' ErrorCode.

ErrorCode union opened (eD6 / eE7): TS forward-compat via the
`(string & {})` autocomplete-friendly hack so downstream consumers
(gbrain-evals etc) don't break their typecheck on every new code.
Three new codes: 'rate_limited', 'extraction_failed', 'fact_not_found'.

OperationContext gains source_id?:string (eD4 / eE2 — TEXT not INTEGER
per schema reality). Resolved once in buildOperationContext from
DispatchOpts.sourceId. Stdio MCP defaults to GBRAIN_SOURCE env or
'default'; HTTP MCP reads it from the per-token sources scope (eE3).

ToolResult gains _meta?: Record<string, unknown> (eD3). Dispatcher
calls a configurable metaHook AFTER op.handler succeeds, wrapped in
its own try/catch so a DB blip degrades to no-_meta rather than
flipping the whole tool call to error (eE4).

New module src/core/facts/meta-hook.ts:
- getBrainHotMemoryMeta(name, ctx) builds the _meta.brain_hot_memory
  payload. Cache key (source_id, session_id, hash(takesHoldersAllowList
  sorted)) (eD10 / eE5). 30s TTL per session. Visibility filter applies:
  remote → world only; local → all. Top-K=10 ranked by effective
  confidence (decay). Skips injection on recall/extract_facts/forget_fact
  themselves. bumpHotMemoryCache() invalidates per (source_id,
  session_id) on extraction event.

D12 (eE1) accepted: serve-http.ts:801 inlined dispatch path REFACTORED
to call dispatchToolCall. HTTP MCP now inherits source_id, _meta
injection, error envelope unification, and OperationContext shape from
the same code path stdio uses. Scope check + mcp_request_log + SSE
broadcast stay in serve-http.ts (HTTP-specific concerns); the dispatcher
returns ToolResult and the HTTP handler reads isError + content + _meta
to fan into the audit + broadcast paths.

put_page compliance backstop (D23): when a conversation-shape page is
written (note/meeting/slack/email/calendar-event/source/writing) with
a substantive body (>=80 chars) on a non-subagent slug AND no
dream_generated:true marker, fire-and-forget enqueue an extraction job
into the bounded queue. Never blocks the put_page response. Skipped
reasons (no_parsed_page / subagent_namespace / dream_generated /
kind:* / too_short / queue_shutdown / backstop_error) are stable
strings consumed by tests.

`gbrain recall` + `gbrain forget` CLI commands (src/commands/recall.ts):
- recall <entity> | --since DUR | --session ID | --today (markdown
  with kind icons 📅🎯🤝💭📌) | --grep TEXT | --supersessions |
  --include-expired | --as-context (prompt-injection-ready) | --json
- forget <fact-id> shorthand for expireFact

Wired into src/cli.ts dispatch table next to takes / think.

Smoke tests confirm: dispatch surfaces (extract_facts → ops →
listFactsByEntity), forget_fact + idempotent re-call, _meta visibility
filter (remote sees world only, local sees all), CLI markdown render
with kind icons + age strings + decayed confidence.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(cycle): consolidate phase — facts → takes promotion (Phase 5)

Phase 5 of v0.31 hot-memory.

New 10th cycle phase `consolidate` between `patterns` and `embed`:

- src/core/cycle.ts:
  * CyclePhase union extended with 'consolidate'
  * ALL_PHASES gets 'consolidate' between patterns and embed (graph-fresh
    after patterns; embed runs after so the new takes get embedded
    same-cycle)
  * NEEDS_LOCK_PHASES gets 'consolidate' (writes takes + UPDATEs facts)
  * CycleReport.totals gains facts_consolidated + consolidate_takes_written
  * runCycle dispatches the new phase via dynamic import

- src/core/cycle/phases/consolidate.ts (new):
  * Scans (source_id, entity_slug) buckets where COUNT(unconsolidated
    facts) >= 3 (uses idx_facts_unconsolidated partial index)
  * Skips buckets where the OLDEST fact is < 24h old (gives signal time
    to settle before locking it into cold memory)
  * Greedy cosine clustering at threshold 0.85; head-element centroid
    keeps it deterministic + cheap. Singletons (no embedding) stay
    unconsolidated this cycle.
  * For each cluster size >= 2: picks the highest-confidence fact's text
    as the take claim (v0.31 deterministic; v0.32 swaps to Sonnet
    synthesis pass). avg confidence → take weight, earliest valid_from →
    take since_date, concatenated source_sessions → take.source.
  * Resolves entity_slug → page_id via pages.slug (per source). Skips
    cluster if page is missing in this source — no auto-page-creation
    in v0.31.
  * INSERT into takes(kind='fact', holder='self') with row_num =
    MAX(existing) + 1.
  * UPDATE contributing facts: consolidated_at = now() +
    consolidated_into = takes.id. NEVER DELETE — facts are the audit
    trail for the resulting take.
  * dryRun honored: pretends the writes happened; counters still tick
    so operators can preview load before the first real run.
  * yieldDuringPhase keepalive between buckets so the Minions worker
    job lock + cycle-lock TTL don't drift on long runs.

Smoke test on PGLite confirms: 4 unconsolidated facts → clustered
(cosine 1.0 since same vector) → 1 take row created → all 4 facts
marked consolidated_into. runCycle({phases:['consolidate']}) wires
through to the report totals. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: 18 facts test files (Phase 6)

Phase 6 of v0.31 hot-memory: comprehensive coverage across the new
substrate. 110 unit tests pass; 5 E2E test files added (skip gracefully
without DATABASE_URL).

Unit tests (PGLite in-memory, no DATABASE_URL):
- test/facts-decay.test.ts (12 cases) — HALFLIFE_DAYS pinned per kind,
  effectiveConfidence math: age=0 / age=halflife (~1/e) / age=2×halflife
  (~1/e²) / expired returns 0 / valid_until past returns 0 /
  preference-vs-event slower decay / belief-vs-commitment crossover.
- test/facts-queue.test.ts (10 cases) — FIFO within session, drop-oldest
  on overflow, per-session in-flight=1 serializes, different sessions
  parallelize, failed jobs counter, shutdown grace + drop_pending +
  external AbortController triggers shutdown.
- test/facts-classify.test.ts (8 cases) — cosineSimilarity edge cases,
  empty candidates → independent, cheap fast-path ≥0.95 → duplicate
  no LLM, threshold-configurable cosine_fallback path.
- test/facts-engine.test.ts (13 cases) — every BrainEngine fact method
  end-to-end: insertFact (insert/supersede), expireFact idempotency,
  list*, findCandidateDuplicates entity-prefiltered + k cap + cosine
  ordering, consolidateFact never DELETE, getFactsHealth shape +
  total_today ⊆ total_week.
- test/facts-multi-tenant.test.ts (6 cases) — cross-source isolation
  on every list method + CASCADE delete on sources.
- test/facts-visibility.test.ts (6 cases) — visibility column private/
  world; remote=true filters to world-only via dispatchToolCall;
  remote=false sees all.
- test/facts-canonicality.test.ts (10 cases) — slugify rules including
  NFKD diacritic strip ("Crème Brûlée" → "creme-brulee"), exact slug
  match, fallback to slugify when no fuzzy match.
- test/facts-extract.test.ts (4 cases) — empty turn returns [], dream-
  generated short-circuit, graceful no-API-key return.
- test/facts-backstop-gating.test.ts (5 cases) — put_page backstop:
  too_short, subagent_namespace, dream_generated, eligible note path,
  non-eligible kind:guide.
- test/facts-anti-loop.test.ts (4 cases) — extractor + put_page both
  respect dream_generated:true marker.
- test/facts-doctor-shape.test.ts (4 cases) — facts_health JSON shape
  pinned for downstream consumers.
- test/facts-mcp-allowlist.serial.test.ts (5 cases) — extract_facts
  write-scope, recall read-scope, forget_fact write-scope, forget_fact
  fact_not_found error code, extract_facts no-API-key zero counts.
- test/facts-context-injection.serial.test.ts (6 cases) — _meta
  injection on success, world-only filter under remote=true, anti-loop
  on facts ops themselves, best-effort degrade on hook error,
  cache-key includes allow-list hash.
- test/facts-separation-pglite.test.ts (2 cases) — Garry's Separation
  Test as primary ship gate, plus expired hidden-by-default contract.
- test/facts-recall-render.test.ts (3 cases) — --today markdown render
  with all 5 kind icons, --json shape with effective_confidence,
  --as-context emits comment-wrapped block.
- test/facts-migration-dim.test.ts (4 cases) — embedding column type
  is HALFVEC/VECTOR (not arbitrary), dim matches gateway-configured
  embedding_dimensions, HNSW opclass agrees with column type, idempotent
  re-init.
- test/cycle-consolidate.test.ts (5 cases) — below-count + below-age
  thresholds skip, happy path 4 facts → 1 take + all consolidated never
  DELETE, dryRun honored, missing page → bucket skipped.

E2E tests (skip gracefully on DATABASE_URL unset; required gates by
CLAUDE.md test policy):
- test/e2e/facts-separation-postgres.test.ts — Postgres parity for the
  ship gate.
- test/e2e/facts-cross-source-isolation.test.ts — cross-source ACL on PG
  + CASCADE delete.
- test/e2e/facts-forget.test.ts — full forget_fact MCP roundtrip.
- test/e2e/facts-context-injection-postgres.test.ts — _meta injection
  end-to-end on PG.
- test/e2e/facts-recall-render.test.ts — recall --today markdown on PG.
- test/e2e/serve-http-meta.test.ts — eE1 regression: HTTP MCP transport
  inherits _meta + sourceId + scope correctness via dispatchToolCall.

Side-effect: src/core/entities/resolve.ts NFKD post-decompose strips
combining marks (U+0300..U+036F) before hyphenating non-alphanumerics,
so "Crème" → "creme", not "cre-me-".

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 feat(operational): kill switch + doctor check + CHANGELOG + README (Phase 7)

Phase 7 of v0.31 hot-memory.

- src/core/facts/extract.ts: new isFactsExtractionEnabled(engine) helper
  reads `facts.extraction_enabled` config row. Defaults to TRUE; flip to
  'false'/'0'/'no'/'off' (case-insensitive) via `gbrain config set
  facts.extraction_enabled false` to kill extraction across the brain
  without binary downgrade.
- extract_facts MCP op short-circuits with zero-counts envelope + a
  'skipped: extraction_disabled' field when the flag is off (clean
  success, not permission_denied).
- put_page facts backstop respects the same flag — eligibility check now
  returns 'extraction_disabled' as the skipped reason.
- src/commands/doctor.ts: new facts_health check (runs after queue_health,
  before index_audit). Probes for the facts table existence (post-v40
  guard), then surfaces total_active / total_today / total_week /
  total_consolidated + top-3 entities for the default source. Pre-v0.31
  brains report "facts table not present (pre-v0.31 brain or migration
  pending)".
- CHANGELOG.md: full v0.31.0 entry in the GStack release-summary voice.
  Headline + numbers-table + what-it-ships + itemized changes + "To take
  advantage of v0.31" upgrade block + out-of-scope. Honest about the
  HALFVEC + serve-http refactor + ErrorCode-open-union complications.
- README.md: cycle phase list updated 8 → 10 (consolidate + purge). New
  "v0.31 Hot Memory" command block under Commands with recall + forget
  variants, kind icons, --as-context surface for headless agents.

Test gates: 28 facts unit tests pass after the kill-switch wiring + doctor
check ride-along. Typecheck clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(migrate): add facts→sources FK explicitly via ALTER TABLE

The inline column-level FK declaration on facts.source_id worked on
PGLite but silently got dropped on Postgres in the v0.31 e2e run —
the migration handler ran via postgres-js's `unsafe()` multi-statement
path and the resulting facts table came back without the
`facts_source_id_fkey` constraint. Same psql input run directly
against the same database produced the FK; the difference was the
unsafe() pipeline, not the SQL itself.

Splitting the FK into a separate ALTER TABLE inside a DO block makes
the constraint declaration explicit and idempotent: the named
constraint either exists or it doesn't, the ALTER is a no-op on
re-runs, and the failure mode is loud rather than silently leaving
a CASCADE-less foreign key behind.

Without this fix, deleting a source row leaves orphaned facts rows
(test/e2e/facts-cross-source-isolation.test.ts CASCADE-on-sources-
delete case caught it). With this fix the constraint is in place,
the cascade fires, and both PG + PGLite e2e suites stay green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: update phase-count assertions for the new consolidate phase

Three e2e/unit tests pinned the cycle phase count or order, all now
updated to reflect v0.31's 10-phase cycle:

- test/e2e/dream-cycle-eight-phase-pglite.test.ts:
  describe rename "8-phase cycle" → "10-phase cycle"; ALL_PHASES
  expectation extended to include 'consolidate' (between patterns +
  embed) and 'purge' (the v0.26.5 addition that was already in
  ALL_PHASES but missing from the test's assertion list). totals
  match adds the new facts_consolidated + consolidate_takes_written
  fields plus the pre-existing purged_sources_count + purged_pages_count
  that should have been added when v0.26.5 landed.

- test/e2e/cycle.test.ts: dry-run full cycle now expects
  report.phases.length === 10 (was 9).

- test/core/cycle.serial.test.ts: yieldBetweenPhases hook count + full
  cycle phases.length both updated 9 → 10. Comments call out the
  v0.31 addition lineage so the next person to add a phase sees the
  precedent.

These are mechanical assertion bumps. The tests pass against the
updated assertions on PGLite and Postgres.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(test): truncate facts table between e2e describe blocks

setupDB() truncates ALL_TABLES between every describe block's
beforeAll() hook. The list missed the new v0.31 facts table, so
facts seeded by an earlier describe block leaked into Garry's
Separation Test on Postgres — listFactsByEntity('travel') returned
2 rows instead of 1 because a prior facts-context-injection test had
also seeded a 'travel' fact.

Adding 'facts' to the truncate list (before 'pages' to respect FK
ordering) makes every describe-block start from an empty facts table.

Pinned by re-running the e2e file ordering that originally caught it
(facts-recall-render → cross-source-isolation → serve-http-meta →
context-injection → separation-postgres → facts-forget) — 13 pass /
0 fail after the fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 test: meta-hook cache + Postgres consolidate phase coverage

Two net-new test files filling real coverage gaps the earlier sweep missed:

- test/facts-meta-cache.test.ts (5 cases) — pins the eD3/eD10 cache
  contract that the dispatcher relies on. 30s TTL hit path, post-bump
  fresh-query, scoped invalidation (bump for sess-A leaves sess-B cache
  warm — closes the cross-source leak risk codex F5 originally surfaced
  on the recall payload), facts-self ops skip injection (anti-loop on
  recall / extract_facts / forget_fact), distinct allow-lists produce
  distinct cache entries.

- test/e2e/cycle-consolidate-postgres.test.ts (3 cases) — Postgres
  parity for the dream-cycle consolidate phase. Mirrors the PGLite
  unit test but exercises the real postgres-engine codepaths: sql.begin
  transactions, advisory locks on insertFact's entity-slug dedup window,
  unsafe('::vector') casts on findCandidateDuplicates ordering,
  addTakesBatch postgres-js unnest path. Happy path (4 facts → 1 take +
  all consolidated_into set), age-threshold skip, dry-run no-write.

All 5 unit + 3 e2e tests pass. Closes the unit-only gap on the
consolidate phase (was only PGLite-tested) and pins meta-cache
invariants the dispatcher depends on.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix: thread auth + sourceId, JSON-shape every error envelope

Three bugs surfaced during the full e2e sweep that all trace back to my
v0.31 dispatch refactor (D12/eE1) silently dropping auth threading +
non-OperationError exceptions emitting plain strings:

1. **HTTP MCP transport lost ctx.auth.** Refactoring serve-http.ts to call
   dispatchToolCall meant auth had to come through DispatchOpts, but the
   field didn't exist yet. Every HTTP whoami call returned
   `unknown_transport` because ctx.auth was undefined. Added `auth?:
   AuthInfo` to DispatchOpts, plumbed it through buildOperationContext,
   and updated serve-http.ts:816 to pass `auth: authInfo` alongside
   sourceId/takesHoldersAllowList. Pinned by sources-remote-mcp e2e
   `whoami reports oauth transport + sources_admin scope`.

2. **Non-OperationError exceptions emitted plain strings, not JSON.**
   The pre-v0.31 serve-http.ts always wrapped errors in JSON envelope
   `{error, message}`; my dispatch refactor missed the unknown-tool +
   uncaught-throw paths and emitted `Error: ${msg}` text content. Every
   caller that did `JSON.parse(content)` (sources-remote-mcp callMcp
   helper at line 104) crashed with `Unexpected identifier "Error"`.
   Both error paths in dispatchToolCall now return JSON-shaped content
   matching the OperationError pattern.

3. **Files→sources FK silently lost on rewound bootstrap path.**
   test/e2e/postgres-bootstrap.test.ts simulates a pre-v0.21 brain by
   `DROP TABLE IF EXISTS sources CASCADE` which removes
   files_source_id_fkey while leaving files.source_id intact. The v23
   migration's `ALTER TABLE files ADD COLUMN IF NOT EXISTS source_id ...
   REFERENCES sources(id) ON DELETE CASCADE` is a no-op when the column
   exists, so the FK never came back on upgrade — and any sources-remove
   afterward stopped cascading to files. Added a defensive
   `IF NOT EXISTS files_source_id_fkey ... ALTER TABLE ADD CONSTRAINT`
   block inside v23's handler. Pinned by `multi-source — cascade delete
   covers every dependent row` after running postgres-bootstrap.

Plus: src/core/preferences.ts now honors GBRAIN_HOME for
`~/.gbrain/migrations/completed.jsonl`. Without this, the doctor
exits-0 mechanical test inherits the developer machine's stale
partial-migration ledger entries (0.21.0, 0.22.4, 0.28.0, 0.29.1
prior dev work) and surfaces them as the [FAIL] minions_migration check.
GBRAIN_HOME-scoped tempdir per test now isolates this state cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 chore: scrub personal references from public artifacts

Per the CLAUDE.md privacy rule on `Garry's Separation Test`, replace
personally-coded references in v0.31 artifacts with neutral examples:

- CHANGELOG.md v0.31 entry: rename "Garry's Separation Test" header to
  "The cross-session test" + drop the "topic-2659/topic-1941, 7 AM/2 PM,
  flying to Tokyo" narrative.
- src/commands/migrations/v0_31_0.ts feature pitch: same scrub.
- test/facts-separation-pglite.test.ts + test/e2e/facts-separation-postgres.test.ts:
  rename describe blocks; replace specific topic-NNNN session ids with
  session-A / session-B; replace personal sample fact with
  "sample event Tuesday".
- src/core/facts/extract.ts extractor system prompt example slugs:
  people/sam-altman → people/alice-example; companies/anthropic → companies/acme.
- src/core/entities/resolve.ts comment: Sam Altman → Alice Example.
- All v0.31 test fixtures: people/sam → people/alice-example,
  Sam Altman → Alice Example, sam-the-cofounder → alice-the-cofounder.
  Test names referencing real-world entities replaced with neutral slugs.

Pre-existing references to "Garry" elsewhere in CHANGELOG (v0.17, v0.19,
v0.21+ entries) are untouched — that's a separate scope from this v0.31
ship.

Plus: the truncate fix for the Bun-script-induced syntax error in
test/e2e/mechanical.test.ts (cliEnv arrow function had ", 30_000)" tacked
onto its closing brace by the bulk-add-timeouts script — repaired to a
clean function definition).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(test): bump E2E phase-count assertions for 11-phase cycle

Two E2E tests still asserted the v0.31 pre-merge 10-phase shape
(consolidate inserted, but recompute_emotional_weight from v0.29 not yet
absorbed). With master's v0.29 work merged in, the cycle is now 11 phases:
lint → backlinks → sync → synthesize → extract → patterns →
recompute_emotional_weight → consolidate → embed → orphans → purge.

- test/e2e/cycle.test.ts: 10 → 11
- test/e2e/dream-cycle-eight-phase-pglite.test.ts: ALL_PHASES + dry-run order

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix(merge): close brace between v44 and v45 migration objects

The v0.30.2 merge resolution stitched master's v40-v44 migrations onto
HEAD's v45 (facts hot memory) migration but lost the closing `},` between
v44 and v45. tsc caught it as TS1136 Property assignment expected at
migrate.ts:2188.

This is a one-line bracket fix; the rest of the merge resolution is
correct and tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 fix: put_page cliHints + buildPlan v0.31.0 in skippedFuture

Two unit-test failures surfaced after the v0.30.2 merge:

1. operations.ts: put_page had `cliHints: { name: 'put', positional: ['stdin'] }`
   from earlier v0.31 development. The parity test enforces that every name
   in `positional` is a real param. Restored master's correct shape:
   `{ name: 'put', positional: ['slug'], stdin: 'content' }`.

2. test/apply-migrations.test.ts: the H9 regression tests pin the exact
   skippedFuture list. Adding v0.31.0 to the registry meant the list grew
   by one. Updated both `expect(...).toEqual([...])` assertions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* v0.31 docs: clarify consolidate is 11th phase + regen llms-full.txt

CHANGELOG.md narrative said "new 10th phase consolidate"; with v0.29's
recompute_emotional_weight already on master, consolidate is the 11th phase
(between recompute and embed). Schema migration is v45, not v40, after the
merge resolution renumbered it to clear master's v40-v44.

llms-full.txt regenerated to reflect the README's 11-phase dream-cycle
phrasing (the build-llms test enforces commit-time parity).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 16:57:47 -07:00
a4df40fe5c feat: v0.15.2 - bulk-action progress streaming (stderr reporter, agent-visible heartbeats) (#293)
* feat(progress): step 1 - shared ProgressReporter + CliOptions

Adds the foundation for v0.14.2's bulk-action progress streaming work:

- src/core/progress.ts: dependency-free reporter with auto/human/json/quiet
  modes, TTY-aware rendering, time+item rate gating, heartbeat helper for
  slow single queries, dot-composed child phases, EPIPE defense (both sync
  throw and async 'error' event), and a singleton module-level signal
  coordinator so SIGINT/SIGTERM emits abort events for all live phases
  without leaking per-instance listeners.

- src/core/cli-options.ts: parseGlobalFlags() for --quiet /
  --progress-json / --progress-interval=<ms> (both space and = forms),
  plus cliOptsToProgressOptions() that resolves to the right mode. Non-TTY
  default is human-plain one-line-per-event; JSON is explicit opt-in so
  shell pipelines don't suddenly see structured noise.

- test/progress.test.ts (17 cases): mode resolution, rate gating, no-fake-
  totals on heartbeat paths, EPIPE paths, SIGINT singleton, child phase
  composition.

- test/cli-options.test.ts (14 cases): flag parsing, invalid values,
  interleaved flags, mode resolution.

Follow-ups wire doctor/embed/files/export/extract/import/sync/migrate/
repair-jsonb/backlinks/orphans/lint/integrity/eval/autopilot/jobs plus
the apply-migrations orchestrators through this reporter, and route
Minion handlers to job.updateProgress instead of stderr. See the plan
in ~/.claude/plans/.

1682 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 2 - wire global flags into cli.ts

Parse --quiet / --progress-json / --progress-interval from argv BEFORE
command dispatch, strip them, stash resolved CliOptions on a module-level
singleton (same pattern as Commander's program.opts()) and on every
OperationContext created for shared-op dispatch.

- src/cli.ts: parseGlobalFlags(rawArgs) at the top of main(); setCliOptions
  once; dispatch sees only the stripped argv. Fixes the "gbrain
  --progress-json doctor" unknown-command case that Codex flagged.
- src/core/cli-options.ts: expose setCliOptions/getCliOptions/
  _resetCliOptionsForTest singleton. Commands that want progress call
  getCliOptions() to construct their reporter.
- src/core/operations.ts: OperationContext gains optional cliOpts field
  so shared-op handlers (and MCP-invoked ops that need a reporter) can
  read the same settings. MCP callers leave it undefined and consumers
  default to quiet.
- test/cli-options.test.ts: +4 cases covering singleton round-trip and
  an integration smoke spawning `bun src/cli.ts --progress-json --version`
  to prove the global flag survives dispatch.

45 relevant unit tests pass (progress + cli-options + cli.test.ts).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 3a - doctor + orphans heartbeat streaming

Doctor on a 52K-page brain used to sit silent for 10+ minutes while the
DB checks ran, then get killed by an agent timeout. Wired through the
new reporter so agents see which check is running and the slow ones
heartbeat every second.

doctor.ts:
- Start a single `doctor.db_checks` phase around the DB section, with a
  per-check heartbeat before each step (connection, pgvector, rls,
  schema_version, embeddings, graph_coverage, integrity, jsonb_integrity,
  markdown_body_completeness).
- jsonb_integrity now scans 5 targets, not 4: added page_versions.
  frontmatter so the check surface matches `repair-jsonb` (per Codex
  review of the plan — the old 4-target scan missed a known repair site).
  Per-target heartbeat so 50K-row scans show incremental progress.
- markdown_body_completeness: wrap the existing query in a 1s heartbeat
  timer. The regex scan over rd.data ->> 'content' can't be paginated
  usefully; this just lets agents see life during the sequential scan.
  No fake totals — the LIMIT 100 query has no meaningful total count.
- integrity sample: same heartbeat pattern around the 500-page scan.

orphans.ts:
- findOrphans() wraps the NOT EXISTS anti-join in a 1s heartbeat.
  Keyset pagination was considered and rejected: without an index on
  links.to_page_id it's no faster than the full scan, and may re-plan
  the anti-join per batch. A schema migration adding that index is the
  right fix and is queued for v0.14.3.

Follow-ups:
- Step 3b: wire embed/files/export (the \r-only stdout offenders).
- Step 5: end-to-end progress test spawning `gbrain doctor --progress-json`
  against a fixture brain, asserting stderr events and clean stdout.

All existing unit tests continue to pass (76/76 in doctor + orphans +
progress + cli-options).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 3b - embed + files + export stderr progress

Replaces the \r-on-stdout progress pattern in the three worst offenders
(embed, files sync, export) with the shared reporter on stderr. Stdout
now carries only final summaries, so scripts and tests that grep for
counts ("Embedded N chunks", "Files sync complete", "Exported N pages")
still work when output is piped.

- embed.ts: runEmbedCore accepts an optional onProgress callback. The
  CLI wrapper builds a reporter and passes reporter.tick(); Minion
  handlers will pass job.updateProgress in Step 4. Worker-pool is
  single-threaded JS so no rate-gate race (per Codex review #18).
- files.ts syncFiles(): tick per file; summary preserved on stdout.
- export.ts: tick per page; summary preserved on stdout.

Also fixes a --quiet flag collision. `skillpack-check` has its own
--quiet mode (suppress all stdout). parseGlobalFlags strips --quiet
globally now, and skillpack-check reads the resolved CliOptions
singleton via getCliOptions() instead of re-parsing argv. Test updated
to match the stripping behavior.

1686 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 3c - extract + import + sync reporter streaming

Extract, import, and sync now stream per-file progress to stderr through
the shared reporter. All three kept their stdout summaries + JSON
action-events intact so existing tests + agent scripts are unaffected.

- extract.ts (4 paths: links/timeline × fs/db): replaced the ad-hoc
  `process.stderr.write({event:"progress"...})` lines with reporter
  ticks. Same channel (stderr), canonical schema now, visible in both
  text and --json modes. Stdout action-events (`add_link` /
  `add_timeline`) untouched — tests grep them.
- import.ts: the logProgress() function that printed every 100 files to
  stdout is now a progress.tick() call per file. Rate-gated by the
  reporter. Stdout still gets the final "Import complete (Xs)" summary
  and the --json payload.
- sync.ts: three new phases (`sync.deletes`, `sync.renames`,
  `sync.imports`) tick per file, so big syncs show each step rather than
  a single end-of-run summary. Phase hierarchy ready to be child()-chained
  into runImport / runEmbed later, per Codex review #26.

Updated the #132 nested-transaction regression test in test/sync.test.ts
to also accept the new hoisted-loop shape — the guarantee (this loop is
not wrapped in engine.transaction) still holds.

1686 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 3d - migrate/repair/backlinks/lint/integrity/eval

Wires the remaining bulk commands through the reporter:

- migrate-engine: phase starts (migrate.copy_pages, migrate.copy_links),
  per-page tick. Old \"Progress: N/total\" stdout logs replaced by
  stderr ticks; final stdout summary preserved.
- repair-jsonb: per-column start + a heartbeat timer while each UPDATE
  runs (minutes on 50K-row tables). CRITICAL: stdout stays clean so
  migrations/v0_12_2.ts's JSON.parse(child.stdout) still works. Per
  Codex review #12.
- backlinks: 1s heartbeat around findBacklinkGaps() (sync double-walk
  of the brain dir).
- lint: tick per page; per-issue stdout output preserved.
- integrity auto: tick per page in the main resolver loop. The separate
  ~/.gbrain/integrity-progress.jsonl resume marker is untouched (its
  role shifts from live progress reporting to resume-only).
- eval: add an onProgress option to core's runEval(), CLI wraps with a
  reporter. Phases: eval.single / eval.ab. Tick per query.

core/search/eval.ts gains a RunEvalOptions type so future callers (MCP
eval op, Minion handlers) can also hook in without the reporter.

1686 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 3e - onProgress callbacks on core libs

- src/core/embedding.ts: embedBatch() gains an optional
  EmbedBatchOptions.onBatchComplete callback, fired after each 100-item
  sub-batch. CLI wrappers pass reporter.tick; Minion handlers can pass
  job.updateProgress.
- src/core/enrichment-service.ts: enrichEntities() config gains
  onProgress(done, total, name) fired after each entity. Same split:
  CLI -> reporter, Minion -> DB-backed progress.

No CLI behavior change on its own. Wiring these callbacks into the
Minion handlers is Step 4.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 4 - orchestrators + upgrade + minion handlers

- cli-options.ts: childGlobalFlags() returns the flag suffix to append
  to child gbrain subprocesses. Empty string by default, " --quiet
  --progress-json" when the parent has them set, so child behavior
  inherits the parent's progress-mode without scattering string-concat
  logic across every execSync site.

- migrations/v0_12_2.ts: each execSync inherits the parent's global
  flags. Phase C (repair-jsonb --dry-run --json) pins explicit stdio to
  ['ignore','pipe','inherit'] so child stderr streams straight through
  while stdout stays captured for JSON.parse. Per Codex review #12.
- migrations/v0_12_0.ts + v0_11_0.ts: same childGlobalFlags wiring at
  each gbrain-subcommand execSync.

- upgrade.ts: post-upgrade timeout bumped 300s → 30min (1_800_000 ms)
  with GBRAIN_POST_UPGRADE_TIMEOUT_MS override. The old 300s cap killed
  v0.12.0 graph-backfill migrations on 50K+ brains; the heartbeat
  wiring added in v0.14.2 makes long waits observable, so a generous
  ceiling no longer means users stare at a silent terminal.

- jobs.ts: the embed Minion handler passes job.updateProgress as the
  onProgress callback, so per-job progress is durable in minion_jobs
  and readable via `gbrain jobs get <id>`. Primary Minion progress
  channel is DB-backed — stderr from `jobs work` stays coarse for
  daemon liveness only. Per Codex review #20.

1686 unit tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 5 - E2E doctor-progress test + CI guard

scripts/check-progress-to-stdout.sh greps src/ for the banned
`process.stdout.write('\r…')` pattern that v0.14.2 removed from the
bulk-action codepaths. Wired into the `bun run test` script so any
future regression that puts progress back on stdout fails fast. An
empty allowlist documents the position: every known call site was
migrated; new exceptions need a rationale in the allowlist.

test/e2e/doctor-progress.test.ts (Tier 1, needs Postgres + pgvector):
- `gbrain --progress-json doctor --json`: stderr carries JSONL progress
  events with the canonical {event, phase, ts} shape, starts + finishes
  for `doctor.db_checks`. Stdout stays parseable JSON — no progress
  pollution.
- `gbrain doctor` (no flag): human-plain progress goes to stderr only,
  stdout stays free of `[doctor.db_checks]`.
- `gbrain --quiet doctor`: reporter emits nothing; doctor still runs to
  completion.

test/cli-options.test.ts: +2 spawning integration tests. One verifies
`gbrain --progress-json --version` keeps stdout clean of progress events
(single-shot commands that don't use a reporter aren't affected). One
guards the skillpack-check --quiet regression — --quiet suppresses
stdout by reading the resolved CliOptions singleton, not re-parsing argv.

Full test matrix:
  bun run test           -> 1726 pass / 184 skipped (no DB) / 0 fail
  bun run test:e2e       -> 136 pass / 13 skipped / 0 fail

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(progress): step 6 - docs + v0.14.2 release bump

- VERSION + package.json bumped to 0.14.2.
- docs/progress-events.md (new): canonical JSON event schema reference.
  Stable from v0.14.2, additive only. Lists every phase name shipped
  in this release, the five event types (start/tick/heartbeat/finish/
  abort), the TTY/non-TTY rendering rules, subprocess inheritance
  semantics, and the Minion DB-backed progress model.
- CLAUDE.md: "Bulk-action progress reporting" section under the build
  instructions; Key files entries for src/core/progress.ts,
  src/core/cli-options.ts, scripts/check-progress-to-stdout.sh, and
  docs/progress-events.md; doctor.ts entry updated to note the v0.14.2
  5-target jsonb_integrity scan + heartbeat wiring.
- CHANGELOG.md v0.14.2: full release summary per project voice rules.
  The "numbers that matter" table, per-command before/after grid,
  backward-compat warnings for stdout→stderr moves, and an itemized
  changes section covering reporter/CLI plumbing/schema/Minion
  handlers/doctor fixes/upgrade timeout/CI guard/tests. No em dashes.
  Real file paths, real commands, real numbers.
- skills/migrations/v0.14.2.md (new): agent migration note. Mechanical
  step is "nothing" since v0.14.2 is purely additive. Walks agents
  through the three new global flags, the 14 wired commands, the event
  schema cheat sheet, Minion progress via job.updateProgress, and
  scripts/verification commands.

Full test matrix:
  bun run test (unit + guards) -> 1726 pass / 184 skipped / 0 fail
  bun run test:e2e (Postgres)  -> 141 pass / 8 skipped / 0 fail

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version to 0.15.2, restore master's [0.14.2] CHANGELOG entry

Master sits at 0.14.2 (reliability wave). This PR lands on top as 0.15.2
(progress streaming wave). Splits the merge-time combined CHANGELOG entry
back into two discrete release sections so history stays honest:

- [0.15.2] = progress reporter, CliOptions, 14 wired commands, Minion
  embed handler, doctor jsonb_integrity 5-target fix, upgrade timeout bump,
  CI guard, progress unit+E2E tests.
- [0.14.2] = master's eight root-cause bug fixes, restored verbatim from
  origin/master.

Touched files:
- VERSION + package.json: 0.14.2 -> 0.15.2 (next patch off master).
- skills/migrations/v0.14.2.md -> skills/migrations/v0.15.2.md (rename
  + rewrite frontmatter + body to v0.15.2).
- CHANGELOG.md: split into two entries; progress-wave refs renamed
  v0.14.2 -> v0.15.2; reliability-wave entry restored from master.
- src/core/progress.ts, src/commands/doctor.ts, src/commands/sync.ts,
  src/commands/upgrade.ts, docs/progress-events.md, test/sync.test.ts:
  progress-wave v0.14.2 references -> v0.15.2. The remaining v0.14.2
  references in test/e2e/migration-flow.test.ts (Bug 3 context) and
  CLAUDE.md (reliability-wave key commands, Bug 3 ledger move) correctly
  point at master's 0.14.2 release.

Test matrix after version bump:
  bun run test -> 1780 pass / 179 skipped / 0 fail

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 17:54:13 -07:00